Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Video Is The Web’s Third Age: AOL’s Martinez

CHICAGO — AOL sees video as spurring the next web age, but eventual manifestations of that will depend very much on consumption context, says an exec from the company. “The third wave of quantum innovation in the web is going to be driven by sight, sound and motion,” AOL video sales head ...

image
‘Addressable TV Will Take Off In 2015′: SMG’s Scheppach

CHICAGO — In 2014, TV began learning the targeting tricks that digital media have deployed for years. In 2015, those techniques will kick in to high gear, says one ad agency exec. “(In) 2014, I see a marketplace that has essentially gone from zero to $300 million of spend,” according ...

image
McDonald’s Puts Video At ‘Core’ Of Digital Focus

CHICAGO — After being named by advertising peers the most creative marketer at this year’s Cannes Lions, McDonald’s is ready to bolster the part video plays in its digital efforts. “We’ve been the definition of  a mass marketer,” the group’s digital brand engagement ...

image
Programmatic ‘Is Not A CPM Crunch’: Digitas’s Zaben

CHICAGO — The collection of new-wave automated ad targeting and trading techniques known as “programmatic” may have initially been seen as helping advertisers drive down inventory costs – but that is no longer the sole way ad buyers see it. “This is not a CPM crunch where ...

image
AOL Tops comScore Rankings of Video Advertising Properties

In comScore’s monthly ranking of video advertising properties, AOL reached the top spot for the time. It  ranked #1 in video ad reach, serving ads to 53.8 percent of the total U.S. population in October. BrightRoll Platform came in second with 52.3 percent reach, followed by LiveRail with 52.1 ...

image
Direct-To-Consumer Video Offerings Slated to Grow, thePlatform’s Roberts

Expect a rash of direct-to-consumer over-the-top offerings from networks and broadcasters in the next year or so, says Marty Roberts, co-CEO of thePlatform in an interview with Beet.TV. “Networks and broadcasters need to establish a direct relationship with the audience,” he says, and ...

image
Tribune Picks Taboola For TV Sites’ Content Marketing: Report

Big media companies are increasingly keen on locking in long-term native advertising commitments, if the latest of two big announcements this week is anything to go by. Tribune Media’s broadcast subsidiary Tribune Broadcasting, whose TV stations include KTLA of Los Angeles and WGN-TV of Chicago, is ...

image
Mediaocean Extends Digital Video Ads With Videology

Mediaocean, which currently helps ad buyers plan, execute and analyze advertising on TV using software, is about to get greater traction in the online video world, through a partnership with Videology. The integration will make online video ad inventory amassed by Videology buyable by Mediaocean‘s ...

image
TV Everywhere Authentication Rates Improving, Marty Roberts of thePlatform

In-home, improved authentication protocols can dramatically improve the consumer experience for TV Everywhere, says Marty Roberts, co-CEO of thePlatform in an interview with Beet.TV. In some cases, log-in failure rates are north of 40% and that’s too high to be sustainable. However, when multichannel ...

image
Video Ad Customization Platform Eyeview Raises $15 Million in New Venture Round

Eyeview, the New York-based company that provides marketers with the technology to personalize digital video ads to specific consumers, has raised an additional $15 million, the company announced on Monday. More on the raise and other developments reported here on Re/code by Peter Kafka. Last month, we spoke ...

image
AOL Presents iJustine In Chicago: the Beet Video Summit w/ McDonald’s,Target, Digitas, Havas and...

iJustine, a pioneering  innovator  in the Web originals space will headlining a two hour video summit on the emerging medium of digital video. Also participating will be  brand marketers  from McDonald’s and Target and media agency execs from Digitas, Havas and Starcom MediaVest Justine will speak ...

image
Digital Means ‘World Of Opportunity’: DigitasLBi’s Shlachter

NAPA, CA — As much as it poses challenges, the ongoing explosion in digital media is also opening up a wealth of new opportunities for marketers to reach consumers, says DigitasLBi North America’s chief investment officer Adam Shlachter. “The numbers are there now – if you look at ...

loader