Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Second Screens Yield Vital Data for TV Networks

The next hurdle for second screen marketing lies in measuring consumer sentiment towards a TV show, says Alan Wolk, Chairman of Second Screen Society, in an interview with Beet.TV. That can be done by tapping into the treasure trove of data available via mobile phones that consumers use to interact with a ...

image
Addressable Tools Will Drive Programmatic TV, Videology’s Scott Ferber

Programmatic TV will be driven first by addressable tools, says Scott Ferber, Chairman and CEO of video advertising technology platform Videology, in an interview with Beet.TV. “The first step where programmatic and addressable will converge in the TV space is on the data side…understanding who ...

image
AOL Sees Growth in Programmatic TV with 58 Ad Campaigns Up

While programmatic TV buying operates in a just a small slice of the overall TV ad market, it is growing and advertisers are allocating budget.  AOL has some 58 advertising campaigns running in test mode, on its platform, says Bob Lord, Global CEO of AOL Platforms in this interview with Beet.TV We sat down ...

image
Out-Stream Gives Slate 10x Video Ad Boost

CHICAGO — Webzine Slate may be producing some video of its own nowadays but, as a text specialist, the outlet may never make enough to satisfy booming advertiser demand for video ad inventory. That’s why the publisher is using InRead, an ad format from vendor Teads which allows sites to inject ...

image
Cars.com’s Kraut Sees Ad Value In Data After Gannett Sale

CHICAGO — A couple of months after US news publisher Gannett finalized its purchase of the remainder of Cars.com, the autos classified site is revving up to combine data points and its new owner’s heft to sell ads. “There’s a lot of data … our advertisers would love to have the ability ...

image
AdTech Banker Kawaja: Video Ad Platforms Need To Differentiate

If you are tired of trying to distinguish one “end-to-end, full-service video targeting and decisioning” vendor from another, imagine how people from outside the digital video industry must feel. LUMA partners CEO Terence Kawaja says 2015 will see ongoing video tech momentum after the ...

image
Nielsen-Adobe Deal on Track for 2015 Digital Ratings

As consumer interest in watching authenticated TV rises, the ad business will look for better ways to measure video across screens, says Jeremy Helfand, VP Adobe Primetime at Adobe, in this interview with Beet.TV. About 10% of all authenticated TV content is viewed through an OTT device, underscoring the ...

image
WPP’s Modi to Deliver “Addressable” TV Ads Programmatically with Videology

Modi, the recently launched advanced TV unit of WPP’s GroupM, is working with video adtech company Videology to deliver television advertising to specific cable and satellite users in real-time, using programmatic ad decisioning technology, explains Jamie Power, Senior Partner at Modi, in this ...

image
Satisfying Readers Is Key To Content Marketing: Digitas’ Mark Book

CHICAGO — DigitasLbi’s “Perceptions Of Care” video campaign for Whirlpool has won rave reviews from ad watchers, after the agency worked with popular web publisher Upworthy to craft the campaign as a pair of videos. The campaign has generated some hefty social media views. So what was ...

image
More Data Needed For Personalized Ads: SMG’s Lichtenberg

CHICAGO — Digital ads can begin to reach individual consumers with specific messages based on unique characteristics – but only if computers know enough to target them, says one ad agency exec. “When I think about the future of video, I think about more personalized experiences,” SMG ...

image
The Problem With Video Pre-Rolls: Teads’ Jim Daily

CHICAGO — It’s only natural that online video advertising takes after television advertising – after all, both media are about moving pictures. But, whilst the forms may be similar, advertiser results can vary across each, according to a tech exec. “It’s a similar experience to what ...

image
Ad Buyers Can Target Cox Viewers In 2015 With AudienceXpress

The sale of TV advertising is about to go a lot more “programmatic” early in 2015, when advertisers via Magna Global start buying slots in Cox Media programming via a new platform. Powered by planning, buying and reporting optimization platform AudienceXpess, the new system will help Magna Global ...

loader