Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Digital Isn’t Boosting Overall Ad Growth Much: comScore’s Fulgoni

Digital advertising has grown fast over the last few years – but its surge is not causing the overall advertising economy to inflate, says an ad measurement veteran. “Digital has skyrocketed – but it’s coming from other budgets,” according to comScore executive chairman emeritus ...

image
Ad Tech Consolidation Doesn’t Mean Walled Gardens: BrightRoll’s Mosher

Ad-tech M&A is in full-swing. Whilst the industry lumascape looks complex today, large internet operators are snapping up some of the bigger ad tech platform vendors? So, as consolidation occurs, will vendors under new ownership be temped to lock customers in to their parents’ business ecosystem? ...

image
eXelate Launching TV-Mobile Ad Solution: CEO Zagorski

It’s been going for seven years with its customer data services. Now eXelate is also one of the companies helping advertisers knit together messaging in the multi-screen era, with some pretty cool-sounding technology. “We’ve partnered with Innovid and connected mobile app data with smart TV ...

image
Retargeting is Huge on Mobile, Video is Next, Criteo’s Deichert

Ad retargeting has become known for serving users with ads for products they have already looked up on the Internet. So could the same work for video? Not yet, according to a man from one of the top display ad retargeting services. “To do a really quality job with video retargeting, you have to have ...

image
IAB’s Rothenberg: The Problem with Viewabilty is the Tech Vendors

While publishers and advertisers are essential in agreement over what constitutes a “viewable” digital advertisement, the technology for measuring ‘viewability” is a big problem, with multiple tech vendors coming with vastly different findings, says Randall Rothenberg, CEO of the IAB ...

image
Marketing Industry Chief: Ad Fraud Must be “Put to Its Death”

Speaking in blunt terms, Bob Liodice, President and CEO of the ANA,  the nation’s trade association for advertisers, says that his group has had enough and wants no part of criminal activity where dollars are being diverted in various ad fraud scenarios.   He says the industry needs to deal with this ...

image
Social Sharing Drives Video Ad Returns: Unruly’s Kosinski

Making your video ad shareable doesn’t just mean a wider audience – it increases its chances of stimulating real marketing outcomes, says Richard Kosinski, president of video measurement agency Unruly Media. That company produces the Viral Video Chart and tracks how videos are watched and ...

image
Innovid Rolls Out New Targeting Tools

Advertising tech platform Innovid is rolling out real-time synchronized ads this year, says CTO Tal Chalozin at the recent Beet.TV executive retreat. As it expands deeper into targeting technology, Innovid has begun working with Cisco to bring tools to second screen apps by targeting ads to relevant ...

image
Small TV Networks Allow Programmatic Buying By Magna Global Advertisers

The US TV business has moved another step closer to selling its advertising programmatically, with 15 small broadcast networks signing up to allow programmatic buying by clients of IPG Mediabrands’ Magna Global investment unit. Its data-driven ad targeting capabilities will be used by small networks ...

image
Teads Outperforms Competitors In Video Ads, Nielsen Commissioned Study

French video ad tech company Teads has lately been shouting about the performance of its ad format InRead, which inserts auto-playing video ads between text paragraphs on major publisher sites. Now it is using a study commissioned from Nielsen to claim it outperforms its competitive video ad format ...

image
Google’s AdX Bigger Than Global Stock Markets: Vanderzee

It is often said that the “programmatic” shift from manual ad-buying to automated mechanisms will do for Madison Avenue what computer trading did to Wall Street. But, on one measure, it’s already bigger than that. “The number of transactions now on the DoubleClick Ad Exchange, if you ...

Martin Sorrell - media rev thumbnail
Making the “Media Revolution:” IPG’s Roth, WPP’s Sorrell, AOL’s Armstrong...

The role of the media agency is dramatically changing with entrance of “biased” tech companies, says Michael I. Roth, CEO of Interpublic Group. He says the role of the unbiased, “agnostic”  media agency is more important than ever. Sir Martin Sorrell, CEO of WPP, says that there will ...

loader