Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
How Addressability Will Change TV Ads: 4C’s Neuhauser

So-called “addressable TV’, the ability to deliver personalized TV ads to individual households, is set to revolutionize the business – if only it can get off the ground in earnest. “There’s a lot of talk about addressable TV. The market is trying to look for what is real,” ...

Ad Tech Has Made Life Harder And More Expensive: Furious’ Swartz

COLOGNE — The sea of advertising technology platforms in which the industry is which the industry is currently swimming is threatening to overwhelm. Whilst each platform promises to make life better and easier, the reality may be the opposite. “There is a proliferation of ad technology – ...

Video Is An Antidote To Mobile Ad Blocking: Mediaocean CEO

Advertisers these days seem worried about a litany of technologies with which bad industry actors and consumers alike can bypass advertising. The latest is ad blocking, which Apple’s iOS 9 now supports on mobile web. But that threat to mobile could benefit another media, one ad tech exec reckons. ...

Clients Want Mix Of Programmatic Tools: Tremor’s Bill Day

Programmatic advertising is booming, as more ad buyers and sellers switch on to technology that automates, targets and better controls the trading of online inventory. So far, the tech has revolutionized web ads, next up is video. “Programmatic has been huge in display, programmatic is a bit different ...

Programmatic Is Changing The Media Landscape: TubeMogul’s Reid

COLOGNE — So-called “programmatic” technologies for automating, planning, targeting and executing online advertising buys have revolutionised the online display ad sector – now they are making in-roads in to video. “Programmatic is changing the media landscape,” according ...

Customer Experience Comes Before Ad Monetization For Belgium’s Proximus

COLOGNE — Many operators out there are busy launching initiatives to super-target digital TV viewers. But, when you’re a telco with paying subscribers, you’d better be sure you don’t do anything that annoys your paying customers, says a Belgian quad-play provider. “As a telco, ...

Belgium’s Medialaan Gears Up For Dynamic Ad Insertion

COLOGNE — Medialaan, the commercial media operator serving the Flemish-speaking Flanders region of Belgium, began making full shows available online last year. Now it is gearing up to better-monetize its full-show assets with online advertising. “It’s been better than we expected,” ...

Biggest Ad Challenge Is ‘Invalid Traffic’: comScore’s Trigg

COLOGNE — As an acronym, it doesn’t seem to make any sense. But “IVT” (that’s “invalid traffic”) is now the new industry jargon for what had been called “non-human traffic”, the scourge of online ads being triggered to defraud advertisers using fake ...

Sky Aims To Capitalize On German Online Soccer Demand: Meininger

COLOGNE — Soccer is big in Germany – and that means soccer is watched a lot, not just on traditional TV but also on new online TV services. That’s what the satellite and telco operator Sky Deutschland is discovering in its multi-platform online viewing service brand, Sky Go. “If you ...

Gabe Greenberg, GABBCON
Greenberg’s GABBCON Aims To Demystify Programmatic TV

COLOGNE — This summer, Gabe Greenberg left his post as general manager of advanced and programmatic television at The Trade Desk to launch a conference about programmatic TV advertising. Dubbed GABBCON (the Global Audience Based Buying Conference), the new LA-based enterprise is more than an ...

Bill Wise, Mediaocean 1
comScore + Retrak Merger “Nice,” But Won’t Replace Nielsen, Mediaocean’s Bill...

The news last week about the acquisition of Rentrak by comScore has been hailed by some as the emergence of a new competitor to Nielsen in the TV ratings space.  For Bill Wise, CEO of  TV media buying platform Mediacean, competition is a good thing, and while the newly combined forces will keep Nielsen ...

Virool is Using Facial Recognition for Ad Targeting

Virool, the video adtech platform that provides publishers with a means to monetize video advertising outside of the traditional video player and into the content stream, is analyzing the facial expressions of consumers to measure reactions to specific video ads, explains CEO Alex Debelov, in this ...