Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
Upfronts Won’t Die In TV Renaissance: Magna Global’s Bayer

FORT LAUDERDALE — If you think the coming age of buying addressable, internet-style ads in the blink of an eye is going to mean an end to the TV ad industry as we know it, think again. “I think we’ve entered …a renaissance of television,” Matt Bayer, SVP of advanced TV at Magna ...

Brienna Pinnow, Experian
Experian Gearing To Launch Connected Addressable TV Platform

FORT LAUDERDALE — “This has really been the year of addressable TV. We’ve done literally triple the amount of work compared to 2014,” according to Experian addressable advertising product lead Brienna Pinnow. Now the data company wants to do more? But what exactly? “To develop ...

More Safeguards Needed For Dynamic Ad Creative: Bank of America’s Paskalis

The concept of dynamic ad insertion – through which, consumers would get served a brand ad with different content, depending on their targeting profile – has been talked about for a couple of years now. The exciting reality may be around the corner, if the industry can persuade brands it is a ...

Context Of Viewing Still Matters To Ads: Videology’s CEO

Now that online video ad targeting technology can reach advertisers’ ideal audiences wherever they are watching, what is the use for the old model, in which buyers place their bets on shows with audiences they believed to be matched? The new doesn’t have to trump the old, says video ad tech ...

Too Soon For Dynamic Creative Ads: DISH’s Gaynor

Super-targeted TV advertising is just around the corner, but don’t expect the content of those ads to be served dynamically just yet. That’s according to DISH media sales and analytics VP Adam Gaynor. In this video interview with Beet.TV, he says: “Advertisers no longer need to use ...

Castree On Videology’s “Sand, Rocks & Boulders”

FORT LAUDERDALE — It’s one of the major ad tech vendors promising to define the emerging video advertising space. But what is going on at Videology lately? The way North America MD Tim Castree describes, Videology is a little bit like geology… “Sand — How we can use our ...

Cut The Fraud Bull From Ad Pricing: GroupM’s Gotlieb

If you want to price digital advertising appropriately, don’t just look at the rapidly-growing supply of inventory, look beneath the surface for what is really real. That’s according to veteran ad exec Irwin Gotlieb, the chairman of ad group GroupM. “Supply measured without taking in to ...

Context Still Matters For TV Ads: FreeWheel’s Rothwell

Does the coming age of one-to-one, super-personalized TV ads make the traditional process of buying ads against shows of a particular type dead in the water? Not at all, says FreeWheel agency and brand relations VP James Rothwell, whose company helps place video ads. “Context still matters,” ...

Data Will Set Context Free: SMG’s Scheppach

Advertisers are getting excited by the possibilities for super-targeted TV marketing promised by new ad tech platforms. In fact, to many, it looks like the days of traditional TV ads, which are bought against the context of demographic audience profiles, are numbered. But things are set for a correction, ...

How Simulmedia Tracks The ROI Of Linear TV: VP Storan

FORT LAUDERDALE — It may be old-fashioned linear TV, but that doesn’t mean you can’t bring internet-like next-gen ad measurement to the ‘ol box. That’s what Simulmedia, the ad technology vendor, is doing. Although the company focuses on helping clients find the best inventory ...

Two Steps To Scaling Addressable TV Heights: Invision’s Marshall

FORT LAUDERDALE — This year, the promise of “addressable TV”, allowing advertisers to target individual TV sets, became a reality. But it remains just a slither of a US TV industry that is worth circa $75 billion. “There’s two parts to scaleability with addressable:,” says ...

TV Ad Super-Targeting Is Ready To Scale: Eyeview CEO

FORT LAUDERDALE — Last year was been characterized by the industry wondering when TV platforms would open up to support addressable TV advertising. This year seems to have been characterized by actual roll-out. Next year could mean larger-scale adoption. “The concept of being able to target ...