Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Setbacks Can Define an Ad Executive, David Moore Says

Success is well and good, but setbacks often define an industry leader, says David J. Moore, Chairman of Xaxis and President of WPP Digital, in this interview with Beet.TV. Take 24/7 Media, an ad tech company he founded that has become emblematic of the ups and downs of the ad business. “The greatest ...

image
Digital First Content Companies Primed for Programmatic TV, SMG’s Delaney

Digital-first content companies such as Netflix, Hulu and AOL are well-positioned for the transition to addressable buying in TV, says SMG programmatic SVP Mac Delaney in an interview with Beet.TV. The recent Digital Content New Fronts mark a transition point in the shift to programmatic buying in TV. ...

image
Ad Tech Is Cyclical: AppNexus CEO O’Kelley

Looked at the ad tech lumascape lately? It’s not pretty; it’s confusing. But it may not be that way forever. The boom in advertising technology is just a natural, cyclical response to a world of complexity, says one of the sector’s top execs. At times, it’s pushed to the background as ...

image
“Convergent TV” Will Take Front Stage at Cannes Lions, Banker Kawaja

The transformation of television and the emergence  “convergent TV,” will take front stage at the Cannes Lions Festival next month, predicts Terence Kawaja, co-founder and CEO of LUMA Partners, a prominent investment bank in the media and adtech sectors. The discussion of addressable TV, and ...

image
New Standards Around Native Units Mean Big Opportunities for Publishers, PubMatic’s McDonald

While much has been said about the opportunities for publishers to create bespoke content for advertisers, for the medium to scale there needs to be standards so the units can be widely shared and sold programmatically,  says Kirk McDonald, President of PubMatic, a global adtech company that serves ...

image
Guardian Labs Mulls Off-Site Branded Content via Programmatic

LONDON — So far, The Guardian division that produces branded content for marketers has published that content only on The Guardian’s own properties. But that could change soon, MD Anna Watkins tells Beet.TV in this video interview. “At the moment, we are looking to confirm one of our first ...

image
Building a Brand Depends on eCommerce, Razorfish’s Denton

The agency of the future needs to be a “data geek,” says Shannon Denton, CEO North America of Razorfish, in an interview with Beet.TV. “They need to understand all kinds of data and how to push it through to the consumer in terms of experiences that show the intelligence and make the ...

Sarah Wood, Unruly
Unruly Brings “Emotional Programmatic” To US Advertisers

If you had tracked two trillion video views since 2006, you would probably know quite a lot about how people respond to video ads. Sarah Wood has, and her video ad tech firm, Unruly, is helping brands understand. “What are the drivers of shareability?,” she says. “Some people think it’s ...

image
Programmatic Audio Ads Get ‘100% Attention': Xaxis’ Bidon

LONDON — So far, the discussion of so-called “programmatic”, automated digital advertising trading has centered on online display and video advertising. But what about advertising in content that no-one can see? Last month, GroupM’s Xaxis programmatic unit announced Xaxis Audio, a ...

image
Tracking From Exposure To Closure Is A Reality: eXelate’s Zagorski

If a tree falls in a forest and no-one was around to hear it, did it really make a sound? Many advertisers have asked themselves that question in years gone by, unable to know whether their analog media spend really made a difference. But digital ads that lead to digital actions mean attribution can ...

image
Teads Touts ‘Outstream’ Video Units for Mobile: Le Monde and Forbes Have Integration

Teads, the adtech company that provides publishers with “outstream,” a solution to insert video advertising into editorial pages, launched a mobile SDK in March and it has been adopted by Le Monde and Forbes, says Bertrand Quesada, CEO, in this interview with Beet.TV Quesada sees robust interest ...

image
Expansion Planned for Guardian’s Programmatic Alliance w/ CNN Intl, Economist, Financial Times...

LONDON — Back in March, The Guardian and four other UK publishers with global ambitions announced they were forming the Pangea Alliance, a shared scheme to pool first-party ad data for ad buyers to target their combined 110 million users in a “programmatic”, or automated fashion. In ...

loader