Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Hulu’s Naylor: Majority of Views Now in the Living Room

FOR LAUDERDALE –  Hulu, the popular entertainment video portal,  is seeing a majority of its views coming via devices that connect to the television, says Peter Naylor, SVP of Advertising Sales, in this interview with Beet.TV He says that some 60% of viewers are now consuming content in their living ...

image
A Wonderful, Unexpected Run/Walk to Fight Cancer with Tracey Scheppach and David Moore

FORT LAUDERDALE –  Cancer has touched so many of us — and sometimes it hits very close to home.  Such is the case for SMG’s EVP Tracey Scheppach whose son Ryan, 9, is battling leukemia.  His official diagnosis is High Risk Pre-B ALL (Acute Lymphoblastic Leukemia).   Tracey has taken to ...

image
The Transformation of Television Advertising via Addressable is Inevitable, Videology’s Castree

FORT LAUDERDALE — It’s going to be gigantic when all the technology is in place – but why let the future get in the way of disrupting the present? “At end of 2015, we’ll be lucky if 3% of the total video universe is transacted programmatically,” ad tech firm Videology ...

image
Deduplicate Audiences With Cross-Media Planning: Nielsen’s Hohman

FORT LAUDERDALE — If you advertise to reach an audience demographic in online video and in TV, you may think you have maximised your reach – but, if some viewers saw your ad in both media, have you overpaid? Nielsen agency solutions MD David Hohman says new measurement tools like his own ...

Screen Shot 2015-01-30 at 12.28.37 PM
Full Ad Viewability Comes With A Price: Google’s Hong

FORT LAUDERDALE — So you’re an advertiser and you only want to pay for ads that humans can actually see? Be careful what you wish for – today, that may be like cutting off your nose to spite your face. “They believe they should have always been paying for the viewable ...

image
MEC’s Shenan Reed: Programmatic Need Demystifying

FORT LAUDERDALE – Programmatic buying is a more automated process, but that doesn’t mean it’s less expensive, says Shenan Reed, President of Digital North America at MEC, at the Beet.TV executive retreat this past weekend. However, that is a popular misconception among some brands. ...

michael glantz revised
Simulmedia’s Glantz: We are Measuring Business Outcome from TV Ads

FORT LAUDERDALE – Advertising technology company Simulmedia is aiming to link business outcomes to TV ads, says Michael Glantz, Manager of Business Development at Simulmedia at the Beet.TV executive retreat this past weekend. TV ads have been measured traditionally on reach and frequency, but layering ...

image
Xaxis and the Wharton Future of Advertising Program Join Beet.TV for Video Series of Industry Influencers

Beet.TV has begun production of a 50-part series of personal interviews with noted advertising and media luminaries. These will be short videos of individuals who have impacted the course of the industry. Titled the “Media Revolutionaries,” the series is being co-produced with WPP’s Xaxis ...

image
It’s Basic: Better Ad Viewability Drives Better Sales: comScore’s Fulgoni

FORT LAUDERDALE — After the discovery that some advertisers were paying for video ads which were not being viewed by actual consumers, the Media Ratings Council last year defined a standard for “viewability”. Now some brands are finding a link between increased viewability and improved ...

image
Content Indexing As Important As Traditional Data Collection: Veenome CEO

LAS VEGAS – The content that a viewer chooses to watch may be just as, if not more, important for targeting as traditional demographic data collected about said viewer, says Kevin Lenane, founder and CEO of Veenome. This concept, which Veenome focuses on, is termed the “viewing moment.” By ...

Screen Shot 2015-01-28 at 11.59.12 AM
AOL is Linking Content and Audience, Martinez explains

FORT LAUDERDALE – One of AOL’s goals these days is to help guide ad partners through the complexity of today’s digital marketplace, says Marta Martinez, Global Head of Video Sales, AOL Inc., at the Beet.TV executive retreat this past weekend. “A lot of my role is education and ...

image
Mediaocean Pursues Convergence Tools in 2015

FORT LAUDERDALE –  Media agency software giant Mediaocean has its sights set on convergence tools in 2015, says Cordie DePascale, VP Product & Partner Solutions at Mediaocean, at the Beet.TV executive retreat this past weekend. The company recently struck a partnership with Videology to integrate ...

loader