Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Horizon Media Tests ‘Advanced TV’ Ad Targeting: Campanelli

AUSTIN — Media agency Horizon Media doesn’t think true programmatic ad buying is coming to television any time soon – but that isn’t stopping it from trying out some of programmatic’s techniques to understand ad targeting better nevertheless. Horizon Media national TV SVP David ...

image
The Economist Shakes Up The White Paper: Sukacheva

AUSTIN — Over the years, when most people have talked about The Economist’s online strategy, discussion has centered around its access model. Right now, that is three free articles a week – but this is not the only horse the publisher has in the race. The Economist Intelligence Unit (EIU) ...

image
Media Agency of the Future is Based on Advisory Model, Interpublic CEO Michael Roth

Keep your head down, stay focused and be competitive. That’s the advice of media industry veteran Michael I Roth, Chairman and CEO of Interpublic in this interview on the changing nature of the advertising business. “Don’t worry about this job, or someone else’s. Keep your head down ...

image
Mobile Video Ads Going Prime-Time: Opera Mediaworks CEO

AUSTIN — The online advertising world is fast moving from the medium screen to the small one. As it does, it can nevertheless recapture some of the advertising benefits boasted by the TV before them, says a mobile ad tech boss. “Mobile is now the first screen,” Opera Mediaworks CEO Mahi ...

image
CNN, the Economist, Financial Times, Reuters, the Guardian Join Programmatic “Coop” powered...

AUSTIN — Earlier in March, The Guardian announced it was leading a coalition including CNN International, the Economist the Financial Times and Reuters to pool their ad inventory for programmatic international sale. The exec whose company powers the so-called Pangea Alliance says such comings ...

image
Online Video Exploding Globally and Starcom’s Amanda Richman Explains Why

AUSTIN – Online video is showing dramatic growth.  It is by far the fastest growing ad medium. In a closely watched quarterly report titled  Advertising Expenditure Forecasts, the big media agency ZenithOptimedia has just reported that online video grew last year at a blistering rate of 34% to ...

image
AOL CEO Armstrong Looks Ahead to a Consumer-Centric Economy

If you heard the economy was about to go in to “reverse”, you would probably be pretty worried. But consumers should actually welcome the coming era with open arms, according to AOL CEO Tim Armstrong – for, what he dubs “the reverse economy” will leave them richer. ...

image
Branded Content Key Focus of Digital Media, BoA Says

AUSTIN — Advertising has moved well beyond the thirteen-week cycles of the past and into branded content as the focus for the future, says Anne Finucane, Global Chief Strategy and Marketing Officer at Bank of America, in this interview with Beet.TV. “What’s here to stay is being able to ...

image
Ogilvy’s Shelly Lazarus on Long-Term Career Success

When advertising veteran Shelly Lazarus began her career more then 40 years ago, marketers dreamed then about truly being able to communicate one to one with customers. Now, that’s a reality and presents an enormous opportunity for the industry and those leading the digital charge. “When I was ...

image
Horizon CEO Koenigsberg on the Transformation of the Media Agency

AUSTIN — The new media agency of the future needs a “white canvas mindset” says Bill Koenigsberg, President and CEO of Horizon Media, in this interview with Beet.TV. “With the way the world is changing so fast, each day you need to come in and paint a new picture. You have to keep ...

image
Starcom’s CEO Donohue & “New Establishment” Brand Airbnb

AUSTIN — One of the benefits of working with “new establishment brands” is they come to media planning without any baggage attached to them that iconic marketers often have to deal with, says Lisa Donohue, CEO at Starcom USA, in this interview with Beet.TV. As an example, she points to ...

image
Can The HuffPost Show Fill Colbert’s Gap, All The Way To Cable?

AUSTIN — With two of broadcast TV’s satirical news-meisters heading to pastures new this year, is online video yet in a position to move in and occupy the space? We may find out on April 3. That’s when Huffington Post launches The HuffPost Show – but, in time, the online media ...

loader