Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
IAB’s Shah: Too Many Viewability Platforms Will Hurt Industry

Viewability, the recently-defined metric that defines whether an online ad slot is really viewable by consumers is a wholesale change for the industry – but its benefits may be blunted as too many platforms are vying to help advertisers capitalize, according to Ziff Davis CEO and IAB chair Vivek Shah. ...

image
Marketers Can Reach ‘Meerkats’ With Mobile: Weather Channel’s Dan Young

LONDON — Mobile video heralds a chance for advertisers to reach consumers who may appear to be distracted or in the outdoors – but who are actually highly engaged and watching TV, according to Weather Channel international yield and programmatic Daniel Young. “It’s personal,” Young ...

image
Ooyala Buying Videoplaza To Enhance Video Advertising Breadth

They are two of the best-known names in online video service provision – and now they are coming together as one. Video streaming service Ooyala, itself now owned by Australian telco Telstra, is acquiring its video advertising peer Videoplaza. The buying company says the deal “allows Ooyala to ...

image
Videology Opens TV Practice To Make Buying Smarter

Videology has established a strong profile for itself by helping TV ad buyers make intelligent decisions to buy online video ads. Now it wants to help them buy good old fashioned TV ads in the same way. The New York-based company is opening a dedicated TV practice. As CEO Scott Ferber tells The Wall Street ...

image
Interstitial Better Than Auto-Play: Jun CEO Mulls New Mobile Ads

CHICAGO – Ad network Jun Group’s CEO Mitchell Reichgut “recoils” when he hears about auto-playing ads, the new format being picked by Facebook and others to deliver video messages. But that doesn’t mean he won’t try introducing interruptive ads of his own. ...

image
Facebook is Redefining the Value of of Video Autoplay, Vivek Shah

Historically, videos which start automatically on Web pages and in mobile apps can be offensive to users, but the approach taken by Facebook by setting videos to autoplay without sound, is an important step forward for publishers and marketers, says Vivek Shah, chairman of the IAB and CEO of Ziff Davis, in ...

image
Mindshare’s Cridlin to Brands: Stop Seeking Fans on Facebook — “Organic Reach Is Non-Existent”

CHICAGO – Put your money where your mouth is. That’s the new mantra of digital ad execs who believe gaining free audiences by building viral social audiences is over. “There is an urban myth of organic reach, of something going viral,” Mindshare digital innovation and strategy MD Jim ...

image
AETNA Allocates Most Ad Spend to Digital; Big Piece is Programmatic Video

For the giant insurer AETNA, a majority of its advertising budget is spent on digital media and at least 10 percent of the total spend going to programmatic video, says Gary Templeton, head of media,  in this interview with Beet.TV With advent of the Affordable Health Care Act, AETNA has been focused  on ...

image
Brand Pair-Ups with Publishers Can Drive Key Metrics, Maxus’ Bahler

CHICAGO — Working with a publisher partner can be a smart strategy for branded content, especially to drive awareness and visibility, says Spencer Bahler, Managing Director-Chicago at Maxus in an interview with Beet.TV. “We have some challenger brands we represent and we look to partnerships to ...

image
Leo Burnett’s Geraghty on Branded Content Breakthroughs for Coke, P&G

CHICAGO — Coca-Cola’s groundbreaking small world machines campaign has won awards and reached around the world, but it wasn’t easy to pull off. The goal of the branded content effort was to showcase corporate responsibility but also bridge borders, says Vincent Geraghty, EVP and Executive ...

image
Attribution Is Changing The Currency Of Media: Adap.tv’s Ackerman

Marketers may be moving in to an era when they no longer pay for advertising based on the likelihood of reaching a number of people of a particular kind – but based on actual end product sales as a result. AOL recently acquired two companies in the “attribution” space which will enable this ...

image
Mobiles Are Now Subscribers’ TVs & DVRs: Comcast’s Strauss

In the age of smart glass everywhere, what’s the difference between a TV and a phone anymore? Comcast Cable wants to give its subscribers value by offering them content from their home TV on devices out of home. In this video interview, the company’s video services SVP and GM Matt Strauss tells ...

loader