Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
SMG’s CEO Desmond on Business Success: “Be Humble and Listen”

Riding the crest of a wave is a pretty good feeling. But the bigger lessons can come from falling off and hitting the bottom. Many a business guru is fond of the ironic value of failure. That’s a notion that the boss of one of the biggest ad groups subscribes to, too. “I’ve always learned ...

image
Xaxis’ Lesser Followed The Paternal Path To Change Advertising

Why did you get in to advertising? For Brian Lesser, it was because his dad showed him the ropes. Now Lesser runs WPP-owned Xaxis, one of the technology companies helping make advertising more targeted and more relevant. “When I was a kid, my father ran advertising agencies … I always liked the ...

519025316_c_648_367
From Mississippi Teenage DJ to Media Mogul, the Amazing Journey of Bob Pittman

Now the chairman and CEO of iHeartMedia, Inc., Bob Pittman got his start in the media business as a 15-year-old disc jockey in Mississippi, where he grew up. Motivated to earn money so he could take airplane flying lessons, he first applied to the local men’s clothing store and then tried to get a job ...

image
Privacy Concerns Slowing Ad Targeting In Europe: Furious Corp’s Swartz

Through the eyes of a US marketing exec, Europeans are, notoriously, relatively more concerned about how ad targeting might impinge on their personal data. A recent report from Belgium’s data protection agency claimed Facebook tracks computers of users without their consent, whether they are logged in ...

image
Research: Digital Video Growing On Live, Long-Form Demand

The old adage used to be that digital video had to be short, that viewers wouldn’t sit and watch it in long sessions. That may have been true on the desktop – but video ad tech company FreeWheel’s latest quarterly market report shows new devices are now driving online video ever more ...

image
Advertisers Lagging Consumers On Mobile: IPG’s Johnson

The data continues to point to a disconnect – consumers have flocked to consume content and engage socially using mobile devices, over desktop alternatives. So why have advertisers still not piled in to mobile at the same rate? “It’s sometimes an afterthought,” IPG Mediabrands global ...

image
IPG Chairman Emeritus David Bell: “An Igniter of People’s Talents”

Interpublic Group’s chairman emeritus David Bell grew up expecting to become a litigator, but when he decided at the last minute not to go to law school and turned his sights to advertising, he got lucky with admission to Leo Burnett’s training program. “I was the only one of 19 that didn’t have an ...

Irwin Gotlieb
The Thrill of the Media Biz Past, Present and Future: an Interview with Irwin Gotlieb

In Irwin Gotlieb’s view, what should make the ad industry exciting to young people is the variety of skills they’ll get to learn and clients they’ll get to serve. “If someone were to watch me in any given day, I still read scripts for television shows, I still make calls on development, and yet in ...

image
Creative Directors Are Warming Up to Data: Annalect’s Matts

Some creatives have continued to view data as an intrusion that undermines their work by valuing algorithms over instincts, but that may be starting to change. One development that creatives are excited about is the availability of programmatic technology that allows them to show hundreds of different ...

image
AdTech’s Torrid M&A Pace Powered by New Buyers, LUMA’s Kawaja

Through January to October 2014, the total value of ad tech deals totalled $3.8 billion, according to a report from investment bank Coady Diemar Partners at the time. And things are about to get bigger. Terence Kawaja, founder and CEO of media M&A advisory LUMA Partners, says his firm is advising on ...

image
AppNexus Replays Video Ambitions, Starting This Summer

It’s four years since ad tech company AppNexus first tried helping advertisers tackle targeted ad buying on digital video, an attempt it later aborted. But now the New York-based outfit is ready to reboot its video plan. AppNexus CEO Brian O’Kelley, in this video interview with Beet.TV, says: ...

image
TV-Synced Digital Ads Next Best To Addressable TV: IPG’s Stoller

We may not yet have reached the point at which so-called “addressable TV”, the ability to target individual connected televisions with unique advertising, is mass-market – but we have reached a close second, thanks to a combination of dumb TVs and smart devices. “Everyone would like ...

loader