Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Publicis’ Lévy Says, In Future, ‘Ideas Will Be King’

The whole of the communications business is being shaken up – and yet, one thing will remain exactly the same. “In 10 years time, as today, ideas will be king,” according to ad agency holding group Publicis’ CEO Maurice Lévy. Speaking with Beet.TV, he envisages a future industry in ...

image
Rubicon’s Stevens Sees Retailers Becoming Publishers

CANNES — If you thought it was just media companies that were selling automatic online advertising, think again. Retailers are also becoming media companies, and they are using programmatic to do it. Programmatic ad tech vendor Rubicon Project’s international GM Jay Stevens tells Beet.TV he ...

image
Video and Mobile Primed for Media Disruption, Havas’ Dominique Delport

CANNES — Video will become ubiquitous and will disrupt many aspects of the media landscape, says Dominique Delport, Global Managing Director of Havas Media Group, in this interview with Beet.TV about the future of video advertising. Consumers are engaging with video on nearly every platform and ...

image
Embrace Programmatic or Be Left Behind, Carat’s Jarvie

CANNES — It’s a given that the massive change in consumer habits and their interaction with new devices is causing great ripples in the ad business too. The byproduct of that change is the widespread creation of new technologies, says Euan Jarvie, Executive Director at Carat Global, in an ...

image
Teads Finds Success with Bet on Video Viewability

CANNES – When Paris-based Teads launched back in 2011, there was very little conversation about the viewability problems around video advertising, but that has changed and the bet by co-founder Bertrand Quesada looks like it is paying off. The company has recently raised a $30 million round and has ...

image
We Are “Short Step” to Full Automation of Advertising Transactions, PHD’s Cooper

CANNES – The  rate of change in the automation of advertising buying and selling and the industry will reach full automation in short order, say Mike Cooper, Worldwide CEO of Omnicom’s PHD media agency unit.   In the interview he also comments on the unprecedented number of media agencies in ...

image
“Xaxis Sync” Reached Short List at Cannes Lions: Caspar Schlickum Explains Why it Matters

CANNES – Powering real-time advertising buying is determined  by a number of signals, including data on which commercials are being aired at a particular time. Xaxis, the automated media buying unit of GroupM, launched a product last year called Xaxis Sync. It  synchronizes TV ads and serves related ...

image
Programmatic Will Transform Outdoor Advertising: VivaKi’s Hopwood

CANNES — Programmatic buying is still almost entirely associated with digital media, but the time might not be far off when creative is routinely served in real time on outdoor advertising displays. That’s what Danny Hopwood, VivaKi’s VP-solutions and platform operations for EMEA, envisions. He ...

image
Dispatch from Cannes: “This was the year of Snapchat,” Mindshare’s Johnston

CANNES – Every year at the Cannes Festival  there is a new player that captures the attention of media world.  “This was the year of Snapchat,” says Norm Johnston, Global Chief Digital Officer of Mindshare, in this interview with Beet.TV He says that Snapchat has proven its value to ...

image
Addressable TV Advertising is Here, Finally, says AT&T’s AdWorks Dunsche

CANNES – No longer pie-in-the sky, the long-anticipated technology enabling brands to deliver TV ads  to specific households is finally becoming a reality for cable and satellite operators including AT&T, says Maria Mandel Dunsche VP of Marketing for AT&T AdWorks, in this interview with ...

image
AdTech Firms Need to Adhere to Attribution Standards, GroupM’s Tilds

CANNES — Agencies like Group M are requiring that technology partners adhere to baseline standards when it comes to the measurement area of attribution modeling, says Cary Tilds, Chief Innovation Officer at Group M, in this interview with Beet.TV. “The one single thing I see in new technology ...

image
Bankoff Wants Vox’s Video On Mobile And Everywhere

CANNES — Vox Media‘s portfolio of eight sites may be pretty popular on their own. Together, the publisher claims around 170 million monthly uniques for the collection, to which it recently added Re/code. But CEO Jim Bankoff is thinking bigger. He wants to spread that content off from just owned ...

loader