Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
IPG Mediabrands To Launch Automated Social Performance Tool

COLOGNE  –  The free ride on social media is over and brands are needing to bolster their social channels with paid ads, says Eric Weaver, Chief Social Officer for Mediabrands G14 Cluster, in an interview with Beet.TV. That’s why IPG Mediabrands is launching in November an automated solution ...

image
Different Media Platforms Demand Different Creative, Havas CDO

COLOGNE  – New platforms and buying patterns demand different forms of creativity, says Sean Lyons, Global Chief Digital Officer at Havas Worldwide, in an interview with Beet.TV. Agencies need employees who understand platforms like Facebook, Twitter, Vine and Instagram, as well as new forms of ...

image
Brands Need to Test Their Own Data in New Creative, Razorfish Cunningham

COLOGNE  – Data can drive new approaches to creative, especially if brands fold in their own insights, says John Cunningham, Chief Technology Officer EMEA at Razorfish, in an interview with Beet.TV. “Data can shake up the status quo and old world way of doing creativity,” he explains. ...

image
Audience Insight Can Drive Creativity, SMG’S Blackborn

COLOGNE  – Even though programmatic advertising is rooted in technology, it has the power to drive more creativity, says Matt Blackborn, President of Emerging Markets at SMG, in an interview with Beet.TV. “There has been such an emphasis on technology and and how programmatic can drive ...

image
Advertising Is Fueling Big Data: Quantcast CEO Feldman

COLOGNE — Big Data technology is key to the future of advertising – but is also being shaped by advertising itself, says media measurement firm Quantcast’s CEO. “Whilst the term ‘ad-tech’ has been around for a while, that term’s been a bit of an oxymoron – ...

image
Reach And Relevance Are Coming Together: Nielsen’s Bradford

COLOGNE — In online advertising, you can either target a specific but small audience like a laser, or reach a broad audience, much of which may be irrelevant. But there is an emerging class of outlet which gives advertisers the best of both worlds, says media measurement firm Nielsen’s UK SVP ...

image
“AdBlocking Goes Mainstream” and Video Publishers are Severely Impacted, PageFair/Adobe Study...

COLOGNE – The use of adblocking software by consumers, which disables banner ads also disables in-stream video ads, is on a rapid increase.   Some 27 percent of U.S. Internet users use some sort of adblocking solution when browsing, according to a study done by Dublin-based PageFair in collaboration ...

image
WPP Ups Investment in AppNexus as “The only alternative to align with Facebook or Google,” Martin...

WPP, the world’s largest adverting holding company is upping its investment in ad tech company AppNexus with a $25 million investment and will use the AppNexus technology to sell ads, the companies announced today. In an interview with the Wall Street Journal, WPP CEO Martin Sorrell says, “The only ...

image
Comcast Acquisition is Powering FreeWheel’s Global Expansion

COLOGNE – Video ad management platform FreeWheel is looking to international and ecosystem growth following its recent acquisition by cable operator Comcast. “We’re powering ad decisioning and management for 90% of broadcast community in the US,” FreeWheel’s global solutions and ...

518422769_2_648_367
Multiple Currencies will Drive Programmatic Video Growth, LiveRail’s Arnstein

COLOGNE — With Nielsen’s OCR, comScore’s vCE and others vying to be the yard stick by which buyers and sellers measure online advertising, is the industry approaching a Betamax moment, in which a single standard will win out? That doesn’t need to happen, says one digital ad exec. ...

image
Death Of TV Is Greatly Exaggerated: GroupM’s Rob Norman

COLOGNE — Some digital media transitions aren’t that pronounced and TV isn’t such a problematic medium after all, says one of the leading ad agencies’ top digital thinkers. “It’s very easy … to take a view that the fast-streaming, uninterruptable, buffer-free, ...

518417644_1_648_367
Brightcove’s New Player Fights Ad-Blockers

AMSTERDAM — Video technology vendor Brightcove last week unveiled Perform, a new service offering publishers its HTML5 video player with analytics, content protection and ad insertion built in. One other feature signifies a war on the ad blockers: “By being able to circumvent ad blockers, we ...

loader