Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Facebook Taps Innovid for Atlas Ad Serving

As one several announcements this week around the relaunch of its Atlas advertising platform, Facebook has tapped video adtech company Innovid for ad serving. Earlier this week, we spoke with Innovid co-founder and CTO Tal Chalozin about the partnership with Facebook. Here is the Innovid press release.

image
Starcom’s Richman: Cross-Channel Attribution is Critical

Data is the foundational playbook for marketing today, says Amanda Richman, President, Investment and Activation, Starcom USA in an interview with Beet.TV. Data provides the power to make marketing more effective and to help brands connect more deeply with audiences, she says. That’s why it’s ...

image
Havas Links with AOL for Programmatic TV Ad Delivery Solution

Media agency Havas has expanded its integration with AOL’s programmatic ad delivery platform to include TV ad buying, the companies announced earlier this week at the AOL Programmatic New Front event. For an overview on the growth of programmatic and ad buying, and the expansion of the alliance with ...

image
Omnicom Links with Facebook for Targeted Ad Delivery via Atlas

Omnicom has signed an agreement with Facebook as the first agency holding company to work with Facebook’s Atlas for its ad serving platform, the company announced earlier this week in New York. At the IAB MIXX conference, we spoke with Jonathan Nelson, CEO of Omnicom Digital about the alliance and the ...

image
AOL Links with Publicis in Latest Programmatic Agency Alliance

AOL Inc. has been working closely with major media agencies including IPG Mediabrands and Havas Media around its programmatic offering. This week, it announced an alliance with the Publicis Groupe.   According to a company statement,  ”The move will provide VivaKi, and effectively all agencies inside ...

image
BSkyB Plans Bigger Footprint For ‘AdSmart’ Addressable TV

LONDON – British satellite TV and telco operator BSkyB hopes its AdSmart technology – one of the world’s largest real-world deployments of addressable TV – will soon extend to more TV channels and more countries. Launched in January after lengthy gestation, AdSmart sees different ads ...

image
Buying TV And Video Improves Campaigns: Amnet’s Hobson

LONDON — In the online advertising industry, many professionals expect advertisers to start moving some of their TV ad budgets in to the fast-growing world of online video, where metrics give them more visible insight in to effectiveness. But not everyone sees the trend as an “either-or”. ...

image
Australia, Europe Lead Private Programmatic Marketplace Adoption, Adap.tv’s Duffield

LONDON — Private-marketplaces, the next generation of programmatic ad-trading platforms, are gaining traction more quickly outside of the United States, as markets work to make up for relatively fewer available ad spots, according to a global video ad tech exec. “A market like Australia is a ...

image
Joining The Dots Is About To Get Hot: Acxiom’s Howe

COLOGNE — So much data, so many new ad tech platforms. Anyone who has seen a lumascape will recognize the confusion that fragmentation wreaks. But there is a way to pull the piece back together, says an ad tech boss. “One of the key buzzwords that we’re going to be hearing about over and ...

image
Programmatic Could Fuel All Media Insights: SpotXchange’s Merwin

COLOGNE — So far, “programmatic” mechanisms for using automation and data to buy better ads have proved useful in the online display segment. Next up, they are destined to change online video ads. But, in the future, Alex Merwin envisages programmatic underpinning entire cross-media ad ...

Amanda Richman, Starcom 1
Starcom’s Amanda Richman Sees Programmatic Expanding To More Channels

One of the biggest opportunities with programmatic lies in using it as a starting point to bring targeting and data efficiency to all channels, including TV, says Amanda Richman, President, Investment and Activation, Starcom USA in an interview with Beet.TV. “Some see programmatic as real-time ...

image
AOL Grows Programmatic Business Year Over Year, Bob Lord

AOL has seen a boost in programmatic buying from 4% a year ago to nearly 34%, says Bob Lord, Global CEO of AOL Platforms, in an interview with Beet.TV. That growth is driven by two trends — more understanding of the benefits of programmatic buying, and the availability of data management platforms to ...

loader