Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Integral Ad Science Raises $67 Million Round to Fund Expansion, Combat Ad Fraud & Viewability

Integral Ad Science, the New York-based tech company that describes itself as  “the leader in quantifying digital media quality” has raised $67 million: $27 million in venture funding from Sapphire Ventures and $40 million in debt financing from Silicon Valley Bank. In this previously published ...

image
14 Chinese Companies Now Among Top 100 Most Valuable Brands: Millward Brown

Going back a decade, when Millward Brown’s annual BrandZ study tracking the 100 most valuable global brands was first released, only one Chinese company made the list. Now there are 14, including the tech giants Tencent and Alibaba. “Most of them are big brands, but they’re not truly globalized brands ...

image
4C Insights Acquires Teletrax to Drive Addressable TV Ad Decisioning

CHICAGO — The latest incarnation of a leading TV-synced online ads enabler is coming in to focus, with the announcement, by ad data company 4C Insights, that it has raised funds to acquire Teletrax, aiming to beef up its activities in social and TV advertising. “We’re playing aggressively in ...

image
Video Ad Platform Jun Group Raises $28 Million In First Venture Round

The Jun Group, a New York-based video ad platform that places ads in custom video players on publisher sites and within mobile apps, has taken its first venture round since being formed in 2005, the company announced today. The company, founded in 205, has taken some debt financing and is profitable, CEO ...

image
Hearst CEO Swartz’ Career Advice: Follow Your Passion To The Top

Steven R. Swartz isn’t the only business reporter to make it big on the business side of reporting. Sequoia venture capitalist Michael Moritz wrote for Time magazine before making tech investments, for example. But he is one of the few to have risen to the very top of the company he has worked in for so ...

image
“It’s Going to Take a Village to Produce Creative”: Tribal’s Guest

CANNES — One consequence of the rise of data-driven marketing is brands creating tens of even hundreds of ads for various audience segments for a single campaign, not just two or three like in the old days. It’s “great from a sales perspective and not so great from a marketing cost perspective,” ...

image
Data is an “Enabler” of Creativity, JWT’s Jeffrey

When J. Walter Thompson’s Non-Executive Chairman Bob Jeffrey got his start in advertising, it was normal for agencies to present a single TV spot to prospective clients in order to prove their credentials. For example, at Chiat/Day, where he worked early in his career, they showed the “1984” Apple ...

image
Addressable TV will Provide Unprecedented Granularity, SMG’s Scheppach

CANNES — So-called “addressable TV”, in which TV sets can now speak back to broadcasters, are tantalizing with the promise of household-level ad targeting. But addressable TV is not just about more refined TV targeting – the device in the living room is one platform that will feed ...

image
Careers in AdTech Will be Powered by Digital Transformation, AOL CEO Tim Armstrong

Less than 10 percent of e-commerce and advertising is powered digitally —  but that will inevitably change. And the looming transformation will provides a twenty to thirty-year “tailwind” for the entire adtech and media industry which will mean vast opportunities for young people ...

image
Big Changes in the Values of Brands Powered by Geography and Technology, Sir Martin Sorrell

In the past 10 years, since the WPP-owned global brand consultancy Millward Brown has tracked the value of brands with its BRANDZ 100 rankings, there have been many changes.  Over that period, nearly half of the companies have changed.  This upheaval  has been powered by two major factors:  geography and ...

terence kawaja
Addressable TV Advertising is Changing the “Lumascape,” Banker Kawaja

CANNES – As addressable TV advertising emerges, so are a number of companies including Visible World (recently bought by Comcast), Clypd, Simulmedia and others.  For the new medium to succeed, one essential will be companies that focus on yield management around addressable TV, says investment banker ...

image
A “Whole New World” of Video Advertising Coming to Comcast/NBCU, Driven by Set-top Data

CANNES – Comcast’s data from its twenty millions set-top boxes, coupled with marketers’ first party data, will create a “whole new world” for NBC Universal, for both advertising and content creation, explains Linda Yaccarino, Chairman, Advertising Sales and Client Partnerships ...

loader