Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Header Bidding Rears Its Head With Yield And Cost: OpenX’s Saifee

PALM SPRING — Look out; there’s a new piece of ad-tech lingo on the block. So-called “header bidding” has emerged as “one of the primary drivers of growth in programmatic for publishers over the last year”, according to OpenX monetization VP Qasim Saifee. So what is it? ...

image
Nielsen Onboarding TAM As Measurement Wars Heat Up

PALM SPRINGS – After comScore’s merger with Rentrak to combine multi-platform media measurement, it’s over to you, Nielsen. Ad agency Group M chairman Irwin Gotlieb last year told Beet.TV media measurement is broken because measurers are using the wrong yardstick in the multi-screen era. ...

image
Deloitte Gets In To TV Tech Tools Game: Ledger

LAS VEGAS — These days, it seems it is no longer enough for a consulting firm to just offering consulting advice – you have to back it up with implementation, too. That’s what Deloitte is doing by partnering with three software providers to wrap up their offerings as one of its own, the ...

image
Programmatic Ad Rates Will Be Higher Than Direct: Meredith’s Schenck

When programmatic ad trading entered the marketplace in the form of real-time buying from open online ad networks, many publishers feared it would devalue their inventory. But, slowly, publishers have begun to exert controls to keep pricing higher than that. Now rates could rise higher again, one exec ...

image
One Screen Good, All Screens Better: Bloomberg’s Caine

PALM SPRINGS — Bloomberg appears to have been undergoing a renaissance lately, with a line-up of strong products and a solid identity running across its many media outposts. In the latest new advertising tie-up, Bloomberg Media has sold a sponsorship for the green room, during its new Bloomberg Go ...

image
Hearst Reaches Maturity With Programmatic: Parker

Time was, newspaper folk sneered with apprehension at the rise of programmatic methods of digital ad trading, fearing automation in real-time open markets would devalue their ad inventory. But times have changed, and those publishers which have dipped their toe in the water are finding growing returns on ...

image
As Merger Closes, Rentrak & comScore Push ‘New Model’ For Media Measurement

PALM SPRINGS — The merger of media measurement houses comScore and Rentrak just completed, heralding what the pair say is a new era in consumer tracking for brands and publishers. According to comScore, the new entity can now see 260 million desktop screens 160 million mobile phone screens 95 million ...

Screen Shot 2016-02-01 at 12.05.34 AM
Turn’s New CEO On Video And The ‘Three-Legged Stool’

PALM SPRINGS — The world of advertising demand-side platforms is so new, most people may only have dealt with one or two. Bruce Falck says he has worked for five. Falck spent just four months as Brightroll’s COO until Yahoo acquired the company, last year joining Turn, the ad tech platform which ...

image
Sourcepoint Barokas’ Melds Ad Transactions To Beat Blockers

If you believe some of the market research about the prevalence of consumer ad blocking, the sky is falling on a publishing industry that has long depended on advertising. Those surveys may or may not be wholly accurate, but Ben Barokas is intent on solving the problem. Whilst working as general manager of ...

image
Genesis Media Figures Out Out-Stream: CEO Yackanich

LAS VEGAS — When should you put an in-article video ad on a page – and when should you hold back? Two-year-old Genesis Media of New York is an ad tech platform aiming to sensitively place auto-playing out-stream video ads, a format that is gaining in popularity. “Video ad formats are easy ...

image
Adobe’s TVMM Is First OTT TV SSP: Videology’s Gaskamp

LAS VEGAS — Back in December, Adobe updated its Primetime ad platform with TV-buying capability, dubbed TV Media Management (TVMM), powered partly by Videology. The feature aims to let broadcasters plan and package up ad slots in their over-the-top (OTT) offerings. So, what is the big idea? ...

image
Netflix Could Make $8bn From Ads: Analyst Broughton

LONDON — What if Netflix started running ads? It is a topic that has previously been debated here on Beet.TV, with one exec betting: “They’re going to have to start to deploy some type of advertising model to recoup some of the revenue that they’re spending on this programming.” A new ...

loader