Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
What Is ‘Premium’ Content Anyway? Ad Execs Discuss

Advertisers and their buyers often say they aspire to advertise only with “premium” publishers. But what does that really mean, and is the notion changing as ad tech evolves around it? Three execs from different parts of the industry debated for Beet.TV… Mindshare’s New York digital ...

image
Ad Groups Look To Reimagine TV On Mobile Screens

Media agencies know that TV viewing is diminishing – but at least growing consumption of video on alternative screens gives them a shot at dreaming up a new kind of moving-pictures advertising stronghold in mobile media. “People are not sitting in front of the television like they used ...

image
Holiday Video Shortage Will Spur Out-Stream Video Ads: Xaxis’ Warner

A new kind of video advertising format could pick up this coming holiday season, as ad buyers look beyond usual pre-roll inventory in a market that still suffers from supply problems. So-called “out-stream” video ads, as introduced by Ebuzzing and Teads, don’t play in video windows before ...

image
Quality Is Key For Branded Content: News Corp’s Chopra

Brands can win an audience for articles and videos they made, even in newspaper websites – if their material is worth consuming, according to a news group executive. Asked why users would consume content made by brands in vehicles designed to convey the news of the day, News Corp SVP and video head ...

image
Slate Sees Videos on All Its Pages

Web magazine Slate says it can make up for a lack of opportunities for its video advertisers by turning every text article in to a video ad opportunity. Sales VP Anthony DeMaio tells Beet.TV in this video that the title recently enlisted French ad tech firm Teads, whose technology inserts auto-playing ...

image
Research: Video Topics Affect Ad Performance, Innovid Finds

The best kind of video to put an pre-roll ad for cars is in, well, videos about cars. But the context that drives video ad engagement is not always so clear-cut. Video ad tech vendor Innovid’s just-published research in to the topic shows that, whilst auto ads placed against auto videos yielded a 40% ...

image
Marketers Need KPIs by Campaign, Channel & Individual, MediaCom’s Lampert

To succeed in a cross-device world, advertisers should develop personalized experiences for each screen with messages tailored to the device and, when possible, to the consumer, says Michael Lampert, Managing Partner and Group Digital Director at MediaCom in an interview with Beet.TV. That kind of ...

518370723_c_600_400
Content Marketing Needs Better Metrics, Rigorous Distribution, MEC’S LaVecchia

Content marketing is on the cusp of breaking through, and needs a few best practices in place to become a powerful marketing tactic, says  Gian LaVecchia, Managing Partner, Digital Content Marketing at MEC, in an interview with Beet.TV. Linking strong storytelling, with metrics and distribution is vital, he ...

image
Radio’s Cumulus Upping Video Strategy In 2015

US radio network Cumulus Media may be best known for its on-air content. But, if JT Batson has his way, the group will soon also be profiting from on-line multimedia presentations. The company’s chief revenue officer tells Beet.TV about plans for Cumulus’ Nash country music station: ”On ...

Screen Shot 2014-08-18 at 6.55.47 PM
Under News Corp, Storyful Put To Work On Brand Content

It may have started life as a tool for editors to validate user-generated content for editorial stories but, following its acquisition by News Corp, Storyful is also being set to work for advertisers’ commercial goals. “We saw a social radar,” News Corp. video SVP Rahul Chopra tells Beet.TV ...

image
BrightRoll Eager For Slice Of Linear TV Ad Business

After signing a deal with Clypd this summer allowing it to buy ad inventory for its clients from TV providers like Cox and DISH, video ad tech vendor BrightRoll reckons it can take a piece of the traditional TV advertising sector. Video is video. whether it’s own on a phone, an iPad, a television set in ...

image
Nielsen: Measuring Consumer Engagement is Key

For marketers to move more money from TV to digital, they need true parity in measurement, says David Hohman, Nielsen’s agency solutions EVP  in an interview with Beet.TV. “Parity is a consistent measurement across digital and TV. You need to be able to consider the devices, and you need to be ...

loader