Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Microsft Dashes To MPEG-DASH Support: Sodagar

LAS VEGAS — DASH by name, dash by nature? The draft international standard for MPEG-DASH, the video standard bringing higher-quality video and audio to digital media by using adaptive bitrate streaming, was first specced out in 2011. Now it is fast being adopted by big-name video server vendors. ...

image
Silverlight’s Chrome Exit Will Spur MPEG-DASH: Bitmovin’s Lederer

LAS VEGAS — Google’s recent decision to end Chrome browser support for Netscape Plugin Application Programming Interface and, with it, for Microsoft’s Silverlight video format, will spur adoption of fast-growing rival format MPEG-DASH, says one online video boss. “We will see a lot ...

image
Akamai Touts Its Video Player, Now Used by Fox News and the Weather Channel

LAS VEGAS – Widely known as a global content delivery network, Akamai has been evolve its media offering to include a video player which is being used by some of the biggest online video publishers including Fox News and the Weather Channel, explains Frank Paolino who heads the video player development ...

image
Yahoo Has Data from 1.5 Billion Mobile Users via Flurry Integration

Consumption of video is moving increasingly to mobile apps and Yahoo can provide brands with deep targeting data around 1.5  billion global users, now that the integration with mobile analytics firm Flurry is complete, says Andrew Snyder, VP of Video Sales at Yahoo, in this interview with Beet.TV Yahoo ...

image
Yahoo’s Big Data is Now “Lit Up” on Brightroll

LAS VEGAS –  Following its acquisition by Yahoo late last year, programmatic video adtech company Brightroll is now using data from Yahoo’s one billion global users, says Brent Horowitz, VP of Business Development at Brightroll, in this interview with Beet.TV. He says that the merger has also ...

image
Programmatic TV Yields More Targeted Ad Outcomes, AOL’s Ackerman

As programmatic buying starts to take root in TV, industry players are finding benefits both in efficiency in the buy as well as in the outcome, says Dan Ackerman, Senior VP of ONE by AOL, in an interview with Beet.TV about the future of TV advertising. “We are seeing the benefits for programmatic on ...

image
TV Ad Firm Sees Growth in Addressable Buying, Invision’s Marshall

Addressable delivery of ads is on the rise by early adopters in recent years, says Steve Marshall, CEO of multi-platform advertising sales software firm Invision, in an interview with Beet.TV. Many MVPDs are moving aggressively to sell their inventory on an addressable basis, he says. “We are focused ...

image
Marketers Want Better Cross-Screen Attribution, Havas’ Keller

One of the biggest goals for marketers today lies in achieving better attribution across screens, says Melissa Keller, EVP, Channel Investments at Havas Media, in an interview with Beet.TV about the future of TV advertising. “As a media agency, our goal is to guide them through the right media mix and ...

image
“TV Everywhere” is Key Weapon for Cable Nets vs. OTT Players

LAS VEGAS – Faced with an onslaught of successful offerings from the likes of Netflix, the cable network industry is pushing back by embracing “TV Everywhere,” explains Will Richmond, editor & publisher of VideoNuze in this interview with Beet.TV He paraphrases the on-stage comments by ...

image
Presidential Hopefuls Are Turning To Programmatic TV Ads: YuMe’s McLernon

If you think it is too soon for so-called “programmatic” advertising buying techniques to jump from the web to mainstream television, consider this – the future president of the United States may already be buying TV ads this way. That is according to video ad tech firm YuMe. Its ...

image
comScore to Release Cross-Screen Video Measurement – Linear TV Measurement Next

LAS VEGAS –  Earlier this year, comScore announced plans to measure video impact across desktop, mobile, tablet and OTT with a product called Video Metrix Multi-Platform.  It will soon be launched.  Next up will be a product to measure how linear television interacts with digital, says Joan ...

image
Intel’s “Thunderbolt” Powers Big Video Workflow Across Devices, Fast

LAS VEGAS – File transfer of large video files between various production and storage devices are moving at 20 GigaBits per second via Thunderbolt 2, a technology platform created by Intel. At the NAB show, Intel demonstrated the technology which it says is 20 times faster that FireWire.  First ...

loader