Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
Ooyala Taps Microsoft Azure for Cloud Solutions

LAS VEGAS – Ooyala, the digital video services company, which has been expanding its offering as a “TV Everywhere” platform for big broadcasters including Univision, has entered an agreement with Microsoft for its Azure, cloud-based platform, explains Jonathan Wilner, VP for Products, in ...

image
Microsoft Unveils Skype TX, a Studio-Grade Solution for Broadcasters

LAS VEGAS – As television, digital, radio and even live events come together, Skype has found a place in the middle of that convergence, connecting people worldwide via live television and digital, says Angie Hill, general manager of consumer marketing at Skype, in this interview with Beet.TV at the 2014 ...

image
UGC-Based Broadcast is #1 New Syndicated Show in U.S.

LAS VEGAS – The manpower and resources behind local news have shrunk but the time frames have not, leading to a lot of repetition in news programs, says Phil Alvidrez, executive in charge of production for RightThisMinute, a TV show that publishes online video content – and tells the story behind it – ...

image
Dolby Readying “Cinema-Like” Audio for Mobile Video

LAS VEGAS – With the advancement in second-screen and mobile technology as companies like Apple and Amazon rolled out mobile devices that could support high-definition video, consumer expectation for high-quality sound on their tablets and phones increased, says JC Morizur, senior director of professional ...

image
Live Video Streaming Format MPEG Dash Embraced by Akamai, Google, Adobe, Microsoft others

LAS VEGAS – MPEG Dash — the emerging live video format for cross-platform streaming, has picked up considerable traction over the twelve months with the participation of major players including Google’s YouTube, Microsoft’s Azure, Adobe Primetime and Akamai, explains Will Law, Chief ...

image
Microsoft says Streaming Olympics was Turning Point for its New Cloud Offering

LAS VEGAS —  Microsoft’s global, cloud-based platform for delivering video, called Azure, is gaining momentum and customer wins in the aftermath of its successful streaming of the Winter Olympics, says Bob DeHaven, GM of Worldwide Communications and Media, in this interview with Beet.TV The ...

image
Microsoft Announces Support for Fast-Emerging Streaming Standard MPEG-Dash

LAS VEGAS  – One of the top stories from the NAB Show was the accelerated support of streaming  video standard MPEG-Dash, perhaps  most notably with the announcement from Microsoft that it will support Dash with its media solutions. At NAB, we interviewed Iraj Sodagar, Principal Multimedia Architect ...

image
Making Content Marketing Accountable To Drive Business, MediaCom’s Sean Black Explains

Content marketing has always been a part of advertising, but the big difference is that content is finally accountable, says Sean Black, Managing Partner and Group Digital Director at MediaCom, during an interview for Beet.TV with Paul Kontonis, SVP Strategy & Sales at Collective Digital Studio. ...

image
Branded Video Must be Tailored to the “Venue,” Mindshare’s Migliozzi

Branded video performs best when it’s tailored properly to the platform, says Joe Migliozzi, Digital Lead at Mindshare NA, in an interview for Beet.TV with Paul Kontonis, SVP Strategy & Sales at Collective Digital Studio. “The consumers are defining the platforms for the brand. If ...

image
Digital Planning will Come to TV Ad Buying, but Not So Fast, Nielsen’s Amit Seth

SAN FRANCISCO – While measurement tools like Nielsen’s OCR allows TV ad buyers to identify online audiences to match TV buys, which are primarily for “make goods,” there are strong headwinds against changing the way that TV ads are sold via upfronts and other conventional means, ...

image
Old Fashioned, “Mad Men” TV Ad Trading Must Stop: Mediaocean’s Woollard

SAN FRANCISCO — Programmatic ad trading techniques could eliminate costs and cut time by bringing ad buying in to the digital 21st century, says one ad tech vendor. “The way that TV is bought has been the same way since the Mad Men days – I pick up the phone, I do a deal,” says ...

image
Ad Buyers Want Linear TV Sales To Go Programatic, Experts

SAN FRANCISCO — IPG’s insight agency Magna Global is using the current TV upfronts, the annual process by which TV networks sell advertising inventory for the year ahead, to push cable networks to adopt programmatic trading techniques. Says Magna Global USA president Kristi Argyilan: ”We ...

loader