Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
SpotXchange Sells Majority Stake to European Broadcaster RTL

SpotXchange, the video ad tech company that provides a technology platform for publishers and broadcasters to sell advertising on an automated basis, has sold a majority of its shares to Europe’s RTL, the companies announced today. RTL, the giant broadcaster controlled by Germany’s Bertelsmann, ...

image
VICE News is “One of the Biggest Newscasts in the World”

In five months since its launch, VICE News on YouTube  is “one of the biggest newscasts in America and possibly the world,” says Brian Dietz, SVP of Sales in this Beet.TV session video.   He says that the network has over 650,000 subscribers on YouTube and has had 70 million views of its ...

image
Video Ads Perform Better On Mobile: Jun Group CEO

The booming video advertising segment stands an even better chance of ringing up success on mobile devices than it has on desktop so far. That’s according to one video ad platform boss. “Seventy percent of the video views we did were on mobile devices,” says Jun Group CEO Mitchell ...

image
Razorfish CEO: Paid Media Should Operate in Real-Time With Social

Paid, owned and earned media can work well on a real-time basis, and brands need to rethink both their allocations for paid media and the timing too, says Pete Stein, CEO of Razorfish, in an interview with Ashley J. Swartz, CEO of Furious Minds, for Beet.TV. “It used to be that paid media was ...

image
Maker Studios’ Jason Krebs: Collaboration Key for Brands & Creators

Collaboration with creators is vital for brands working with digital producers and talent, says Jason Krebs, Head of Sales at Maker Studios, in an interview with Ashley J. Swartz, CEO of Furious Minds, for Beet.TV. Maker Studios creates and distributes video content focused on millenials and its videos ...

image
Havas Dish Latino Digital Campaign Jumps to TV

Some brands are now migrating successful digital ads to the TV set, in a reversal of what was once the trend of repurposing TV spots for online video, says Rori DuBoff, Global Head of Strategy at Havas Digital, in an interview with Beet.TV. Havas recently launched a campaign for Dish Latino around the World ...

image
‘The Big Ideal’ Fuels Cross-Platform Content: Ogilvy’s Abby Marks

Nevermind your product, what do you stand for? That’s what OgilvyEntertainment’s senior strategy and operations director Abby Marks tells her clients, channeling her agency’s late founder. “David Ogilvy was always about research and treating the cons as an equal, thinking about ...

image
JWT’s Branded Content Approach Starts with Participation

Branded content is not an “if” for marketers; it’s a must-have, says Eric Weisberg, Executive Creative Director at JWT, in an interview with Beet.TV. To be effective, branded programming needs to deliver on the right combination of entertainment and utility. As an example, he points to work ...

image
Multi-Channel Network Collective Digital Studio Launches New Content for Purina

Los Angeles-based Collective Digital Studio (CDS) launched a new series of programming for its popular YouTube channel Petcentric as part of a recently minted deal with Purina, say Paul Kontonis, SVP in an interview with Beet.TV. That channel is now managed by CDS, which works with YouTube influencers to ...

image
Millennials Are Cool With Brands Changing All The Time: Initiative/IPG Study

Brands need to mimic the characteristics of millennial-age consumers to curry their favor, according to research, The Reset Generation, conducted by media agency IPG’s Initiative group. “Their expectations of brands are what they think people expect of them,” says Initiative’s US ...

image
If “Content Is King,” then “Distribution is Queen” as Paid Media is Essential,...

Mindshare, the media agency within WPP, has created some 250 branded campaigns for video and television over the past 9 years, says David Lang, Chief Content Officer and President of the agency’s entertainment division,  in this interview with Beet.TV  On the subject of viral video hits and paid vs. ...

image
Hearst is Building a “Netflix Model” Around Magazine Video Content

Hearst has just launched “CosmoBody,” a daily fitness show which is advertiser-free and is  being offered on a subscription basis.  This is the first of what will be a number of Hearst titles and Hearst partners who will be part of a new “Netflix model,” explains Chris Grosso, SVP ...

loader