Google to Whining Publishers: Use a Robot to Block our Spiders

By on 05/14/2009 1:36 PM @beet_tv

MOUNTAIN VIEW, CA — It must not be not easy for the Google crew to be called "parasites or tech tapeworms in the intestines of the internet" by Robert Thompson, the managing editor of the Wall Street Journal

Well fine, if publishers don't want their content crawled, they can easily tag their content with a "Robot" which blocks Google's spiders from crawling and indexing their pages, says Google spokesman Gabriel Stricker in this interview with Beet.TV

Blocking search engines is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol.  Google explains how Robot.txt is used here.

Gabriel told us that the reports coming out of a recent meeting of newspaper publishers was "confusing," and that Google sends 1 billion clicks per month to the world's newspapers.  He says the company hopes to work with newspapers in helping them make money in an expanding online universe.

Andy Plesser, Executive Producer

Here is our story on the Huffington Post.

Video Transcript

Gabriel Stricker:  From all Google properties we send about a billion clicks per month to newspaper publishers. So imagine that's the equivalent of roughly every single second of every single day of every single month, a newspaper article sort of being opened up and focused on for a user somewhere. It's a billion times a month. And that's what our role is in this is–to be the conduit that connects the user who's sort of looking for something in the news space, in that case, finds the answer that they're looking for in the news space, and connecting them to the original content.

The law Fair Use allows us to show a snippet view; and the snippet view, again, is just to headline and a little bit of text, a teaser of text for what the complete story represents. And so for users, they see just that snippet and they click on it and it takes them through to the original source. The news was kind of confusing. I think that there, that it was a week of sort of conflicting stories and so…I think, so for example, and this was an interesting one, there are some reports about different internet and I suppose there were different reports about the role of search engines and the different responsibilities that they have in terms of the content of newspapers and, and journalism as a whole, and I think you'd find that all search engines fundamentally occupy that space of being the conduit connecting users to the end publisher. As far as questions about well who's using this content fairly or not fairly, are people taking that content unfairly, I think there's an important thing to remember as far as that's concerned. Today there are web standards including one that's called Robots.txt. I know it's a little bit technical, but basically what that entails is there's this standard that allows a publisher of any kind, it could be a website, it could be a newspaper, to just put a little tag on a site, on a section, on an individual article, which sends the message to search engines, "Do not call me." "Don't go here" is in essence what it it says. Think of it as sort of a key that search engines give to publishers to allow publishers to lock up their information if they don't want it to be crawled and indexed.

I think that what you're seeing is news publishers in, going through really really serious times and trying to figure out, kind of, what the essence of the problems are so they can come up with solutions. Now, we actually view ourselves as being part of the solution, namely sending all this traffic their way. And one thing that I think has to, that you really have to acknowledge in all this, by the way, is just because, so the Seattle PI, for example, has gone purely online. They don't publish offline. One thing that that you have to understand is just because you do absolutely see certain papers that are no longer publishing offline, and there are struggles in that industry. From what we see you should not interpret that as, as that that material that that journalistic content is unpopular; it's wildly popular. People are going and consuming it in droves and a lot…there are numerous sites that have seen increases in their traffic. The challenge is working together with, with news publishers to figure out ways to make money off of that content that's on line once all those readers get there.

To just put a little tag. On. Site on a section on an individual article. Which sends the message to search engines. Do not call me. Don't go here is in essence what it it says think of it as sort of a key. That search engines — to publishers. To allow publishers to lock up their information if they don't want it to be crawled and indexed I think that what you're seeing is is is. News publishers in going through really really serious times and trying to figure out. And of what the essence of the problems are so they can come up with solutions now. We actually you ourselves being part of the solution. Namely sending all this traffic airway and one thing that they passed it. You really have to acknowledge and all this by the way is. Just because so that the Seattle.

One thing that that you have to understand is just because you do absolutely see certain papers that are no longer Republican offline. I'm and there are struggles and an industry. From what we see. You should not interpret that is as that that material. — that journalistic content is unpopular it's. Wildly popular people are going in consuming intro was in a lot there are numerous sites that is seen increases in their traffic. It challenges working together. With with news publishers to figure out ways to make money. Off of that content that's on line once all those readers got there.

Recent Videos
image
The Viral Video Is Not a Business Model

While the viral video hit can be golden for brands, it can’t be done consistently and it’s “not a sustainable business model” says Mitchell Reichgut, CEO and founder of the Jun Group, a digital video ad services firm that secures views for branded content on hundreds of publisher ...

image
TubeMogul Crushes Wall Street with 64% Rise on First Day of Trading

Whether the dramatic increase of the company’s stock price on its initial day of trading, which closed at 11.50, up $4.50 or 64%, can be sustained or not, time will tell.  For TubeMogul co-founder and CEO Brett Wilson, he is focused on “building a big, transformative company,” he told ...

image
Native Video Advertising on Mobile Devices Powers Growth for Jun Group

The dramatic rise of mobile, opt-in video advertising has helped fuel revenue growth for the New York-based adtech firm Jun Group by 120% in 2013 and the company is on pace for that kind of growth this year as well, says CEO Mitchell Reichgut  in this interview with Beet.TV. In the last few years, Jun ...

image
Simulmedia’s Morgan: TV Finally Embracing Ad Tech

Tacoda and Simulmedia founder Dave Morgan is just back from a fun week chewing over the TV ad tech revolution at Cannes Lions with Beet.TV He is interviewed on stage by digital media investment banker Terence Kwaja. In this AdAge article, Morgan concludes channel fragmentation, a desire for accountability ...

image
Comcast Serves up 13 Million Video Streams of World Cup: “TV Everywhere is Ready for its Close-up”

Comcast has provided  13 million video streams of World Cup matches to subscribers watching via digital devices.   That’s up  55% over the number of streams for the Soichi games, the cable giant says. The enormity of the digital consumption represents something of a watershed moment for “TV ...

image
Sling Rolls Out New Box With Added Mobile Features

Echostar-owned Sling Media introduced a new box to deliver pay TV channels to other devices. The “Slingbox M1″ and “SlingTVTM” are designed to be mobile friendly and are slated to retail for $149.99.  Here is the company press release. In the last year, Sling Media has been focusing ...

image
Digital Targeting Enables Consumer Choice in Storytelling, MediaCom’s Savic

CANNES, France — In a digital world, where transparency is the name of the game, brands can’t hide anymore. What works and doesn’t work is apparent every day, says Sasha Savic, CEO of MediaCom USA, during an interview with Beet.TV at Cannes. “We used to love our ideas. Now, it ...

image
Britain’s Mail Online, Growing Fast in U.S……and Making Waves

The British tabloid the Daily Mail, known as Mail Online,  has become the biggest English language newspaper destination site in the world.  It is growing quickly in the U.S., with a 30 percent  increase over last year, and monthly unique visitors up to 36,4 million, according to comScore.   That growth ...

image
Video Conference Platform ‘Vidyo’ Raises $20 in New Round

Vidyo, the live video platform solution that powers Google Hangouts and a number of other enterprise-level implementations. has raised an E round of $20 million, the company announced today.   Last year we sat down with VP Ray Glynn about the company and the opportunities to implement live streaming across ...

image
Web Originals from Netflix, AOL, Crackle Get Emmy Nods

Netflix, AOL and Crackle were among the digital-first video producers to get recognized with a slew of Emmy nominations which were announced yesterday.   Netflix received several nominations for its big budgeted series.    In the category of Outstanding Short-Form, Nonfiction Program, Crackle’s ...

image
DMEXCO Expects 30,000 Attendees for Giant Digital Media Expo in Cologne

It may not have the sunshine and parties of Cannes Lions, or the glittery nightlife of CES, but DMEXCO (digital media expo) in Cologne is where business gets done for the digital media industry, says Christian Muche, co-founder of the giant two-day conference in September which will attract  a ...

image
Mobile Ratings Next for Nielsen

The TV ratings giant is prepping to add mobile ratings to its product mix. The goal for Nielsen? To weave mobile into TV ratings in time for the fall season, said Steve Hasker, President of Nielsen Global Products, in an interview with Beet.TV this spring. The addition of mobile to the TV ratings set can ...

loader