New YouTube Auto Captioning Aids Search, Usability, and the Hearing Impaired

By on 12/17/2009 10:40 PM @daisywhitney

MOUNTAIN VIEW, CA — YouTube recently added automatic captioning to its videos, in a move that has far-reaching implications for the deaf and hard-of-hearing, international users, and publishers who seek increase search optimization.

I traveled down to Google’s headquarters yesterday to meet with Ken Harrenstien, a software engineer at Google who helped develop the system.

While YouTube has offered captioning capabilities since 2006, the new features make the process of adding them easier on some videos and also lets creators time up the captions to the spoken words in a video, Harrenstien explained through an interpreter. (His wife Lori served as his interpreter during our visit.)

In addition, the captions can be translated into 51 languages making many English-language videos accessible around the world now.

But there’s also a strong business case to be made for captions because they improve the searchability of videos. “Because captions are text — guess what? You can search them. And at Google we can use that to find out where exactly in a video there is a short snippet,” he explained.

Adding captions can help grow views for publishers, he said. “Hopefully it will make it easier to add captions and I think very soon we will start to notice that it really makes a difference in how many people watch their videos.”

The technology works by linking Google’s automatic speech recognition technology with the YouTube caption system, Harrenstien explained. The automatic captioning is only available on a few channels, including PBS, National Geographic, and Duke University. In time, the goal is to expand that capability to all videos, we are told.

But any YouTube creator can use the automatic timing tool, which lets a user upload text when they post a video. YouTube then matches the spoken words to the written ones creating captions, Harrenstien explained.

From a personal standpoint, Harrenstien said he has dreamed about this technology for a long time. Because he has three young children, he doesn’t have much time to watch videos, but now he said he can understand the ones they are watching.

Here’s a story on the new captioning program by Miguel Helft of The New York Times.

Daisy Whitney, Senior Producer

Update 12/21:  We have created this transcript below and uploaded it to YouTube which created the automated captions in the YouTube video we have posted here.  Came out really well.

Video Transcript

Daisy Whitney:  Hey there, I’m Daisy Whitney reporting for Beet.TV and NBC Bay Area here at Google’s headquarters. Google recently introduced automatic captioning for its videos on YouTube and we’re going to talk about what that means and what it means for users with a software engineer.

So Ken, you have some interesting updates with what’s happening with YouTube and captions. Can you walk us through some of the new products, updates that you have?

Ken Harrenstien:  Sure. Two things we just announced a few weeks ago: 1) automatic captioning and 2) automatic timing. And to me it’s some of the most important things that we’ve done so far. Obviously the captions themselves are very important for many reasons. Would you like me to explain the reasons?

Daisy Whitney: Yes I would.

Ken Harrenstien:
  Sure. Okay. Well obviously we need captions for someone like myself when I watch video. Two, there’s other people who speak a different language but can understand English if they read it or another language, and the captions allow us to translate the language to theirs. So it’s not just access to me, a deaf person, but it’s access for the world. The third, we have people who, for whatever reason, like to have the sound off. Maybe their busy, maybe they’re on a train, on a bus, maybe it’s at night and they don’t want to wake up somebody.

And forth, because captions are text, guess what, you can search on that text. And at Google, we can use that, for example, to find exactly where in the video you have a special snippet and jump straight to that part of the video. And for a short video maybe it doesn’t matter, but for a movie, ooh wow, it really helps a lot. And if you’re searching for a class lecture on physics, maybe you want to skip the boring parts and go straight to what you need to know.

Daisy Whitney:  So it improves searchability on YouTube?

Ken Harrenstien:  Yes. Now, I need to follow up. Two things I want to add. Basically it makes it easier to create captions for videos because we have so much content now. The number is every minute we have 20 hours of more video being uploaded to YouTube. That’s a lot. I mean, a lot of videos are already captioned, but it’s just a very small amount compared to what’s coming in.

So I figured, fortunately the timing is right to hook up a lot of important things at Google and hook them together to make this happen. We linked speech recognition to the technology. Suppose you have a video; it doesn’t yet have captions. The captions are uploaded by the people who own the video. I think it’s easy, but for many people it’s not easy. They have to type the words, they have to find the time, they have to upload the file, so some people don’t bother. But now, if we can recognize the speech in the video, we can generate captions for you. That’s early–it’s only available on a few educational channels and government channels and we’re trying to roll it out as fast as we can.

But the second thing, automatic timing. Suppose okay you’re willing to type in the words for the transcript–that’s all you have to do. You upload that file and we will use speech recognition to find out when those words were spoken and change them into captions. We actually use that technique ourselves at Google for a lot of YouTube videos.

Daisy Whitney:
  So if I upload this video to YouTube, what do I need to do as a user? Do I need to create a transcript and upload that transcript so that there can be captions for this video for instance?

Ken Harrenstien:
  Okay, there’s two things you can do. First, if you were one of the partners for whom this is enabled, then we will do speech recognition for you. It won’t be perfect, but you can take that, fix it up, and upload it as real captions and it will be perfect. Or the second thing we can do. Often you can write down the transcript of what people say in the video, upload that file, and we will do the rest. And I hope you do!

Daisy Whitney:
  Who are your partners right now for whom you do the automatic captioning? And is your goal at some point to do automatic captioning for every video? Is that even possible?

Ken Harrenstien:  Obviously because of our, I mean Google’s, mission statement, we want to make everything accessible to everyone. And we will get there someday, maybe. But for now, the list of partners, we have that on our blog post. We just recently added a lot more. Sorry I forget the list.

Daisy Whitney:
We’ll get it. We’ll look it up. What do you see as the value for publishers and content owners?

Ken Harrenstien:  Hopefully we’ll make it so much easier for them to add captions and then they will. And if they can’t, for whatever reason, we will do the best we can. And I think very soon we will start to notice that it really makes a difference in how many people watch their videos.

Daisy Whitney:  More views. What does it mean for you personally?

Ken Harrenstien: Well I’ve dreamed about this for many years of course. It’s a funny thing, I don’t have a lot of time to watch videos myself because I have three small children, but if they want to watch something, I like to understand what they’re watching too. So it’s great.

Daisy Whitney:
That is great. Thank you so much.

Recent Videos
image
Vice Sees Huge Upside for News Operation, Brian Dietz Tells Beet.TV

In just the past six month, Vice has become “one of the biggest newscasts in the country and possibly the world,” says Vice SVP for sales Brian Dietz in this segment on Beet.TV.  Dietz tells our Ashley Swartz that Vice news has some 650K subscribers on YouTube and has had 70 million videos ...

Ashley Swartz, Furious Minds 2
VOD Could Shepherd More Programmatic Buying in TV, Ashley Swartz

Video-on-demand inventory is the next logical entry point for programmatic style advertising into linear TV, says Ashley J. Swartz, CEO and Founder of Furious Minds, in an interview with Beet.TV. “VOD is the first inventory that will come online for dynamic ad insertion,” she says. “The ...

image
Razorfish Media Chief Kathuria on Scale, TV Dollars, and Big Data

Moving TV dollars to online remains a challenge says Vik Kathuria, Global Chief Media Officer at Razorfish in an interview with Beet.TV. “It’s very hard to find an apples to apples comparison in terms of TV money and where to move it. If you are a TV buyer, you want to scale. To effectively move ...

Steve Parker, SMG UK
Programmatic Poised to Impact TV Buying, SMG’s Steve Parker says

LONDON — Programmatic advertising not only has the potential to influence digital advertising, but also TV, says Steve Parker, co-CEO of Starcom MediaVest Group in London, in an interview with Beet.TV. “What we’re learning around programmatic video allows us to transform our relationship ...

image
More Agencies to Launch Private Exchanges, Zenith’s Zilberbrand

As programmatic buying expands, look for agencies to build more private marketplaces in the next 12 to 18 months, says Julian Zilberbrand, EVP Activation Standards, Insights, & Technology at ZenithOptimedia Group in an interview with Beet.TV. That’s because private exchanges provide an ecosystem ...

image
Programmatic TV Buying on the Horizon

Programmatic advertising has flourished digitally, but in time it will spread further to TV, says Brendan Condon, CEO of Media Property Holdings, which operates like an ad network but for TV buying by aggregating TV spend across multichannel video providers, syndicators and local broadcasters. But there are ...

image
Ad Tech Lets Advertisers Run Their Own Show: MediaMath’s Cox

Larger advertisers are beginning to realize that, with a range of new online technology, they can disintermediate their agencies to take more control of their own campaign spending, says an ad tech exec. “We’re seeing an increase in operators wanting to run their own marketing programs directly using ...

image
2014 Is ‘The Year Of Private Marketplaces’: BrightRoll’s Avila

The next stage in the adoption of  automated ad-trading tools, private marketplaces are helping so-called “programmatic” practices flourish, says an exec in the space. “If 2013 was the year of real-time bidding and programmatic coming in to reality, 2014 is the year of private ...

Stefan Maris, Civolution
Publishers May Soon Sell TV-Synced Ads: Civolution’s Maris

Nowadays, advertisers are starting to buy ads in social media streams that are synchronized with TV ads. Soon, they may also get to buy display ads on premium publisher sites in the same way. “2014 was all about getting the pipes in place,” says Stefan Maris, global product manager of Civolution, ...

image
“Utopia” Of Programmatic TV is Getting Closer: TubeMogul’s Eadie

The holy grail of mass-market TV through which advertisers can target individual viewers is coming nearer as better and better ad tech improves prospects, says one company playing in this emerging area. “We obviously don’t have that utopic state in linear TV right now,” says TubeMogul CMO Keith ...

image
Key Focus for Programmatic Should Be Business Outcomes, Razorfish’s Kathuria

The key focus of programmatic buying should be on delivering business outcomes for a marketer, says Vik Kathuria, Global Chief Media Officer at Razorfish in an interview with Beet.TV. If an agency can do that, then it makes sense to keep programmatic buying in house, he says. In the coming year, he predicts ...

image
Amazon to Buy Live Video Portal Twitch for $1Billion, report

Twitch, the live video platform that allows gamers to interact over  the Internet, will be acquired by Amazon for over $1 billion, the Wall Street Journal reports.   the site was formerly known as Justin.tv, a pioneering platform for personal live streaming. Last year, we spoke with chief revenue officer ...

loader