IDEA #60 - Video Search from the Audio

By Steve Poland • May 31, 2007

I’m assuming this has to be happening somewhere. Basically, there should be a video search engine that uses speech-to-text technology (which isn’t perfect, but even if it’s 90% right, that’s better than nothing) and starts indexing videos based on what is being said in the audio. So the software would have to listen/watch every video it’s indexing — which could be a very time consuming task, given the amount of video content out there these days.

Pluggd is doing this for podcasts, so it seems it’d be fairly easy to do this for videos. It’s really neat technology — shows a heatmap in the podcast based on the keyword you’re searching for (so that you can see where in the podcast that topic is being spoken about). My only reservations are what the terms of service are on the video websites.

You could also have this technology see if it recognizes any text or images in the video — thus it might notice a picture of an apple, or a logo, or text that says something.

One problem with this search is that the video might be a comedy, but there’s no way of this technology to know it’s a comedy — it would just know the speech in the audio.

IDEA #36 - Social Audio / Video Recommendations Problem: I’m out and want to rent a movie, or buy a movie / CD / book — I’d like recommendations from friends or people with similar tastes as mine. Solution: Dial a phone number...
IDEA #42 - Audio Ads Network Provider As many of you are aware, I have posted several ideas that utilize VXML technology. I write ideas that incorporate this technology, because I feel it’s under-utilized, has great utility (the phone is something everyone...
IDEA #5 - Video & Audio Ratings of Transactions (Businesses/Products/People) How many millions of transactions are happening each day? Whether I’m looking to buy something from a local retailer, or an online retailer, or an individual on Craig’s List — I’d like to know the...
IDEA #7 - Video Comments Plugin for Blogs Michael Arrington, TechCrunch 1/14/07: “For the blogger crowd, it would be very useful to have a plugin that works directly with the standard comments feature that allows users to leave text, audio, or audio/video comments...
IDEA #47 - Video Trains and Tracking OK, so I’m still obsessed with Vidmeter — I think there’s a market for tracking videos and other ranking/rating features for videos. If you can get people to copy the embed code for a YouTube/Metacafe/Revver/etc...

Related posts brought to you by Yet Another Related Posts Plugin.

Comments

6 Responses to “IDEA #60 - Video Search from the Audio”

Colin Dowling on June 1st, 2007 9:34 am (perm link)

Interesting idea, and one that wouldn’t seem that hard to implement either through software or super cheap labor. IIRC, google uses the closed caption transcripts to index video. That said, most user generated content won’t be captioned, so there is definitely a space for this service. No idea what the actual “market” would be outside of licensing the product to video portals, but a solid idea nonetheless.
Chandra Bajpai on June 1st, 2007 10:54 am (perm link)

Great idea Steve, I’m surprised no one has done it yet, esp. with voice recognition being so advanced today. It might even be able to be cobbled together using open source components. One side benefit you could get is a transcript of the recording…which you could use to drive traffic to your video site via search engines.
James D Kirk on June 1st, 2007 1:39 pm (perm link)

You assumed correctly, Steve! I thought this was a widely known capability. People must check out the technology that nexidia.com has come up with. Simply said it’s pretty amazing. Not perfect mind you, but compared to what’s available, pretty amazing nonetheless.

Nexidia provides the technology for an Atlanta news station. Check out the “Video Now” feature at http://www.11alive.com/ and you’ll likely be blown away.
Michael on July 3rd, 2007 2:47 am (perm link)

This website is a ghost town….what happened?

..mp
Sean Scott on July 17th, 2007 9:46 am (perm link)

I would argue that the technology for speech to text is not there yet. I’ve been looking for software/asps that do speech to text well and its hard to come by. You have to remember that videos have a lot of embedded static bg noises. And that even some of the most advanced speech recognition software have an accuracy of 70 to 85%. You will still need some kind of human transcription to get it where you want it.

There are a ton of companies out there that use proprietary software to do similar service, however i’d be interested at having a good look at that proprietary process.

If it was that easy, don’t you think google video search would have come up with it by now
Don Clark -Atlanta on August 7th, 2008 1:02 am (perm link)

A year later, and this is the process we use:

Blinkx.com has been using speech recognition for some time and it works pretty well, except most of the video is SUPER compressed FLV’s.

So I use Blinkx to find the source, and then Bit Torrent and Miro to find better quality versions.