Xbox Project Natal and Lip Sync

[kenmead]

O.K....so, I just checked out the Xbox Project Natal. Very Cool stuff! I'm curious if any of you know whether they are working on lip sync for live animation. The system seems to be primarily centered around full body motion-capture, facial recognition, and voice-recognition. I'd love to see this technology take the facial recognition one step further into lip sync. This would allow one to perform digital puppetry/animation in front of a live audience with body movement and lip sync...much like Jim Henson's digital puppetry, but far less expensive. Do any of you have info or comments regarding this capability? I produce media for a children's ministry in a rather large church. Live 2D or 3D animation would be an incredible addition to my work and would cut down tremendously on my production time, since I produce at least one video short (1 to 3 minutes each) per week.

[Professor S]

I've called every contact I still can get in touch at Xbox, and trolled most of the usual reliable gaming websites (used to be press for the games industry) and I can't find anything like what you are talking about. However there have been several rumors for some time that lip synching in real time was coming for the avatars. Although I'd be highly skeptical of something like this happening. The closest game to do this is "Your in the Movies" which uses the xbox live vision camera to put you in pre-scripted 'movies' and it did not sell very well - so I don't see a major developer putting time, money and resources into it. However, Project Natal could totally change the market for who is playing and buying games/apps for the xbox so it is possible that within the next few years it could happen. But don't look for an easy way to get anything you'd do this way on youtube (owned by google and thus VERY UNLIKELY to partner with MS)

[Rudiger]

I wonder if you couldn't do something like that with the new video tracking feature in Anime Studio 6?

I haven't actually tried it out yet, but it might be possible to place a group of bones placed around the mouth and use them to track the edges of the mouth. You could even put white dot stickers on your face if that made it easier. You could then use a script to threshold the relative bone positions to generate switch keyframes for the lip sync.

[InfoCentral]

Live 2D or 3D animation would be an incredible addition to my work and would cut down tremendously on my production time, since I produce at least one video short (1 to 3 minutes each) per week.

I don't know about "live" animation but there are several "real time" animation programs out there. iClone comes to mind as one of them. Final productions have the game look to them which some hate and other really relate too. It will have your X Box look and feel. For doing kids stuff I think this may be perfect match and greatly speed up your work flow especially once you start accumulating your assets.

iClone 3 Flim Making Made Easy

If you buy the boxed version it includes the Sketch Up importer for your models. It also has lip sync but you may want to look into CrazyTalk while your there to add more impact to your characters. Perhaps Crazy Talk will be all you need with its new advanced features...

[InfoCentral]

If your looking for facial motion tracking software that you can use in your productions than you may want to check out this cheap solution.

Make your own motion capture files with Zign Track!