Project NATAL, an integrated speech/emotional/body posture/face recognition system that utilizes a new(ish) technology that effectively means you have a MOCAP model WITHOUT green screens and ping pong balls and only one camera.
Project Natal for XBOX 360
As you can see from the video above, it certainly seems very interesting. This tech has been around for years, yet NOW it is only being intergrated into one item for the mainstream.
Now, lets see some of this in action.
E3 2009 – Project Natal – Milo Demo with Peter Molyneux
Milo is perfect example of Game AI, he dosent understand EVERY word said to him, only keywords, but there are enough to superimpose meaning
Below is an interview between EuroGamer and Peter Molyneux, CEO of Lionhead Studios, a game maker that has produced another AI centric game, Black&White and Black&White 2.
Eurogamer:
How long have you been working on Milo?
Peter Molyneux:
There are two answers to that. We started work with the Natal stuff in
December, and the first thing we did was go round all the Microsoft
people – the handwriting recognition people, the facial recognition
people, the motion recognition people. We brought the technology
together and put it in there.
So
the world you see created here has been in development since December.
Before that, we’d been working on this thing called emotional AI since
we finished the first Black and White.
Eurogamer:
Is this what you used to call Project Dmitri?
Peter Molyneux:
Yes.
Eurogamer:
I’m trying to understand how much of this comes from Lionhead and how much comes from Microsoft…
Peter Molyneux:
A lot of stuff, like the voice recognition stuff, is based on things
like Windows 7 technology. We just went round and took all that stuff
and fitted it together. The interesting thing is, a lot of that stuff
existed without reason – and when you bring it all together with
something like this, it kind of works.
Eurogamer:
So what can Milo do?
Peter Molyneux:
Milo can recognise the emotions on your face and the emotions in your
voice. He can recognise certain words you say. You can have
conversations with him, you can read stories to him. We’re trying to
bring all these things together. Some of them are tricks – I’ll be
absolutely honest with you – to make you believe Milo’s real.
He can recognise what you’re wearing. If he notices you’ve got dark bags under your eyes he will say, ‘You look tired today.’
Eurogamer:
Really?
Peter Molyneux:
Absolutely, all of that works. We’re combining all that together to
make you really believe that he understands what you say.
Let’s
try an experiment. When a human voice says something funny, there’s a
different tone in the voice. Even though Milo’s not trained to
recognise your voice, if you say something funny to him, he should
recognise it as something amusing. Try it now.
Eurogamer:
Tell him a joke, you mean?
Peter Molyneux:
Yeah.
Eurogamer:
OK. Milo?
Milo:
[Looks up, smiles and nods]
Eurogamer:
Bloody hell. Er, OK. A Times New Roman walks into a bar. The barman says, ‘We don’t serve your type.’
Milo:
[Giggles]
Eurogamer:
Bloody hell.
Peter Molyneux:
Now, he didn’t really understand every word you said, but from the tone of your voice he guessed you were telling a joke.
Eurogamer:
But I put it to you, Peter, that was not a very good joke. So the fact
he laughed at it demonstrates a serious flaw within the software.
Peter Molyneux:
Well, that wasn’t a real laugh. That was a polite giggle. Let’s move
on. The game is called Milo and Kate and you play through a story.
There is another character called Kate. Kate is a dog.
Eurogamer:
But of course! Wait, are you sure it’s not a parrot?
Peter Molyneux:
No, it’s not a parrot. Milo can recognise your writing – you can write
words, write numbers, draw pictures, and put them into his world.
Eurogamer:
Can you draw a nob? Because most people, given the opportunity to draw something, will draw a nob.
Peter Molyneux:
That’s the interesting thing, you see. We’ve been very, very clever
about this. Although you can put stuff in his world, you’ll notice he
never shows you the stuff. So although you could do obscene stuff,
he’ll just look at it and he won’t understand it. He won’t pin that
picture up on the wall, because I’m fully aware people will do things
like that.
Eurogamer:
Can you explain more about the pocket money system and how that works?
Peter Molyneux:
You can buy stuff for Milo’s world, like a bicycle or a trampoline.
He’ll come back from school one day and say, ‘Oh, Alex’ – Alex is this
character at school who always does a bit better than Milo – ‘Alex has
got a new bike. When can we get a new bike?’
To
get that bike you need to earn money by doing activities. There are
three activities you can do, and the amount of time you spend on each
activity sculpts your Milo in different ways – so everybody’s Milo will
be completely unique to them.
If you do lots of work, your Milo
will be very studious. His hair will have a side parting. He’ll be
quite worried about his appearance and he won’t like to get dirty.
Whereas if you do more of the play stuff with Milo, he’ll be more of a
kid who goes out and scratches his knees.
Your character doesn’t have to be a boy, it can be a girl. At the start you can choose whether to be play as Milly or Milo.
Eurogamer:
One of my colleagues did want me to ask why you made him a 12 year-old boy, and not a nubile 17 year-old lady acrobat?
Peter Molyneux:
If we were making a porn game, I probably would do that. He’s not 12,
he’s about 10, and that’s before he’s hit puberty. Part of the amazing
impact of this is he can remind you of your childhood.
Eurogamer:
My colleague pointed out that if it was a 17 year-old acrobat, instead
of things like ‘Have you done your homework?’ you could say, ‘Will you
take your bra off?’
Peter Molyneux:
Yeah, you could do. You could make a great porn game with this stuff,
that’s absolutely for sure. But I’d love the idea that you’ve got this
character who you are inspiring. It is such a wonderful feeling that to
inspire anything, whether it’s a dog or a person or a kid. When you see
and feel that emotion, it’s pretty emotional.
Eurogamer:
You said he only understands certain words. So presumably you can’t have a conversation about the situation in Palestine?
Peter Molyneux:
The number of words he understands is built up over time. For Claire
[the lady who demoed a conversation with Milo during Microsoft's
conference], it’s something like 500 words.
But
we haven’t cracked the real problem, which is him understanding the
meaning of it all. He’ll give you the illusion he does that. The
interesting thing is you can only talk to him when the Talk icon
appears at the bottom of the screen. That’s when he’s listening to you;
the rest of the time, he’s not. He’s listening to you because there’s a
context in which you can talk to him.
One of the journalists who came in before you had obviously read up on the Turing test.
He asked Milo one of the questions in the test – ‘Do you remember when
we met yesterday?’ Well, of course, we haven’t cracked the Turing test.
If we had, then applying it to a computer game would be the last of the
solutions we’d use it for.
Eurogamer:
Looks like my time’s up, so briefly: are you going to change your name to Gepetto Molyneux?
Peter Molyneux:
As in Pinocchio? I could do, I suppose. I have to tell you, it is
amazing. You do feel, in a way, that you are creating something that
has never existed before. When you show it to people, especially
non-gamers, it does promote this incredible emotional reaction.
Eurogamer:
I was nearly in tears during the E3 conference. But that was partly
because I am a girl and I have jetlag and the internet wouldn’t work
properly for the Live Text. But yes, I was nearly in tears. I didn’t
want Shane Kim to see me cry though.
Peter Molyneux:
We have had people in tears, because there are times when this is quite
an emotional journey. It’s very different. It’s very ambitious. But
we’re going for it.
http://www.eurogamer.net/articles/e3-project-natals-molyneux-and-milo-interview