Exclusive: How Does Microsoft Xbox Kinect Work?

Microsoft invited T3 for a UK exclusive, all-access tour round the labs at the company's HQ in Redmond, Seattle to see exactly how Kinect works, and the work that’s gone into it. Kinect is made up of three distinct subsystems that each marries hardware and software.

[CrzyFooL]

You know, I never thought to ask, lol.

[AAACE5]

That doesn't look technologically advanced enough... I got it... slap the name "SONY" on it, and it will be the greatest invention of the decade!

LOL, Just kidding guys!

[Army_of_Darkness]

The same way as the PS eyetoy?!?!?
Boom-chicka-bow-wow!

[DelbertGrady]

Please read.

"Kinect’s optical setup is what allows it to track your movements in real time. It’s ridiculously complicated and made up of tech that's been around for about 15 years, but allows for effects and functions that have only been available at huge expense up until very recently.

It’s made of two main parts: a projector and an IR VGA camera. The former bounces out a laser (don’t worry, Microsoft insists it’s safe) across the entire field of play, which the camera picks up to separate you from your sofa on what’s called a ‘depth field.’ It’s essentially all the pixels that Kinect gets back as IR noise measured in varying colour dependant on how close they are to the system. That way bodies appear a bright shade of red, green etc, and things further away appear grey.

The software takes this image and runs it through a host of filters so that Kinect can work out what’s a person and what’s not. The system follows a basic system of guidelines, such as ‘a body is from x-foot tall to x-foot tall’ and ‘a person has two arms and two legs’ to work out that your coffee table or dog aren’t extra players. It’s also taught to be able to pick you out if you’re wearing baggy clothes or have hair coming over your shoulders. When we saw this as the developers see it, it was impressively accurate at sussing out each body part (right shoulder, for example) from not much information.

Once that’s sorted, it converts body part identification into a skeleton with moving joints. Kinect is preloaded with 200 common poses, so that it can fill in the blanks if you make a move that obstructs the cameras view of your entire skeleton. The only downside we could see was that fingers aren't mapped individually on the skeleton, meaning that those dreams holding a pretend gun and pulling the trigger for Kinect FPS games are over.

The system does all this continuously at 30fps.

What about that promo trailer where Kinect signs in players just by looking at them? We saw that work in real life. The reality is that you’ll need to go through an ‘enrolment’ process in for that to happen. It’s a short one, but works by mixing your skeletal measurements with some basic facial recognition software. Microsoft says that if you drastically change your appearance you’ll need to reenrol."

Personally, I haven't seen something that has interested me enough to consider buying it. But it has a lot of potential and I think some developers will utilize it in cool ways we haven't thought of yet.

[commodore64]

Thanks, Soda.

By the looks, this is nothing at all like the eyetoy.
The eyetoy is obviously just a webcam, while kinect is really so much more.

In light of what Soda described above, to say otherwise is foolish indeed.

[webeblazing]

so just because it has ir and some and software for it its not tyhe same as the eyetoy. that like saying move aint like the wii mote because it use cam, better gyros, and software. now you get what we saying it would be a different story if they showed a game that we didnt see for the eyetoy. or maybe if yall would of been like ok its like it but they up the tech a lil hope the make better games then the eyetoy had instead of saying it new been done b4 and this is gonna change the world.

im really not seeing the logic yall be making about this thing. like i said i see it when i believe it. ill try it, it dont mean ill buy it. n by the way the eyetoy sucked and that was by sony.

[Triella]

Oh look that's exactly how I said it works...

MS : "t’s ridiculously complicated and made up of tech that's been around for about 15 years"

Me : "MS didn't spend any time in developping kinect, they licenced Gesturetek motion control software that existed for 15 years (same as the Eyetoy)";

--------------

MS : "It’s essentially all the pixels that Kinect gets back as IR noise measured in varying colour dependant on how close they are to the system."

Me : "Normally Kinect should have been equiped with a single-chip custom-silicon solution like in the diagram below. In this configuration part of the data processing (determine depth based on the pattern's deformation as captured by the infrared, and its transcription into points cloud) could have been carried out by the cam itself providing the console with ready to use datas."

"The PrimeSense depth camera projects a known infrared pattern onto the scene, like a grid-like pattern, and determines depth based on the pattern's deformation as captured by the infrared CMOS imager"

--------------

MS : "The software takes this image and runs it through a host of filters so that Kinect can work out what’s a person and what’s not."

Me : "In MS' version (Prime sens RGB-D camera stripped down from its internal chip) the software, and therfore Xbox360's CPU, does all the work of sorting raw data, converting them and processing them in the game...that's why Kinect is laggy and less accurate than what it could be"

"Apprently MS decided to cut costs once again by getting rid of the internal chip, and instead licenced Gesturetek's movement recognition software to transcribe what the cam sees in raw datas, thus resulting in the 360 processor carrying out all the workload (this is were the lag comes from)"

"But it's a truncated truth since the motion recognition software installed in the 360 is the one analyzing and processing the data and therefore greatly gnawing at the CPU"

-------------

MS : "Once that’s sorted, it converts body part identification into a skeleton with moving joints. Kinect is preloaded with 200 common poses, so that it can fill in the blanks if you make a move that obstructs the cameras view of your entire skeleton."

"What you see in the "freestyle" part of the Dance central choreographed demo (= not representative of what gamers will get out of it) and the Your shape demo is not an avatar but only a shape formed by the raw datas (points cloud) the RGB-D cam sends to the processor so the lag is minimal while still clearly visible, but when you have to map this "shape" and its movements to a real avatar in a game you get :

1. even more lag because the datas will then need to be interpreted first and then retranscribed so that your avatar matches your movments.

2. Jerking avatar limbs and so on because when you move around alot and there is latency the processor is not able to take in all the datas and restore them properly fast enough. "

-------------

[PandemicPrawn0]

I have to admit it is a clever liitle device.

It's a real shame that finger tracking is not an option but I would not be surprized to see Microsoft update that sort of thing via software.

[tiamat5]

Badly

[WLPowell]

badoomtish!!!

[LordStig]

It works when IT wants to

[WLPowell]

badoomtish!!

[dizzleK]

if you don't like kinect, please go discuss move in one of the myriad of articles on n4g. all the game sites are talking about it.

i couldn't even hold a straight face while i typed that ;p. move is out next month and nobody cares.

[WLPowell]

Aaaand I'm spent!!!! That last sentence was the funniest of them all, dizzleK. Way to use that last bubb. "Nobody cares" cause it works. We don't have to worry about sony sending bags of money and free consoles to sell faulty hardware. They just put the tech on display and let the genuine fun do the talking. I'll be testing it out when I go to Lollapalooza this weekend. I'll be sure to keep you updated on motion tech that real gamers will actually play.

edit: good one xbreakx, hopefully the N4G community will keep this going. I'm off to bed.

[LeonSKennedy4Life]

DizzleK is buying Kinect.

I'm not sure who you really are, but I advise you as a human being not to buy Kinect. That's $150 you'll regret spending! I'm not even kidding. That money could feed a child from the Feed The Children Foundation for 15 months. Don't just spend money to prove a point.

Take care, man!

[edhe]

The only thing out next month is reach.

What is this 'Move' you talk about?

[DigitalRaptor]

The positive hands-on impressions are doing all the talking so far for PS Move. Whilst only the hype and public stupidity is what's keeping Kinect in the light.

You just wait for the TV ads, billboards etc. The promotion for Move should be pretty good, and they will be emphasising the good value, affordable entry price, the fun and the accuracy.

And then there's the Coca-Cola promotion. Just wait for that. ;)

[rdgneoz3]

And no lag, plus casual and hardcore games (Move). Can't wait to see the RTS R.U.S.E with Move.

[DJMarty]

Kinects crap, only blind sheep are talk about Kinect.

EyeToy says hi, that was like 7 years earlier. This is far from new.

There is a reason both Nintendo & Sony ditched Kinect tech. They saw the flaws and both went with far superior tech.

[edhe]

Yes - because controlling interfaces without hands-on devices is so .. stupid.

Because having a tech that will instantly recognise you from any other human being is ... daft.

Because expanding technology into true natural interfaces such as voice and gesture is.. moronic.

We all want to avoid being those sheep - and just stick with the thumb twiddling herd!

Stop comparing kinect to the eyetoy - it's like comparing the ps3 to the ps2.

[CaJuaLFeatPuFF]

Kinect sucks.

[kasasensei]

Interesting. Really interesting. I learned a few things with this article.

[dtrain21]

Interesting...i'm buyin it

[jazzking2001]

interesting but wont get at current price

[DJMarty]

Pew Pew Pew, need I say more. lol

[xavier-tooth]

silly question. get it to work first before asking that question.

[Hades1337]

Great article, can't wait to give it a try.

[stuna1]

Do you want the quick answer to that question? It doesn't.

[midgard227]

i didnt know it even worked