On the current gaming trade show E3 in Los Angeles Microsoft presented Project Nadal, which allows completely new forms of interaction with computer or console. In this article I would like to analyze what technologies are combined in this project and what I would like to see coming next from the gaming industry.
First let take a look at the hardware, which was used for the project. Beside the XBOX 360 console the most interesting part was a camera, which is able to catch the third dimension of the person or other object in front of it. The technology reminds me of ZCam developed by an israel-based start-up company 3DV. This company was bought by Microsoft few month ago, but Microsoft denies that the technology for their cam comes from 3DV, the reason why the company was bought were patents. But anyway the technology must be quite similar. ZCam from 3DV uses infrared rays to measure the distance to the object, so it is a kind of radar with a resolution of 1-2 cm, which should be sufficient for most needs. The cam has also a microphone, which is needed for the voice recognition capabilities.
But the most interesting part is the software. Microsoft research center integrated so much goodies, which complement each other, that it is hard to separate them and view at each of them independently.
Let start with face recognition. The software seems not only recognize the user of the console, but also the mimics and make assumptions about his current mood. This requires very advanced pattern recognition. The number of users in a home-based console is probably not that large, so the recognition should be quite easy, but finding out the feelings of a person based on expression on his face isn't that simple. It is interesting, if the software needs any calibration in advance. As can be seen in the video, the mimics are used for gameplay (fire balls out of monster's mouth), advanced KI can change the game, so the dialogs can be more personal, music can be played, or films from the online video store can be suggested based on the mood of the user.
Scanning of the objects, which can be then used in the game is another great improvement in the game play. In the demonstration of a new game, which uses capabilities of Nadal, Peter Molyneux demonstrated how the camera scanned a drawing on the paper, which was used for the continuation of the story. Again a very advanced pattern recognition is needed for that. So I think first games will use this feature, just to get the pattern without recognizing it, so things like skateboards or cloths can be customized. Of course this technics can be used for something what Second Life has promised but never delivered, that is the possibility to become one with the avatar, so that it becomes a truly virtual mapping of yourself. This is actually a dream for all fashion companies, so that the customer can try all cloths (even with combining them with cloths he already has) before ordering them. Probably the resolution is still not high enough for tailoring new cloth, which fit exactly, but with the time the resolution will improve, so it becomes possible to send orders for individual manufacturing of clothes.
Voice recognition is an game controlling element, which hasn't been used too much, because it is too slow to control a game with words and the variety of expressions is too large, so KI must be very advanced to be able of handling it. But there is one type of games, which are perfectly suited for voice recognition, these are all quiz shows. Computer must recognize just the right answer, which is much less complicated then handling free speech. However the demonstration of Milo showed that Milo seemed to understand what the person was saying. The answers reminded me of ELIZA, but recognition of free speech and conversion into format for ELIZA is very big achievement, if it really works as promised.
Recognition of players moves. Sony's EyeToy could detect moves, but without support of measurement of 3rd dimension, the recognition was quite inaccurate. Now Microsoft is promising that the whole body will be recognized and the resolution will be much higher without need of any controller. Peter Molyneux is absolutely right when he is saying that controllers with more and more buttons prevent a natural interaction with the console and Nintendo's success with Wii only proves it. So now even the Nunchuck is not required, which should increase the community of console players even more, because the entry barrier is very low. Just stand in front of the TV and start playing. One often criticized point is that it is unrealistic of driving a car with an air wheel, but player can use every object as a wheel, if he wants to have something in his hands.
So what is missing now for a perfect gaming experience? The input is perfect, but the output still lacks some important features for complete diving into the virtual environment. The visual output is with introduction of HDTV much better, but now the format of the screen is not optimal. Of course a VR-cave would be the perfect solution, but it will remain expensive and consumes too much space, so it is impractical. All VR-helmets failed so far and they do not allow social interaction with other persons in the room. So a solution would be a screen, which has height of human body. A fight is much more realistic, if the opponent has approx. the same size as the player himself, so are all sport competitions. For different games, which need a bright view angle like flight simulators, the screen should be rotatable.
3D-cinemas have a new revival with the new digital systems, so this technology should become affordable for home users as well. Either the user could wear shutter-glasses or there are special monitors, which are able to show 3D even without this. A console must generate double as many pictures, but I don't think it is a big problem.
The biggest issue is the force feedback. A boxing fight without having physical contact with the opponent is not realistic. So while gaming industry is offering vibrating controllers or special seats, these solutions do not work if there are no controllers and the gamer is moving in the room. So fresh ideas are needed here.
As a conclusion I can say that project Nadal is revolutionary for the gaming industry, it combines several very advanced technologies into a solution which makes lot of sense and is very intuitive for the consumer. It is interesting to see what kind of games will be using this technologies and how much effort it will be to create such games. Microsoft promised to deliver the cam in 2010, so it remains to be seem, if all promises can be fulfilled. However there are still lot of wishes open, which could make gaming even more realistic.