Application of Depth Sensor in the Design of Hybrid Robotic Gaming Environment

This project combines depth sensor, virtual game platform, and mobile robots to create an environment where user can engage in a game (i.e. a game of air-hockey) by using simple hand gestures to control physical mobile robots against another user using the virtual counterparts of mobile robots in a virtual environment. The mobile robots move on an open field using DC motors, and each of them is equipped with a unique reflective marker. The overhead camera feeds the image of the field into the game program which utilizes image-processing algorithm to read the positions of the reflective markers and displays the results in the virtual environment. The depth sensor provides the skeleton models of the players which in turn give the hand positions and gestures to the program. Through the combination these information, players can then interface with the virtual environment. In the virtual environment, the real mobile robots which play against virtual robot models of similar design are controlled by the players to move the puck into their respective goal to score. Through this system, users can experience robot sports game in a hybrid gaming environment using real mobile robots and virtual robots.