The mixed reality process presented is based on two technologies: in the real world, eight cameras, four pairs each, capture the scene from all sides and generate depth maps of it at up to 30 hertz. Gestures and dynamic movements are also recognized. This data is then summarized by algorithms, encoded and transmitted to the VR station in real time with the corresponding 3D textures.

In the virtual scenario, on the other hand, another 3D camera records the VR user. Thanks to the Fraunhofer HHI algorithms for 3D body detection and gesture interpretation, the user can interact naturally in the VR scene without any interfering controllers or markers. He is represented there as a moving full-body avatar, so to speak, and sees his own body and gestures in virtual space. "Only the combination of the two technologies enables a unique solution for new mixed reality interaction and collaboration scenarios," says Paul Chojecki, project manager at the Fraunhofer Heinrich Hertz Institute (HHI).

The feedback from the virtual world is displayed in the real scene by means of a projection. Special image processing algorithms, also from the Fraunhofer HHI, are used for this projected augmentation. These ensure that the information and control elements are displayed correctly, even if the surfaces onto which they are projected move or are tilted. The fields of application for the X-Reality solution are diverse: in addition to remote assistance, the process can be used in rapid prototyping, human-robot interaction and telecommunications or in telepresence and gaming sessions. For example, two physically separated people can play a board game together.

At the Fraunhofer stand at Cebit, visitors were able to try their hand at a 3D puzzle and get help from the virtual world. In the demo, the real object was projected live into the VR environment and explained to the VR user. They can then react to it virtually and mirror these reactions back to the real situation.