Dementia is on the increase. Patients who are particularly seriously ill usually only react through facial expressions and gestures. Communication with those around them is becoming increasingly difficult. Demographic change means that more and more people will be living in such facilities. The Fraunhofer Institute for Reliability and Microintegration has taken this fact on board and, in the DAYSI project, has developed a gesture-controlled table that communicates in writing, image and sound together with partners from the fields of care, furniture and software production. It can be used in occupational therapy for people with dementia as well as for individual care.

The table as an interactive user interface

The basis is a simple table equipped with radar and communication technology. For example, voice control can be built into an artificial flower or a small projector can be integrated into a picture frame. An invisible database integrated into the table can provide the nursing staff with pictures, songs and other information to address the person in need of care individually.

Such a table can be used as early as the admission interview: The system recognizes frequently recurring patterns of behaviour, such as addressing children, using speech recognition and responds with an adapted response. This ensures that care staff are given the option of enabling or disabling sensitive information via a security query.

Integrated radar sensors for gesture control

Fraunhofer IZM is developing the entire surface in the project. This includes the hardware for the radar sensor, wireless communication and wireless charging of the individual components. By integrating these elements, the table becomes an interactive interface on which commercially available objects become gesture-controlled means of communication. The individual components are the contact points to a mini-computer integrated into the tabletop. Additional external components of the system, e.g. projectors, cameras and a voice recognition system, can be connected via the table's wireless interfaces and, if required, integrated into everyday objects such as vases or picture frames. All these objects are to be connected using automatic playback technology so that they connect directly to the computer when in use.

In addition to various wireless communication interfaces and a charging device, gesture recognition will also be integrated in collaboration with software partner Creonic. Other partners in the project are Contag AG for setting up the hardware according to the specifications of Fraunhofer IZM's AVT technology, the Böhm Group for installing the necessary hardware in the table and Evangelische Altenhilfe Duisburg and Charité Universitätsmedizin Berlin for testing and using the table in a real application environment.

Development of a 3D gesture control for multi-touch environments

About halfway through the project, the project partners agree: "The DAYSI interface will revolutionize the field of patient care. Such systems can significantly simplify care. Other fields of application are also conceivable, such as the gaming industry," says project manager Christian Tschoban from Fraunhofer IZM.

Demonstrator for test purposes

The consortium is currently working on the hardware for a prototype of the interface. The demonstrator will then be installed in the table by the project partners and the feasibility, user acceptance and possible communication errors will be tested by Charité Universitätsmedizin Berlin and Evangelische Altenhilfe Duisburg in various application scenarios. For example, a projector is used to simulate family members or friends who can communicate and interact with the patient via the table. Gesture recognition is used to assess the calming effect and, if this is not successful, the care service is informed. To ensure that the system can also support activating animation, game options have been integrated that can be controlled via the coils embedded in the table, the projector and gesture recognition.

The extent to which it helps people with dementia to suddenly experience their previously passive environment interactively is also being investigated. Meanwhile, the Fraunhofer researchers are testing the functionalities of the integrated radar technology and the miniaturized system. Until now, such systems have only been available in a much larger form. Fraunhofer IZM's expertise in the miniaturization and reliable performance of the smallest systems is impressive.

Operating concept to revolutionize care

The DAYSI project is being funded by the Federal Ministry of Education and Research for three years and will run until the end of 2021. Until then, all project partners will be working on the interactive, everyday assistance system for people with dementia, which is to be used in care facilities and the home environment at the end of the project.

In addition to elderly care, the Fraunhofer researchers and the Technical University of Berlin are also aiming to collaborate with Garamantis. While the company's multi-touch tables still function capacitively like a cell phone, the interactive worlds are to be operated using gesture control in future. Initial project applications for the development of 3D gesture control for multi-touch environments using innovative multi-radar sensor architectures for virtual reality environments have been submitted. These offer a basis for further fields of application such as autonomous driving or as a relief for older and impaired people such as the visually impaired.