The assistant robot Garmi is becoming increasingly versatile and intelligent. As researchers from the Munich Institute of Robotics and Machine Intelligence (MIRMI) at TUM will be demonstrating at the ICRA 2024 robotics trade fair in Yokohama, Japan, the robot not only understands various commands via ChatGPT, but also autonomously performs a variety of tasks and skills such as grasping objects, maneuvering safely and communicating with patients. It also arranges live conversations with doctors who carry out telemedical examinations. "Garmi' is now able to perform various individual skills that we have taught him over the last few years safely and on demand via ChatGPT," explains Dr. Abdeldjallil Naceri, head of the geriatronics project.

Human-robot interaction

To achieve this, the researchers are combining various technological innovations: a digital twin is used to try out the robot's movements in a simulation before the real interaction with humans and thus avoid collisions. Artificial intelligence (AI) helps Garmi to grasp cups and glasses and pass them over without spilling liquid. And ChatGPT acts as a link for communication between robots, patients, physiotherapists and doctors.

The TUM researchers have made great progress in three areas in particular:

1. dexterity - grasping and moving precisely from a distance
To enable Garmi to grasp in a similar way to humans, the researchers have brought together a camera, a robotic arm with seven joints and an artificial hand, as well as artificial intelligence. First, the camera takes a picture of the object that Garmi is supposed to grasp and identifies it as a cup, bottle or ball using neural networks. As the camera only sees the object from one side, the system adds non-visible areas, such as a cup, by comparing what it sees with other images and reconstructs the complete 3D object. The researchers display the probabilities with which the object is represented as it looks in reality in a color-graded graphic, a 'heat map'. This makes it possible to determine the ideal position for the hand to grasp a cup, for example. The complex system is now able to do this in nine out of ten cases. "If it works with one cup, our system can transfer this to all other cup shapes," says Naceri.

The researchers devised a special experimental setup to investigate whether doctors can work together with patients via teleoperation. They drew simple shapes on a digital graphics tablet. Garmi was also equipped with a pen in one hand and a camera in the other. One room away, Garmi was asked to transfer what the researchers had previously drawn onto a screen - in other words, to project a simple drawing onto a complex robotic system. It turned out that the best circles, squares and triangles were created when Garmi used the camera autonomously, a finding that will be incorporated into future collaboration between doctors and patients. For example, during an ultrasound examination, it is important to position the ultrasound probe as precisely as possible, or to perform movements correctly during a rehabilitation exercise.

In a new research project, the researchers show how tools can be maneuvered past an object. The challenge is to keep an eye on the distances on the one hand and to correctly assess the mobility of the robot arm with all its joints on the other. If this is successful, the robot can even avoid balls flying in the opposite direction.

2. safety: a tactile robot reacts within 1 ms

Garmi processes information with a cycle time of 1 ms. This applies equally to perception, interaction and navigation. The force sensors of the robot arms register the slightest contact and react immediately. If a person accidentally bumps into the robot's arm, it stops within 1 ms for safety reasons. To prevent accidents, humans and robots initially meet as digital twins in a virtual environment. This is essential, as the assistance robot can theoretically reach speeds of up to 20 km/h in a care home. In the computer simulation, Garmi uses the Safety Motion Unit to register via its sensors if a person comes too close and moves more slowly. If the person moves away, it moves faster again.

3rd language: ChatGPT uses a list of commands

The AI tool ChatGPT functions as a translator between technology and humans. It has learned various commands such as "Start rehab", "Show me tomorrow's weather" or "Call the doctor", and Garmi uses this tool to communicate with patients. The researchers currently have a list of 15 to 20 commands that trigger certain actions. "Potentially, we can expand it as much as we like," says robotics researcher Naceri, "making MIRMI one of the first institutes where robots and humans interact with each other using ChatGPT."

Dealing with the human environment

The new universal Garmi is now becoming active in a model apartment in Garmisch-Partenkirchen. The main field of research will be the further development of hands that can take on ever finer tasks. It will be several years before Garmi is ultimately used in care homes. "It's like the autonomous car," says Naceri, "a lot of progress has already been made, but a few details are still missing before it can be used in the human environment."