The web interface is easy to use: Doctoral student Martin Schuck selects a music track and enters a text field in which he wants the system to suggest a choreography. Additional instructions can be given to the swarm of drones via another 'prompt' in the ChatGPT tool before an algorithm checks whether these trajectories are feasible. The screen in the 'Learning Systems and Robotics Lab' now shows a simulated airfield with six drones flying in circles to the music. If the scientist from the Chair of Safety, Performance and Reliability for Learning Systems at the Technical University of Munich (TUM) likes this choreography, he logs it in and a short time later six palm-sized drones take off from the floor of the robotics lab.

In Prof. Angela Schöllig's lab, the research team has installed six cameras on the ceiling of a room measuring around 40 m2 and three meters high. Crosses are marked on the floor with insulating tape - the starting positions of the drones. Once the computer has verified a possible choreography, the flight can begin. The cameras detect the position of the quadrocopters, which are equipped with four propellers and motors, 200 times a second. The system compares them with the desired position. The flight shows in the Learning Systems and Robotics Lab, which the research team performs with up to nine drones, are 100% safe today. Without the special safety filter, only one in four flight shows goes off without an accident.

Additional algorithm makes flying robots safe

For the 'Dance of the flying robots', Prof. Schöllig combined ChatGPT with the safety filter. "The ChatGPT AI tool was primarily created to generate texts, but it can also suggest choreographies," says the professor. "However, it initially knows nothing about the characteristics of drones and physical limits for the trajectories. So it is clear that ChatGPT makes mistakes." The additional safety algorithm closes this gap and plans the flight paths for the proposed choreography precisely so that the drones do not collide with each other in the air. This even makes it possible for two drones to fly diagonally towards each other. Prof. Schöllig calls the overall concept of ChatGPT and safety filter designed for the use of several flying robots 'SwarmGPT'. On the one hand, the tool generates the processes in the air and, on the other, serves as an interface between robots and humans, who do not require any expert knowledge.
When Angela Schöllig began her research on drones almost 15 years ago, choreographies were developed by hand. It took more than three years before the first six choreographies for six drones were developed and working. "ChatGPT has brought about a quantum leap here," says Prof. Schöllig. In the last three months, the researchers experimented with over 30 choreographies for up to nine drones. Today, the researchers only need around five minutes to develop a reliable choreography for individual 30-second music clips with three drones. The more drones are added, the longer ChatGPT has to calculate, and the longer it takes to propose a choreography. But Schöllig is certain: "The concept is scalable."

Robotics: ChatGPT as an interface for non-experts?

Can other robots with such an interface also be used via ChatGPT? For robots that use voice control to grasp things, lay cables or open doors, the success rate of the scenarios is currently only 63%, 56% and 80%. This means that the use of robots in other robotics scenarios has so far been rather unreliable. For Prof. Schöllig, this is an incentive: "I assume that our approach will also work better and better with other scenarios." It may soon be possible to reprogram suction and industrial robots simply by voice command, without expert or programming knowledge.