The Fraunhofer IPA tracking system automatically documents hand movements with 3D image processing, evaluates them intelligently and thus eliminates the need for manual documentation during tests.

© Fraunhofer IPA

When documenting experiments, every detail counts. In addition to obvious information such as the name and quantity of substances or the temperature, unconscious movements and habits can play an important role. Documenting this is not only extremely time-consuming, but errors can also creep in or information can be omitted. "We have a customer who had a habit of mixing the samples when pipetting. To do this, she intuitively pressed the pipette quickly and repeatedly. In the end, this was not recorded in the protocol, but it was essential for the experiment to be successful. The experiment could not be repeated by her colleagues. Until the decisive movement was recognized, we had to repeat the experiment many times and have it demonstrated," recalls Marc Andre Daxer, scientist at Fraunhofer IPA.

The Fraunhofer IPA's tracking system aims to make things easier for laboratory staff: An intelligent 3D camera mounted above a sterile bench seamlessly records their hand movements and forwards the data live to an information system. Here it is evaluated using motion recognition algorithms, classified and transferred to a log. The system thus records the individual process steps precisely. As the tracking system requires only simple software and hardware, it is also suitable for small laboratories.

The laboratory robot

Another development from the Fraunhofer Institute called 'TeachIT' makes it possible to teach laboratory robots automatically. Normally, teaching takes place in such a way that an employee stands next to the robot and follows the gripping movements by hand. The robot saves the coordinates in its database and thus learns the movement. "It takes one to two days for a robot to master a new process, depending on the task. That's not bad in itself, but the processes in laboratories vary so often that it's usually not worth it here," says Daxer. With the TeachIT solution, the robot masters its new task in just a few minutes: To do this, the multi-titer plates in the work area are equipped with barcodes. A 3D camera on the robot arm recognizes the marking and shows the robot where it needs to reach. Normally, the robot would have to be retrained every time a work set-up is changed. With TeachIT, however, it recognizes the new objects and their position independently.

Planning laboratories virtually

Mixed reality can also be put to good use in the laboratory. For this purpose, the Fraunhofer IPA equipped Microsoft Holo-Lens glasses with CAD data from laboratory equipment and systems. This allows employees to view things in the laboratory that have not yet been built. This works on site, but also from anywhere in the world. This technology is useful, for example, when planning a new facility: "We can run through the typical work processes with the customer in advance and make optimizations at the planning stage," explains Daxer. Mixed reality also makes it easier to maintain and repair laboratory systems, as technicians can work out in advance how to replace a defective component. This reduces the time required on site.