In order for robots to work autonomously, i.e. as independently as possible without human supervision, they are equipped with sensors and electronics that enable them to perceive their environment and cope with unforeseen situations on their own. The more complicated the task, the more intelligent and agile the machine needs to be. This means that more and more different sensors - for example for distance measurement, motion detection or pressure determination in the event of contact - are being combined. At the same time, the electronics and computer technology used to record and process the data must be ever more powerful. This trend is accompanied by a significant increase in energy consumption. For mobile systems in particular, this leads to a shortened operating time or range and, according to current forecasts, will even reach the limits of global energy generation in the coming decades.

To counteract this, the Fraunhofer Institutes involved in NeurOSmart are relying on a neuromorphic in-memory accelerator that is tailored to the respective sensor. The human brain serves as a model for the electronics to be developed, as it is very energy-efficient when making decisions despite its enormous computing power.

"This type of data processing, i.e. thinking, is implemented using a new type of analogue computer memory technology, which is also able to perform computing operations when new data is recorded in the system," explains ISIT scientist and project manager Dr. Michael Mensing: "In practice, this is used to recognize objects and their behaviour precisely and in real time." Until now, several separately developed components in computers and particularly energy-intensive communication between them have been necessary for this mode of operation.

In parallel, particularly small and efficient models for object recognition and classification are being developed that are specially adapted to the sensor, the new possibilities of directly integrated electronics and their applications.

Over the course of the four-year project with a funding volume of 8 million euros, this approach will be combined for the first time with a complex LiDAR system developed by Fraunhofer and tested in an application-oriented environment. This sensor system is a key component of autonomous systems, as it uses detailed distance information to recognize its surroundings even in poor weather and over a wide range of distances. As a first test of the new sensors, they will be integrated into robotic systems over the next few years to support their human colleagues in production environments, for example by moving heavy loads or handing over components.

Within the project, Fraunhofer IPMS is developing the circuitry of the neuromorphic accelerator. Modern HfO2-based crossbars form the core of the analog accelerator. This allows the complex, often iterative digital calculations of AI models (for example in CNNs), which otherwise have to be calculated on generalized graphics cards with high energy consumption, to be mapped to energy-efficient memory operations in the circuit. For the internal control of data flows, the RISC-V processor core EMSA5 is implemented with direct interfaces to the analog accelerator module and higher-level systems as well as error protection mechanisms. Research into a software-programmable system architecture will enable flexible use of the circuit in a wide range of applications.

Fraunhofer IPMS is also working with Fraunhofer IAIS to train the neuromorphic accelerator circuit. In a first step, the neural network model for LiDAR data evaluation is being researched. The aim is to automatically map the available hardware topology and transfer it to a circuit design. In the system, the circuit analyzes pre-processed LiDAR data. Communication between pre-processing (FPGA) and the accelerator circuit takes place via a real-time capable high-speed Ethernet interface.

Finally, Fraunhofer IPMS is involved in the validation of the new memory cell approaches for future accelerator implementations together with the Fraunhofer Institute for Silicon Technology ISIT.