One of the ways in which robots are 'revolutionizing' production is through collaboration with humans. This is not just about programming, but also about allowing robots to observe and imitate the actions of humans with the help of deep learning. Among other things, this is made possible by a system developed by Nvidia called 'Nvidia Jetson AGX Xavier', which is based on deep learning technology. Using neural networks, it is able to derive and execute a program that can be understood by humans from specific physical processes. The focus here is on the sequence 'register action, generate program, implement program'.

The neural networks are trained exclusively on the basis of synthetic data in simulation processes. With synthetic data generation, almost unlimited pre-labeled and classified data is available so that the corresponding algorithms can be easily developed. The decisive practical tests take place in the form of a Baxter robot that picks up various household items and places them elsewhere. This method was developed specifically to optimize human-machine communication - and, in a broader sense, to enable the two to work together seamlessly in the future.

The live instruction

Thanks to Nvidia's new method, robots are learning to work side by side with humans.

© Nvidia

The system answers the question of how a human can instruct a robot. The necessary communication requires two components: the goal of the instruction and information on how best to achieve it. The 'live demonstration' is effective here, as the instructor can both define the goal and show suitable ways to achieve it. As a rule, a single demonstration is sufficient to instruct the robot.

Another challenge is to enable the robot to deal with external or technical uncertainties, changes and disturbances. Production robots usually work according to the open-loop principle, which means that they are dependent on the reliable repetition of all processes and a stable environment. Every now and then, however, irritations occur to which the robot has to react appropriately - even if it is just an object that it wants to grasp but which is a few centimetres away from its usual position.

The Nvidia system (re)acts in such cases by continuously observing its environment and paying particular attention to changes detected by the camera. With the help of a deep neural network (DNN), all objects can be classified and evaluated in real time according to their position and behavior. In this way, changes are registered and incorporated into the robot's behavior.

The potential of DNNs

Deep learning is the science of training large neural networks that can contain hundreds of millions of parameters and are therefore capable of mapping complex functions such as non-linear dynamics. The DNNs form status representations of dimensionally accurate, multimodal raw sensor data, as commonly found in robotic systems. Unlike other machine learning methods, they do not require humans to manually develop feature vectors of sensor-generated data during the design process.

The next step in the use of AI is autonomous machines. The robotics platform 'Nvidia Isaac' is intended to give a further boost to the spread of such solutions in many industries. The 'Isaac SDK' supports developers in the construction of robots - among other things with libraries, frameworks and tools that cover the entire functional spectrum of a robot. 'Isaac Gems' are a collection of robotics algorithms for registration, navigation, guidance and control that are compatible with 'Isaac SDK' and 'Isaac Sim' as turnkey applications. With 'Isaac Sim', developers can train and test their robotics software using realistic, simulated environments. Replays and test runs can be initiated in minutes.

Author:
Eddie Seymour is Director Embedded Business at Nvidia in Munich.