The introduction of image processing has significantly increased the productivity of robots. Vision systems use cameras, sensors and algorithms to imitate human vision so that robots can perceive their surroundings and act precisely on this basis. Vision systems already make it possible to inspect, identify, count, measure, read barcodes and control robots.

Potentials and limits of image-based control systems

Image-based control systems are a form of vision technology used to control robots in dynamic environments. A typical setup for this type of technology is a robot arm equipped with one or more cameras. These serve as sensors that provide a secondary feedback signal to the native robot controller. This allows the robot to move more precisely to a variable target position and to locate and manipulate objects more accurately. This is crucial for applications such as placing or joining small components in electronics assembly, for example.

Conventional vision systems for control based on 2D or 3D cameras have significantly expanded the range of applications in robotics. Nevertheless, these systems also reach their limits. To fully understand these challenges, it is helpful to look at how conventional vision systems work.
A basic 2D approach is naïve pattern matching, where each image pixel is compared to a predefined pattern. More advanced methods use filters to highlight image features, making position determination independent of rotation and scale. 3D problems use point clouds generated by stereo systems or time-of-flight cameras that emit infrared light and measure its reflection time.

Even the simpler 2D technologies use structured light, which makes them susceptible to changes in brightness and color. Complexity reduction is often achieved by ignoring colors or using a stable reflection point. Challenges such as sunlight, contrast changes, extreme viewing angles or unexpected objects in the image make target identification more difficult. Targeted masking of information is necessary to protect the systems from distractions, which requires extensive engineering knowledge and workspace modifications. This often makes automation in dynamic factory environments difficult and expensive, which is why complex tasks are often performed by humans.

A more sensible approach would be to use systems with algorithms that analyze in a similar way to humans. Humans intuitively consider all salient information - such as colors, shapes, brightness and reflections - and know what information can be ignored if it is irrelevant to the intended goal. This is where AI comes into play.

AI-supported vision systems for control

In contrast to conventional vision systems, AI-based solutions are able to handle transparency, reflections and highlights robustly.

© Micropsi Industries

The integration of artificial intelligence, in particular deep learning, into vision systems significantly improves their capabilities. The strength of deep learning lies in the use of artificial neural networks. These networks are algorithms that are modeled on the biological structure of the human brain and enable pattern recognition, grouping and classification of objects in images.

AI-controlled vision systems analyze visual data and use these neural networks to extrapolate beyond the input data. This means they can use visual input data as input to identify generalizations and similarities. This enables them to derive appropriate responses to new scenarios after just a few examples. Rather than relying on predefined visual features or the exact replication of scenarios, AI-driven vision enables adaptable robots that can work in dynamic environments with changing lighting conditions. Other benefits of these systems include

• Real-time movements: AI-based vision systems have the computing power to process and analyze data quickly. This enables them to make decisions in real time and make continuous corrections to the robot's movements.
• Simplified commissioning: Users can train the systems with simple hand movements, eliminating the need for extensive development work and the need to modify the workspace.
• Cost efficiency: Flexibility, lower labor costs, increased safety in the production process and higher quality through the use of AI-based vision systems lead to a reduction in costs in the long term.

In contrast to conventional vision systems, AI-based solutions are able to deal robustly with transparency, reflections and highlights.

How an AI-based vision system is put into operation

AI-based vision systems such as 'Mirai' from Micropsi Industries learn through training. During the training phase, the user familiarizes the system with the desired movement and the variances that can occur during this process. Cameras on the robot arm record the scene and the recorded images are converted into data and transferred to a computing cloud. There, a learning algorithm trains a mathematical model to guide the robot.

Unlike conventional systems, vision systems such as 'Mirai' do not require CAD data or 3D cameras. During training, they learn how they should react in certain situations. The AI identifies relevant visual features and develops solutions based on the training data. During the execution of a task, the system does not compare the live scenario with a rigid 2D or 3D template. It searches for the defined features itself and recognizes the target, for example a workpiece, even from new angles, as long as it looks similar enough to the training data.

AI-based vision systems supplement the existing robot control system. When executing an application, the robot's native controller takes over the programmable movements, while the AI controller takes over for the sections with unpredictable variances.

Areas of application for AI-based vision systems

Thanks to AI-based vision systems, users can easily show the robot variance - for example, different-looking pipes in the leak test.

© Micropsi Industries

Robots controlled by AI-based vision systems can be used for complex tasks such as cable insertion, leakage testing, rack assembly and screwing. These areas of application are characterized by a large number of unpredictable variances.

In leakage testing, for example, the pipes at the back of a refrigerator through which cooling gas flows are checked for leaks. The numerous sources of variance that occur in this process are special: The material composition of the pipes can vary and give them a different appearance. Pipes may be soldered, resulting in shiny 'drops'. Some tubes may have been manually clamped, with the clamped tubes bending in different directions.

However, thanks to AI-based vision systems, users can easily show variance to the robot - for example, different looking pipes in the leak test. To start the leakage test, a robot must position a test probe near a solder joint or the point where the pipe has been sealed. Neither the exact position of the pipe nor the position of the solder joints is known in advance. In addition, the background often differs between the test episodes. The lighting conditions can also differ and cause reflections and mirroring - a task that is impossible for a conventional system.

AI not only extends the robot's capabilities to include the perception of its environment, but also enables it to understand it and adapt to changing conditions. This enables it to deal with variations in position, shape, color, lighting and background.

The author: Maximilian Mutschler is Vice President Sales at Micropsi Industries.