Image-based vision systems are used in manufacturing to control robots. Typically, a robot arm is equipped with one or more cameras that transmit secondary feedback to the native controller. This enables the robot to control variable target positions more precisely and manipulate objects more accurately.

However, traditional vision systems based on 2D or 3D technologies reach their limits in numerous applications. The main reason for this lies in the rigid assumptions that these systems have to make about tools, workpieces and their working environment. This is now changing with the spread of new systems based on machine learning, a branch of AI. AI-based systems enable eye-hand coordination in real time that is comparable to the human sense of sight.

Overcoming boundaries with AI

But what are the specific limitations of 2D or 3D technologies?

AI-based vision systems can cope with variances in the appearance of workpieces, for example hand-soldered tubes, such as those used in the inspection of refrigerators. © Micropsi Industries

Problems with CAD and 3D data or 2D matching systems: Systems that are based on 6D pose estimation and use CAD models and 3D cameras often have difficulties with (rotationally) symmetrical parts. Recognizing and precisely handling such parts is a major challenge, especially for demanding joining tasks. In contrast, AI-based vision systems, such as Micropsi's 'Mirai' software, do not rely on CAD or 3D data and can effectively deal with symmetries. Simple 2D matching reaches its limits when confronted with variable lighting conditions, transparency or reflection. Such systems fail when the three-dimensional position and orientation of the workpieces are important. AI-based vision systems overcome these hurdles by using robust algorithms that deliver reliable results even under difficult conditions.

Dealing with variations in shape and appearance: Both 2D and 3D vision systems typically have difficulty processing workpieces that vary in appearance. They assume that the workpieces are completely rigid and do not change in shape or appearance from cycle to cycle. Even if the underlying CAD model is exchanged in a 3D vision system, the user must commit to a single, fixed geometry. There is no option to switch flexibly between different variants, which makes it difficult to handle components from processes with many variants. Variances occur, for example, when new component types are introduced, when a system has to process not only white square plugs but also black round plugs in random order. The challenge is even greater, for example with hand-soldered tubes in the quality inspection of refrigerators, whose shape and color change in every cycle. An AI-based system can be trained by the user to deal with a wide and gradually increasing range of variances. This makes it possible for a single robot station to efficiently process different variants of a workpiece, even in random sequence.

Geometry detection of moving parts: Perhaps one of the most serious limitations of traditional systems is that they can usually only determine the position of rigid parts in the robot's working area. Without additional measurements, they cannot detect the geometry of the parts moved by the robot, be it the tool itself or gripped components in the tool. Therefore, these systems must assume that the geometry of the moving parts is completely rigid and that their position relative to the robot remains unchanged. However, these assumptions are violated in many practical applications, which makes automation impossible and uneconomical. This particularly affects gripping and joining tasks with flexible parts or parts that cannot be gripped uniformly due to process variations. In contrast, AI systems constantly correct the relative alignment between the tool or the gripped component and the workpieces in the workspace in real time. This dynamic adjustment makes it possible to carry out precise tasks that could not be automated with traditional systems.

Lifecycle management of the installation: If vision systems initially function satisfactorily, even the smallest changes in upstream processes can lead to disruptions. A reaction to such changes is often only possible if the newly occurring variance is introduced as a one-off and comprehensive adjustment, whereby the vision system must also be adapted to the new situation. Problems arise when the new variance occurs as an additional case and the variety of objects to be recognized is increased - for example, by introducing a new component type while residual stocks of the old type are still being processed. In such scenarios, traditional systems quickly reach the limits of their adaptability. AI systems, on the other hand, can be adapted directly by the user and learn to handle both the old and the new components. This is even possible if sufficiently accurate CAD models for the new variant are not available, as training is carried out directly on the real object.