As the 'eye of production', industrial image processing (machine vision) observes and monitors production processes in real time. The digital image data generated in the process is processed by integrated machine vision software and made available for a wide range of tasks in the process chain. For example, objects can be accurately recognized and precisely positioned based on optical features. The technology also detects faulty products and enables them to be automatically sorted out, which optimizes quality assurance processes.

Today, more and more modern artificial intelligence (AI) processes are being incorporated into machine vision systems. Deep learning methods based on convolutional neural networks (CNNs) are particularly noteworthy here. Here, very large amounts of digital image information are used for an extensive training process, on the basis of which the software can later classify new objects independently. As part of the training process, specific features and characteristics that are typical for a certain object class are automatically learned. This allows new image data and the objects depicted on it to be precisely assigned to their respective class, which enables very high and robust recognition rates. These deep learning algorithms are also suitable for the precise localization of objects and defects.

Suitable for deep learning

Deep learning can be used to precisely detect objects, as the example of 'Halcon' from MVTec shows.

© MVTec

Deep learning technologies are predestined for certain areas of use within machine vision applications. These primarily include classification, object detection and semantic segmentation. However, deep learning is only suitable to a limited extent for other applications in machine vision. As extremely large amounts of data usually need to be analyzed, training usually requires very large computing capacities and appropriately dimensioned hardware. A standard CPU is therefore not sufficient in many cases, especially for highly time-critical applications.

This applies, for example, to high-speed applications such as high-precision measurement tasks or the localization of objects with an accuracy in the millimetre or even micrometre range. On a standard CPU, deep learning algorithms require calculation times in the order of 50 to 100 ms. In many applications, however, only a few milliseconds remain for exact positioning. In such applications, deep learning would only be feasible in combination with a powerful GPU, which is often not available in industrial hardware.

Deep learning is also not the optimal choice in industrial applications in which the objects to be recognized or inspected only have a small variance, such as in the electronics and semiconductor industries. As the corresponding components usually look very similar here, only a few sample images are required for training. Often, a single image is sufficient to accurately recognize and localize the objects.

Minimum 100 training images per object

However, the use of deep learning algorithms only makes sense if there is a high variance of objects and at least 100 training images per object are available. Image processing tasks with a low variance of objects to be recognized can therefore be solved using conventional methods such as rule-based software technologies. Heuristic methods are also better suited to reading codes than deep learning algorithms. Heuristic methods such as sub-pixel-accurate contour extraction are also used for the metric measurement of objects.

However, if there are use cases in which deep learning can unfold its full potential, certain challenges must be taken into account: Deep learning is a comparatively young technology for which the market still offers hardly any common standards. In addition, the entire handling of deep learning is extremely complex and requires in-depth knowledge and many years of experience in dealing with artificial intelligence, programming and industrial image processing. Companies often reach their limits here as they do not have the necessary specialists.

Pre-trained deep learning networks

Companies can still benefit from the advantages of deep learning technology at a reasonable cost, for example by using pre-trained deep learning networks. The market offers a selection of free open source solutions for this purpose. However, their use is associated with a number of pitfalls: It should be noted, for example, that licensing problems can arise, as several hundred thousand sample images are often required for the precise identification of objects. This large number is required because many different characteristics such as color, shape, texture or surface structure are decisive for the recognition process. When selecting this large number of images, it must be ensured that they are free of third-party rights, which is often not guaranteed with open source products. Another challenge when using open source tools is that they usually only solve certain machine vision tasks in isolation and are difficult to integrate with other applications or into existing frameworks. This is because typical problems in image processing always involve several steps: First, the digital image data must be fed from the image acquisition device into the respective application. The second step involves pre-processing. The images are optimally aligned in order to bring the objects into the desired position. Finally, the processed data is passed on to other components, such as a PLC, so that the results are seamlessly available for further process steps. Open source systems usually reach their limits here.

Proprietary standard software

A more practical solution is offered by proprietary standard software solutions for industrial image processing that already have pre-trained networks. These include, for example, the 'Halcon' toolbox from MVTec, which is equipped with all the important features for training deep learning networks. Thanks to its wide range of functions and specially preconfigured tools, it can be easily combined with other applications. It also makes licensing issues a thing of the past. The solution contains several networks that have been pre-trained on the basis of around one million carefully selected, license-free images from the industrial environment. The advantage for companies: They only need a few additional images to finally tailor the training of the networks to their own specific applications. This significantly reduces the amount of training required, saves costs and does not involve any risk with regard to image rights.

There is also another challenge to be aware of when using open source tools: Deep learning applications usually contain several 100,000 lines of programming code. In order to function without errors, this code must meet certain quality criteria. The use of open source code developed by an unknown community entails certain risks in this respect. To be on the safe side, companies should check the quality of the code internally. This entails enormous effort and barely manageable costs due to the high volume. With a proprietary, commercial solution, on the other hand, companies benefit from high-quality, quality-tested and secure code. In addition, professional contacts and experts are available in the event of support queries, which is not necessarily guaranteed with a free community.

Author:
Johannes Hiltner is Product Manager 'Halcon' at MVTec Software in Munich.