Machine vision is often described as the eye of production. This is because the technology automates manual tasks, teaches robots how to see and performs quality and inspection tasks with unparalleled accuracy. Artificial intelligence and deep learning methods in particular are giving the technology additional momentum. More and more applications are benefiting from machine vision that could not previously be automated accordingly. In addition, the performance of existing applications can be significantly improved. Virtually all industrial sectors, electronics and semiconductor manufacturing, even agriculture, benefit from the advantages.

Bosch is also relying on the advantages of machine vision in conjunction with deep learning. In this specific case, the mobility division of the technology and services company uses machine vision to check the quality of electrical connections between circuit boards and sensors. The global company also has sites in Portugal. Among other things, systems and functions in the areas of vehicle safety and dynamics, driver assistance, automated driving and car multimedia are developed and manufactured there. Quality inspection has now been changed in the production of its electronic components for various customers in the automotive industry. "We already had an automated inspection process. To further optimize this, we decided to use machine vision software with deep learning methods. We want to increase our productivity and reduce the amount of work on the image processing application," explains João Paulo Silva, inspection expert from the 'Center of Competence Optics and Mechanics' department at Bosch Automotive Electronics in Portugal.

Robust detection of different defects with little effort

Once trained, the system recognizes errors such as missing labels or incorrectly placed or missing components.

© MVTec

The application involves checking metal springs for defects. These metal springs form the electronic connection between the main circuit board and a copper feed-through on the cover of a sensor. As the processing is carried out manually, various defects can occur on the metal spring during production. These must be reliably detected in order to achieve the high quality standard of an automotive sensor. Until now, the inspection process has been carried out using rule-based industrial image processing methods. João Paulo Silva and his team decided to optimize the process with new methods and components, relying on modern deep learning technologies. "We pursued three goals with the retrofit: Firstly, to improve the overall quality of the inspection. Secondly, the new solution should also be more cost-effective and thirdly, it should reduce maintenance work for the application."

In its search for new software, Bosch found what it was looking for in the 'Merlic' machine vision software from MVTec Software. MVTec is a leading international software manufacturer for industrial image processing. "We have been working with MVTec for a long time. Merlic has the advantage that it is particularly user-friendly, highly flexible and at the same time has the most modern functionalities. We can also contribute to the MerlicC roadmap. This means that our input for the further development of the software, for example on functionalities that we need, is taken into account," explains Silva.
The technology from Merlic that Bosch required for the redesigned machine vision application is called Global Context Anomaly Detection. This deep learning-based technology is a further development of classic anomaly detection. The advantage of Global Context Anomaly Detection is that it is able to recognize completely new variants of anomalies, such as missing or incorrectly arranged components. This means that fault detection is no longer limited to local defects, but also enables context-dependent and logical inspection. This opens up completely new application possibilities. For example, missing or incorrectly installed components or missing labels can be identified or completeness checks carried out.

Neural networks check for local and logical defects

The structure of the inspection system

© MVTec

Image processing in this application is as follows: A five-megapixel camera captures an image of each component from above. Polarized flat dome lighting is used as the light source. Global Context Anomaly Detection is used to inspect the recorded images with the metal springs. The deep learning technology has two neural networks. The 'local' network checks for small defects such as scratches, cracks or soiling. The 'global' network goes one step further and checks for logical errors. For example, whether the metal springs are bent, missing completely or whether other components around the metal springs are missing. Global Context Anomaly Detection determines an anomaly score from the interference of the two networks. This value is then compared with the anomaly threshold value defined in advance. If the anomaly score is higher than this, it is by definition a faulty component, which is then rejected as non-OK (NOK). The anomaly threshold value can be set manually within the Merlic software. This means that the image processing specialist can individually determine how severe the anomalies may be before a component is classified as NOK. This is useful for processing different materials, for example.

Back to the system at Bosch. In the Merlic front end, each image can be viewed again after the inspection. Particularly helpful: A heat map can be used to transparently trace which areas of the image are the cause of the anomaly determination. The images can also be easily saved out of MerlicC.

An important aspect of deep learning is the training of the neural networks. Bosch also benefits from the technology in that only 'good images' are required to train the deep learning application. These are easy to obtain in practice. Of course, the Global Context Anomaly Detection model also benefits from bad images if they are available. With classical methods, on the other hand, all possible types of defects must be extracted individually using 'bad images'. This makes the application less flexible, the maintenance effort significantly higher and unknown defects are not recognized. The neural networks are trained using MVTec's deep learning tool. The tool makes it easy to train data, even without programming knowledge. After training, the networks are simply loaded into Merlic and operation can begin.

Integrating machine vision software into the machine control system

But how is it possible to integrate machine vision software into an existing production process? This question was particularly exciting for Bosch, as the production process and the integrated quality inspection were not to be changed. The metal springs are still enclosed in a machine by an upper and a lower cover. The lower assembly is inserted into the machine by hand. Even if the risk is low, this can cause damage to the metal springs. This is why the inspection must be carried out at precisely this point, namely before the upper component is mounted. The image feed, i.e. the image capture, still takes place from above.

As the production process remained unchanged, the main focus was on integrating the image processing software into the machine control system. The software had to be connected directly to the machines, as the system does not have a programmable logic controller (PLC). The MQTT protocol integrated in Merlic provides the necessary machine-to-machine communication. This allows the image processing software to be easily integrated into the process via standard IoT communication protocols. The development of the calibration program for the image processing system can be accelerated using the easy-to-use software.

Further projects with industrial image processing planned

"We successfully completed the proof of concept at the end of 2022. We achieved all of our goals in terms of detection rates, system maintenance and costs. A new production line will therefore be commissioned in mid-2023. The rollout to other existing lines is then planned," explains Joao Paulo Silva. Given the potential, Bosch is planning to automate further automotive electronics plants with the help of deep learning in the future.

Interview: 3 questions for...

Christian Eckstein, Product Manager & Business Developer at MVTec in Munich

© MVTec

Deep learning has been experiencing a lot of hype for some time now. What advantages does deep learning offer the manufacturing industry?

Artificial intelligence (AI) is indeed currently experiencing a real boom and is attracting a lot of attention in business and society. AI, and in particular deep learning in machine vision-based applications, also offers great potential for the manufacturing industry.

The big advantage: while traditional approaches explicitly describe and process the properties of an image, deep learning - as the name suggests - focuses on a learning process. Neural networks are comprehensively trained with the help of image data sets. The relevant image properties can then be specifically identified and evaluated on this basis. This method enables outstanding identification rates and paves the way for completely new inspection applications that could not previously be realized with machine vision.

Which deep learning technologies are particularly suitable for inspection applications?

There are various deep learning methods for quality control, such as classification, segmentation and object detection. Errors or defects can be reliably found using the 'anomaly detection' method. Particularly relevant for practical suitability: Only images of workpieces without defects are required to train the model - and these are much easier to obtain than images that have to show all possible defects. At MVTec, we launched the 'Global Context Anomaly Detection' feature on the market in 2022. This takes deep learning-based anomaly detection to a new level: previously, it was possible to detect localized, structural anomalies. The enhanced technology is now able to accurately detect completely new types of defects, such as missing, deformed or incorrectly arranged components. This means that defect inspection is no longer limited to the mere identification of local anomalies, but can now also comprehensively 'understand' the logical content of the entire image.

What additional applications can be implemented with this?

Let me give you an example from the electronics industry: The new 'Global Context Anomaly Detection' technology can reliably identify missing or incorrectly positioned components on printed circuit boards as well as missing labels and check the products for completeness. The feature uses training data to independently learn the correct quantity of screws and nuts in a pack, for example - and immediately sounds the alarm if the wrong number of items are present. Although such applications were previously possible, they required a great deal of programming effort.