Artificial neural networks trained with large amounts of data enable intelligent cameras to solve increasingly demanding applications in industrial environments. The integration of deep learning algorithms in image analysis and processing software makes it possible to automatically recognize, inspect or classify trained objects or features. As a result, thanks to the intelligent functional specialization of sensors in areas such as food and wood processing, it is possible to increase material utilization, stop wasting resources and improve the quality of products and processes. In logistics automation, deep learning cameras can check sorter trays for the presence of flat shipping bags and their actual occupancy by evaluating the taught-in image base or recognize objects lying next to or on top of each other on a conveyor belt as individual units. Thanks to deep learning, sensors provide intelligence services that were previously reserved for humans - for example, the recognition and evaluation of structures or features that are detected by the sensor in this form for the first time during operation. This makes deep learning, as a sub-area of machine learning, probably the most important future technology within the field of artificial intelligence and, in the long term, a driver of Industry 4.0.

Structured deep learning workflow

In logistics automation, deep learning cameras can, for example, recognize objects lying next to or on top of each other on a conveyor belt as individual units by evaluating the taught-in image base.

© Sick

Most of the deep learning projects that Sick is currently working on come from manufacturing applications or optical quality inspection. In order to be able to assess the various applications efficiently, the company has developed a multi-stage process standard as part of an internal deep learning initiative involving both Sick's deep learning experts and the customer's process and quality experts.

Even though modern 2D and 3D cameras are becoming ever faster and more powerful, classic image processing algorithms are still being pushed to their current limits. From a machine learning perspective, the only question that arises is the uniqueness of criteria: Can these be recognized and interpreted with sufficient clarity across a large number of images, good as well as bad examples? From the customer experts' point of view, what is good or bad, yes or no, tolerable or not, or okay or not okay with regard to certain criteria? Can the experience- or knowledge-based assessment capability of a human being be provided by the sensor as intelligence at all? If these questions are assessed positively by the respective application engineer, the recording and annotation of many images by an experienced person creates the training data basis for the subsequent deep learning algorithms in the sensors.

Neural networks consist of layers. The degree of abstraction increases from concrete image details to coarser concepts in the layer sequence. This ensures that the network can process previously unseen data correctly with a high degree of probability. A neural network learns to solve a given task using suitable training data.

In general, open frameworks are available for deep learning development in order to define and train neural networks. However, these frameworks were developed without any specific reference to sensor technology or image processing. This is precisely where Sick begins to explore the limits of sensor technology with deep learning.

Deep learning functionalities in the sensor

In contrast to the process of classic algorithm development, which is mainly characterized by the manual design of a suitable feature representation, a neural network learns optimal features for its task on its own. It can be retrained again and again with suitable data in order to adapt to new conditions. Sick uses an independent computer and IT basis as the executing unit both for building up the training data set by capturing thousands of images and examples and for training the neural networks. The extensive calculation of the complex operations of the deep learning solution for training is carried out on specially equipped, in-house computers with high GPU (Graphics Processing Unit) performance. The new deep learning algorithms generated from this are provided locally on the sensor and are therefore available immediately and fail-safe, for example on an intelligent camera.

An example of a deep learning application was recently implemented in wood processing. Here, the position of the annual rings was trained using a large number of images of rough-cut boards with tree edges. The aim was to use a programmable camera from the 'Inspector P65x' product family to recognize the rotational position of the annual rings. Based on this training, the camera can evaluate new, unfamiliar images and assign them to a result. Deep learning was used to teach the camera how best to use the wood - a task previously performed by experienced people. As a result, thanks to the camera, the wood is now positioned in the machine in such a way that optimum processing and material utilization is achieved.

Expansion of the sensor portfolio

With the implementation of deep learning in selected sensors and sensor systems, Sick is launching a sensor software concept that creates adaptable and future-proof solutions for automation applications. The upcoming deep learning products include other image-processing sensors and cameras. In principle, the concept of the specialized sensor with artificial intelligence can also be applied to simple sensors such as inductive proximity switches, retro-reflective photoelectric sensors, ultrasonic sensors and others. In addition, system solutions such as vehicle classification at toll stations offer potential for deep learning-supported classification and categorization of vehicles into toll classes.

Authors:
Andreas Behrens is Head of Product Management Barcode RFID Vision at Sick in Reute,
Klemens Wehrle is Head of Track and Trace Systems, Research & Development at Sick in Waldkirch.