For a long time, hard real-time requirements in terms of industrial image processing only worked on a hardware basis - for example using digital signal processors (DSP) and field-programmable gate arrays (FPGA). However, thanks to instruction set extensions such as SSE, AVX and multithreading, PC processors are now also at a sufficiently high performance level to implement hard real-time properties for industrial image processing.

undefined

The 'Realtime Suite' enables the development of real-time applications and their hardware-related implementation on external systems. In addition to the area of image processing, these are primarily automation and communication protocols.

© Kithara Software

A prerequisite for actual real-time properties in PC-based approaches is the execution of the application code at kernel level. The preemptive multitasking system of Kithara's 'Realtime Suite', for example, ensures that the most important tasks are executed with priority control. Tasks can also be distributed to dedicated CPU cores in order to further improve real-time properties by bypassing the influence of Windows without having to do without its user interface. In terms of functionality, the Windows extension is on a par with a fully-fledged real-time operating system. Programming languages such as C/C++ and Delphi are used to generate native machine code.

Relevant interfaces as well as external devices and systems are connected to the real-time environment via corresponding function modules. The focus here is on meeting time-critical requirements through strictly defined response times.

The right camera interface

Image processing always starts with the camera and the corresponding interface. The 'Realtime Suite' uses specially developed and optimized network and communication drivers, which form the basis for achieving real-time image acquisition with the corresponding camera interfaces. For example, a real-time connection to GigE Vision cameras is established via powerful Ethernet drivers or to cameras with the USB3 Vision standard via XHCI drivers. This makes it possible to use cameras from all manufacturers with the real-time system and exchange them flexibly with each other. To ensure that communication between the camera and the real-time system runs smoothly, large buffer memories are available to prevent data packets from failing. The buffer size and number can be adjusted according to requirements, especially for streaming.

Immediate processing

Machine vision does not stop at image capture. The actual image processing takes place via special libraries, which evaluate and further process the captured image data using a variety of ready-made algorithms. In order to execute the entire vision process in a single real-time context, the corresponding image processing library is also loaded by the real-time system in kernel mode. This ensures that the reaction times between image acquisition and processing are kept to a minimum, fixed period of time. The Realtime Suite supports the free 'OpenCV' library and the 'Halcon' tool from MVTec.

As a link between real-time image processing and control response, developers can directly merge the image processing and machine logic in their application code. Using the Ethercat protocol, thousands of I/O signals with frequencies of 1 kHz or higher can be recorded, evaluated and processed. The I/O topology is flexibly scalable. As a result, a single development environment can be used to convert incoming processed image data into complex sequences that can be used to control machines in real time.

An application example

The 'Metrix Analytic Sorter' from VMEK uses a waterfall method for quality control: The bulk material is evenly distributed by vibration via a hopper device and each individual part is recorded by two cameras while still falling.

© Kithara Software

The importance of real-time for the entire image-supported automation process is illustrated by the operation of a sorting machine from US company VMEK with integrated functions from the Realtime Suite. This is a machine solution for classifying and sorting bulk goods such as agricultural products or large quantities of small workpieces. Two cameras capture the material to be tested from different angles using the waterfall method. Each individual part is inspected at a drop speed of around 4 to 5 m/s for individual criteria such as size, color, impurities or the weight determined by volumetric calculations and sorted out according to the set exclusion parameters using an air blast. The real-time system ensures that the image data received from the GigE Vision cameras reaches the Halcon-supported image processing system within a few microseconds and that the evaluated data triggers the air blast in real time. A prerequisite for a precise sorting process is the short distance of just a few centimeters between the camera and the air blast. Without 'hard' real-time properties, this would not be feasible.

Authors:
Martin Ebert works in Marketing & PR at Kithara Software in Berlin;
Uwe Jesgarz is Managing Director of Kithara Software in Berlin.