Most users are aware that there are problems with conventional image processing in conjunction with real-time under Windows and the ever faster and more efficient machines, and some have already experienced this themselves. Using an imaginary example, various solutions and avoidance strategies are discussed below.

Delayed image acquisition

Currently available image processing systems - such as Halcon, Matrox Imaging Library (MIL for short) or VisionPro - preferably run under Windows. As all Windows users know, this operating system was developed for desktop applications. This results in delays or latency times of up to 500 ms due to the Windows scheduler, which determines the current order in which tasks are processed. Every user is familiar with the behavior where the mouse does not react for a short time and 'virtually' stands still before suddenly moving somewhere uncoordinated. This uninfluenceable behavior can lead to delays during image acquisition - i.e. the transport of image data to the memory via GigE Vision, for example. If the machine control or the process is dependent on an image being available at an exact time, this can lead to an unacceptable time difference, which can result in a machine standstill.

For example, when opening a can of peanuts, everyone expects it to contain only edible nuts. However, it happens time and again that you bite into peanuts that taste strange (= bad). This can be recognized by their color, as the bad nuts are usually much darker. Sorting could be done by dropping the peanuts over a defined distance with simultaneous image recording and evaluation. Of course, no image must be lost, an inedible nut must be reliably detected and removed by blowing it out within a defined period of time. However, if this image is taken too late due to latency times, the nut has already fallen too far for it to be removed.

Complex image algorithms

The algorithms for shape or color recognition are normally applied to every image read into the Windows memory. These image processing algorithms can be very complex and may require different calculation times. This can occur, for example, with iterations (functions whose number of passes cannot be predicted exactly). If these take too long, termination mechanisms can be defined. Every algorithm running under Windows control can be interrupted or delayed during processing. As a result, the result of the calculation may not be available at the required time, which in turn can lead to a malfunction of the machine.

Again, the peanut example: Ideally, the image acquisition should always take place at the correct time without interruption and no image should be lost. The machine requires a signal at a defined and precisely adhered to time (= deterministic) with which the blow-out valve can be controlled to remove the 'defective' peanut. If this signal arrives too late or not at all due to the non-definable latency times of Windows, the valve may be opened and a good peanut removed. In the meantime, the 'faulty' peanut has already fallen a few centimetres further due to the delay or has already arrived in the delivery box.

What remedies or strategies are now available to deal with this dilemma?

The frame grabber strategy

A common solution is frame grabbers, on which not only the image acquisition but also the complete calculation can be carried out directly. In this case, the algorithms must be created in advance as a VHDL program, compiled with the appropriate VHDL compiler - which can take up to a day per compilation process - and then tested. This means a great deal of effort and additional development time. Without programming, such frame grabbers cost several hundred euros each - without including the required interface (GigE, CameraLink or CoaXPress) or the camera. Whether this can be a solution with regard to self-configuring and autonomous machines under Industry 4.0 remains to be seen.

PCI card developed in-house

Another strategy is the complete in-house development of a PCI plug-in card based on a DSP or an FPGA. This means that image capture and all real-time processing takes place on the plug-in card and therefore runs separately from Windows. However, the interface must be programmed and the real-time processing must be developed and tested. In addition, an operating system is required for the DSP or for the FPGA, which may need to be adapted. The image processing algorithms must also be adapted to the chip used.

It is also unavoidable to develop your own hardware with the associated circuit board developments, error iterations and test scenarios. And last but not least, there is the need for stock-keeping for service and breakdowns. The possible discontinuation of a component - in the worst case the central, active component - and the fact that the board is only designed for a limited performance, with the result that a new board has to be developed if more performance is required, are also problematic.

The image processing PC

From these two challenges, 'resourceful minds' devised a strategy that at first glance appears to solve the problem, but on closer inspection turns out to be a sham: the solution is a dedicated PC for image processing only. Nobody is allowed to move anything on this PC, for example the mouse. In this environment, there are also systems in which various Windows services have been deactivated in order to avoid any disruption to the image processing system. However, this can lead to an unstable Windows. And even with a stand-alone PC just for image processing, the tape may stop several times a day for reasons that are not clear. However, in the absence of cost-effective alternatives, this strategy is a common scenario in today's automation world.

In order to get all Windows latency times under control, it would be necessary for image acquisition and image processing to run independently of Windows. Depending on the camera interface used, image acquisition would have to run as an independent process on a separate core to which Windows has no access. In this case, the image transmitted by the camera or the associated image data would not be stored in the Windows memory, but in a separate memory that is independent of Windows.

If the image algorithms could also be applied to the image data independently of Windows, it would be possible to implement a complete machine, including the controller and fieldbus connections, on a PC using modern multi-core processors.