The interaction between robots and machine vision software has taken a huge leap forward in recent years. The Irish Manufacturing Research (IMR) provides an example of the growing application possibilities of robots and industrial image processing software. The IMR is a research and technology facility with a broad portfolio of research, training and consulting services in the field of Industry 4.0. Together with Multipix Imaging and MVTec, a fully automated robot application with machine vision was recently developed that can process components for knee implants.

From manual to automated removal

The sequence of the individual process steps as they are carried out one after the other.

© IMR

The 3D image processing application developed by IMR enables the fully automated identification and localization of randomly aligned, complex-shaped parts. The robot can pick them up precisely and repeatably and also put them down again safely. Until now, the application was carried out manually. However, in order to increase efficiency and for cost reasons, the application should be robot-based in future. However, due to the highly polished, reflective surface and the complex shape, such applications are difficult to implement.

As project partners, the IMR chose the companies Multipix Imaging from Petersfield in the UK, a distributor of image processing components with a focus on promoting image processing in automation and manufacturing processes, and MVTec, based in Munich, as a software manufacturer for industrial image processing.

The image processing software was a crucial component in the implementation of the application, as the nature of the implant surfaces is very challenging - the difficulty lies in the wide variation of surfaces from matt to highly reflective as well as in the complex shapes. There are also challenges arising from the process environment: Parts are concealed by container walls, are arranged randomly and have to be picked and placed from containers of different sizes.

At the same time, the demands on the application as a whole were also high: the complex-shaped, highly reflective parts must be machined in the six degrees of freedom (6DoF) with an accuracy of +/-3 mm. In addition, the cycle times had to be less than 15 seconds. The customer requires a unique system that can be used across all polishing stages. And last but not least, it should be possible to operate and work with a robot system with components in a semi-structured configuration.

Interaction between hardware and software

At the center of the implemented application is a 6-axis UR3 robot equipped with an end-of-arm tool (gripper). A ring light was installed in the robot cell to optimize performance and ensure uniform ambient lighting. A 2D industrial camera is used for image acquisition. The application is controlled by a laptop with 'Halcon' software installed, which uses a 2D image camera to localize the object. The coordinates of the localized parts are then sent to the robot controller via TCP/IP.

In use, the application works as follows: The container with the components is introduced into the robot's work area. It then independently picks up all the workpieces, without touching other implants, and sorts them according to where they belong. To do this, the robot must know which component it is currently picking up and where to place it.

To enable the robot to 'see' the components, it uses the 'Halcon' image processing software from MVTec: the relevant technology for the application was shape-based 3D matching. This shape-based matching technology finds objects precisely and robustly - even if the parts are rotated, scaled, distorted in perspective, locally deformed, partially covered or outside the image, or subject to non-linear lighting fluctuations.In practice, the image recognition process works by first loading the 3D CAD models of the objects to be captured into the software in order to create the 3D shape object model function. The image processing software first creates the 3D object model, which is then used for the subsequent matching process. The tool is able to calculate the different views of the 3D CAD model based on the surface viewing directions of the parts specified by the user. The views are generated automatically by placing virtual cameras around the 3D CAD model and projecting the model into the image plane of each virtual camera position. For each view obtained in this way, the software calculates a 2D object representation. This means that no real images of the object are used to generate the 3D object model, only the 3D CAD model. The object representations of all views are saved in the 3D object shape model, which is returned by the operator and saved in a file for later comparison.

The author: Kevin Dooley is Project Manager at IMR.

© IMR

When the part is removed, a 2D camera image provides the profile of this part, which is compared with the 3D CAD profiles stored in Halcon. Based on the comparison between the stored profiles and the 2D camera image, a score between 0 and 1 is generated to determine the optimal part profile. The 3D coordinates of the part are then generated and sent to the robot.

An initial company has already developed its own solution based on IMR's research application. IMR is therefore already developing further projects with robots and the Halcon image processing software, for example for the detection of liquid reagents in a biomedical application.