In today's production facilities, the workspaces of humans and robots are still generally separated from each other by fences, light barriers or similar safety technology. As soon as this strict separation is lifted and people share the workspace with fixed manipulators or mobile robots, interactions inevitably occur. These can be both unintentional interactions - for example when the paths of humans and mobile robots cross by chance - and intentional physical interactions such as the handover of parts or tools. As far as the latter is concerned, in view of ever-increasing product variability and shorter product life cycles, the primary aim in future will be to interlink routine tasks that robots can handle efficiently with demanding or specific tasks that humans are better suited to handling.

It is crucial for the success of such concepts that the robot always behaves safely in the shared workspace. The Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB has developed an appropriate method for this based on environment detection. People and objects in the workspace are detected by sensors so that a three-dimensional model of the robot's environment can be created. This makes it possible to calculate the distance between the robot and the next obstacle, and the robot can slow down and stop in good time before colliding with people or other obstacles in the workspace.

Various sensor principles can be used to obtain 3D information: time-of-flight principle (laser scanner/LIDAR, PMD cameras), triangulation principle (pattern projection, 'Kinect' sensor) or cameras based on the stereo principle. The sensors can be permanently installed in the workspace or mounted on the robot. The second variant is particularly useful for mobile platforms and mobile manipulators, as it allows their potentially very large workspace to be covered with a reasonable number of sensors.

Due to limited sensor fields of view and possible occlusions, several sensors are usually required to adequately monitor the workspace. This ultimately led to the development of a process that can be used to merge 3D information from different sensors. In other words, it combines the data from the individual sensors to create a 3D model of the local robot environment (Fig. 1). In addition, hidden areas that cannot be seen by the sensors because they are behind an object (Fig. 2 above) are explicitly considered and taken into account when calculating the current distance between the robot and obstacles. Occlusions are particularly critical if the obstacle is located between the sensor and the robot, because in this case only the object surface facing the sensor is detected, while the object parts closer to the robot are not perceived by the sensor. If occlusions were not taken into account, the obstacle distance would be underestimated, which must be prevented for safety reasons. By merging sensors with overlapping fields of view, the extent of occlusions can be reduced (Figure 2 below).

Figure 2: The coverage of the working area can be increased by merging sensors. In addition, the extent of occlusions is reduced in the case of overlapping fields of view.

© Manfred Zentsch / Fraunhofer IOSB

A further challenge arises if the robot itself can be seen in the sensor data in addition to the obstacles. Before calculating the distance, a distinction must first be made between robot and obstacle points. This is done by calculating a 3D model of the robot in its current state and filtering out the corresponding area from the sensor data so that only the actual obstacle points remain. In addition to the current joint angles of the robot arm, the objects with which the robot is to interact must also be taken into account. For example, objects gripped by the robot are added to the model so that they are not considered obstacles during transportation.

In addition to the 3D detection of the immediate robot environment, moving obstacles are detected and predicted in the entire workspace. The current speed and direction of movement of people, vehicles or other objects are estimated from the time history of their movement. From this, predictions can be made about object positions in the near future. The areas likely to be occupied by obstacles are compared with the robot's planned path. If there is a risk of collision, the robot can slow down or stop at an early stage. As soon as the obstacle has left the path again, the robot continues its journey.

Cooperate intuitively

Ensuring that the robot's path planning is safe for humans and the environment allows it to operate autonomously in the human's immediate workspace. For human-robot cooperation, the cooperating employee must also be detected, the situation analysed with regard to their activity and the interaction commands recognized interpreted and executed.

In order to enable humans to interact intuitively with the robot, methods should be supported that humans can use without consciously thinking and without consciously shifting their attention. The most obvious methods are those that humans use in everyday dialog with other people - such as speech, gestures, facial expressions, gaze or posture.

Against this background, the researchers at Fraunhofer IOSB are developing and implementing processes for camera- and video-based human-robot interaction that support these communication methods and also enable the robot to perceive and understand its cooperating employee, their intentions and the context of their actions. The basis for this is the same fused 3D depth data that is obtained - as for workspace monitoring - from environmental detection, for example using Kinect sensors in the environment or on the mobile platform itself. Special methods for detecting people and capturing body poses are used to localize people in the robot's environment and capture their posture in the form of a simplified skeleton model.

Figure 3: Schematic representation of the captured skeleton (green) during a pointing gesture.

© Manfred Zentsch / Fraunhofer IOSB

Figure 3 shows a schematic representation of such a skeleton model. The skeleton model can be used to examine the features that are important for the interaction, such as the orientation of the face, pointing gestures in the form of outstretched arms or dedicated hand poses: Relevant body parts are analyzed from the depth data with additional classifiers and placed in a temporal context. When identifying bystanders, this ensures the correct assignment of an interaction command to the person who originally made it.

Tracking individual arm movements also allows the detection of intended gestures for interaction - for example, pointing at certain objects or tools - as opposed to a general arm movement. And the assignment of head orientations to bystanders makes it possible to anticipate intended interactions as soon as attention is clearly directed towards the robot. The number of detection components to be used and evaluated in real time depends directly on the complexity of the underlying use cases or interaction scenarios: In the case of a dedicated human-robot interaction where only a single person is present, methods for distinguishing between people become obsolete. Similarly, in scenarios where gestures and speech clarify explicit interaction intentions, temporal context detection can be minimized. This is because a person's interaction can always be assumed to be given by the given combination of the two modalities and does not have to be detected first. At Fraunhofer IOSB, this necessary modularity is already taken into account in the basic design of the methods in order to be able to react flexibly to different fields of application and use cases.

Can also be used in dynamic environments

As soon as the robot receives a command, it plans a collision-free path to its target based on the described procedure, taking into account a 3D model of the static obstacles and the obstacles currently detected by the sensors. This means that the robot can also be used in dynamic environments where the position of objects such as chairs or parked vehicles is not known in advance. The transfer of objects is controlled by the robot recognizing the force exerted by the human when gripping the object and then releasing the object. This intuitive behavior allows even inexperienced users to pick up objects without any problems.

In a nutshell: while some alternative methods can only react once a collision has already occurred, the Fraunhofer IOSB's approach attempts to avoid collisions between the robot and obstacles from the outset on the basis of 3D environment detection. In addition, the sensors can be mounted on board mobile robots so that their use does not have to be restricted to specially equipped work areas.

However, the method described cannot be considered a safety concept for the approval of robots because the 3D sensors used do not meet the requirements (safety integrity level/SIL) from the relevant standards in terms of reliability, robustness, response time, 2-channel evaluation, etc. A different argument must therefore be used for approval: in the case of the demonstrator described, for example, the platform's safety laser scanners and the proven low risk of injury in the event of collisions with the lightweight arm. Against this background, 3D workspace monitoring should be seen as a 'comfort system' that also prevents minor collisions, reduces the robot's downtime to a minimum and thus increases the usability and acceptance of human-robot interaction.

Implementation using the example of assembly

The overlapping detection areas of the 2D laser scanners (blue/purple) and the 3D depth sensors (yellow/green) enable the detection of obstacles (red) around the robot (white/orange).

© Manfred Zentsch / Fraunhofer IOSB

A Kuka Omnirob mobile manipulator is available at the Fraunhofer IOSB for the development and demonstration of the methods described. This is an omnidirectional mobile platform with a 7-axis lightweight arm. The platform has two laser scanners that detect obstacles in a plane above the floor. In addition, two triangulation sensors (Kinects) were installed to capture the working area of the lightweight arm in three dimensions. This makes it possible to avoid collisions between the arm and obstacles without restricting its ability to reach onto a table or shelf. The described workspace monitoring system merges the data from the four sensors mentioned.

Assembly support serves as a demonstration scenario: the robot picks up tools or parts that can be selected using gesture control, for example, and hands them over to the worker or places them at the workstation. The safety of people interacting with the robot or unintentionally entering its workspace is guaranteed at all times thanks to environment detection.

As part of the EU project SAPHARI (Safe and Autonomous Physical Human-Aware Robot Interaction), the concept for 3D workspace monitoring was also integrated on another robot and successfully demonstrated in the application scenario of screw logistics for robot production at Kuka Roboter. An Omnirob takes small load carriers with components from a shelf, shows them to an employee for quality control and places them at a predetermined target point (depending on whether the quality control has been passed or not).