Prototype of the camera-based joint sensor system at the Hannover Messe 2018. The joint angles are measured and the robot arm (center) is controlled solely by image processing of the recorded camera images (top right).

© RoVi RobotVision

Sensor technologies for environmental perception and the physical interaction of a robot are key technologies for such robot systems. Sensors for measuring joint angles, force-torque sensors and tactile sensors, all of which are now realized as dedicated electronic modules, are crucial in this respect. Cameras are also commonly used for object detection and localization as well as for process control.

RoVi's technology replaces complex hardware-based sensor systems with its sensor software, enabling robots to 'feel by seeing'. In technical terms, this involves the robust measurement of joint positions, positions, gripping forces, contact profiles and other tactile/haptic contact information using image processing and cameras mounted externally - i.e. next to or above the arm or on the gripper.

This sensor concept can be vividly explained using a human analogy: With their eyes closed, a person can only position their arm inaccurately, as their perception of the position of their limbs and their joint positions is not very precise. Although classic industrial robots generally work 'blind', unlike humans they use high-precision sensors together with a rigid construction to achieve highly accurate positioning of the end effector.

Rendering of a fully integrated gripping system equipped with an industrial gripping module, two cameras and camera-based tactile sensor technology.

© RoVi RobotVision

Humans, on the other hand, also use visual perception and thus multimodal information processing to enable the precise manipulation of objects. In a similar way, RoVi's software-based sensor technology uses cameras to record the joint positions of a robot arm, positions and gripping forces, thereby enabling the robot to interact precisely and autonomously with objects.

The software calculates the joint positions of a robot arm in three-dimensional space using image analysis algorithms. Force-torque and tactile sensors are replaced by simple passive flexible elements - for example, inexpensive foam. Contact forces lead to characteristic deformations of these elements. The software also measures this deformation using image analysis methods and calculates the applied forces and torques based on a material model. The sensor software can be applied to complete robot systems, robot arms, grippers or mobile robot platforms and enables the intelligent control of these systems.

Industrial robot arms with conventional hardware sensors have very high absolute accuracies (<0.1 mm) and joint speeds, as they are specially designed for this purpose. Hardware sensors can deliver sampling rates of >1 kHz. The aim of RoVi technology is not to compete with these values, but rather to open up relevant new areas of application based on the relative accuracy between the robot and the target object. Like any industrial arm (if augmented with a camera), the technology is limited by the camera-based position estimation of the target object. In typical set-ups with simple cameras (less than full HD resolution), a positioning accuracy of around 1 mm can be achieved, with UltraHD cameras up to <0.3 mm. The accuracy of the tactile and force/torque sensors depends heavily on the camera setting in terms of distance and resolution. In typical cases, an accuracy of less than 5% can be achieved, which is comparable to the accuracy of hardware sensors in the mid-price segment.

Affordable automation

The sensor software is intended to enable new types of robotic arms and grippers that act sensitively and autonomously, but are technically simplified and therefore cost-effective to manufacture. Cameras are becoming standard in more and more robotics applications, as they are used to recognize the environment or objects. In addition, camera modules are becoming increasingly powerful and inexpensive due to their widespread use, for example in smartphones. RoVi's sensor software uses these cameras to replace a large number of sensor components in robotic systems.

Illustration of the possible applications of the various camera-based sensors. They replace the entire electronic sensor system on current robot arms and thus reduce the amount of cabling required, for example.

© RoVi RobotVision

As the current robot configuration is always measured via external cameras and software, connecting elements do not necessarily have to be designed to be as rigid as possible and manufactured with high precision. Instead, flexible elements and materials as well as simpler manufacturing processes with lower precision can be used.

Small and medium-sized companies in particular, which today often still use few robots, benefit from the lower costs of automation. With their technology, the founders of RoVi are primarily targeting the growth market of service robotics.

The fundamental changes in the design of the robot arms also make applications possible in the private sector, for example. One application, for example, would be robotic vacuum cleaners equipped with small arms that could clear obstacles such as toys out of the way or vacuum on skirting boards. However, a number of hurdles still need to be overcome in research into robotics in general for widespread use in the home.

Author: Stefan Lochbrunner is co-founder and chief developer at RoVi in Munich.

From the idea to the start-up

The idea for the camera-based sensor technology for intelligent robots originated from the doctoral thesis of co-founder Nicolas Alt, whose dissertation topic was the 'visuo-haptic' perception of robots. Inspired by observing how robot vacuum cleaners deliberately and repeatedly encounter obstacles for tactile mapping, he developed a tactile sensor for mobile robot platforms - at the time still part of a research project. This consisted of a simple foam bar that was observed by an existing camera on the robot and also served as a soft bumper. The same concept applies to the tactile sensors that RoVi attaches to the fingers of grippers.
Later, the concept of using an external camera in combination with passive elements was extended to develop force-torque sensors and joint angle sensors for robots.

In 2016, the research project evolved into a spin-off project with the intention of further developing and commercializing the technology. The start-up is currently being financed by Exist Forschungstransfer, a funding program of the Federal Ministry for Economic Affairs and Energy specifically for high-tech start-ups from research.