The importance and purpose of sensors vary greatly between the different types of robots. While the arms of classic manufacturing robots, for example, make little use of sensors and generally only follow a pre-programmed path, on the other side of the spectrum there are mobile robots that work in predominantly human environments, for example robotic trolleys that transport laundry in hospitals and rely heavily on sensors. The traditional approach is effective when everything around the robot is fixed, i.e. it works largely isolated from humans and the objects handled by the robot are always in the same place. Robots with appropriate sensors, on the other hand, move in dynamic environments in which unexpected movements by a human colleague can suddenly block the robot's path. In order to work safely in such environments, robots must be equipped with 3D cameras, lidars, sonars and the like to be able to perceive and adapt to their dynamic environment.

The requirements for a robot solution for a logistics warehouse lie somewhere in the middle: The environment doesn't change too much, but the objects that need to be handled change with each task and their positions are often uncertain too. The use of 2D and 3D cameras helps to find these objects in particular.

Perceive surroundings

3D cameras can be used in pick-and-place systems to capture detailed images that help the robot develop a comprehensive understanding of its environment. Image processing algorithms process this data and enable the robot to perform its pick-and-place tasks accurately and efficiently. By continuously analyzing and evaluating its environment, the robot can effectively perform different tasks simultaneously.

360° barcode scanners help to optimize warehouse processes through intelligent route planning. This enables fast reading from any angle and ultimately minimizes the need for manual adjustments.

Picking up and putting down

The integration of computer vision into robotics enables object recognition, navigation and precise task execution.

© Smart Robotics

When a robot picks something up in response to a corresponding request, it simultaneously sends an image capture request to each 3D camera. Using depth image analysis, the algorithm then identifies the exact container from which something is to be picked up and updates the robot's world model to ensure accurate positioning and avoid collisions. If objects need to be placed in a bin, a new 3D image is generated to recognize both the bin and its contents. A special stacking algorithm then determines the optimum stacking sequence and also takes distance calculations into account to ensure gentle handling.

Detailed work

2D cameras can be used to identify the items to be picked, complemented by computer vision and deep learning algorithms to determine their position and quantity within the bin. Using this information, the robot can update its collision model and carry out the removal of items without collisions. In addition, the deep learning algorithms can also recognize the material type of the items, for example whether they are made of cardboard, paper, hard or soft plastic, so that the robot can select the correct suction cup for the respective material.

Meanwhile, a weight sensor integrated into the gripper can detect the mass of each object. This helps to determine if the robot has inadvertently picked up two objects instead of one and adjust its speed of movement accordingly to prevent it from dropping them.

As there may be several objects in the bins, which may interfere with each other, the robot may not know all the dimensions accurately at first. Therefore, a new image of each removed object is required to accurately capture its dimensions. This task is facilitated by the 3D camera, which captures a new image of the object that the robot is holding.

It is also possible for the robot to determine the ideal gripping position by precisely analyzing the orientation of the objects. If an object is lying at an angle, for example, the robot must adjust its gripper so that it tilts its suction cup towards the correct surface.

Increased reliability

The Smart Parcel Picker is an intelligent parcel picking robot for depalletizing and unloading parcels.

© Smart Robotics

It goes without saying that improved vision and recognition functions significantly increase the reliability of robots. As the robot derives object information directly from the 3D images, tedious SKU teaching is no longer necessary. SKU (Stock Keeping Unit) is a stock unit that can be uniquely identified by a code and is stored in a company's system. The robot learns continuously, learning from every 3D image it analyzes. This improves its capabilities and leads to faster and more accurate item recognition. Thanks to the robot's continuous analysis of the environment, which includes the position of the containers and the dimensions of the items, the likelihood of collisions is also significantly reduced.

Developments in camera technology

There are currently various interesting developments in the field of camera technology: In 2D cameras, for example, a rapid increase in resolution can be observed. While resolutions of 12 to 48 MP for smartphone cameras are not uncommon in the area of consumer cameras, the resolution in robotics was often limited to 1 MP. In recent years, however, manufacturers have launched resolutions of 5, 8 and more MP on the market. This means that smaller objects and details can now be better recognized.

The development of 3D cameras has gone even further. In the last year in particular, the number of suppliers of 3D cameras with a range of technical specifications, a significantly higher resolution and consequently improved accuracy has increased considerably. Of particular importance here is the integration of deep learning to improve the quality of 3D data. For most 3D cameras, it is problematic to capture complete and accurate data for all surfaces within an image, often resulting in gaps or areas of high uncertainty. Cameras that utilize deep learning algorithms step up to intelligently fill these gaps based on sample images.

Image processing, movement and task planning

Robots need to handle a wide variety of objects with precision and continuously adapt and improve their efficiency - in complex environments. To achieve this, careful orchestration of image processing, motion and task planning is required. The integration of computer vision into robotics has revolutionized the way robots interpret their environment through digital images or video, enabling object recognition, navigation and precise task execution. However, the development of safe and efficient motion planning systems remains a challenge, as sophisticated algorithms are required to precisely control a robot's movements.

The author: Heico Sandee is CEO and co-founder of Smart Robotics in Best, the Netherlands.

© Smart Robotics

Task planning and execution software complements these systems and enables robots to understand, organize and execute tasks autonomously. While movements account for the majority of a robot's cycle time, motion planning is used to guide the robot's path from A to B. Task planning, on the other hand, includes all the actions that a robot has to perform, such as moving to the removal position, creating a 3D image or switching off the gripper, and ensures that they are executed at exactly the right time to achieve the goal, for example the removal and placement of an item.