With solutions such as the Hololens 2 from Microsoft, the range of AR design options is now expanding.

Augmented and mixed reality (AR/MR) help to increase the efficiency of the workforce by providing information where it is needed most: in the real context of the daily working environment. In addition to smartphones and tablets, the first Hololens data glasses from Microsoft, available since 2016, already offered an effective way of projecting digital twins of physical products or machines or even purely digitally existing prototypes of new developments into the room or onto a table and enabling several distributed people to work on them together. One of the advantages over other output media was the free hands of the employee, which enabled quick guided assembly in production, for example, or quick maintenance and repairs in service, while the product or the machine to be serviced was superimposed with digital displays, instructions and performance data in front of them. However, the rather small field of vision, the fact that finger control does not always work perfectly, ergonomic aspects such as weight and center of gravity and, last but not least, the rather hefty price of 3,000 US dollars ensured that the Hololens 1 did not gain widespread acceptance.

Improved wearing comfort: Among other things, the Hololens 2 has a more balanced center of gravity than its predecessor.

© Microsoft

With the Hololens 2, Microsoft launched the second generation of mixed reality glasses at this year's MWC in Barcelona - with a variety of new design options such as more intensive 'immersion'. This makes holograms look even more vivid and realistic. Compared to its predecessor, the device also has a field of view that is more than twice as large, while the resolution of 47 pixels per degree of view has been retained. This increase in performance is made possible by a new display system with improved 2K displays, which also consumes very little energy. In addition, the interaction with holograms has been fundamentally revised: Thanks to a new time-of-flight depth sensor, built-in artificial intelligence and integrated semantic analysis, interacting with holograms is as intuitive as with objects in the real world. This means that hands can now also be used to control objects that were not yet recognized by the first model. Last but not least, the current version has integrated eye tracking sensors. These enable biometric authentication by iris recognition via 'Windows Hello', for example, so that the device can be used easily and securely by several people in a company.

One of the biggest criticisms of the first Hololens model was its wearing comfort. Microsoft has taken this into account and ensured that the new generation has a balanced center of gravity so that the device is comfortable to wear even when used for long periods of time. Thanks to the use of carbon fibers, the current model is also lighter than its predecessor and another new mechanism allows the glasses to be comfortably adjusted to any head shape when put on. In addition, the visor can be folded up, making it possible to switch from mixed to physical reality in just a few seconds. At USD 3,500 for the basic model, the price is in the same range as the previous model, but the price-performance ratio has improved significantly.

Microsoft already offers numerous templates, instructions and services for quickly creating your own AR experiences with a variety of mixed reality programs and tools. One of these interesting tools is Azure Spatial Anchors, for example, which can be used to create cross-platform MR applications with a spatial reference. This allows users to anchor specific dashboards with status or process information as well as virtual buttons and switches to individual machines or devices in the production environment. These can then be accessed using Hololens 2 as well as iOS and Android devices. Route guidance systems can also be created in this way.

Digital system data as a valuable basis

Step-by-step instructions also help new employees with error-free installation.

© Microsoft

When it comes to the sources of AR content, the companies themselves come into play as well as their existing product, customer or process data from the CAD, PLM, MES, ERP or CRM system. AR platforms such as 'Vuforia Studio' can then be used to quickly and easily create your own AR applications from the existing product and system data. More specifically, Vuforia Studio uses the 3D CAD data of products, machines or entire systems to create virtual images that can be projected freely into the room or onto their physical counterparts. Even real-time data - from sensors, for example - that is collected via IIoT platforms such as ThingWorx (PTC's industrial innovation platform) can be incorporated and displayed using this technology if required. No programming or coding skills are required for this. Using a simple and intuitive drag-and-drop process and pre-built widgets, the time to create AR content is reduced from days and weeks to hours or even minutes.

It all starts in product development

The possibilities of IoT technology alone, which is also still relatively new, are already providing enormous added value for product development by making previously unknown field data available - provided, of course, that this data is collected during ongoing use at the customer's premises. This means that developers and engineers no longer have to rely solely on customer surveys or the analysis of replaced devices and machines in order to gain insights for new and better series.

Howden's customers also receive predictive maintenance alerts via the digital twin.

© PTC

AR is now bringing a further efficiency boost to product development. The focus here is on collaboration between development teams often working at several locations. First of all, AR can ensure that complex physical prototypes no longer need to be built, especially in the initial development stages of a product. The developer simply projects a digital model of the prototype into the room or onto a table. This model can now be viewed from all sides and modified according to the developer's ideas, for example with the help of voice commands or gestures. The developer can even move around in it, display different assemblies or zoom in on details. A colleague at another location does the same, with changes being immediately displayed to the other team member.

In this way, car manufacturer Ford, for example, enables its engineers to communicate in real time about the latest developments and even goes one step further in the mixed reality world: Ford uses virtual reality (VR) technology to create a virtual workshop in which all engineers involved can exchange information from different locations in real time on 3D holograms of full-size vehicle prototypes. All details such as the position of the steering wheel, the angle of the dashboard or the position of the various instruments and controls can be clarified quickly and easily.

In addition to the type of product development, AR is also changing the products themselves. With augmented reality, for example, fewer physical controls may become necessary, with mobile device apps enhancing or even completely replacing the operation of products. For example, engineers at Ford may soon no longer need to worry about the position of certain controls, as every customer can have them displayed in their desired design using data glasses. With increasing AR personalization and the elimination of traditional buttons, switches and control panels, products can also be produced more cost-effectively and offered at a lower final price, as the AR user interface is purely software-based and provided via the cloud - all at little additional cost.

AR in a production environment

As soon as the development of the product has been completed, it is time for production. Here too, AR devices can display production and assembly instructions in the real working environment and thus optimize existing processes, as the error rate is reduced in addition to the need for training for certain production series. With 'Vuforia Expert Capture', PTC presented an AR solution at this year's Hannover Messe that relies on the experience of employees in the company as a data source, among other things, by recording their knowledge and expertise in certain workflows and processes. This means that this knowledge is retained for future generations even after experienced employees have left the company.

BAE Systems uses AR to train new employees in the assembly of battery cells.

© PTC

In practice, this works as follows: Experienced workers carry out their assembly tasks with the help of a Microsoft Hololens or a head-mounted tablet from Realwear, for example. Both images and speech are recorded so that the worker carrying out the task can also provide tips or explanations on certain procedures. The recorded procedures can then be refined in a secure, SaaS-based environment by processing each individual sequence with text, image and video material and, if necessary, enriching it with additional information to aid understanding. The result is self-created tutorials for all possible work tasks and process steps, which are then available to employees throughout the company on the data glasses, among other things.

At the same time, AR systems can visualize important monitoring and diagnostic data for every machine and every process in order to warn production managers and maintenance technicians of impending, costly downtimes in good time. With the help of the aforementioned Azure Spatial Anchors, for example, all important machines and systems can be provided with corresponding displays that can be quickly viewed with the data glasses.

Service and maintenance - the killer application

The service sector was one of the first killer applications for AR in the industrial environment and still is today - for example at Howden. The global engineering company manufactures industrial products ranging from mine ventilation and wastewater treatment to heating and cooling. The Scottish company was able to significantly improve its service quality by creating scalable, step-by-step mixed reality service manuals based on existing 3D models using Vuforia Studio.

By integrating real-time and historical IoT data from ThingWorx and Microsoft Azure, Howden is improving its customers' experience of their typically process-critical assets and enabling more inexperienced staff to work on them. As soon as they step in front of such a system, it is recognized by the Hololens 2 and superimposed with a digital twin. If, for example, a specific component needs to be replaced, the digital twin can be used to display step-by-step instructions. In addition, supplementary displays are shown that inform the employee at all times about the performance status of the system and other predefined system and environmental parameters in real time. This provides customers with on-site insights into the operating conditions and performance of the equipment that they did not have before.

BAE Systems, one of the largest aerospace and defense companies, has also recognized the benefits of AR for training new colleagues and upskilling existing staff. As the demand for efficient and low-emission electric motors has risen rapidly in recent years - and with it the pace of production - new employees are to help meet this demand on time. However, with such a complex product - especially the batteries - this is easier said than done: BAE Systems had to look for innovative solutions to make the time-consuming and cost-intensive training as efficient as possible so that new employees could be fully deployed quickly. Previous learning methods were often limited to passive watching of work processes, the use of learning videos or reading a manual - interactive learning: no way!

With the help of Vuforia Studio, BAE Systems was able to create interactive mixed reality experiences for the Hololens within a few hours and at a tenth of the cost and, based on these guided step-by-step work instructions, speed up the assembly of battery cells and ultimately make the training of new employees 30 to 40% more efficient.

These examples show that augmented reality has long since arrived in practice. Nevertheless, we are only at the beginning of a big wave. With the increasing digitalization of company data and processes as the basis for IoT applications, the spectrum of AR applications is also broadening.

Author:
Markus Hannen is Technical Sales Vice President at PTC.