TMD Friction has been developing brake pad technology for 140 years. The company is always striving to modernize and optimize not only its products, but also its production processes. This is why TMD commissioned the IT consulting company Acando and its subsidiary Brickmakers with a proof of concept for augmented reality.

The subject of this feasibility study was the safe digitalization of the mixing and weighing process for brake pad components. Using the Microsoft Hololens and modern augmented reality visualization, this process is to be simplified and made more precise - without creating serious safety gaps. TMD manufactures brake pads for cars, trucks and racing cars from 200 different raw materials and 250 different components. As a result, there is an almost unmanageable number of pad formulations and mixing variants, the correct application of which is to be solved more quickly, precisely and pragmatically by the application developed by Acando and Brickmakers.

The management and IT consultancy Acando already had experience of working with Microsoft's Hololens glasses and their AR applications. Jörg Neumann and his IT team first took part in a workshop at the TMD plant and were introduced to the complex production processes. Weighing and mixing processes were previously implemented in three separate steps. The industrial PC in the mixing department starts one or more orders, the materials for which are then retrieved from the warehouse. A second PC is located there, which also displays the orders, as well as a hand scanner to check and enter the raw materials.

This is where Acando and the app developers from Brickmakers planned to use the Hololens. These mixed reality glasses, which are used without a smartphone or computer, enable 3D projections to be displayed in the user's immediate surroundings.

Game engine for visualization

With the help of the game engine Unity, originally used to increase immersion in gameplay, the user interfaces as well as the weighing and mixing processes were digitized. Unity was chosen for two reasons: Firstly, it made sense to write the 3D graphics and animations in a suitable program that particularly supports 3D development, and secondly, the workload could be significantly reduced by using the Unity editor. Acando developed the storyboard and took over the organization with TDM, while Brickmakers was primarily responsible for the technical implementation.

First, the team created a UWP app and started two separate projects (user interface and business logic). With the Universal Windows Platform (UWP), apps can be developed for various Windows devices, in this case for the Hololens. The developers implemented these applications and the Microsoft Dynamic Link Library (DLL) via a plug-in in Unity and familiarized themselves with the many possibilities of the data glasses.

From programming to holographic visualization

The biggest challenge was the interaction with the DLL. The Hololens runs as a UWP under a Microsoft .NET application that is not yet supported by Unity. The necessary transfer of the various program libraries (DLLs) presented the developers with a problem: in order to enable the interaction of the DLL files, he had to program a 'wrapper' that acted as an interface between the two programs and enabled the data transfer of the applications.

Emil Wasilewski, Software Developer at Brickmakers: "To control the Hololens, all you have to do is integrate the toolkit into Unity. This then enables automatic tracking".

© Acando

Especially for applications with different programming languages, it makes sense to work with wrappers to avoid conflicts between them. Using a wrapper that generated 'fake data' as an interface, the developers were able to write the scripts in 40Unity for the Hololens, although both use different DLLs.

In a second step, the images created in Unity could be transferred to the data glasses' visualization software using a Visual Studio solution. As these Visual Studio solutions are in turn compatible with the DLLs from the UWP, there was no need to resort to working with the 'fake data'. After a cloud interface was set up via Microsoft Azure, the actual control of the AR glasses and the virtual weighing process from the TMD plant could now be combined and simulated. The gesture and voice control did not require any additional technical calibration.

Once the Acando and Brickmakers teams had given their technical approval, the Hololens with the programmed AR visualization naturally had to be tested in reality. Do voice and gesture control also work with dialect or work gloves?

Virtually tested, tried and tested in reality

The Saxon dialect - initially a small hurdle - can be easily overcome using the integrated Cortana language learning software. As with other voice recognition assistants, Cortana learns the user's linguistic idiosyncrasies with regular use if an internet connection is available. The mixing and weighing process with the help of augmented reality visualization is still multi-step, but greatly simplified by the data glasses.

First, the employee receives an order via the glasses, which then virtually shows him the materials in the respective weighing. With this image in front of them, they then enter the warehouse, select the raw materials and complete the selection using the voice function (command: 'Picking Done'). The employee then returns to the mixing hall to the scales, places the mixing container on them and tares them. Next, he scans the label of the materials with the data goggles, which confirms that it is correct by comparing it with the recipe. When filling the container, the Hololens also checks the weight and sequence of the raw materials and creates a final balance with regard to the specifications. The weighing process is again completed by voice control (command: 'Finish').

The advantages and disadvantages

The advantages of this method can be seen quite literally: The employee no longer has to memorize material codes and has their hands free during the entire process. Work can be carried out more quickly and safely, material mix-ups are virtually eliminated and the Hololens can be learned quickly.

However, the glasses themselves still have some disadvantages: At more than half a kilo (579 g) and with a battery life of only two to three hours, the Hololens in its current version (Development Edition) cannot yet be used in an eight-hour working day. However, the developers are certain that augmented reality and data glasses have great potential.

As the proof of concept showed, more than just immersive gameplay is possible with the pioneering technology of the Hololens - the visualizations can simplify, optimize and make everyday work in industry and production safer.

Author:
Anja Günther is Client Relationship Manager at Acando.