Twelve years ago, Apple's iPhone caused a sensation with a new operating concept. The keyboard was replaced by a touchscreen and the entire front of the device was filled with a huge screen with an unprecedented pixel density. It is operated via a touchscreen with a projected capacitive field, which can also recognize and evaluate several touch events simultaneously. This technology can only be used sensibly with a completely new type of user guidance via software. Tapping, sliding and other gestures with one or two fingers control different functions.

Since this product was introduced, the world of input and output has changed. The point touch of the touchscreen, which triggered exactly one action, has been transformed into gestures with one or more fingers that trigger complex actions. 'Drag and drop' or 'pinch' work from the wrist without having to laboriously select objects via menus and assign an action. This change requires completely new underlying software and therefore a new operating concept.

However, actions do not always have to be initiated by touching a touchscreen in front of the screen; it is also conceivable to evaluate movements in front of the screen. Several technologies make use of this idea: 3D touch systems are suitable for gestures without any particular requirement for accuracy: they determine the positions by means of a change in an electrical field. The resolution of these systems is sufficient for a qualitative determination of 'louder/quieter', 'higher/lower', 'zoom in/zoom out'.

Alternatives to the touchscreen

If the position needs to be evaluated more precisely, cameras come into play that follow the operator's hands, for example. The control units for AR/VR glasses, which allow sensitive control in the system, offer perfection in terms of position resolution.

Eye tracking' takes a different approach: after a short learning phase, the system is able to track the operator's pupils with the help of cameras. The 'mouse click' is made by blinking. After a familiarization phase, the operator can operate the system without using their hands. Applications are everywhere where the hands are not free - for example during an operation or when assembling a device - or must remain sterile.

Another trend is voice control, as demonstrated by Apple, Google and Amazon: Calling up information from the internet or controlling devices in the living room at home works on demand. Behind this is an artificial intelligence that is located in the provider's server farm.

Voice input, which follows more or less strict syntax rules, is somewhat simpler. It can be found, for example, in car navigation systems in the style of "Please enter the address in the form city, street and house number". Parsing - i.e. assigning the words entered to fields in the database - and executing the desired action can be carried out by a local controller.

The visualization

On the other side of the interaction is the output, which is usually handled by a display. Here, too, technology is advancing. Again driven by mass market quantities, OLEDs are appearing with a brilliant display and excellent image quality thanks to the enormous contrast. In certain applications - such as the display of high-contrast images in medical imaging procedures - they appear to be outstripping LCD technology. However, they are subject to an ageing process, which becomes visible through a reduction in brightness. TFT counters with quantum dots, which are entering the market under names such as 'QLED' and promise a high color gamut with vibrant hues. Screens with micro-LEDs are still at the laboratory stage and are helping LED technology to enjoy a renaissance, moving it from being a pure light source for TFTs to a frontline image source. The production of large screens with high resolution has not yet been solved for series production, but prototypes show the great potential of this technology.

But back to the Human Machine Interface: It is not one technology alone that leads to success, but the combination. An HMI must offer a coherent input function that appeals to many sensory organs. In addition to the touch systems, there is haptic feedback, which directly signals successful input to the operator via the sense of touch. It could also be provided acoustically via a beeper, although this may be drowned out by the noise level in a noisy environment. The display shows the required information ergonomically and in high resolution. However, the user guidance provided by the software or the graphical user interface (GUI) plays a crucial role. The GUI determines how information is presented, through colors, shapes, arrangement on the screen and presentation in relation to each other.

The next revolution

The human-machine communication channel is multisensory: see and be seen - with image output and eye tracking; hear and be heard - with sound output and voice input; touch and feel - with touchscreen and haptic feedback.

© Adobe Stock / undrey, Hy-Line Computer Components

The term HMI 5.0 therefore describes the comprehensive interaction of people with the system using all their senses. New methods are coming onto the market, such as 3D gesture recognition, speech recognition and eye tracking for input. For output, the importance of haptic feedback, which compensates for the shortcomings of conventional touchscreens, and three-dimensional visualization with holograms or glasses is increasing. More powerful graphics cards have made glasses for AR and VR affordable. Initially driven by computer games, VR glasses have already found their justification in simulator training, where they visually and acoustically transport the operator into a realistic scene. Overall, the communication channel between man and machine has become broader: seeing and being seen - with image output and eye tracking; hearing and being heard - with sound output and voice input; touching and feeling - with touch screen and haptic feedback. Only smell and taste are still missing.

So is touch control still state of the art? The answer to this question is a clear 'yes'. Because despite many new technologies, the touchscreen will continue to set the tone for some time to come. In addition, comprehensive, multi-sensory communication between humans and the computer will play an important role, which is not limited to the keyboard/touch screen and monitor. Even if some technologies have not yet achieved a broad breakthrough, they are ready as tools to provide the user with a new user experience. With their support, they are well prepared when it comes to implementing software for artificial intelligence or machine learning.

Author:
Rudolf Sosnowsky is Head of Technology at Hy-Line Computer Components in Unterhaching.