Shortly before 10 p.m. on March 18, 2018, Elaine Herzberg was pushing her bicycle across the four-lane Mill Avenue in Tempe, a suburb of Phoenix, Arizona. She was wearing a black top and was not using the designated crosswalk. The algorithms of the self-driving test car from Uber, which was heading straight for Herzberg, were at odds for a long time as to what they were dealing with. The car ran over the 49-year-old homeless woman without braking. She died a short time later in hospital. The safety driver of the test car had not intervened. Instead of paying attention to the traffic and keeping her hands above the wheel ready for action, she had her eyes down, as the video from a surveillance camera shows.

If the car had slowed down and Herzberg had been able to cross the road unharmed, no one would probably have asked how the self-driving car's algorithms had arrived at their predictions and probability calculations. But because they sometimes make mistakes, it is a serious problem that their solutions are usually hidden. This is because modern machine learning (ML) algorithms are like a black box. They no longer follow predetermined if-then rules, but generate a complex model from the input data, the design of which humans can only influence indirectly at best.

"The more complex the neural network is, the more precise but unfortunately also the more difficult it is to understand the results," says Professor Marco Huber, who heads the Center for Cyber Cognitive Intelligence (CCI) at the Fraunhofer Institute for Manufacturing Engineering and Automation IPA in Stuttgart. He has set himself the task of putting an end to this constant trade-off between the performance and interpretability of an algorithm. In future, artificial intelligence (AI) will make explainable decisions and comprehensible forecasts.

Data protection requires explainable AI

Explainable Artificial Intelligence' - or xAI for short - is the name of this branch of computer science research, which was initiated by the Defense Advanced Research Projects Agency (DARPA), an agency of the US Department of Defense. For a long time, industry's interest in it was limited. However, since AI no longer only recommends books on Amazon or films on Netflix, but also directs machines and robots in production halls or helps drivers to steer their cars safely through busy city centers and park without accidents, this has changed fundamentally. After all, why should people entrust their fate to self-driving cars or collaborative robots when their algorithms make opaque and sometimes even wrong decisions?

However, it is not only the human need for security and an explanation for unexpected or far-reaching events that makes xAI relevant, but also the European Union's General Data Protection Regulation (GDPR). According to Article 12, companies that process personal data are obliged to provide information to data subjects "in a concise, transparent, intelligible and easily accessible form". According to Article 13, this also applies to "automated decision-making", where data subjects are entitled to "meaningful information about the logic involved and the scope and intended effects".

A 52-member group of experts from the European Commission, which presented 'Ethics guidelines for trustworthy AI' last April, is even more fundamental. It declares the EU treaties, the EU Charter of Fundamental Rights and international human rights to be the basis for the development, introduction and use of AI systems and formulates four ethical principles based on these: Trustworthy AI must respect human autonomy, prevent harm, and be fair and explainable. This is because only comprehensible decisions can be "properly challenged".

Explainable models

This legal and moral framework raises a number of questions that xAI researchers need to address. First of all, what methods and procedures are there to ensure that black box algorithms not only tell us their results in future, but also the solution? The answer to this question depends on which model the AI process is based on. This is because there are inherently explainable or whitebox models that users can easily understand, and there are inherently inexplicable models for which an alternative path to explainability must be found.

As already mentioned, explainable models work towards comprehensibility, but their accuracy is often not sufficient for an AI result. These include linear models. The data is assigned to a class depending on which side of a straight line it lies. They therefore have a comprehensible decision limit.

Another explanatory aid for ML results are decision trees. These are particularly suitable when a linear decision boundary is no longer sufficient and represent possible rule-based decision paths of the algorithm based on their branching. The decisions are structured hierarchically and thus lead from the initial question to a classification result according to the intermediate answers. However, decision trees can also generate such complexity that they become incomprehensible.

As a third explanatory aid, rule-based systems that make decisions based on predefined if-then rules are in principle easy for users to understand. "Although a decision tree is already a rule system, I can break these rules down again separately," explains Huber. "For example, rules can be prioritized or all weighted equally. If there are contradictions, these must be resolved. And here, too, it is important to ensure that the number of rules remains manageable. Otherwise there is no added value compared to an automated model."

Aids to explainability

If an AI result is based on a model that cannot be explained by nature, such as a deep neural network, it requires a kind of translation service for users. This is because, in contrast to explainable models, there is no clear decision boundary for obtaining results. The translation service can either be globally oriented - i.e. explain the model as a whole - or locally oriented and explain why a certain input x leads to a certain output y. Three methods are listed as examples.

The first method is the creation of a surrogate model or surrogate. This whitebox model simulates the blackbox model and makes largely identical predictions. If a decision tree is to be extracted from a neural network for a surrogate, for example, the complexity of the network can be influenced by regularization or by forcing certain properties of a network. It is also possible to automatically set the weight of as many edges as possible in the network to zero so that they are omitted. This leads to so-called sparseness. It not only makes the network leaner and therefore more comprehensible, but also enables faster evaluation because the computing time is reduced.

The second tool is counterfactual explanations. These break down which detail of the input data actually produced a result. They therefore serve the purpose of local explainability. For example, an algorithm that decides on loans could explain: "Your loan application was rejected because your income is too low to service the interest." This tool not only makes ML applications easier to understand, but also includes a recommendation for action. If the user knows that from a sum of input data exactly input x led to result y, the smallest possible change to the input data can produce a different output.

Finally, a third option is explanatory representations. Models can be visualized or explained with the help of narratives, virtual reality, animations or voice output. "In image processing, for example, I want to know why a photo was classified as a cat and not a dog. We use so-called heatmap technologies for this. These highlight which parts of the image were important or unimportant for the decision," explains Huber. Or a word processor breaks down why an e-mail was declared as spam.

AI for safe human-robot collaboration

Every HRC workplace is different. Safety officers must therefore carry out a separate risk and hazard assessment for each one. Algorithms could support them in this, but they must deliver comprehensible results.

© Fraunhofer IPA / Rainer Bez

In human-robot collaboration (HRC), global explanations that explain the model as a whole are indispensable. This is because reinforcement learning (RL) - a machine learning process in which a robot is 'rewarded' for correct actions - is increasingly being used in the mandatory risk and hazard assessment in accordance with ISO 12100. This means that robots receive points for correct actions and rise in a ranking. They therefore strive to collect as many points as possible and reach the top of the ranking.

Using RL, a robot at a virtual HRC workstation learns how to complete its assembly task without injuring anyone, despite the completely unpredictable movements of a human. However, because RL uses a neural network that a safety officer cannot penetrate, they will never take responsibility for safety implementations suggested by an algorithm. That would be grossly negligent and could end up in court.

So far, there are at best local explanations for the RL for specific input data. In the KLEAR research project, Huber, together with HRC expert Ramez Awad from the Robot and Assistance Systems department at Fraunhofer IPA and ten other partners from science and industry, would therefore like to work on the global "traceability and explainability of the risk and hazard assessment obtained using ML methods in a human-robot cooperation application for CE documentation". The problem here is that global explanations have so far only ever been approximations of the complex solution paths of a neural network. The aforementioned decision trees, for example, offer an easily comprehensible form of representation. However, they are so simplified that a lot of information is not taken into account. With so-called ontologies - diagrams similar to a mind map on which individual terms are placed in relation to each other - the opposite is true. They depict reality completely, but are difficult to understand.

Huber and his team of researchers now want to extract comprehensible explanations for the security implementations suggested by the neural network while keeping the loss of information to a minimum. In this way, they want to enable security officers to check the protective measures provided by the algorithms and then confirm or reject them. With their consent, the system automatically derives a code for the programmable logic controller (PLC) of the planned HRC workstation.

Maintenance at the right time

If wear parts are replaced too early or too late, this results in unnecessary costs. Deep neural networks should therefore evaluate the sensor data of a machine and determine the right time for maintenance.

© University of Stuttgart IFF, Fraunhofer IPA/Rainer Bez

Another area of application for xAI is predictive maintenance. This is because in traditional production plants, wear parts are currently usually replaced too early or too late. Both lead to unnecessary costs. Sensor data and its automatic evaluation should enable maintenance to be carried out at exactly the right time.

Complex event processing (CEP) approaches using comprehensible rules are currently often used for real-time analysis. In practice, however, it is often the case that the short live data section observed is not sufficient to identify potential problems. By extending the observation period using complex ML approaches, meaningful patterns can be identified in the data, but their origin cannot be traced. However, this is crucial for refining and improving CEP rules and for improving the quality of predictions. Surrogates should help to improve the understanding of the rules in future. - A topic that is specifically addressed in the research project 'Extraction of explanations for supervised learning methods' - EVE for short.

Authors;
Karin Röhricht is an editor at Fraunhofer IPA,
Hannes Weik is an editor at Fraunhofer IPA.