It can compose speeches and tell stories with a high degree of linguistic precision - and in a matter of seconds. The text robot ChatGPT, a language software with artificial intelligence (AI) developed by the US company OpenAI, is currently the talk of the town. The program, which is fed with huge amounts of data, is causing quite a stir, but also scepticism.

Scientists and AI experts in Germany are warning of data protection and data security loopholes, hate speech and fake news. "There's a lot of hype at the moment. I have the feeling that there is hardly any critical reflection on this system," says Ruth Stock-Homburg, founder of the 'Leap in Time Lab' research laboratory and business administration professor at the Technical University of Darmstadt.

ChatGPT has a very wide range of applications. Among other things, you can ask the program questions in a kind of chat box and receive answers. Work instructions are also possible - for example, writing a letter or an essay based on basic information.

In a project together with TU Darmstadt, the Leap in Time Lab has now sent thousands of queries to the system over seven weeks without personal data in order to find vulnerabilities. "These systems can be manipulated," says Stock-Homburg.

Manipulation possible, data not traceable

In a presentation, Sven Schultze, TU doctoral student and expert in language AI, shows the weak points of the text robot. In addition to anti-Semitic and racist statements, sources are simply incorrect or come to nothing. In the case of a question about climate change, a given link leads to a website about diabetes. "It is usually the case that the sources or scientific papers don't even exist," says Schultze. The software is based on data from 2021, when Chancellor Olaf Scholz is still Finance Minister and the war in Ukraine is unknown. "Then it's also possible that it's simply lying or inventing information on very specific topics."

In the case of direct questions with criminal content, for example, there are security notices and mechanisms. "But you can use tricks to bypass the AI and the security warnings," says Schultze. With another approach, the software shows you how to generate a fraudulent email or throws up three variants of how con artists can proceed with the grandchildren trick. GPT also provides instructions for a home burglary. If residents are encountered, weapons or physical force can also be used.

However, Ute Schmid, who holds the Chair of Cognitive Systems at Otto Friedrich University in Bamberg, believes that the main challenge is that it is not possible to find out how the text robot arrived at its information. "A deeper problem with the GPT3 model is that it is not possible to trace which sources were included in the respective statements, when and how."

Despite this serious shortcoming, Schmid is in favor of not only looking at errors or possible misuse of the new technology, for example when examinees have their term papers or exams written by the software. "I think we should rather ask ourselves: what chance do we have with AI systems like this?" Researchers generally argue that AI expands our skills, perhaps even enhances them, but does not restrict them. "This means that I also have to ask myself in the field of education - as I perhaps did 30 years ago with calculators - how can I shape education with AI systems such as ChatGPT?"

Nevertheless, there are still concerns about data security and data protection. "What we can say is that ChatGPT collects, stores and processes a wide range of data from the user in order to then train this model accordingly at a given time," says certified Frankfurt data protection expert Christian Holthaus. There is the problem that all servers are located in the USA.

Critical consideration necessary

"That's the real problem if you don't manage to establish the technology in Europe or have your own," says Holthaus. There will be no data protection-compliant solution in the foreseeable future. Stock-Homburg also says the following about EU data protection regulations: "This system should be viewed rather critically here."

ChatGPT was developed by one of the leading AI companies in the USA, OpenAI. Software giant Microsoft had already invested one billion dollars in the company in 2019 and recently announced plans to pump billions more into the company. The Windows company soon wants to make ChatGPT available to customers of its own cloud service Azure and the Office suite.

At the moment, ChatGPT is more of a gimmick for private use, says Stock-Homburg. But at the moment, it is in no way something for business or security-relevant areas. "We have no idea how to deal with the still immature system."

Oliver Brock, Professor at the Robotics and Biology Laboratory and spokesperson for the 'Science of Intelligence' cluster at the Technical University of Berlin, does not see ChatGPT as a "breakthrough" in artificial intelligence research. For one thing, development in this area is continuous rather than erratic. Secondly, the project only represents a small part of AI research.

However, ChatGPT could be seen as a breakthrough in another area, namely the interface between humans and the internet. "The way in which these huge amounts of data from the internet are made accessible to the general public intuitively and in natural language with a great deal of computing effort can be considered a breakthrough," says Brock.