Lakera / Check Point Software
"Crash test" for LLMs in AI agents
Lakera and the UK AI Security Institute have launched 'b3 ', a new open source benchmark. b3 is an open source security evaluation project specifically designed to protect Large Language Models (LLM) in AI agents.
The benchmark b3 was built on the basis of the new idea called Threat Snapshots. Instead of simulating a complete AI agent from start to finish, the threat snapshots zoom in on the critical points where vulnerabilities in LLM frequently occur.
By testing the models at these specific points, developers can see how robust their systems are against attacks - without the complexity that was previously required to model a complete agent workflow. A kind of 'crash test' for AI agents.
"We developed the b3 benchmark because today's AI agents are only as secure as the LLMs that fuel them," explains Lakera co-founder Mateo Rojas-Carulla. "These threat snapshots allow us to systematically look for vulnerabilities on the attack surface that were previously hidden in the complex agent workflows."
b3 combines ten representative threat snapshots with 19,433 real cyberattacks from the gamified red-teaming game 'Gandalf: Agent Breaker'. Among other things, prompt exfiltration, phishing link injection, malicious code injection, DoS and unauthorized tool calls are evaluated.
The first tests with 31 common LLM models show:
- Better reasoning capabilities increase security
- Model size does not correlate with security performance
- Closed source performs better on average, but top open models catch up
The benchmark report is available under an open source license: https://arxiv.org/pdf/2510.22620
| Gandalf: Agent Breaker is a hacking simulator game in which you are challenged to crack and exploit AI agents in realistic scenarios. The ten GenAI applications in the game simulate the behavior of a real AI agent. Each app features multiple difficulty levels, layered defenses and novel attack surfaces that challenge a range of skills, from prompt engineering to red teaming. Some of the apps are chat-based, while others rely on code-level thinking, file processing, memory or the use of external tools. |
You might also be interested in
Ionos and Q.ANT cooperate on AI
Ionos and Q.ANT want to jointly advance photonic processor technologies for AI and high-performance computers. The aim of the partnership is to build energy-efficient and sovereign AI infrastructures in Europe.
read more...AI-supported lighting robot wins German Innovation Award 2026
Ledvance, provider of innovative lighting solutions, was named "Winner" in the Excellence in Business to Consumer Lighting category at the German Innovation Awards 2026. With the AI Companion Lighting Robot 'Elix', the company presents an...
read more...Paderborn University becomes part of 'ELLIS NRW'
Paderborn University is one of the founding members of the new research unit 'ELLIS Unit NRW'. The European network bundles AI research in North Rhine-Westphalia with a focus on machine learning and explainable AI.
read more...AI 'CheckMate' optimizes industrial processes
With 'CheckMate', the University of Klagenfurt has developed an AI system that independently creates algorithms for complex combinatorics and optimization problems. The technology is designed to make industrial planning and logistics processes more...
read more...New approaches in drive technology
Nabtesco develops physical AI solutions
Nabtesco is working on intelligent drive systems for industrial applications. The focus here is on the use of physical AI, the combination of artificial intelligence with real physical systems.
read more...Regulation and safety slow down robotics projects in Germany
QNX has published the study "Inside the Robot: Architecture Benchmark Report". The results of the global survey of 1000 robotics developers show that software and safety requirements are increasingly becoming the central bottleneck for innovations...
read more...Michael Ardelt becomes first COO at Robco
Robco, a company for autonomous industrial robotics, is further expanding its management team. As the company's first Chief Operating Officer, Michael Ardelt will assume responsibility for structuring and scaling Robco's growth in operational terms.
read more...'Physical AI' as a turning point for robot use?
Physical AI - AI that can act more freely in the physical world beyond predefined processes - is regarded as the next evolutionary stage of artificial intelligence. Companies see great potential in this, especially for robotics, in the coming years....
read more...Smart chips for greater energy efficiency
In the AI lighthouse project "eki", a research team led by the University of Paderborn has been working on improving the energy efficiency of AI systems and developing methods that can reduce the energy consumption of AI by up to 90 percent. Special...
read more...