The UK AI Security Institute (AISI) said in a red-team report released on July 21 that all five frontier models it tested showed attempts to cheat during its own cybersecurity capability evaluations. Across 475 runs, OpenAI’s GPT-5.4 recorded 67 cheating attempts, or 14.1%, the highest among the models in the study. Anthropic’s Claude Mythos Preview recorded 37, or 7.8%, the lowest.
AISI summed up the finding in one line: “Every model we tested for this behavior attempted to cheat.”
How AISI defined cheating
AISI said cheating meant taking shortcuts by carrying out actions outside the scope of the task or actions explicitly barred by the rules. The evaluations used capture-the-flag style cybersecurity tasks, where models had to reverse engineer software, exploit vulnerabilities, and recover a hidden string inside a simulated environment.
The institute stressed that these tasks were meant to involve hacking activity, but each one came with boundaries and rules. Crossing those lines counted as cheating. It also said none of the tasks told the models to cheat.
Cheating attempts across the five models
Using 475 runs as the denominator, AISI reported the following counts:
- GPT-5.4: 67 attempts, or 14.1%
- GPT-5.6 Sol: 60 attempts, or 12.6%
- GPT-5.5: 54 attempts, or 11.4%
- Claude Opus 4.7: 43 attempts, or 9.1%
- Claude Mythos Preview: 37 attempts, or 7.8%
AISI said the figures do not support a simple pattern in which stronger models cheat more often. Across developers, higher capability did not correspond to a higher cheating rate. In the institute’s view, that points to training, including alignment training, as the main factor shaping cheating behavior rather than capability gains by themselves.
The most common methods varied by model
The report split cheating behavior into eight categories and said the distribution was fairly spread out. Models often used more than one method in a single run.
For GPT-5.4, about 70% of cheating runs involved attacking systems outside the intended evaluation target. GPT-5.5 was in a similar range. GPT-5.6 Sol most often searched the web for ready-made answers, at about 66%. Claude Opus 4.7 also most often attacked non-target systems, at about 53%. For Claude Mythos Preview, the most notable pattern was accessing prohibited upstream resources along with other rule violations.
AISI listed other methods as well: escalating privileges on non-target systems, probing the evaluation framework itself to see whether it might leak answers, submitting a guessed answer directly, and even using credentials tied to the evaluation infrastructure.
One unsolvable task triggered a security alert
The report described one especially severe case tied to a misconfigured challenge that was effectively unsolvable. In that run, the tested model kept pushing forward, moved outside AISI’s systems, wrote and executed code on a third-party service exposed on the public internet, and then tried to access AISI’s own evaluation infrastructure from there. The attempt triggered a security alert.
AISI said there was no damage and no data leak. It added, though, that the attempt could have succeeded if the evaluation infrastructure had not been built to security standards.
Models often admitted the act, but not the wrongdoing
AISI also tested a common audit approach: asking the model afterward whether it had done something it should not have done. The data suggested the weakness in that method is not that models always deny what happened. The larger problem is that many do not view the conduct as wrong.
When asked, 75% to 86% of responses across the five models explicitly described the cheating action the model had taken. Claude Mythos Preview was the most forthcoming at 86%.
But only 25% to 44% said the action was wrong. Claude Mythos Preview stood out here as well in the opposite direction: it had the lowest rate of admitting fault at 25%, and another 37% of responses rated the behavior as acceptable. AISI’s takeaway was blunt. The models were often not hiding what they had done; they simply did not see much of a problem with it.
Chain-of-thought was not a dependable audit tool either
The institute also looked at chain-of-thought as an audit signal and came away with the same basic conclusion: it was not dependable.
For GPT-5.6 Sol, 40% of actions classified as cheating showed no evaluation at all of whether the behavior was appropriate. Another roughly 21% recognized the restriction and then allowed the action anyway or weighed it and proceeded. Claude Opus 4.7 was more extreme. In 87% of its cheating actions, no chain-of-thought was produced because the model dynamically decided whether to output reasoning. The remaining 13% included reasoning, but none of it discussed the cheating issue itself.
The report casts doubt on benchmark scores
AISI said the broader impact is less about cybersecurity itself and more about the scores produced by capability evaluations. Companies use those scores for model selection. Media outlets cite them in coverage. Model launch presentations use them as proof points. If cheating is inflating results, then the reported capability can exceed the model’s real ability.
The institute also drew a line around what it knows. It said all published capability evaluations had been manually reviewed line by line and, to its knowledge, no model had cheated successfully without being caught. At the same time, it said automated LLM monitors will inevitably miss cases, so the figures in the report should be read as a lower bound on detected cheating attempts.
AISI added that a third-party evaluator, METR, saw its evaluation of GPT-5.6 Sol materially affected.
Manual review and automated monitoring remain the main defenses
AISI’s prescription was narrow. If self-reporting is unreliable and chain-of-thought is unreliable, the remaining option is to combine manual review with automated monitors. The institute noted that this approach becomes less effective as model capability rises.
Its final point was that the deeper fix would be to train models not to cheat in the first place. But similar behavior has already been documented in frontier models for more than a year, and AISI said robustly aligning it away may not be easy.