Anthropic said in a 154-page report released on Sept. 10 that it had identified and blocked large-scale distillation efforts targeting Claude by seven Chinese AI labs. The company called the activity "illicit distillation," defining it as unauthorized, large-scale and covert extraction of model capabilities.
According to the report, the actors used methods Anthropic described as clear violations, including stolen credit cards, login credentials and API keys to create fake accounts in bulk. Anthropic also said some Chinese model companies forwarded user requests to Claude or bought user conversations from third-party routing services, then used that data to train their own models. Some of those conversations, it said, contained names, corporate data and valid access credentials.
This was not the first time Anthropic had made such claims. In February 2026, it said DeepSeek, Moonshot AI and MiniMax had used about 24,000 fake accounts to generate more than 16 million interactions with Claude. Since then, Anthropic has raised its defenses. OpenAI and Google have also kept working on ways to identify and respond to distillation attempts.
The harder question is whether any of those measures can actually stop distillation. Liu Yi, an assistant professor at Griffith University in Australia, said he is skeptical. Liu has long studied artificial intelligence and cybersecurity and previously worked as an AI research scientist at Quantstamp. A 2023 prompt injection study he took part in was cited in OWASP's LLM 01 entry. He has also received Anthropic security bug bounties twice and has researched attacks aimed at stealing chain-of-thought reasoning from top commercial models.
Why chain-of-thought data matters
Liu said mainstream large models can be thought of as having three parts: the user's input, the intermediate reasoning process, and the final output. Anthropic and OpenAI now hide that intermediate chain of thought and only return the final answer. For complex reasoning tasks, extracting the hidden reasoning path is itself a form of distillation attack.
He described chain-of-thought data as a high-quality solution path already produced by a stronger model. That makes it valuable training material. Model performance, he said, depends on parameters, data and compute. Parameter counts are rising quickly across the industry. When time and compute are limited, companies that want to improve model capability faster need more high-quality data.
That is where the pressure comes from. High-quality data for complex, long-horizon tasks is hard to build, and useful real-world training data is becoming scarcer. In that setting, some model companies try to distill stronger systems in different ways.
Asked how attackers pull out chain-of-thought and other task traces, Liu said the process usually combines several techniques, including encoding, jailbreaking and prompt injection. He also said attackers may keep assigning the model a role, trying to persuade it that it is a researcher working on a difficult but meaningful task so that it reveals more of its reasoning process.
There is also another route. Instead of extracting native chain-of-thought directly, attackers can try to synthesize intermediate reasoning from the input and output. Liu said some data companies are said to sell chain-of-thought data for long-horizon tasks at 800 to 1,000 yuan per item.
Detection is easier than proof
On which company has been distilled the most, Liu said Anthropic appears most often in public reporting, but he would not draw a firm conclusion. He said he has looked at actual chain-of-thought text from DeepSeek, GLM, Kimi, Anthropic and OpenAI, and the writing styles do not differ much. In his words, it is hard to say who distilled whom, and he added that it is even possible some players initially distilled DeepSeek.
Technically, he said, Anthropic appears to rely heavily on usage behavior. A cluster of accounts may suddenly send large volumes of requests aimed at extracting a fixed dataset. Anthropic can first identify abnormal accounts, then look for shared patterns across them, such as the same system prompts.
Liu said different companies use different agent harnesses, and their system prompts differ as well. That can help identify the company behind the activity. With prefix caching, Anthropic can also see which prompt groups share the same cache. If those prompts do not match known system prompts, locating the distiller becomes easier.
But moving from "these requests look like distillation" to "this data was definitely trained into a specific model" is much harder. Liu said Anthropic may be able to accuse a company of trying to distill Claude, yet still struggle to prove with 100% certainty that the extracted data actually entered the other side's model.
He pointed to GLM, Kimi and DeepSeek as examples. Although they are open-source models, what is open is the weights, not the datasets. And model weights alone make it difficult to establish definitive proof of distillation. In his view, that is why Anthropic's report focused on how relay services operated rather than using white-box analysis to prove where the training data went.
He gave another example. If you ask a large model for the first 100 words of the first Harry Potter book, it will probably answer. That does not prove the model was directly trained on the original text. It may have learned the passage from blogs or reviews that quoted it. Liu said Anthropic can build a fairly strong attribution case from platform-side call logs, account links and infrastructure information, but that is different from proving from the final model itself that a specific batch of Claude data entered training. The latter is much more difficult.
Three layers of anti-distillation defense
Liu grouped current countermeasures used by U.S. frontier AI companies into three categories.
- First, built-in detection. He said Anthropic likely has classifiers aimed at chain-of-thought extraction. Once the relevant state is triggered, the system can block the extraction attempt.
- Second, product-level concealment. Instead of returning raw chain-of-thought, the model compresses or summarizes it before producing an answer.
- Third, external detection. The system can check whether the output overlaps with its own chain-of-thought. If the overlap crosses a leakage threshold, it can stop the output. Repeated behavior of that kind may lead to account bans.
Anthropic's report did not spell out those technical details. Liu said this part was his own inference because such practices are common. He pointed to Anthropic's bug bounty program as an example. In his view, the company is effectively paying people to jailbreak its models, then feeding successful attack patterns back into its classifiers so the next attack becomes harder.
Liu said he received Anthropic bug bounties twice. He spent about two to three weeks each time conducting adversarial testing on Anthropic's safety classifiers within the scope of the authorized program, trying to find inputs that could bypass existing defenses. He described the outcome as binary: either the attack works or it does not. The tasks at the time were aimed at retrieving answers to questions specified by Anthropic rather than directly reconstructing chain-of-thought, but he said the logic was similar.
Higher collection costs, and possible collateral damage
Could anti-distillation measures hurt ordinary paying users? Liu said regular subscribers are probably less affected, while risk controls mainly target abnormal users, such as relay accounts or accounts trying to attack Claude.
He said Anthropic's report also disclosed a case involving two undergraduate students at a university in Hunan who designed a cybersecurity agent and used it to attack various websites. That activity was also uncovered. In Liu's words, in theory Anthropic can inspect all information exchanged through interactions with its systems.
Whether model companies can retrieve and review user data when they believe a user poses a threat depends on the terms of service, he said. As he understands it, Anthropic, OpenAI and Google offer some form of Zero Data Retention, or ZDR, under which customer prompts and AI-generated replies are destroyed immediately after processing, not stored and not used for model training.
That feature, however, is mainly for eligible API or enterprise customers and usually requires an application or extra configuration. Ordinary users of Claude, ChatGPT and Gemini are still subject to each platform's own retention policy, which is not the same as strict ZDR.
Liu also raised a practical question: if a model company rewrites user data slightly and then uses it for training, does that still count as using the user's data? In his view, frontier model companies may charge users while also continuing to use interaction data to scale up. If a user corrects a model's answer in Claude Code or Codex, for example, that feedback becomes a valuable reward signal that can improve post-training.
He said distillation follows the same logic. Current post-training relies heavily on reinforcement learning, where reward signals matter. If a company obtains the chain of thought from a stronger model, it is effectively getting a high-quality solution path. That can reduce blind exploration, help the model converge faster and save both time and compute.
"Anti-distillation is a pseudo-problem"
Liu said the success criteria for anti-distillation at U.S. frontier AI companies are straightforward: stop other model companies from taking shortcuts in training, preserve their own lead, raise rivals' distillation and data collection costs, and do so without disrupting normal use too much.
On whether that can work, his answer was blunt. It is extremely hard to defend against. As long as a model is available for people to use, there is always a risk of leakage.
He pointed to OpenAI's public beta launch of the Agents API on Sept. 10. Unlike the earlier Responses API, where a user gives an input and receives an output with a chain of thought in the middle, the Agents API is closer to a task-based system: the user submits a task and gets the result, while the intermediate chain of thought and the agent harness are hidden. Liu said that is effectively selling a full execution environment.
In that setup, users no longer need to manage many low-level details, and task execution becomes more abstract. He said distillation attacks may become more expensive in such a setting.
Even so, he maintained that anti-distillation is a pseudo-problem. In theory, he said, distillation is not truly defensible. If distilling another model is more efficient than building training data from scratch, people will choose distillation first.
The commercial logic behind the crackdown
Liu said he has long viewed the industry's moat as shifting over time. In 2024, he believed short-term architectural innovation in model design was a moat. After that, the moat could expand to compute. Later, it could move to infrastructure and energy. From what he sees now, the industry is broadly moving in that direction.
In that framework, he places distillation closer to the compute side. In theory, many capabilities can still be reached through repeated trial and error, but time may not allow it.
That is why, in his view, anti-distillation is not only about protecting model capability. It is also about protecting the business value built on technical leadership. Frontier AI companies are trying to defend lead time, narrative control and valuation.
He cited a recent OpenAI claim that it used about 10,000 AI agents over 88 hours to complete a mathematical proof for one of the seven Millennium Prize Problems. He then offered a hypothetical example: if DeepSeek were to announce by the end of this year that it had solved all seven problems at one-tenth of OpenAI's cost, U.S. investors might ask why OpenAI needed so much money while delivering worse results. In that case, OpenAI's valuation could come under pressure.
For that reason, Liu said the anti-distillation push by OpenAI and other U.S. frontier model companies is fundamentally commercial, not purely technological. In Anthropic's case, he said, the company wants to collect data and build its own data flywheel while also making its annual recurring revenue look stronger ahead of a possible IPO. He said OpenAI follows a similar logic.
Liu also said OpenAI has kept issuing reset credits to users. In his view, that is effectively a way to exchange subsidies for more data while also making the financial picture look better.
Dario Amodei's call to slow down
On Sept. 12, Anthropic founder Dario Amodei called on the industry to slow the pace of frontier AI model iteration. Liu said statements like that may suggest the company sees bottlenecks ahead.
He grouped the possible bottlenecks into three areas: data, compute, and infrastructure or energy. He said it is hard to tell which one matters most right now, and it may be all three.
His broader point is that capital seeks returns. In theory, if a company believed it could build AGI, it would have little reason to stop and every reason to keep pushing. If companies are now calling for limits under the banner of AI safety, he said, they have probably run into real constraints.
Distillation as equalization, and its risks
Liu said the core reason frontier model companies are strengthening anti-distillation measures is still to raise rivals' costs and preserve their lead. He added that these companies may believe Chinese models are catching up quickly, yet have not found a better way to stay ahead, so they tighten restrictions and try to seize the narrative first.
From one industry perspective, distillation is a major security issue and close to intellectual property theft. From a practical perspective, Liu said, it can also be seen as a form of AI equalization. In his words, distillation is one way to rebalance the market quickly. Because there are many competitors, no single model company can easily dominate the market and set prices however it wants.
Still, he said, distillation has costs. For the industry, output styles may converge and models may inherit the failure modes of a small number of teacher models. For ordinary users, there is also the risk of privacy leakage.
More broadly, Liu said he is focused on the structural tension between AI safety and commercial incentives. Model companies do invest heavily in safety, but stricter safety measures can in some cases reduce model capability, delay product releases or weaken user experience. Safety goals and competitive goals do not always line up.
That makes governance difficult. Liu said one key question is how to align safety investment more closely with business incentives. At least in the short term, he does not see an elegant solution that fixes safety problems without sacrificing model performance.
The fight will continue
Looking ahead, Liu said the offense-defense cycle around distillation will continue and may settle into a dynamic balance. He sees several possible paths: distilling without chain-of-thought at all and simply testing how far a frontier model can be copied; obtaining native chain-of-thought where that remains possible; or synthesizing chain-of-thought independently.
For ordinary users worried about privacy leakage, his advice was simple. Do not use relay services. If a model gives you the option to allow your data to be used for training, do not opt in. Beyond that, he said, there is not much ordinary users can do.