DeepSeek has pushed model distillation into the center of the AI debate. According to the source material, after releasing its 671B-parameter R1 model, the company also introduced seven distilled versions ranging from 1.5B to 70B parameters. The pitch is clear: keep as much reasoning ability as possible while cutting deployment costs. Pricing is part of that argument. The article says DeepSeek-R1 charges $0.55 per million input tokens and $2.19 per million output tokens, compared with $2.50 and $10 for GPT-4o.
Distillation starts with Hinton’s idea of “dark knowledge”
The basic framework goes back to 2015, when Geoffrey Hinton, Oriol Vinyals, and Jeff Dean published “Distilling the Knowledge in a Neural Network.” Their key insight was that a large model does more than produce the right answer. It also produces a probability distribution across alternatives, including tiny values that reveal how the model relates one concept to another.
Those small probabilities are what Hinton described as dark knowledge. In standard training, a model learns from hard labels, the single correct answer. In distillation, a smaller student model learns from soft labels, meaning the richer output distribution produced by a larger teacher model. That gives the student more signal than a plain right-or-wrong target, even if the student is much smaller.
High-temperature training makes the transfer work
The source also points to temperature scaling as a technical hinge. During distillation, the temperature is raised so the softmax output becomes smoother and carries more information. Once training is done, the temperature is lowered again for inference, letting the student return sharper and more precise answers.
The classroom analogy in the article is simple but effective: a teacher does not just say the answer is A, but explains why A fits better than B or C. That is what the student model absorbs. The piece notes that this logic has spread widely through the industry, citing OpenAI’s GPT-4o Mini, distilled versions of Meta’s Llama models, and Google’s Gemini Flash.
DeepSeek turns a research method into a pricing strategy
What made the market react was not just the method, but the performance-per-cost ratio. The source says DeepSeek’s distilled 7B model beat OpenAI’s o1-mini on math reasoning benchmarks, while DeepSeek-R1-Distill-Qwen-32B outperformed OpenAI o1 across several benchmarks. A smaller student model was no longer just cheaper. It was also showing competitive results in direct comparisons.
That split reflects two different business models. One approach is to spend heavily on compute, data, and training to build the smartest possible teacher model, then rent access through APIs. The other is to find a strong teacher and compress its capabilities into smaller models that are cheap to run almost anywhere. The article adds that Nvidia lost nearly $600 billion in market value in the days after the DeepSeek news, as investors reconsidered how much advantage comes from simply owning more GPUs.
The legal dispute is about whose model can be distilled
The same technique is now at the heart of a growing fight. The source says Anthropic accused DeepSeek, Moonshot AI, and MiniMax in February of conducting an “industrial-scale distillation attack” against Claude. Anthropic claimed the companies used about 24,000 fake accounts and made more than 16 million queries in order to systematically extract outputs for training. OpenAI also sent a public letter to US lawmakers saying it had seen signs that DeepSeek was distilling its models in an obfuscated way.
The hard part is that the technical process itself looks similar whether it is internal optimization or disputed extraction. AI companies routinely distill their own systems into smaller and cheaper products. The argument begins when one company is accused of using another company’s outputs as the teacher. The source notes that US courts are still handling cases that may define where model output, training labor, and property rights meet. Until those decisions arrive, the fight over distillation is unlikely to slow down.
In the source material, distillation is not framed as simple copying. It is described as a way to reconstruct behavior by observing outputs rather than transferring parameters directly. That distinction is exactly why the technology has become so hard to contain and so important to the economics of AI.