The Sequence Knowledge #920: The Physics of Teaching: Distillation Scaling Laws

Every field becomes a science at the moment it stops collecting anecdotes and starts fitting curves.
For most of its history, distillation was an anecdote field. It worked, often spectacularly, and nobody could tell you in advance by how much. Should the teacher be as strong as possible? Folk wisdom said yes; practitioners kept tripping over cases where a stronger teacher produced a worse student. How much data does distillation need? Depends who you asked. Was distilling ever actually cheaper than just training the small model longer? Shrug. The field ran on vibes and ablations, which is a fine way to write papers and a terrifying way to spend ten million dollars on a training run.
Meanwhile, right next door, pretraining had undergone exactly the transformation distillation lacked. The Kaplan scaling laws, then Chinchilla, turned “how big a model should I train, on how much data?” from a matter of taste into a matter of arithmetic. Loss became a predictable function of parameters and tokens. Budgets became optimization problems. The single most consequential number in the industry — twenty-ish tokens per parameter — fell out of a fitted curve.
The obvious question hung there for three years: where is the Chinchilla of distillation? If a student’s loss is a function of its size and its data, it must also be a function of its teacher. What does that function look like? In early 2025, a team at Apple led by Dan Busbridge answered it, with the most compute-intensive controlled study of distillation ever run — students from 143 million to 12.6 billion parameters, teachers spanning a similar range, up to 512 billion training tokens. The resulting paper, Distillation Scaling Laws, is the closest thing the field now has to physics. This essay is about what the curve says, and what it quietly settles.

Facts Only

* Distillation historically relied on anecdotes rather than fitted curves.
* Practitioners found that stronger teachers did not guarantee better student performance.
* The cost-effectiveness of distillation relative to long training is unclear.
* Pretraining evolved via scaling laws like Kaplan and Chinchilla.
* Loss became a predictable function of parameters and tokens in pretraining.
* A consequential relationship, twenty-ish tokens per parameter, emerged from fitting curves in the industry.
* Apple led a controlled study on distillation: students ranged from 143 million to 12.6 billion parameters with teachers in a similar range.
* The Apple study involved up to 512 billion training tokens.
* The resulting paper is titled Distillation Scaling Laws.

Executive Summary

The field of distillation has historically relied on anecdotal evidence, where results were unpredictable regarding the optimal teacher strength. Pretraining, conversely, has evolved by adopting scaling laws, such as Kaplan scaling and Chinchilla, transforming model training from an intuitive process into a mathematical optimization problem. This shift is reflected in the discovery that loss becomes a predictable function of parameters and tokens. A major unsolved question arose: extending this predictability to distillation required understanding how the teacher influences the student's outcome. A controlled study by Apple researchers investigated this relationship across a wide range of model sizes and training tokens, yielding "Distillation Scaling Laws." This research provides a framework for understanding the relationship between model size, data, and teacher influence in distillation.

Full Take

The narrative traces a transition from qualitative, experience-based practice to quantitative, physics-based understanding within the field of machine learning scaling. The core implication is that complex phenomena in AI can be governed by underlying mathematical laws when sufficient data and rigorous experimental control are applied. The tension between distillation's historical reliance on "vibes" and pretraining's establishment of arithmetic scaling suggests a philosophical shift: moving from intuitive craft to verifiable science. The search for the "Chinchilla of distillation" represents the desire to find the underlying physical law governing knowledge transfer, positing that teacher influence must also conform to predictable mathematical functions. The effort to establish these scaling laws is an attempt to impose scientific rigor on what was previously an area governed by heuristic wisdom, raising questions about who benefits from the established, yet unquantified, shortcuts in the industry. What remains unstated is the social and practical cost of delaying this mathematical formalization, and whether the pursuit of perfect curve-fitting ultimately serves the goals of human agency or merely optimizes computational expenditure.

Sentinel — Human

Confidence

The text reads like an analytical essay blending technical concepts with historical context, displaying the reflective pacing typical of expert commentary rather than automated synthesis.

Signals Detected

Sentence length variance is erratic; uses conversational framing ('Shrug', 'fine way') mixed with highly academic language.

Clear argumentative progression that weaves historical context (anecdotes vs. science) into a contemporary technical discussion, displaying an idiosyncratic focus on the narrative tension.

The argument flows logically from historical problems to mathematical solutions and culminates in citing specific research ('Distillation Scaling Laws') without boilerplate attribution.

Specific technical references (Kaplan scaling laws, Chinchilla, 20 tokens per parameter) and the mention of a specific team/paper context suggest grounding in real-world discourse.

Human Indicators

The piece employs a narrative voice that juxtaposes philosophical musings about 'vibes' against rigorous mathematical results, which is a characteristic of thoughtful, non-purely algorithmic writing.