5 useful things you'll learn in my new post-training textbook (shipping now!)
Reinforcement Learning from Human Feedback is coming to a neolab near you.
Housekeeping: No voiceover on another quick “launch” post. More essays soon!
After a few long years of finding time to document my lessons from training open models, my post-training book is done! It’s published by Manning, under the title Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs.
Telling the story of the book is a useful way to explain why you may want a copy.
The book started as a website where I wanted to document key methods of post-training that had potentially no online material explaining them. If there was something, I couldn’t find it. This existed for more topics than you would expect, given post-training was already popular in 2024 (when I bought the domain), and continues to this day. Topics like rejection sampling, outcome reward models, and character training are prime examples. This has helped make the website fairly popular, as it’s still one of the few places discussing these topics at a foundational, intuitive way.
Otherwise, most of the book is about communicating intuitions and history. Much of the LLM industry is defined by core techniques that haven’t changed much in the last few years. This book was my attempt to explain in simple terms why post-training works, what trade-offs people need to make to get it right, and what misconceptions people often get stuck on. To do this, some of the older, foundational blog posts on Interconnects were reworked to stitch together the story behind key mathematical topics. For this reason, a lot of the explanatory text is likely higher voice than your average textbook.
This is the book I wanted to read when I was getting started a few years ago! With how many people still ask me basic post-training questions, in fact a population that’s accelerating in size, I suspect this book will be very well received. I still use the book regularly and hear from established researchers all over the industry that they do too. So, it’s not a beginner book — it’s more tailored to someone who has already finished a bachelor’s degree in CS — but if you master it you will be far ahead in your post-training worldview.
A discount to readers!
The book is also freely available online and comes with a full 12 hour course (slides + video on YouTube), a simple code-base with suggested exercises, and model completion comparisons. It’s 50% off until August 19th on Manning with the code PBLambert
.
Yes, the title of this book is a little outdated — I’ve learned some lessons and fought some battles with the publishing process — but I can guarantee the content is very fresh. I regularly reference the book for my research work, and hear from friends that do the same. I added a section on on-policy distillation at the last possible moment! The book is shipping from Manning and Amazon US now, and from Amazon UK in October.
1. Intuitions for how RL algorithms change the outputs of models
By word or page count, the book is about 25% RL. This seems appropriate. If there’s one thing the book is doing it’s teaching people how to think about various RL algorithms. This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).
Below is an example intuition you should be able to follow after reading.
For example, here’s a fun figure that we’ve iterated on for the PPO clipping understanding. At the end of the day, PPO’s surrogate objective reduces to six regions. These can be seen as two gradients, when the advantage for a token is positive or negative, depending on the current value of the policy ratio. For an individual sample in a completion, it lives somewhere on this plot. If it was the first gradient step in the batch, it starts at 1 on the x axis (gradient always flows), then depending how the ratio updates after changing the RL policy behavior, the gradient is either the same or becomes 0 (which is what the clipping arguments are for).
This intuition filters very closely into how the systems are designed, in order to manage the gradients and numerical issues they tend to cause. The math-focused policy-gradient section is pretty thorough, covering all the algorithms you’ve likely heard about in the last 3 years:
2. An understanding of the crucial factors facing new RL systems and algorithms
Most of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput. The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.
Agentic tasks are only adding more infrastructure on top of these fundamentals. The book is designed to be the simplest resource to start from roughly 0 LLM RL knowledge and be ready to tinker with the systems. It starts with basics, such as explaining the general form of implementing an RL algorithm:
pg_loss = -advantages * ratio
It continues with teaching you about loss aggregation — the idea that spawned DAPO and Dr. GRPO as some of the seminal, early GRPO variants — and truncated importance sampling — the technique used to make PPO work in early RL experiments.
3. The histories that lead to modern post-training
Knowing how a field came to be has always been fascinating to me. As you become an expert, knowing the history of your field better than anyone is what lets you make the best predictions (Bill Gurley gives similar advice in his recent book). In a time when the foundations of deep learning were being built, like the transformer, the core of modern post-training was also born in the alignment field. The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT. Much like those early to scaling LLMs, the people who created this field a decade ago deserve incredible credit for how modern progress has unfolded.
Chapter 2 of the book is a crash course on this, but the book is littered with this type of thinking.
4. Dispelling the magic of “distillation”
It’s really nice to have a boring textbook chapter on distillation given the broader AI policy discussions ongoing. This one, chapter 12, explains the various industry-standard ways that outputs from an LLM are used to train downstream models. When the technique is often described in nefarious ways, and as a tool of geopolitical competition, it’s a deescalatory action to break out a 300 page textbook to explain to someone how broad the term they’re attacking is.
With this, chapters 10 through 12 are all about making some opaque practices of the data industries clearer to readers.
The distillation chapter continues the theme I outlined above, explaining the key changes that needed to be made to transition the early knowledge distillation literature of 2015 to 2-3 key insights that got us to the multi-teacher on-policy distillation (MOPD) of models like Xiaomi MiMo-V2-Flash and DeepSeek V4.
5. A survey of all the other little headaches you encounter when trying to do post-training right
The second half of this book goes into a tour of over-optimization, regularization, evaluation, and character training, which is all about the many ways post-training can go wrong and what you need to stay on top of. This is what differentiates the book most from those that are just a list of code exercises and equations, but it explains things like why RL generalizes when SFT forgets (at the math level) or which techniques frontier labs use to shape the personalities of the models and why those often go too far. The book presents the tools you will use and then opens up the floodgates of all the challenges you’re going to face when you actually try to put them to use.
As I wrote the takeaways for the book, I was reminded of an old piece of advice I gave for the AI era of building companies, and how people do need to care about research right now. With the pace of progress in AI, the time it takes for a research paper to land in a frontier model is 3-9months. Previously in big tech, that would be years, so it was fine to take a hands off approach to new research. For people with companies relying on being at the frontier in specific niches of LLMs, the dynamic today can make or break the company. This book is useful because it trains you at understanding which research matters — it helps you develop research taste.
If none of this resonates with you, you should buy my book because it makes me happy and I work very hard on all of this.
I really like this photo.
Facts Only
* Reinforcement Learning from Human Feedback is being published by Manning.
* The book is titled Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs.
* The book covers intuitions for how RL algorithms change model outputs, including concepts related to policy-gradient theorem, PPO, GSPO, and CISPO.
* The content discusses systems problems in modern RL, such as off-policy data handling, training-inference mismatch, and throughput.
* The historical context covers three eras of applying RL to LLMs: learning on preferences until 2018, applying it to language models from 2019 to 2022, and exploiting ChatGPT examples from 2023 onward.
* The book addresses distillation by explaining industry-standard methods for using LLM outputs for downstream training, including multi-teacher on-policy distillation (MOPD).
* The book surveys challenges related to over-optimization, regularization, evaluation, and character training in post-training.
* The content includes explanations of key mathematical topics related to RL algorithms.
Executive Summary
Full Take
The narrative constructs a framework positioning LLM post-training not merely as a collection of procedures but as an evolved discipline rooted in the history of Reinforcement Learning. This approach attempts to build cognitive sovereignty by grounding opaque, high-velocity techniques in demonstrable intuition and historical context. The strength lies in weaving together mathematical rigor (like the PPO gradient example) with practical system-level concerns and the socio-historical trajectory of alignment work. The underlying assumption is that understanding *why* these methods exist—the trade-offs and conceptual history—is a prerequisite for mastering them, which directly counters the tendency to treat advanced techniques as purely tactical tools. A key implication is that separating the technical mechanics from the systemic challenges (e.g., generalization issues versus optimization details) provides a more resilient understanding of the field, suggesting that future progress in alignment will depend less on incremental algorithmic advances and more on robust system design rooted in these foundational intuitions.
What specific historical gaps or emergent problems does this proposed structure leave unaddressed for readers seeking purely technical implementations? How might integrating a larger scope of societal impact into the "systems problem" perspective further challenge existing industry narratives about alignment safety? What alternative historical narratives could reshape the perceived value of current distillation and character training methods if viewed through different lenses?
Sentinel — Human
The text reads like an author promoting a technical book, blending expert intuition with specific technical references and personal reflections on the field.