Real deployments rarely fit the topic-level picture. The same base model may be adapted for a general assistant, an educational product, an enterprise system, or a public-sector service, and each setting needs different boundaries within the same topic. A civics tutor and a public-sector assistant can share a model yet require opposite behaviour on politics: both should answer factual questions about an election, but only one may need to refuse a request to write targeted political manipulation. A topic-level guard cannot express that split. LlamaGuard-3, for example, covers elections only as "factually incorrect information about electoral systems and processes," which excludes persuasion and manipulation and, at the same time, excludes the factual prompts a deployment must keep answering.
Our latest paper, Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal, studies this narrower problem directly. The question is not whether an entire topic should be refused, but which subset of that topic is incompatible with a given deployment policy, and how to train and measure a model against that boundary.
Narrow-boundary safety
We formalise the setting as a topic universe, all political prompts in our experiments, that contains a target-harmful subset the deployment wants to refuse. The intended policy is not to refuse all of politics, but to refuse the harmful subset while continuing to answer the benign complement. The ideal behaviour is a sharp step: refuse inside the subset, answer everywhere else in the topic.
The narrow-boundary setting. A deployment may need to refuse only the political prompts that ask for manipulation or targeted persuasion, while still answering other political prompts, rather than refusing all of politics. A trained model's refusal is smoother than the ideal split and can spill into benign territory near the boundary. Source: paper Figure 1.
A trained model never learns that sharp step. It learns a refusal probability that only approximates the target, and cross-entropy training that raises refusal inside the harmful subset can also push refusal outward into the benign complement. So the real problem is not only raising refusal on harmful prompts, but shaping the behaviour near the boundary itself. We operationalise that boundary as pairs of prompts that share a topic anchor and differ only in intent, one that should be refused and one that should be answered.
We use political persuasion as the testbed, since manipulative persuasion can cause real harm while factual political information stays legitimate, which is exactly the case where topic-level refusal is too blunt.
Where self-generated safety tuning breaks
The natural way to build training data here is self-generation: take the target model, steer it toward a refusal on each harmful prompt, and keep the traces a guard model verifies as genuine refusals. This is the recipe behind methods like ThinkSafe, and we adopt it as our reference, applied to political prompts and measured component by component. Framing the problem as a boundary rather than a topic exposes three weaknesses in that standard pipeline.
The first is a coverage gap. A single steering attempt does not always produce an accepted refusal, and those prompts are silently dropped from the training set. In our audited pool, single-shot generation drops 19.88% of prompts, 8,009 of them, and these failed prompts may well be the hardest examples. We repair this instead of discarding it: an escalating retry strategy, resampling the same prompt through progressively stronger steering, brings the residual failures down to 0.20%, or 79 prompts. Coverage repair leaves 40,293 harmful training prompts where the naive pipeline would have thrown thousands away.
The second is downside reactions. Safety tuning tends to produce false refusals on benign prompts that look superficially dangerous. To compensate, we build in-distribution benign data, including 11,955 verified surface-dangerous benign prompts across 18 semantic types, so the model sees safe prompts with dangerous-looking wording during training rather than only at evaluation.
The third is that ordinary harmful and benign splits do not measure the shape of the boundary at all. A model can improve its harmful-refusal rate simply by expanding refusal into nearby permissible prompts, and a topic-level metric will call that an improvement. Held-out harmful-benign pairs, 1,539 per side, let us measure both sides of the boundary directly.
The trade-off, and a trap it hides
Training on political refusal data works in the obvious sense. On Qwen3-8B, the escalated-coverage model raises in-distribution political refusal from 9.47% to 84.75%, and it also transfers: the mean unsafe-response rate across three broader harmfulness benchmarks, HarmBench, StrongREJECT, and WildJailbreak, scored by LlamaGuard-3, falls from 26.26% to 0.14% in the strongest configuration.
Reported alone, those numbers look like a clean win. They are not. At the same checkpoint, over-refusal on XSTest rises from 2.00% to 74.00%. The configuration with the lowest harmful-response rate is also the one that refuses nearly three quarters of plainly safe prompts. It is a blunt refusal machine, not a safer model, and you cannot see that unless you measure the benign side. This is the central message: data composition decides where a checkpoint sits in the space of safety against over-refusal, so the two axes have to be reported together.
Two of our data components pull the over-refusal number back down without giving up the safety gain. Replacing externally adopted compliance responses with verified responses generated by the target model itself lowers XSTest over-refusal from 15.20% to 5.20% under single-shot generation, at a modest harmfulness cost. And the harmful-benign boundary pairs do the most precise work of all.
Left: over-refusal on the comply-worthy side of the held-out boundary, lower is better. Runs with the benign boundary data (PB) fall to 0.03 to 0.08; without it, the number rises toward 0.49. Right: refusal on the harmful side, higher is better, which falls only slightly. Source: paper Figure 6.
Concretely, adding the benign boundary data reduces over-refusal on the comply-worthy side of the held-out pairs from 32.94% to 4.16%. Refusal on the harmful side drops only from 91.88% to 87.72%. In other words, most of the false refusals near the boundary disappear while almost all of the genuine refusals survive. There is a real recall cost, and it is small and measurable, which is the point: you can only trade it off deliberately if you are measuring both sides.
What this changes
The practical takeaway is that safety tuning should not be assessed by harmful-refusal rate alone. A model that refuses more is not automatically safer, and on a narrow boundary the same move that raises refusal on harmful prompts can quietly make the model useless on the legitimate prompts right next to them. Composition of the training data, coverage repair, in-distribution compensation, and boundary pairs are what control that trade-off, and both sides of the intended boundary have to be evaluated for the numbers to mean anything.
This work is part of Multiverse Computing's research into making model behaviour controllable and measurable at the level real deployments care about, rather than at the level of broad topic categories. The same generation pipeline extends to other topics beyond politics, and the paper reports the full set of data-composition ablations behind the results above.
Want the full technical details, including the coverage-repair strategies, the loss routing that separates harmful cross-entropy from benign forward-KL preservation, and the complete held-out boundary evaluation? Read the full paper, or get in touch with our team to talk about deployment-specific safety for your own models.
Facts Only
* A model can be adapted for various settings (e.g., assistant, educational product) requiring different boundaries within a single topic.
* Topic-level guards cannot express specific behavioral splits between different applications sharing the same topic.
* The objective is to refine safety refusal to target a specific harmful subset of a topic rather than refusing the entire topic.
* A target behavior involves refusing only the harmful subset while answering the benign complement within the topic universe.
* Self-generated safety tuning methods face coverage gaps where single-shot generation drops prompts, leading to failed examples in training.
* Safety tuning tends to produce false refusals on benign prompts.
* Measuring only harmful-refusal rates fails to capture the shape of the boundary between harmful and benign requests.
* Training on political refusal data can lead to over-refusals on safe prompts, as demonstrated by increased refusal across broader harm benchmarks.
* Incorporating in-distribution benign data alongside harmful-benign pairs reduces over-refusal on the comply-worthy side of the boundary from 32.94% to 4.16%.
Executive Summary
The research investigates the challenge of applying topic-level safety refusal policies to real-world deployments where different applications within the same topic require divergent behavioral constraints. The core problem identified is that a general, topic-level guard cannot differentiate between acceptable and unacceptable requests, as demonstrated by examples like a civics tutor versus a public-sector assistant. The study focuses on establishing "narrow-boundary safety," aiming to refuse only the harmful subset of prompts within a larger topic while still answering benign complements. This approach is tested using political persuasion as a testbed because it separates factual information from manipulative intent.
The methodology addresses weaknesses in standard safety tuning pipelines, such as coverage gaps and downside reactions. The researchers use self-generated data generation alongside boundary pairs of prompts to measure the model's behavior near the refusal boundary. Findings indicate that focusing solely on harmful-refusal rates can lead to over-refusals on benign prompts. To mitigate this, incorporating benign boundary data is crucial; it shows that reducing over-refusal on the permissible side while maintaining genuine refusals requires measuring both sides of the boundary simultaneously. The central conclusion is that safety assessment must account for the composition of training data and the shape of the safety boundary rather than relying only on aggregate refusal rates.
Full Take
The tension in this research arises from the conflict between macro-level policy setting and micro-level deployment reality. The framework suggests that safety alignment is not a monolithic state but a spatially defined property of the model's response space. The failure of traditional metrics highlights an underlying systemic assumption: that any refinement to safety is additive or monotonic, when in fact, modifying the input distribution during tuning inevitably shifts the decision boundary unpredictably. When safety tuning inflates refusal on benign prompts, it demonstrates a lack of awareness regarding the proximity of valid responses—a manifestation of system opacity where increased statistical performance masks the loss of fine-grained behavioral control.
The practical implication is that safety engineering must evolve from post-hoc metric checking to proactive geometric mapping. The proposed solution, involving boundary definition and dual measurement of both sides of a transition, forces an acknowledgment that utility involves navigating ambiguity rather than imposing arbitrary absolutes. The cost discussed—a small recall trade-off—is reframed not as an unavoidable loss, but as the necessary price for achieving contextual control; the system must be engineered to account for this deliberate partitioning of response space, ensuring that safety aligns with operational goals rather than just superficial compliance metrics.
BRIDGE QUESTIONS:
If the boundary is inherently dynamic based on deployment context, what mechanisms should govern the selection of these boundary definitions across disparate applications? How can continuous monitoring systems track and flag when a model's learned boundary has drifted into an unsafe region for a specific deployment scenario? If the cost of false refusal is measurable, how should human safety oversight integrate this cost into the decision-making process when balancing utility against caution?
Sentinel — Human
This analysis presents a sophisticated argument about the limitations of current LLM safety tuning methods, focusing on boundary definition and data composition rather than simple outcome metrics.