Classifiers are Better Experts for Controllable Text Generation

Steering a model you refuse to retrain

A language model is controllable when we can sample from $p(x\mid c)=\prod_i p(x_i\mid x_{<i},c)$ for a condition $c$ we care about: a sentiment, a topic, the absence of toxicity. Retraining a conditional model for every new $c$ is the brute-force answer, and CTRL-style training is exactly that. We wanted the opposite: take a frozen model and bolt the condition on at sampling time.

Bayes gives the recipe in one line: reweight each candidate token as $p(x_i\mid x_{<i},c)\propto p(x_i\mid x_{<i})\,p(c\mid x_{\le i})^{\alpha}$, where the exponent $\alpha$ sets how loudly the classifier speaks. The catch is cost. Scoring $p(c\mid x_{\le i})$ means running the classifier once per candidate token, and a GPT-2 vocabulary offers tens of thousands of candidates at every step.

Around that bottleneck the field grew workarounds. GeDi and DExperts distill the condition into small conditional language models that score the whole vocabulary in one pass, but those models must themselves be trained. PPLM instead nudges hidden states by gradient ascent at each step, which is slow and temperamental. Meanwhile, when we wrote this, Hugging Face hosted roughly ten thousand ready-made text classifiers and only about two dozen conditional language models. That asymmetry is the motivation: we asked whether the plain classifier is enough.

Classify only the tokens that could win

One observation unlocks it: sampling strategies throw almost every token away. If we sample from the top $k$ after reweighting, a token buried deep in $p(x_i\mid x_{<i})$ cannot claw its way back with any realistic classifier score, so classifying it is wasted work. CAIF therefore scores only the $j$ most probable candidates, with $k<j\ll|V|$; across all our experiments $j=100$ was enough.

Concretely, at each step the frozen model proposes its distribution, we keep the top $j$ candidates, append each one to the context, ask the classifier for $p(c\mid x_{\le i})$, add $\alpha\log p(c\mid x_{\le i})$ to the token logits, and sample from the resulting top $k$. Guidance becomes a drop-in replacement for the sampling loop.

Point the exponent the right way

For detoxification the classifier outputs a toxicity probability, and there are two ways to hold it against a token. The route earlier systems defaulted to weights tokens by $(1-p(c\mid x_{\le i}))^{\alpha}$ with $\alpha\ge 1$, the inverse-probability score. We found the restriction to positive exponents unnecessary: weighting by $p(c\mid x_{\le i})^{\alpha}$ with $\alpha<0$ is equally valid and behaves very differently.

The difference lives in the curve. The negative-$\alpha$ score $-\log x$ starts punishing the moment toxicity probability $x$ rises above noise, while $\log(1-x)$ barely moves until $x$ approaches one half. Head to head on a thousand prompts, negative $\alpha$ detoxified significantly better while keeping lower perplexity, so every later experiment uses it.

One classifier against three specialist systems

Our main test follows the DExperts protocol: ten thousand non-toxic prompts from RealToxicityPrompts, twenty-five continuations each, GPT-2 Large as the base model. CAIF is guided by an off-the-shelf toxic-bert classifier with $\alpha=-5$ and judged by a different, independently trained offensive-language classifier, with perplexity measured under GPT-2 XL, so no model grades its own homework.

The result is not a tradeoff but a dominance. We reach the lowest toxic-sample probability, 3.3 percent against 6.4 for the strongest DExperts variant and 11.2 for GeDi, and simultaneously the lowest perplexity, 15.0, where every specialist pays fluency for control. Binned by prompt toxicity, we beat DExperts on every bin below 0.75; on the most hostile prompts DExperts edges ahead on toxicity only by giving up far more perplexity.

Sentiment control tells the same story. On both neutral and negative prompts from OpenWebText, guiding with a public sentiment classifier produced more positive continuations at lower perplexity than PPLM, GeDi, and DExperts, with the same recipe and no retraining. The honest fine print is the distinctness column above: top-$k$ sampling repeats slightly more unigrams than the top-$p$ baselines.

Guide only where the model hesitates

Guiding every step still costs $j$ classifier calls per token, so we asked when the calls actually matter. A periodic criterion guides every $p$-th step and is obviously blind: with period two, a toxic token is free to appear on every unguided step. The entropy criterion is the interesting one. When the model's next-token entropy is low, the continuation is nearly forced, so we skip the classifier; we intervene only above a threshold $e$, where the text can still fork.

The empirical entropy distribution lets us match budgets exactly: a threshold of 3.2 guides about half the steps, like period two, and 5.0 guides about a fifth. At equal budget the entropy criterion wins on both perplexity and toxicity, and on sentiment control entropy-3.2 matched or beat plain CAIF while paying for half the classifier calls.

Speed is the one axis where distilled systems keep an edge. A free-form classifier re-reads the whole prefix, so each call grows quadratically with length, while GeDi and DExperts cache their way to linear cost; we measured CAIF faster up to twenty generated tokens, comparable to fifty, and slower beyond. For pipelines that otherwise sample many candidates and filter afterward, trading raw speed for far fewer rejected samples is the better bargain.