Implicit Unlikelihood Training: Improving Neural Text Generation with RL

01 / repetition-trap

Greedy decoding walks in circles

Language models trained purely by maximum likelihood have a famous failure mode, documented by Holtzman and colleagues: under maximization-based decoding they repeat themselves. Our baseline reproduces it on cue. Fine-tuned on WikiText-103 and decoded greedily, GPT-2 medium turns a biography prefix into the same clause forever: promoted to the rank of CB, promoted to the rank of CBE, four words rotating until the token budget runs out.

We measure the loop with sequence repetition, $\operatorname{seq\_rep}_4(\mathbf{x}) = 1 - \tfrac{\#\text{unique 4-grams}}{\#\text{total 4-grams}}$ , the fraction of duplicate 4-grams in a continuation. The looping sample above scores 0.65. Human text almost never repeats a 4-gram, so the target is near zero, and perplexity has to stay put while we get there.

The loop, verbatim

Degeneration

Two continuations of the same WikiText-103 prefix by GPT-2 medium, quoted from our samples. Plain fine-tuning loops one 4-gram; the same model after i-UT fine-tuning moves on, and the repetition metric falls from 0.65 to 0.019.

02 / why-implicit

When you cannot write down the negatives

The strongest training-time fix is unlikelihood training, from Welleck and colleagues: alongside likelihood, minimize $\mathcal{L}^{t}_{\mathrm{UL}}=-\sum_{c\in\mathcal{C}^{t}}\log\left(1-p_{\theta}(c\mid x_{<t})\right)$ , which pushes down the probability of negative candidates $\mathcal{C}^{t}$ , in practice the tokens that would extend an already-seen n-gram. It works, but only after $\mathcal{C}^{t}$ has been spelled out by hand.

That requirement is the crack we aim at. For repetition the negatives are easy to enumerate. For a property like toxicity, scored by an external classifier, there is no natural candidate set to punish. So we make the unlikelihood implicit: fine-tune with policy-gradient reinforcement learning against the metric itself, with the reward of a sampled continuation defined as $R = 1 - \operatorname{seq\_rep}_4(\mathbf{x})$ . Anything you can score, you can train against.

03 / mechanism

Three losses take turns

Training alternates three updates, gated by an update rate $r$ . With probability $1-r$ we take an ordinary likelihood step on real text, which anchors perplexity. Otherwise a fair coin picks either a sequence-level unlikelihood step or a policy-gradient step, so each runs at rate $r/2$ . Unlikelihood training is exactly this scheme with the policy-gradient path removed.

One gate, three updates

Training mixer

The full i-UT recipe over 5,000 updates. Removing the green path gives standard unlikelihood training; removing the middle path gives plain policy-gradient fine-tuning. We test all three.

The policy-gradient step is deliberately plain REINFORCE. We greedily decode 100-token continuations for a batch of $m$ fifty-token prefixes, score each sequence with $R_j$ , and subtract the batch mean as a baseline, $\Psi_j = R_j - \tfrac{1}{m}\sum_{k} R_k$ , giving the loss

\mathcal{L}(\theta,\mathcal{D}_m) = -\frac{1}{m}\sum_{j=1}^{m}\Psi_j\cdot\frac{1}{T}\sum_{t=k+1}^{k+T}\log p_{\theta}\!\left(x^{(j)}_{t}\mid x^{(j)}_{<t}\right).

Each token's log-probability is weighted by how much its sequence beat the batch average. A coefficient $c$ scales this loss against the other two; we sweep $c\in\{3,9,15,30\}$ . The greedy rollouts are load-bearing: every other sampling scheme we tried made the repetition metric converge worse.

04 / results

Several times less repetition at the same perplexity

Perplexity against greedy repetition

greedy seq_rep_4, lower is better

i-UT (ours)ablations and baselines

Small GPT-2 at update rate 0.5, five seeds per point. Lower-left is better. Every i-UT setting beats every ablation on repetition, and c = 15 also has the best perplexity of all fine-tuning schemes, 19.17.

The components compound rather than compete. Unlikelihood alone leaves greedy repetition at 0.056 with perplexity 19.44; policy gradient alone reaches 0.032; run together inside i-UT, repetition lands between 0.005 and 0.009 while perplexity improves to 19.17 at $c=15$ . The count of unique generated tokens rises too, from 11,210 to 11,432.

Repetition across decoding strategies

seq_rep_4

unlikelihood trainingi-UT, c=30 (ours)

Small GPT-2, validation prefixes. The advantage is largest exactly where degeneration is worst: greedy decoding. Where sampling already randomizes away repetition, at top-p 0.9, the two match.

Beam search is where models loop hardest

seq_rep_4

plain fine-tuningunlikelihood trainingi-UT (ours)

Beam-search continuations on validation prefixes. Plain fine-tuning repeats about two thirds of its 4-grams; i-UT cuts that to 0.03 on the small model, ahead of unlikelihood training in every configuration.

On held-out test data with GPT-2 medium the wins are consistent rather than cosmetic: softmax perplexity improves from 13.83 for plain fine-tuning to 13.16, the Jensen-Shannon divergence to the ground-truth distribution is lowest for i-UT under every sampling scheme we evaluated, and the model produces the most unique tokens, 20,493 against 19,932.

05 / negative-results

What did not work, and what opens up

We report the dead ends because they shaped the recipe. Proximal Policy Optimization destabilized validation perplexity and never produced comparable results. Replacing the one-reward-per-sequence estimator with per-token rewards, with or without generalized advantage estimation, always lost to a single sequence reward minus the batch mean. And rollouts sampled any way other than greedily made the repetition metric converge worse.

The point of making unlikelihood implicit was never repetition alone. The same loop optimizes any property of generated text you can score, with toxicity and bias the objectives we want next. Repetition was the controlled experiment showing the loop works without paying in language-model quality.