The Differences Between Direct Alignment Algorithms are a Blur

One recipe, many logos

DPO started a small industry. Skip the reward model and the reinforcement-learning loop, train the policy directly on preference pairs, and alignment becomes a supervised objective. Dozens of successors followed, each modifying the loss and claiming an edge: IPO, SimPO, NCA, Cal-DPO, APO-Zero, and the one-stage odds-ratio family of ORPO and ASFT. Reading their results side by side is nearly impossible, because each method arrives with its own training protocol, its own hyperparameter conventions, and its own favorite benchmark.

We wanted the comparison the literature kept not making: fix the pipeline, fix the search budget, and ask which design choices actually move alignment quality. Two axes organize every method we tested, and the axis nobody advertises turned out to be the one that matters.

Two axes hide inside every loss

The first axis is the scalar score a method optimizes. The DPO family scores a completion by how far the policy has moved from a frozen reference, $r^{\mathrm{ref}}_\theta=\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}$. ORPO and ASFT instead use the odds of generating the sequence, $r^{\mathrm{odds}}_\theta=\log\frac{\pi_\theta(y\mid x)}{1-\pi_\theta(y\mid x)}$, with no reference policy at all.

The second axis is how the score enters the loss. Pairwise objectives, DPO, IPO, SimPO, ORPO, optimize the gap between chosen and rejected, in the classic form $-\log\sigma(\beta r_w-\beta r_l)$. Pointwise objectives, APO-Zero, NCA, Cal-DPO, ASFT, push each side independently: raise $r_w$, lower $r_l$, no comparison required. Learning-to-rank research treats this split as fundamental; DAA papers rarely mention it.

Putting every method on the same footing

Two obstacles blocked a clean comparison. ORPO and ASFT are one-stage recipes: they bolt an alignment term onto the SFT loss and train straight from the base model. And they ship without the temperature $\beta$ that every DPO-family method tunes. Neither trait is essential. The alignment terms already contain the SFT signal, ASFT's is exactly a binary cross-entropy, and $\mathcal{L}_{\mathrm{ORPO}}\le\mathcal{L}_{\mathrm{ASFT}}$, so the explicit SFT term is redundant.

So we unify. Train SFT first, then apply only the alignment term, tempered: $-\log\sigma(\beta r^{\mathrm{odds}}_\theta(y_w,x))-\log\sigma(-\beta r^{\mathrm{odds}}_\theta(y_l,x))$ for ASFT, and $-\log\sigma(\beta r^{\mathrm{odds}}_\theta(y_w,x)-\beta r^{\mathrm{odds}}_\theta(y_l,x))$ for ORPO. Setting $\beta=1$ recovers the originals, so nothing is lost, and every method now lives in the same two-stage pipeline and the same hyperparameter space.

An explicit SFT stage is worth ten points

The conversion is not a formality. On Llama 3.1 8B with UltraFeedback, one-stage ORPO trained from the base model reaches 14.8 length-controlled win rate on AlpacaEval 2. The same objective, run as alignment-only on top of an SFT checkpoint, reaches 24.1, right next to DPO's 23.4. Keeping the combined loss but merely starting from SFT is worse than either at 13.4, so the separation itself does the work, not the initialization.

ASFT follows the same pattern at a lower level, 14.5 to 16.4. The SFT stage itself is also cheaper than folklore suggests: across every method we tested, training the SFT checkpoint on just 5 to 10 percent of UltraChat already delivers at least 95 percent of the full-data alignment score.

The temperature they shipped without

Tempering is the second missing piece. On the Reddit TL;DR setup the effect is dramatic: a tuned $\beta$ lifts ASFT by 43.4 GPT-4 win-rate points, from 43.8 to 87.2, and ORPO by 7.0. On the harder 8B UltraFeedback setup the gains are 8.27 and 3.46 length-controlled points. A knob the original methods did not expose turns out to be a first-order control on their quality.

The axis that survives unification

With everything unified, we ran the comparison properly: at least six $\beta$ values times four learning rates per method, 651 GPU-days in total. The advertised differences between scalar scores dissolve; $r^{\mathrm{ref}}_\theta$ and $r^{\mathrm{odds}}_\theta$ methods land on top of each other. What separates the field at 8B is the ranking axis: every pairwise method beats every pointwise method on both AlpacaEval 2 and ArenaHard.

The split is statistically real and robust. Permutation tests over all 70 pairwise-versus-pointwise group assignments give $p=0.014$ on every 8B metric, and ranking by each method's top three configurations instead of its single best preserves the ordering, so no lucky hyperparameter is responsible. Mistral 7B reproduces the separation, and on Qwen 2.5 math reasoning it reappears at 14B, including under AlphaPO and forward-KL score families.

Why ranking style matters more than the score

Every score family defines a per-prompt average $b_\theta(x)=\mathbb{E}_y[r_\theta(x,y)]$, a bias the model inherits from its data. Pointwise losses keep pushing chosen scores up and rejected scores down even on pairs they already order correctly, which amounts to spending capacity unlearning $b_\theta(x)$. Pairwise losses are structurally indifferent to such uniform shifts, their total score gradient per prompt is zero, so the same capacity goes to the examples that still need it.

That account predicts where the gap should live: nowhere at the easy or hopeless extremes, and squarely at intermediate difficulty. It is exactly what we observe. On easy TL;DR summarization every method wins roughly 90 percent of GPT-4 comparisons, ASFT trailing at 87.2. At 3B on UltraFeedback the methods blur into overlapping confidence intervals. At 8B, where capacity suffices to unlearn bias but not to also master the hard examples, the families separate.

The scope is deliberate: offline alignment on static binary preference pairs, with judge-based results cross-checked on verifiable math benchmarks up to 14B. Online preference optimization may behave differently. Within that scope our advice is blunt: before believing a new direct alignment algorithm is better, check whether it changed the ranking objective or just redrew the logo.