ESSA: Evolutionary Strategies for Scalable Alignment

Alignment without a backward pass

Online alignment of large language models usually means gradient-based reinforcement learning: PPO, GRPO, and their relatives. The pipelines work, but they are heavy machinery. They roll out long trajectories, backpropagate through them, keep optimizer state in sync across devices, and punish every hyperparameter mistake. The larger the model, the more the engineering dominates the research.

We take the opposite bet. Evolutionary strategies need nothing but forward passes with perturbed parameters and a scalar score per candidate, which makes them almost embarrassingly parallel and indifferent to sparse or noisy rewards. Their classic weakness is dimensionality: naive evolution over billions of weights goes nowhere. ESSA is our answer to that weakness, and it is mostly an exercise in shrinking the search space until evolution becomes practical.

Shrink the problem until evolution works

We start from parameter-efficient adaptation. Each attention projection gets a low-rank update $\Delta W = BA$ with $r \ll \min(m,n)$, and a short supervised fine-tuning stage initializes these adapters with task-aware weights. Then we compress further: we decompose each trained factor separately, $A = U_A \Sigma_A V_A^{\top}$ and $B = U_B \Sigma_B V_B^{\top}$, freeze all four orthogonal matrices, and keep only the singular values trainable.

The result is a candidate vector whose length is the number of adapted matrices times $2r$, a fraction $\alpha$ of which we actually optimize. For a 7B model this is thousands of numbers, not billions. The SVD basis matters: those frozen singular vectors point along the directions SFT already found useful, so evolution searches inside a subspace that is small and aligned with the task.

A training loop made of inference

We drive the search with CMA-ES, which maintains a full covariance model of promising directions: candidates are drawn as $x^{(k)}_{i+1} \sim \mathcal{N}(m_i, \sigma_i^2 C_i)$, evaluated, and folded back into updates of the mean $m_i$, the step size $\sigma_i$, and the covariance $C_i$. Each worker reconstructs its candidate adapters from a shared random seed, runs forward evaluation on a reward, and sends back a single scalar.

Because the loop never computes a gradient, the model only ever runs inference, and inference tolerates quantization. On Qwen2.5-32B trained on PRM800K, peak validation accuracy moves from 0.847 in BFLOAT16 to 0.844 in INT8 and 0.838 in INT4: less than a point for a four-fold memory cut. Under INT4, even a 72B backbone fits on a single GPU, one candidate per device.

Time-to-quality scales with the cluster

This is the thesis of ESSA in one plot. Candidate evaluations are independent, so extra devices turn directly into extra population throughput; the only synchronization is the seed broadcast and the reward gather. GRPO also speeds up with more hardware, but each step still all-reduces gradient tensors, and that communication floor rises with model size. On 16 GPUs we are already twice as fast to target accuracy; on 128 GPUs the gap grows to roughly six.

The advantage is not only about clusters. On a single GPU with the smaller Qwen2.5-3B, ESSA still converges faster than GRPO in our runs, because forward-only evaluation is simply cheaper per unit of learning signal in this regime.

Quality holds against the gradient baseline

Speed would mean nothing if quality collapsed, so we compare against GRPO, the de facto standard for online alignment. ESSA improves test accuracy of Qwen2.5-Math-7B by 12.6 points on GSM8K and 14.8 on PRM800K, and lifts LLaMA3.1-8B on IFEval by 22.5 points, where GRPO plateaus near 0.45 while ESSA holds around 0.6. On HelpSteer2 with a learned reward model, the two methods track each other closely.

The strangest result came from making the comparison maximally fair. We gave GRPO exactly our parameterization, updating only the SVD singular values, so both methods search the same tiny space. That SVD-GRPO variant stalls near 0.5 accuracy on PRM800K even at rank 16, while ESSA at rank 2 reaches about 0.72. Compressing the space concentrates the curvature into a few tightly coupled coordinates; a diagonal first-order optimizer like Adam struggles there, while CMA-ES adapts its covariance to exactly that structure.

Hyperparameters, meanwhile, are forgiving. Populations between 24 and 96 all work, ranks from 2 to 32 change little, and any $\alpha \ge 0.4$ of trainable singular values reaches full quality. GRPO's convergence degrades visibly as rank shrinks; ESSA barely notices.

The warm start is load-bearing

ESSA is not a full training pipeline; it replaces the expensive online RL stage on top of a cheap SFT warm start. That start is not optional. The frozen singular vectors are the coordinate system evolution searches in, and they are only as good as the SFT that produced them.

The honest boundary of the method sits here too. ESSA is capped by what fixed-rank LoRA can express, weak warm starts plateau early, and very large populations trade communication savings for raw FLOPs. What we claim is narrower and, we think, more useful: once the search space is compressed to the right few thousand coordinates, gradient-free alignment stops being a curiosity and becomes the fastest tool on the shelf.