Steering LLM Reasoning Through Bias-Only Adaptation

The whole intervention is one vector per layer

Reinforcement learning with verifiable rewards is the recipe that turned base language models into reasoning models: sample chains of thought, check the final answer, reinforce what worked. The standard version updates every parameter of the network, which is expensive, and it leaves a scientific itch: after all that training, nobody can point at what actually changed. We wanted the smallest intervention that still gets the result.

Our answer is almost embarrassingly small. We freeze every base weight and insert one learnable steering vector $s_\ell\in\mathbb{R}^{d}$ at the end of each of the $L$ transformer layers, added to the residual stream at every token position: $h_{\ell,t}\leftarrow h_{\ell,t}+s_\ell$. On an 8B-parameter model those $L$ vectors amount to roughly 0.0016 percent of the network.

Steering vectors are usually computed, not trained: the interpretability literature builds them from activation differences on contrastive prompts and treats them as amplifiers of features the model already has. We instead optimize them directly with reinforcement learning, at the scale of a real reasoning-training run, and ask how much of the fully fine-tuned model's gain this bias-only family can recover.

Trained with rewards, not demonstrations

Training is a standard online loop in the DeepSeek-R1 style. For each prompt we sample $N$ candidate solutions from the current policy $\pi_\theta$ and give each rollout a binary reward for a correct boxed answer. RLOO turns this into a low-variance update: the baseline $b=\tfrac{1}{N}\sum_i r_i$ is the mean reward of the group, each rollout gets advantage $a_i=r_i-b$, and the vectors follow the policy gradient $\mathbb{E}\left[a\,\nabla_\theta\log\pi_\theta(y\mid x)\right]$.

We train on the DeepScaleR problem set with 128 prompts per step and evaluate on six mathematical benchmarks: AIME24, AIME25, and AMC23 averaged over 32 samples because they are small, plus MATH500, MinervaMath, and OlympiadBench at pass at 1. Every number we report is a mean over three evaluation seeds, and each model family is compared in three variants: the untouched base, full RL fine-tuning, and our steering-only training.

It matches full fine-tuning almost everywhere

On Qwen2.5-14B steering actually edges ahead, 42.3 against 41.3; on Qwen2.5-Math-7B it is 43.3 against 43.5; on the instruct-tuned Llama the two are identical at 26.1. The one clear exception is base Llama3.1-8B, where steering reaches 9.1 against 11.5 for full tuning, recovering roughly seventy percent of the gain. We come back to what that exception teaches below.

The economics follow directly from the parameter count. Optimizer state for 245 thousand parameters is 240 kilobytes instead of 13.8 gigabytes, and pushing fresh weights to the sampling engine after every update takes a tenth of a second instead of ten. Over the 314 steps of an epoch that is 34 seconds of overhead where full fine-tuning spends 52 minutes.

A logit lens on what training added

Because the entire effect of training lives in $L$ vectors, we can ask directly what they do. We read each $s_\ell$ through the unembedding matrix with the logit-lens trick: for every vocabulary token $v$ with unembedding row $u_v$, the cosine $c_\ell(v)=\langle s_\ell,u_v\rangle\,/\,\lVert s_\ell\rVert\lVert u_v\rVert$ says whether the vector pushes the residual stream toward or away from that token. We collect the fifty most-aligned tokens per layer and have GPT-o3 translate and cluster them.

The depth profile tells a story. At layer 2 the vector aligns with source-code and test-harness vocabulary plus named entities, structure the model seems to borrow from programming for formal manipulation, and the names that populate word problems. At layer 17 it shifts to correctness, necessity, and confirmation, the vocabulary of checking a step. At layer 30 it promotes Because, Therefore, and However, in several languages at once, the connective tissue that assembles verified steps into an argument.

What one vector can and cannot do

A bias per layer cannot write new circuits into frozen weights; it can only shift the operating point of circuits that already exist. That such a shift recovers full RL performance is strong evidence for the view that reasoning training mostly amplifies behaviors pretraining already built. Our results tighten the parameter budget on that claim considerably: not millions of adapter weights, but thousands of bias dimensions.

The base Llama3.1-8B exception sharpens the picture rather than blurring it. A steering vector is the same at every token position, and in a follow-up on GSM8K and MATH we found that replacing it with a rank-4 LoRA on the MLP down-projection, in effect a token-dependent steering vector, closes the remaining gap. Where a single global direction is not enough, a slightly position-aware one is.

We are careful about scope. Our sweep covers one online-RL setup and mathematical reasoning only, and the logit lens reads each vector in isolation, ignoring how later layers transform the injected signal. But the practical recipe stands: if you want an RL-tuned reasoning model and a budget measured in kilobytes of optimizer state, one vector per layer is very likely all you need.