F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

RLVR mostly sharpens what the base model already knew

Reinforcement learning with verifiable rewards is now the default way to post-train reasoning models: sample a group of $N$ rollouts per prompt, check each one, and push probability toward the correct ones. A growing body of evidence says this process rarely adds new solutions; it concentrates the distribution on solutions the base model could already reach. The symptom is a model whose pass@1 rises while its pass@256 falls below the base model's.

Group-relative methods make the sampling structure explicit. GRPO scores each rollout against its own group, $\widehat{A}_i=(R_i-\bar{R})/(\sigma_R+\epsilon)$, so when every rollout in the group is right or every one is wrong, the advantage is zero and nothing updates. The group size $N$ therefore controls which prompts speak on each step, and the field offers conflicting advice about it: two rollouts suffice in one analysis, while another argues for scaling rollouts broadly. We wanted to know what finite $N$ actually does to rare correct behavior.

The tail-miss event: active updates that never see the rare answer

Fix one prompt and call a subset of its correct rollouts rare when its probability $\tau$ is a small slice of the total success probability $\mu_{\mathrm{pos}}$. The dangerous event is not sampling failure in general. It is the conjunction: the group contains both successes and failures, so the update fires, and yet no sample lands in the rare region, so the update can only reinforce the common solutions.

This event has a closed form. We show its probability is exactly $(1-\tau)^N-(\mu_{\mathrm{pos}}-\tau)^N-(1-\mu_{\mathrm{pos}})^N$, and the shape of that expression is the story: small groups are usually inactive, huge groups eventually cover the tail, and intermediate groups combine frequent activity with poor coverage. The curve peaks exactly where practical compute budgets live.

Total correct mass can grow while unsampled correct mass shrinks

Missing the tail once would be harmless if the update left unsampled solutions alone. It does not. In the categorical abstraction, where a softmax policy over a finite action set makes every quantity observable, we derive the one-step change of the unsampled-correct mass $Q_{\mathrm{u,pos}}$ and show it can be negative even while the total correct mass $Q_{\mathrm{pos}}$ increases: a direct drift term pushes unsampled logits down whenever the batch is reward-positive, and softmax normalization drains further mass toward the sampled winners.

We simulated this with 128,000 actions of which 10,000 are correct. Total correct mass rises for every group size, which looks like healthy training. The fraction of initially correct actions that keep their probability tells another story: it collapses at intermediate $N$, stays high for tiny $N$ because updates rarely fire, and stays high for enormous $N$ because the tail gets sampled; at $N=131{,}072$ retention holds near one throughout. Sharpening is not a mystery of deep networks. It falls out of finite sampling plus softmax bookkeeping.

Price the update by how much the group already succeeds

The rare set is invisible during training, but a good proxy for its danger is visible in every group: the empirical success rate $\widehat{\mu}_{\mathrm{pos}}=X/N$, where $X$ counts correct rollouts. High success means the sampled correct set is broad and the reward-positive drift on everything unsampled is strong. So we borrow the shape of Focal loss from object detection and scale each group's advantage by $g(x)=(1-\widehat{\mu}_{\mathrm{pos}})^{\gamma}$, giving $\widehat{A}_i^{\mathrm{F\text{-}GRPO}}=g(x)\,\widehat{A}_i^{\mathrm{GRPO}}$.

The design is deliberately minimal. The weight multiplies the group-relative advantage without touching rewards, clipping bounds, or importance ratios, so it composes with any group-relative objective; $\gamma$ is the only new hyperparameter and there are no extra rollouts, networks, or reference models. We select $\gamma$ per setup from a small sweep, and the winner is always 0.5 or 1.0.

The sweep that shows the dip, and the fix at fixed cost

The sweep matches the theory qualitatively: $N=2$ keeps pass@256 high mostly by rarely updating, intermediate sizes trade the tail for pass@1, and $N=32$ buys the tail back with compute. Our diagnostic agrees. On correct trajectories that were rare under the base model, the negative log-likelihood increase is 0.19 at $N=2$, 0.68 at $N=8$, and 0.52 at $N=32$; F-GRPO at $N=8$ sits at 0.46, deviating less from the base distribution exactly where deviation hurts.

One reweighting, three optimizers, three models

Because the weight only rescales the group advantage, it drops into DAPO and CISPO unchanged. Across nine method-model combinations, Qwen2.5-7B, Qwen3-4B-Base, and Llama-3.2-3B-Instruct at $N=8$, focal weighting improves both math and out-of-domain pass@256 in nine of nine cases, by 3.9 and 4.1 points on average, while math pass@1 is preserved or improved in all nine, by up to 2.8 points on Qwen3-4B-Base.

A single-solution Maze setting from concurrent work sharpens the interpretation: with exactly one correct sequence per prompt, F-GRPO at $\gamma=0.5$ still improves pass@1 to between 74.4 and 93.6 against 65.6 to 75.8 for GRPO across group sizes, so the effect is not only about preserving alternative solutions within a prompt. It also protects coverage across prompts.

Where the analysis stops

The theory is prompt-local and one-step: the tail-miss probability and the mass-redistribution result describe regimes, not quantitative predictions for full LLM training, and the concentration-zone boundaries in our simulation depend on the initial distribution. We probed group sizes 2 through 32, not an exhaustive map, and $\gamma$ still has to be chosen; our runs use one value rather than a schedule.

The claim we stand behind is the mechanism. Finite groups make active-but-blind updates common at exactly the group sizes people can afford, and a success-conditioned weight is a near-free way to keep those updates from erasing what they never sampled. Anything that estimates the unseen correct mass better than $X/N$ does should improve on us.