Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success

Excellent describers, poor actors

A modern vision-language model can caption almost anything and still freeze when asked what to do next. The gap has a data explanation: step-by-step interaction data with images is expensive to collect, so training corpora are static image-text pairs, and models become describers rather than actors. Simulators are the affordable source of interaction, if the learning recipe survives contact with them.

The recipes mostly did not survive. RL4VLM splits each response into thought and action tokens and rescales the thought half by a coefficient $\lambda$, scoring $\lambda\log\pi_\theta(a^{\mathrm{thought}}\mid s)+\log\pi_\theta(a^{\mathrm{action}}\mid s,a^{\mathrm{thought}})$; the right $\lambda$ changes with every model and environment. LOOP hands the same noisy sequence-level return to every token it generated. ArCHer needs a large replay buffer and dense rewards to keep its critic alive. Each knob is one more thing to retune per world.

Two signals, two granularities

VL-DAC, our decoupled actor-critic, keeps each learning signal at the granularity where it is informative. The policy is updated token by token: writing $r_{t,i}=\pi_\theta(a_t^i\mid s_t,a_t^{<i})/\pi_{\theta_{\mathrm{old}}}(a_t^i\mid s_t,a_t^{<i})$ for the importance ratio of the i-th token of step t, we minimize

The advantage $A_t$ is estimated once per environment step with GAE, so every token of a step shares one honest measure of how good that step was. The value head is a small MLP on the backbone features, $V_\phi(s_t)=\mathrm{MLP}_\phi(\mathcal{F}_{\mathrm{VLM}}(s_t))$, trained toward the discounted return; crucially, its gradients stop at the backbone, so a noisy critic can never corrupt the representation the policy depends on.

Three stabilizers from classical RL, still underused at this scale, complete the recipe: a per-token forward KL penalty $\mathbb{D}_{\mathrm{KL}}(\pi_\theta\,\|\,\pi_{\mathrm{old}})$, a value warm-up before any policy update, and that stop-gradient. There is no thought-action coefficient, no replay buffer, and nothing left that must be retuned per environment.

The coefficient we removed was the failure mode

The payoff is robustness. On MiniWorld's OneRoom task, RL4VLM with Qwen2-VL-7B wants $\lambda=0.35$ and reaches 0.98 success; at $\lambda=0.5$ it slips to 0.93. Switch the backbone to Gemma3-4B and the same $\lambda=0.35$ collapses to 0.55 with a standard deviation of 0.38 across seeds, while $\lambda=0.5$ recovers to 0.82. The best coefficient moves with the model, and it fails silently.

VL-DAC, run off the shelf with identical settings, reaches 0.98 on Qwen2-VL and 0.93 on Gemma3 with low variance, and reaches high success on five of six simulator tasks overall. Our ablation shows every stabilizer earning its place: KL, value warm-up, and stop-gradient successively cut variance on top of RL4VLM, and swapping in the token-level objective adds the final margin of stability.

Where sequence-level credit runs out

Long horizons separate the surviving baselines. On four sparse-reward MiniWorld tasks, LOOP plateaus after 15,000 to 30,000 steps: its leave-one-out return hands one noisy number to every token in the trajectory, and once the early successes are absorbed, the gradient is mostly variance. Our step-level critic keeps delivering usable advantages, so success keeps climbing, ending up to 34 points higher without any extra tuning.

Skills learned in toy worlds show up on real benchmarks

The transfer result is why the recipe matters. We train one model in one cheap simulator at a time, 50,000 environment steps, then evaluate zero-shot on real-image benchmarks that the simulators only loosely resemble.

The same ALFWorld run adds 1.5 points on VSI-Bench route planning, one to two points on MuirBench and the spatial slice of VideoMME, and training in OneRoom instead shifts the gains toward relative-direction reasoning. Meanwhile GQA, MMBench, MME, MMStar, and the rest of the general suite stay flat or tick upward, 62.02 to 62.35 on GQA, 56.53 to 57.51 on MMStar. Nothing we measured got worse.

Twenty GPU-hours, honestly bounded

The whole run costs 20 GPU-hours on a single H100 for 50,000 environment steps. Even a 2,000-step WebShop session, all our budget allowed there, moved VisualWebBench: web captioning rose from 27.81 to 29.31, element OCR from 82.36 to 83.49, heading OCR from 75.62 to 76.34. Cheap worlds plus a tune-free algorithm make this kind of casual skill injection practical.

The honest boundary: rewards must not be too sparse, every environment we used is screen-based with discrete actions, our models span 4B to 7B parameters, and training requires strictly parsable, coherent multi-step outputs. Continuous control, multi-agent credit assignment, and long abstract memory, where MiniWorld's WallGap already strains the model, remain open.

We read the result as an argument about scaling environments rather than scaling realism. A learner with nothing to retune turns every new cheap world into a usable skill source, which is exactly what an open, growing gym of small simulators needs to compound.