01 · Signed feedback
One-Batch REINFORCE
Center rewards across the next fresh batch of independent prompts. Successes and failures both contribute, with no value network and no sibling-rollout group baseline.
Overview
One rollout per prompt. One update per batch.
Group-relative RL spends several rollouts on the same prompt and waits for sibling trajectories. FlashREINFORCE uses one rollout per prompt, preserving prompt coverage and naturally fitting asynchronous long-horizon agents — without learning a critic.
02 · Drift control
Sequence Trust Region
Use token importance sampling for behavior correction, then screen complete trajectories with a sequence-level sampled-action KL proxy.
03 · Length control
Sample-Mean Optimization
Average within each trajectory before averaging across the batch so long failures do not receive an automatic length multiplier.
Method
A one-pass asynchronous policy-gradient update.
A rollout worker stores the probability that actually generated every action. The learner recomputes the current policy, corrects the action distribution locally, admits or rejects the full trajectory, and performs exactly one update on the fresh batch.
Collect
One rollout / prompt
Workers submit trajectories independently as soon as they finish.
→
Correct + trust
IS + sequence gate
Correct sampled actions and screen accumulated trajectory drift.
→
Optimize
Sample mean
Take one full-batch step, then discard the batch rather than replay it.
1. One-Batch REINFORCE
For the next \(B\) completed trajectories, center scalar rewards across independent prompts:
\[\bar R=\frac1B\sum_{j=1}^{B}R_j,\qquad A_i=R_i-\bar R.\]
The batch mean is a critic-free control variate. It creates signed feedback while keeping the data path single-rollout.
2. Behavior correction + Sequence Trust Region
For each sampled token, use the learner-to-behavior ratio
\[\rho_{i,t}=\exp\!\left(\log\pi_\theta(a_{i,t}\mid h_{i,t})-\log\mu_i(a_{i,t}\mid h_{i,t})\right).\]
Token IS corrects actions at stored histories. The remaining history-distribution mismatch motivates a trajectory-level trust decision. FlashREINFORCE averages a sampled-action Bernoulli KL proxy over the sequence and admits trajectory \(i\) when \(\bar D_i\le\delta\).
3. Sample-Mean Optimization
Each admitted trajectory receives equal outer weight, independent of response length:
\[\widehat{\mathcal J}_{\mathrm{FR}}(\theta)=\frac1B\sum_{i=1}^{B}\frac{m_iA_i}{T_i}\sum_{t=1}^{T_i}\frac{\pi_\theta(a_{i,t}\mid h_{i,t})}{\mu_i(a_{i,t}\mid h_{i,t})}.\]
This retains signed feedback without letting long negative trajectories dominate simply because they contain more tokens.
What the base method does not need
No learned criticAdvantages come from fresh-batch reward centering.
No ratio clippingBehavior correction is paired with divergence-based trajectory admission.
No reference-model forward passThe base update is deliberately lightweight.
No sibling-rollout waitCompleted trajectories can enter the learner immediately.
Important: store the behavior probability used by the sampler itself. Recomputing an “old” probability later is not guaranteed to reproduce the sampling distribution, especially when the training and inference stacks differ.
Results
Stable under policy lag, competitive with fewer rollouts.
Experiments cover long-chain mathematical reasoning, multi-turn Python tool use, a 30B MoE model, and interactive ALFWorld agents.
6,000
stable updates on DeepSeek-R1-Distill-Qwen-1.5B at policy lag ≈4
38.0
five-benchmark mean on Qwen2.5-Math-1.5B
½
the rollouts of the reported GRPO comparison: 256k vs 512k
98.3%
ALFWorld seen success; unseen success is 96.5%
| Setting | FlashREINFORCE | Comparison / observation |
|---|---|---|
| Long-CoT · DeepSeek-R1-Distill-Qwen-1.5B | AIME24/25 mean 21.7 → 33.7 | Stable through 6,000 updates at lag ≈4. |
| Reasoning · Qwen2.5-Math-1.5B | 38.0 mean, 256k rollouts | Reported GRPO baseline: 36.3 mean, 512k rollouts. |
| Python tool · Qwen2.5-7B-Instruct | 37.0 mean, 3.25 calls / trajectory | Compared GRPO run: 30.3 mean and 0.00 tool calls at step 600. |
| Python tool · Qwen3-30B-A3B | Stable at lag ≈8 | Leads the matched-budget GRPO comparison by 6.8 points. |
| ALFWorld · Qwen2.5-7B-Instruct | 98.3% seen / 96.5% unseen | Evaluated at step 200 after 12.8k training trajectories. |
Practical principles
Asynchronous RL is an algorithm–systems co-design problem.
Prompt coverage over sibling multiplicityWith a fixed rollout budget, one rollout per prompt covers more distinct tasks and removes group synchronization barriers.
Fresh means unused, not necessarily youngA trajectory can be stale in policy age but still fresh for optimization. The trust region handles drift; one-batch training avoids repeated updates on the same collected data.
Token IS is not full trajectory correctionIt corrects the sampled action at a stored history. The history itself still came from the behavior policy, motivating sequence-level drift control.
Support still mattersExact importance correction assumes common support. Aggressive top-k or top-p truncation can violate that assumption.
Paper & code
FlashREINFORCE
Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models.
@article{hu2026flashreinforce,
title = {FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models},
author = {Hu, Jian and Zhang, Yifan and Zhang, Hao and Xu, Binfeng and Zhang, Shaokun and Peng, Hongqing and Yu, Zhiding and Molchanov, Pavlo and Kautz, Jan and Dong, Yi},
year = {2026},
url = {https://github.com/yifanzhang-pro/FlashREINFORCE}
}