VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction

Overview

Train first on what the model is sure about.

Reinforcement learning with verifiable rewards (RLVR) needs an external verifier, which is expensive or simply unavailable for open-ended tasks. Verifier-free rewards such as majority vote or entropy remove that dependency, but their noise turns into destructive gradient variance and training often collapses. VI‑CuRL keeps the reward you already have and changes which prompts drive each update: a curriculum built from the model's own confidence, annealed back to the full data.

34.2%

average Pass@1 for VI‑CuRL with a majority-vote reward, where TTRL collapses to 0.1%.

Qwen2.5-Math-1.5B · 6 math benchmarks

58.6%

without any verifier, higher than standard GRPO with a ground-truth verifier (50.8%).

DeepSeek-R1-Distill-Qwen-1.5B · average Pass@1

20–45%

lower action and problem variance on the kept subset early in training, as Theorem 2 predicts.

measured kept / full variance ratio at β_t ≈ 0.2

Abstract. Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a dominant paradigm for enhancing Large Language Models (LLMs) reasoning, yet its reliance on external verifiers limits its scalability. Recent findings suggest that RLVR primarily functions by eliciting latent capabilities, motivating the development of verifier-free algorithms. However, in such settings, standard methods like Group Relative Policy Optimization face a critical challenge: destructive gradient variance that often leads to training collapse. To address this issue, we introduce Verifier-Independent Curriculum Reinforcement Learning (VI‑CuRL), a framework that leverages the model's intrinsic confidence to construct a curriculum independent from external verifiers. By prioritizing high-confidence samples, VI‑CuRL effectively manages the bias-variance trade-off, specifically targeting the reduction of action and problem variance. We provide a rigorous theoretical analysis, proving that our estimator guarantees asymptotic unbiasedness. Empirically, VI‑CuRL promotes stability and consistently outperforms verifier-dependent/independent baselines across math and general reasoning benchmarks with/without verifiers.

Video

VI‑CuRL in eight minutes

A narrated walkthrough of the problem, the method, the theory and the results.

Method

A verifier-independent curriculum

VI‑CuRL plugs into a standard GRPO loop. The reward source is unchanged (verifier, majority vote or entropy); only the prompts that drive the gradient update change over training.

Conceptual overview: standard RL has high-variance gradients; VI-CuRL filters prompts by confidence, trains on a stable biased subset first, and converges to the unbiased objective as the retention rate anneals to one. — Unlike standard RL, which treats all samples equally, VI‑CuRL selects high-confidence prompts to stabilize early training and gradually returns to the full distribution.

Confidence: a prompt-level, verifier-free score

For each prompt we average the normalized entropy of the behavior policy's next-token distribution over sampled responses:

u(x)=\mathbb{E}_{y\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)}\Big[\frac{1}{T}\sum_{t=1}^{T}\frac{H\big(\pi_{\theta_{\mathrm{old}}}(\cdot\mid s_t)\big)}{\log|\mathcal V|}\Big]

c(x)=1-u(x)\in[0,1]

Confidence never acts as a reward and receives no gradient. It only gates which prompts are used to control variance.

One training step

Sample B prompts and G responses each; compute rewards and group-relative advantages from any reward source.

Score each prompt by confidence \(c(x)\) under the current behavior policy.

Keep the top \(k_t=\lceil\beta_t B\rceil\) prompts and re-weight them by \(1/\hat\beta_t\).

Anneal the retention rate \(\beta_t\) linearly from 0.2 to 1, so training ends on the full, unbiased objective.

Interactive playground

Play with the two ideas behind VI‑CuRL

Both widgets are illustrative toy simulations that run in your browser. They explain the mechanism; they are not paper results.

Theory

A principled bias–variance trade-off

Training on a filtered subset introduces bias. The two results below show that the bias vanishes as the curriculum anneals, while the variance terms that destabilize training shrink.

Theorem 1 (Consistency of the objective)

If the per-prompt surrogate loss is bounded, \(|\ell(\theta;x,y_{1:G})|\le L_{\max}\), then

\big|\mathcal L(\theta)-\mathcal L_t(\theta)\big|\;\le\;2L_{\max}\,(1-\beta_t).

As \(\beta_t\to1\) the curriculum objective converges to the true objective. The bound does not require the confidence score to be correct or calibrated; the retention rate only has to reach one.

Theorem 2 (Variance decomposition)

\operatorname{Var}(\hat g_t)=\underbrace{\frac{\sigma^2_{g,t}}{\beta_t}}_{\text{action}}+\underbrace{\frac{V_{\mathrm{prob},t}}{\beta_t}}_{\text{problem}}+\underbrace{\frac{1-\beta_t}{\beta_t}\big\|\nabla_\theta\mathcal L_t(\theta)\big\|^2}_{\text{masking}}

Action variance is the spread of gradients across responses to the same prompt; problem variance is the spread of mean gradients across prompts, both measured within the selected set. Masking variance is the cost of partial retention and vanishes as \(\beta_t\to1\).

Does VI‑CuRL really reduce variance?

We measured both variance terms during training on the kept subset and on the full batch (oracle reward). Early on (\(\beta_t\approx0.2\)) the kept subset has 20–45% lower variance; as \(\beta_t\to1\) the ratio returns to one, as the theory predicts.

Variance ratio kept/full over training steps for Qwen2.5-Math-1.5B: both action and problem variance ratios are 0.55–0.8 while beta is 0.2 and rise to 1 as beta reaches 1. — (a) Qwen2.5-Math-1.5B

Variance ratio kept/full over training steps for DeepSeek-R1-Distill-Qwen-1.5B: ratios around 0.55–0.62 early, rising to 1 as beta anneals. — (b) DeepSeek-R1-Distill-Qwen-1.5B

Results

Explore every number in the paper

Four backbones, six math benchmarks, three reward settings, Pass@1 and Pass@8 (16 samples, mean ± std over 5 seeds). Pick a view below; every chart has a table view.

Ablations

Why confidence, and why annealing?

Citation

BibTeX

@inproceedings{cai2026vicurl,
  title     = {{VI‑CuRL}: Stabilizing Verifier-Independent Reinforcement Learning
               Reasoning via Confidence-Guided Variance Reduction},
  author    = {Cai, Xin-Qiang and Sugiyama, Masashi},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}