RL for LLMs, Part 2: Important Papers in RLVR

July 21, 2026

In this part 2 of the blogpost we would look at a lot of papers that have been proposed (mostly after GRPO) claiming to be a better algorithm for doing RL on LLMs, especially for RLVR and understand the differences between them. Most of the approaches modify one of these 4 elements:

advantage estimation
token vs. sequence-level importance weighting
clipping/trust-region behavior
sampling and credit assignment

Important papers and methods

Year	Paper name	Contribution	Weakness
2017	Proximal Policy Optimization Algorithms (PPO)	Directly clips the IS ratios which is easier than TRPO's approach for constraining the KL divergence	Need to train a separate value prediction model, quite expensive
2020	Approximating KL Divergence	This short note prescribes 3 ways of estimating KL-divergence. Let $p = \pi_{\theta}, q = \pi_{ref}$ and suppose $a \sim p$ . We want to estimate $D_{KL}(p \|\| q) = E_{a \ \sim \ p}[\log \frac{p(a)}{q(a)}]$ . $k_1$ : ordinary sampled log ratio is this term inside $\log \frac{p(a)}{q(a)}$ . It is unbiased estimate of the KL-divergence, but it can be negative (which should not be the case as KL-divergence is always positive) and its variance can be high. $k_2$ : squared log-ratio approximation is defined as $k_2(a) = 0.5 (\log(\frac{q(a)}{p(a)})^2$ . This is non-negative and has lower variance, but this is only an approximation of the KL-divergence and is only accurate when $p$ and $q$ are close. $k_3$ positive, unbiased estimator is defined as $k_3(a) = \frac{q(a)}{p(a)} - \log\frac{q(a)}{p(a)} - 1$ . This is always non-negative when $\frac{q(a)}{p(a)} > 0$ and is an unbiased estimator of KL-divergence. Also there are two related terms, forward and reverse KL. When we want to minimizwe the KL-divergence between p and q, there are two directions, $D_{KL}(p \|\| q) = \sum_{a} p(a) \log \frac{p(a)}{q(a)}$ and $D_{KL}(q \|\| p) = \sum_{a} q(a) \log \frac{q(a)}{p(a)}$ . The first term is called reverse-KL and the second one is called forward KL. Reverse KL asks where teh current policy goes: $D_{KL}(p \|\| q) = E_{a \sim p}[]$ , so actions receive weight according to the current policy $p$ , therefore it strongly penalizes the policy for placing probability on actions that the reference policy $q$ considers unlikely. But if current policy $p$ ignores some reference-likely action, that actions receives almost no weight because $p(q) \approx 0$ , this means reverse KL can tolerate dropping some reference modes. In forward KL: $D_{KL}(q \|\| p) = E_{a \sim q}[]$ , actions receive weight according to reference, therefore if the reference considers an action likely but the current policy $p$ ignores it, the penalty is large. Forward KL discourages dropping reference modes. Therefore, forward-KL is mode-covering and reverse KL is mode-seeking. Using $k_3$ produces gradient for minimizing the forward-KL, using $k_1$ and $k_2$ produces gradient for minimizing the reverse-KL.
2022	Training Language Models to Follow Instructions with Human Feedback (InstructGPT)	RLHF with PPO	Same weakness as PPO
2023	ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models	Goal is to remove the expensive critic from PPO. Switches to REINFORCE algorithm but to reduce its variance it does two inferences from the model (normal and one greedy) and uses the reward of the greedy inference (G) as the baseline. Policy-gradient estimate becomes $(R - G)\sum_{t=1}^T \nabla_{\theta}\log \pi_{\theta}(y_t \mid x,y_{<t})$ . For a fixed prompt and current policy, greedy does not depend on the stochastic response y, therefore the greedy does not bias the baseline. It gets rid of all machinery of PPO (critic model, GAE, token-level advantages, clipping)	There is no token level credit assignment, every token receives the same scalar $R-G$ and the greedy inference is not an estimate of the expected return of the prompt.
2024	DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO)	This is the paper that introduced GRPO, a way to get a baseline using mean of several rollouts for the same prompt without the need of training a value (critic) model. For a question q, sample G rollouts and the advantage is calculated as $\hat{A}_t = \frac{r - \bar{r}}{\sigma(r) + \epsilon}$ . Each token receives the same advantage, so $\hat{A}_{i,t} = \hat{A}_{i}$ . The full GRPO objective is similar to PPO, except with critic model and GAE replaced: $J_{GRPO} = E[\frac{1}{G}\sum_{i=1}^{G} \frac{1}{T_i}\sum_{t=1}^{T_i} \min(\rho_i\hat{A}_i, clip(\rho_i, 1-\epsilon, 1+\epsilon)\hat{A}_i - \beta \hat{D}_{i,t}^{KL}]$ . Unlike PPO, GRPO doesn't add the KL into reward, but adds it into the loss directly. The way GRPO estimates the KL-divergence per-token is using a $k_3$ estimator. For sampled token it defines $\zeta_{i,t} = \frac{\pi_{ref}(y_{i,t} \mid x, y_{i, <t})}{\pi_{\theta}(y_{i,t} \mid x, y_{i, <t})}$ , $\hat{D}_{i,t}^{KL} = \zeta_{i,t} - \log\zeta_{i,t} - 1$ or equivalently $\frac{\pi_{ref}(y_{i,t} \mid x, y_{i, <t})}{\pi_{\theta}(y_{i,t} \mid x, y_{i, <t})} - \log\frac{\pi_{ref}(y_{i,t} \mid x, y_{i, <t})}{\pi_{\theta}(y_{i,t} \mid x, y_{i, <t})} - 1$ . This is always non-negative because $x - \log x - 1 \ge 0 \text{ for } x > 0$ . Therefore when $\pi_{ref} == \pi_{\theta}$ , KL penalty is 0, and as they diverge the KL penalty increases. In expectation $k_3$ formulation is an unbiased estimator of reverse KL, i.e., $D_{KL}(\pi_{\theta}( . \mid s) \|\| \pi_{ref}( . \mid s))$ The paper also describes having process rewards which can give rewards for each step and thereby allow computation of reward-to-go and assign credit to each token.	Main weaknesses are (a) large rollout cost, (b) no signal if all rollouts receive same reward, (c) due to division by standard deviation, rare successes on hard quesiotns can receive very high advantages which can alter training landscape.
2024	Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (RLOO)	Uses sequence level reward (all tokens in a rollout get the same advantage) and average of multiple rollouts as the baseline similar and drops the critic model (all similar to GRPO). Unlike GRPO, when calculating the mean of reward they exclude the particular run in question (for unbiased estimate), also do not divide by the standard deviation, they even remove the old-policy importance ratio and ppo clipping update, and sequence averaging (so longer sequences get more reward). Therefore advantage, $A_i^{RLOO} = \frac{k}{k-1}(R_i - \bar{R})$ and the policy gradient esimate becomes $\frac{1}{k}\sum_{i=1}^k A_i^{RLOO} \sum_{t=1}^{t_i} \nabla_{\theta}\log \pi_{\theta}(y_{i,t} \mid x_i, y_{i,<t})$ . The reason they do away with most of the PPO's machinery, is the paper's bet that the variance and other instability claims are overblown when training from a well SFTed LLM.	The claims of the paper are verified for the RLHF not RLVR setting, that too with small models and limited tasks.
2024	VinePPO: Unlocking RL Potential for LLM Reasoning Through Refined Credit Assignment	VinePPO proposes another way to compute the token-level advantages which PPO uses a critic model for and GRPO does away with. The idea is to use monte-carlo estimate of the advantage at certain tokens. They use the prompt before and after that token as prefix and sample multiple completions from the LLM, the percentage of correct samples gives the value of that state, $V^{\pi}(s_t)$ and $V^{\pi}(s_{t+1})$ and the difference between them gives them the advantage, because $A_{t} = V^{\pi}(s_{t+1}) - V^{\pi}(s_t)$ (assuming no intermediate reward and $\gamma = 1$ ). After the advantages is computed at some tokens, we do the normal PPO update $J_{VinePPO} = \sum_{i,t} \min(\rho_{i,t}\hat{A}_{i,t}, clip(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_{i,t} )$ . More continuations are more expensive, but decrease variance. Since doing this approach at every token basically requires $\mathcal{O}(N^2)$ completions, they only do this at a few tokens (like newlines or reflective tokens). So the approach gets rid of the expensive critic model while sitll having differentiated credits for tokens which GRPO does away with. They also found that PPO's critic gets worse with longer reasoning chains (as it is hard for the critic to generalize on unseen patterns) whereas monte-carlo estimates are more reliable.	1. The paper's demonstration of this technique was limited to math domain. 2. The auxiliary sampled trajectories are not used to update the model (as advantages for them are not calculated) and they get discarded. 3. Much more expensive than even GRPO in terms of inference cost.
2025	REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models	This is also a variation to PPO, making it critic-free and its also a variation of GRPO, replacing its prompt-based normalization with a global one. The advantages are calculated for each token based on the terminal reward and the KL-divergence terms are subtracted from it, i.e, $A_{x, y_t}^{raw} = r - \beta \sum_{j=t}^T \log \frac{\pi(y_j \mid x, y_{<j})}{\pi_{ref}(\pi(y_j \mid x, y_{<j}))}$ , and define $\mu_{batch} = mean_{\ x, \ y_t \ \in \ batch}A_{x,y_t}^{raw}$ and $\sigma_{batch} = std_{\ x, \ y_t \ \in \ batch}A_{x,y_t}^{raw}$ . Finally $A_{x, y_t}^{norm} = \frac{A_{x,y_t}^{raw} - \mu_{batch} }{\sigma_{batch}}$ and this is inserted in the PPO style objective, $\min(\rho_{i,t}A_{x,y_t}^{norm}, clip(\rho_{i,t}, 1 - \epsilon, 1 + \epsilon)A_{x, y_t}^{norm})$ (over all tokens). Therefore the advantage is critic-free like GRPO, but unlike GRPO the advantage normalization comes from the completions in the entire batch not just for that prompt. Therefore, this algorithm can even work when you only sample one rollout per prompt instead of a group requirement like GRPO (although a variant exists in the paper in which they subtract the group mean from the advantage in case there are multiple rollouts per prompt)	The main limitation is the advantage normalization assumes reward scores are comparable across prompts, which is not true is prompts are of a varied difficulty level. Empirical valiation is limited in scope and scale.
2025	Kimi k1.5: Scaling Reinforcement Learning with LLMs	This is broad model training paper. They first SFT a base model on a dataset that has long-CoT in order to enable it for RLing. They have a completely different RL algorithm compared to all other papers. They use online mirror-descent objective. So at any iteration, their objective is $max_{\pi}[E_{a \ \sim \ \pi}[r(y \mid x)] - \tau D_{KL}(\pi \|\| \pi_{\theta_i})]$ . The reverse-KL term has the same role as clipping in PPO, prevent the update policy from going too far from the sampling policy. This problem has a closed-form optimum $\pi^{}(y \mid x) = \frac{\pi_{\theta_i}(y \mid x)\exp(r(y)/\tau)}{Z(x)}$ , where $Z(x) = \sum_{y'} \pi_{\theta_i}(y' \mid x)\exp(r(y')/\tau)$ . Taking logs, gives $r(y) - \tau\log Z = \tau\log\frac{\pi^(y)}{\pi_{\theta_i}(y)}$ . Divide both sides by $\tau$ gives, $\log\frac{\pi^(y)}{\pi_{\theta_i}(y)} = \frac{r(y)}{\tau} - \log Z$ . The relative log-probability change equals reward divided by $\tau$ (and a factor $-\log Z$ ). For two responses $a$ and $b$ , subtract their equations, $\log\frac{\pi^(a)}{\pi_{\theta_i}(a)} - \log\frac{\pi^*(b)}{\pi_{\theta_i}(b)} = \frac{r(a) - r(b)}{\tau}$ , i.e., difference in log-probability changes = difference in rewards/ $\tau$ . Computing Z(x) over all possible completions is not possible, so for each prompt they sample several completions $(y_1, y_2, y_3, y_k) \sim \pi_{\theta_i}$ . The theoretical estimate would be $\tau\log Z \approx \tau\log[\frac{1}{k}\sum_{j=1}^{k}\exp(r_j/\tau)]$ . In practice they just take the sample mean $\bar{r} = \frac{1}{k}\sum_{j=1}^{k}r_j$ and they train $\pi$ so that $\tau \log \frac{\pi_{\theta}(y \mid x)}{\pi_{\theta_i}(y \mid x)} = r_j - \bar{r}$ . Which means they regress the sequence log-ratio toward the group-relative reward. The final gradient is $\frac{1}{k}\sum_{j=1}^k[(r_j - \bar{r})\nabla_{\theta}\log\pi_{\theta}(y \mid x) - \frac{\tau}{2}\nabla_{\theta}\big(\log\frac{\pi_{\theta}(y_j \mid x)}{\pi_{\theta_i}(y_j \mid x)} \big)^2]$ The first term is REINFORCE with group-mean baseline and the second term is the proximal penalty (approximation to reverse KL). Note that the log-probability is sequence-level, not token-level, i.e. $\nabla \log \pi_{\theta}(y \mid x) = \sum_{t} \nabla\log\pi_{\theta}(y_t \mid x, y_{<t})$ . The regularizer (penalty to sampling policy) is also sequence-level: $\sum_{t}(\log\frac{\pi_{\theta}(a_t \mid s_t)}{\pi_{\theta_i}(a_t \mid s_t)})$ . It constrains the sum of token log-ratio changes, not every token's change independently (cross-coupling across tokens). There is no critic, no GAE, no process reward, no token-specific advantage, no PPO importance ratio, and no PPO clipping. They reject a value function for the same reason as GRPO: expensiveness and inaccuracy at partial completion levels. They also have length reward and penalty, but that depends on the prompt (its not absolute). So for a group of samples for a prompt, short responses receive a positive reward and long receive negative reward. They also have curriculum learning by starting with easier and moving towards harder prompts and prioritized sampling uses harder prompts more often. Similar to Pipeline-RL, they use partial rollouts and complete them later with updated trainers.	Note that both TIC-GRPO and Kimi use sequence level token-ratios, however Kimi has them in the log-space, so they don't blow-up whereas they can for TIC-GRPO
1983	Mirror Descent	Kimi uses mirror-descent as unlike gradient-descent whose default geometry is euclidean, mirror descent can optimize in user-selected geometry which here means not going too far in the KL-divergence term of the probability distribution over tokens for the two policies. Suppose we want to minimize $f(x) \approx f(x_t) + \langle \nabla f(x_t),x-x_t\rangle$ , gradient descent chooses the next point by solving $x_{t+1} = arg min_{x}[\langle\nabla f(x_t), x\rangle + \frac{1}{2\eta}\|\|x - x_t\|\|_2^2]$ , which gives the familiar update of $x_{t+1} = x_t - \eta\nabla f(x_t)$ . This moves in the direction of linearized objective, but not too far. Mirror descent changes the notion of distance, it allows the user to choose a convex function $\psi(x)$ , called the mirror map or distance-generating function, it induces bregman divergence $D_\psi(x,x_t)= \psi(x) - \psi(x_t) - \left\langle\nabla\psi(x_t),x-x_t\right\rangle$ , mirror descent performs $x_{t+1} = argmin_{x \in \mathcal{X}}[\eta\langle\nabla f(x_t), x\rangle + D_{\psi}(x, x_t)]$ , the first term is the improve the objective and the second term is do not move too far. Gradient descent is a special case of $\psi(x) = \frac{1}{2}\|\| x \|\|_2^2$ , so gradient descent is euclidean mirror descent. The canonical example of application of mirror descent is in probability distributions where using euclidean descent can produce negative entries or entries that don't sum to 1, using mirror descent is much easier.
2025	DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning	This doesn't propose a new RL algorithm, it just demonstrated the use of GRPO on base-models (no SFT) to enable them for reasoning with RL (model called R1-Zero). The reward was correctness and format (thinking inside tokens). Then did generations from R1-Zero, bunch of filtering for correctness, language consistency, human re-writing was done to create a massive SFT dataset and then the baseline model was SFTed on this (R1-Dev1) and then again RL using the same correctness reward + language-consistency reward (R1 model). Then they sampled bunch of correct and readable generations from R1 and use this new SFT dataset to finetune the original base model producing R1-Dev3. RL is then done on this model with a combination of rule-based reasoning, language-consistency, a helpfulness reward model, safety reward model, and format reward. The RL stage lasts 1700 steps, but reward models are only used for 400 beyond that, they start reward hacking. The traces from this paper was the first to show "aha" moment and more reflections like "wait" or "wrong". They also observed that RL caused a gradual increase in reasoning length.	No weakness of the paper, here are the negative findings from it: process reward models were hard to scale because of (a) hard to train them as collecting human annotations is expensive, and (b) reward hacking.
2025	Process Reinforcement through Implicit Rewards (PRIME)	This approach is similar to a mix of GRPO and PPO. Like GRPO, for each prompt, it samples multiple responses and calculates the advantage of a response's token as the outcome reward minus the mean of the sampled responses. Like PPO, it estimates the advantage of each token by training an implicit reward model that is trainable. Overall there is a policy model from which the responses are generated and is trainable, a implicit reward model that is also trained and a reference model that is usually not trained. The reward model is trained in the following way: for a sampled response, we compute the ratio of log probs from the reward model and the reference model and sum it over all the tokens in the response. If the overall response received a positive reward from the verifier we train the reward model to increase the log ratio of the probablity of the token (w.r.t the reference model), if the overall response received a zero reward -- the reward model is trained to decrease the log ratio of the probability of the token (w.r.t the reference model). $S_{\phi}(x, y) = \beta \log \frac{\pi_{\phi}(y \mid x)}{\pi_{ref}(y \mid x)} = \sum_{t=1}^T \beta \log \frac{\pi_{\phi}(y_t \mid x,y_{<t})}{\pi_{ref}(y_t \mid x,y_{<t})}$ , and the loss for the implicit reward model is the binary cross-entorpy loss: $L_{PRM} = -(z\log(\sigma(S_{\phi})) + (1-z)log(1-\sigma(S_{\phi})))$ . If z = 1, training pushes the total log-ratio $S_{\phi}$ higher and vice-versa when z = 0. Once the reward model is trained, we can estimate the per token advantage as the sum of the outcome reward + the log ratio of the reward and reward-to-go which is computed using the sum of the log ratios of the tokens from the reward and reference model. $A_t^i = G_t^i + A_{out}^i$ , where $A_{out}^i = z - \bar{z}$ and $G_t^i = \sum_{j=i}^T r_{\phi}(j)$ . Both the outcome information and the implicit reward model information only comes from the verifier, there is no additional supervision signal, the main gains comes from sharing information across multiple trajectories across multiple prompts (which GRPO does not, it only shares information across trajectories of one prompt, but PPO does). In some sense this is very similar to PPO, both share information across trajectories and prompts, the only difference is the objective, the critic's goal is to predict success given prefix, while the implicit reward model is to ask whether is prefix is more likely under its success identifier trained reward model.	Main weakness is the cost, the PRM adds another language model almost the same cost as the policy model (similar to PPO's critic model).
2025	DAPO: An Open-Source LLM Reinforcement Learning System at Scale	This paper tried to reproduce the results for GRPO on Qwen model and could not do it and identified 5 issues that they proposed solutions to. 1. Clip-higher: they found that GRPO training rapidly reduces policy's entropy which can lead to getting stuck in local minima. For e.g., consider a token whose $\pi_{old}(a \mid s) = 0.01$ , with $\rho_i = 0.2$ , the $\pi_{\theta}(a \mid s) \approx 0.012$ , so only a very slight increase in rare token's probability. Rare tokens may represent backtracking or reflecting paths, so we don't want to get rid of them. Therefore, they use asymmetric clipping, i.e., $\rho_{low} = 0.2$ and $\rho_{high} = 0.28$ , higher clipping on the upper side to encourage rare tokens. However the lower bound is not changed in order to not rapidly decrease the probability of tokens with negative-advantage (which will also reduce entropy). (b) Too easy or too hard prompts generate no gradient signal if all rollouts get the same reward (all -1 or all 1), DAPO discards such prompts from the batch (not permanently discard). This helps use an effective constant batch size, reduce gradient variance, and uses examples that contribute gradient signals therefore accelerating improvement. (c) in the GRPO objective we first average over tokens in a sequence and then over all sequences, however DAPO argues this leads to weaker reinforcement for longer sequences (good trjaectory increased weakly and bad trajectory decreased weakly), and changes the order of averaging, now the average is over all tokens equally (it does not make the advantage per-token (the naming is confusing as its called Token-level policy-gradient) and does not introduce a critic model). (d) it removes the rollouts that have not completed so their rewards cannot be ascertained (earlier approaches counted them as failed, but that's wrong, this is unknown). It also plays with length reward (longer length gets negative if it crosses the prefered length). (e) it gets rids of the KL penalty to the reference model as the authors argue that RL is supposed to move the model substantially away from the base model. $J_{DAPO} = \frac{1}{\sum_{i=1}^G T_i} \sum_{i=1}^{G} \sum_{t=1}^{T_i} \min(\rho_{i}\hat{A}_i, clip(\rho_i(\theta), 1-\epsilon_{low}, 1+\epsilon_{high}))$ subject to 0 < #(correct solutions in group G) < G	Apart from asking to do more experiments, there are no as such weakness in ideas except that the clipping $\rho$ can be made adaptive of the rare token's probability $\pi_{old}(a \mid s)$ . If this is very low to begin with the clipping should be high and it grows the clipping should become tighter. Similarly on the negative side. I thought of it independently, but I later found it to have been proposed in the DCPO paper.
2025	Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO)	This paper also has critiques of the GRPO paper. (a) length normalization weights the tokens in shorter sequences more which more strongly reinforces tokens in shorter sequences compared to longer, so similar to DAPO, they get rid of sequence length normalization. (b)advantages in GRPO divide by standard deviation, but this gives a very high advantages to a rare correct or rare wrong response, both of which are harmful. Therefore this paper removes dividing by standard deviation. (c) shows that including the particular run in the baseline averaging (which RLOO proposed against) just changes the baseline by a constant that can be absorbed in learning rate, so no need to exclude that. (d) it gets rid of the KL penalty to the base model similar to DAPO. Dr. GRPO retains the group sampling, group-mean baseline, same advantage for all tokens in a sequence, importance ratios, ppo-style clipping, and absence of critic model similar to GRPO. Experiments show that both GRPO and Dr. GRPO improve reward and increase response length, however after reward plateus, GRPO's length keep on increasing whereas Dr. GRPO's stabilizes and the most of gains come from the (a) change. The paper also had some discussion on removing the template from the base model as it misleads it.	Experiments focus on Qwen and math problems.
2025	Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model	This is the open source replication of Deepseek-R1-zero paper that claimed RL can improve even a base model, but other folks were not able to replicate it. This paper uses PPO algorithm, and trains a critic model (which is a separate model initialized with the same model as policy except with a random head for scalar output). It is trained to predict the value of a prefix which is trained using the terminal loss (all prefixes of correct completions receive reward 1 and all prefixes of incorrect completions receive reward 0). The paper also uses $\gamma = 1$ and $\lambda = 1$ . Therefore $\hat{A}_{i,t}^{GAE} = r(i) - V(s_t)$ and $V_t^{target} = \hat{A}_{i,t} + V(s_t) = r(i)$ (both for the critic and actor). This mean that the value target is not bootstrapped from an inaccurate critic (similar to VAPO), but unlike VAPO the actor also does not use future critic predictions (only at $V(s_t)$ ). The paper normal PPO for updating the model, but weirdly they only use one rollout for one update, so the importance sampling ratio is always 1, which means PPO clipping is never used. Unlike PPO, the paper removes KL loss to the reference model.	(a) expensive critic training, (b) high-variance training of the actor as critic's estimates are not uses for advantage estimation, (c) strictly on-policy PPO means using a rollout just once, unnecessarily expensive.
2025	VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks	This paper aims to improve the issues with PPO rather than discarding the critic model completely. It implements several changes for this. (a) First thing, it wants to make the critic more accurate, so instead of directly training the actor (policy) and critic together, they freeze the actor, generate a bunch of rollouts and train the critic on these rollouts. Another change they do in order to train the critic more accurately is to use a $\lambda = 1$ , which means that the critic is not bootstrapped from its own outputs. It completely depends on the Monte Carlo returns, which means that the critic is more accurate. This is called critic pre-training. (b) After this both actor and critic are co-trained. Critic is always trained with $\lambda = 1$ , while the actor uses a different $\lambda$ which utilizes the now better trained critic and reduces variance. (c) Instead of using a fixed $\lambda$ for the actor, they determine the it based on the length of the completion, about 5% of the length. This makes each response have its own adaptive $\lambda$ . If instead we used a fixed $\lambda = 0.95$ , then this has an effective token horizon of 20 tokens in the GAE computation. For a completion of 100 tokens, that is 20% of its length, whereas for a completion of 1000 tokens, that's very small. Length-dependent $\lambda$ solves this issue. (d) similar to DAPO, VAPO adapts clip-higher for tokens and token-level aggregation of loss. (e) It adds a NLL loss for the completions that are correct -- which means that the tokens in correct completions are reinforced both through PPO loss and normal NLL loss. (f) Unlike vanilla PPO that samples one completion per prompt, VAPO like GRPO samples a group of completions per prompt which means the update sees diversity of completions leading to lower noise (although it doesn't use a group-level advantage).	The first issue is the cost of the critic model. VAPO could have also mixed the group-level GRPO advantage with critic's advantage since it already has the group rollouts. It could also train the critic to have a confidence predictor, so $\lambda_{actor}$ can also depend on critic's confidence instead of just depending on the completion length.
2025	Absolute Zero: Reinforced Self-play Reasoning with Zero Data	This paper proposes to do RLVR using the self-play setting where the model plays the role of problem proposer and solver, both are trained using different losses (often complementary) and with training both become stronger over time. It does uses a python verifier for the training signal. However the algorithm used here is just critic-free PPO (almost reinforce), so not a novel RLVR algorithm -- so skipping much of the details here.
2025	DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization	This paper says that GRPO's objective has a implicit weight based on prompt difficulty. Consider a prompt whose probability of getting solved is $p(q)$ , GRPO's normalized advantage is $\frac{r - p(q)}{std()} = \frac{r-p(q)}{\sqrt{p(q)(1-p(q))}}$ . Therefore for binary reward, when r = 1, the advantage is $A_{+} = \sqrt{\frac{1-p(q)}{p(q)}}$ and when r= 0, advantage is $A_{-} = -\sqrt{\frac{p(q)}{1-p(q)}}$ . Since the correct completions occur with probability $p(q)$ , its expected coefficient is $p(q)A_+ = \sqrt{p(q)(1-p(q))}$ , and for the incorrect completions that is $(1-p(q))A_- = \sqrt{p(q)(1-p(q))}$ . Therefore the expected GRPO objective for a prompt can be written as $J = \sqrt{p(q)(1-p(q))}[E_{o^+}(s(o^+, q)) - E_{o^-}(s(o^-, q))]$ , where $s$ is some score. The term inside the brackets try to increase the score correct completions and decrease the score of incorrect completions. But there is an outside term $\sqrt{p(q)(1-p(q))}$ which has dependence on prompt difficulty. For $p(q) = 0.5$ , that term is 0.5, but for $p(q) = 0.01$ , that term is 0.1, so a hard prompt gets one-fifth the weight of a prompt that gets solved 50% of the time. (Dr. GRPO which removes division by the standard deviation, actually makes their weight even more skewed.) DisCO proposes to remove that weight. The scoring function that is used inside for the discriminative training is either the log-likekihood score (which increases the log-probs of each token in the correct completion and decreases the log-probs of each token in the incorrect completion) and likelihood-ratio score. Neither scores are clipped, but there is a KL-divergence to the old policy (not reference, the sampling policy) that is used to prevent too much drifting and this only activates when KL-divergence becomes larger than a threshold. Finally inorder to focus on harder negatives, they replace the vanilla average of the negatives with the logsumexp of the negatives	Since the prompt difficulty weight is removed, the main advantage can turn into a disadvantage even with a slight noise as the model will get updated even when the completion for a hard prompt was correct due to verifier noise or just lucky guess. The paper lacks empirical validation on MoE models, long completions, beyond maths.
2025	DCPO: Dynamic Clipping Policy Optimization	This paper introduces 3 changes from the original GRPO: (a) GRPO has fixed clipping for each token, i.e, $1-\epsilon \le \rho_{i,t} \le 1+\epsilon$ , where $\rho_{i,t} = \frac{p_{i,t}}{q_{i,t}}$ Consider a common token with $q_{i,t}=0.5$ , with $\epsilon=0.2$ , the upper bound allows $p_{i,t}$ to become 0.5 + 0.1 = 0.6, an absolute increase of 0.1. However a rare token with $q_{i,t}=0.001$ can only become $p_{i,t}=0.0012$ , a mere increase of $0.0002$ . Therefore a fixed ratio clipping, kind of, freezes rare tokens. DCPO proposes to do this adaptively, so $\|(\rho_t-1)p\| = \|(\rho_t-1)\rho q\| \le \epsilon$ . This gives upper and lower bounds that are based on the original probability of the token and not fixed. Overall, this leads to successful rare tokens to grow much faster and unsuccessful rare tokens to shrink much faster (compare to GRPO) (b) in a group, if all rollouts are correct or all incorrect, then we don't get any gradients. DAPO solves this by sampling more rollouts for this prompt until we get mixed batch. DCPO instead keeps a historical reward statistics for that prompt and uses that. It creates two possibilities $S_{new} = \frac{i-1}{i}A_{new} + \frac{1}{i}A_{total}$ and $S_{total} = \frac{i-1}{i}A_{total} + \frac{1}{i}A_{new}$ and the advantage of a token is the smaller of the two. So even when $A_{new} = 0$ , the advantage used is $\frac{1}{i}A_{total}$ that can produce some gradient signal. This means that if relative of the running advantage if a batch produces some failures, they can still be reinforced. (c) the last change is DCPO re-introduces the sequence level averaging of loss, critizing DAPO which introduced token-level averaging (which favors longer responses). It also removes the division by group size, but that is just the same as increasing learning rate by G. The paper had another interesting observation that after around 60 training steps on Qwen2.5-Math-7B, 95% of sampled tokens have $q_t > 90\%$ and after 100 steps this rises to around 97% which means most tokens are very high probability and RL reinforces that. But it also shows that most tokens are not decision makers, once a direction is selected, most subsequent tokens are almost forced until we reach a forking point, therefore the reasoning decisions are sparse.	The paper chooses a specific form of dynamic clipping and doesn't discuss the possible variants. Removal of division by G is the same as scaling LR.
2025	BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping	This paper has a similar motivation to DCPO, which is adaptive clipping. Their empirical observation is different though, they say they find that positive tokens with high importance ratios are the ones with low-probability and upper-bound clipping doesn't let them increase much. And they also find that its mostly the negative tokens that are contributing to the gradient updates as they still remain active (everything empirical here). Therefore, the authors want the clipping to be adaptive in order to have the a certain percentage of gradients to be from the positive tokens. The algorithm works like this, use the default $(\epsilon_{low}, \epsilon_{high})$ and calculate the fraction of loss contribution from positive tokens. If it is less than a threshold, iteratively increase $\epsilon_{high}$ (in some fixed step sizes) until either the positive contribution fraction reaches the desired threshold or you hit upper limit of $\epsilon_{high}$ , in which case you now start increasing $\epsilon_{low}$ so that the contribution of the negative tokens to loss starts to decrease, overall increasing the fraction of positive token loss. So unlike DCPO where the bound for each token can be different, here they are the same, but adjustible.	The main weakness of the paper is that they did not check how generalizable is the phenomenon they observed, they only test a few models and only AIME dataset. Second they do not state why this loss balancing is an important thing to achieve -- they have some empirical observations like initially most trajectories are unsuccessful, so most loss is from negative tokens and high importance ratio positive tokens are removed by upper clipping -- which they want to tackle. Won't recommend this approach. The good idea is the adpative bounds itself, but DCPO already has that.
2025	Group Sequence Policy Optimization (GSPO)	This paper says that the advantage in GRPO is the same for all tokens, then why is the importance ratio that we optimize over different? They say that introduces noise, especially for MoE models that might have even different experts being used during the multiple reuses of the same rollout, which means that the importance ratio of each token is less meaningful. They want to use the likelihood ratio of the entire sequence, but as discussed earlier this can become extremely small, so they propose to use the geometric mean of the token-level importance ratios, i.e, $s_{i}(\theta) = (\frac{\pi_{\theta}(y_i \mid x)}{\pi_{old}(y_i \mid x)} )^{ \frac{1}{T_i} } = \exp[\frac{1}{T_i}\sum_{t=1}^{T_i}\log \rho_{i,t}(\theta)]$ . Overall objective becomes $J_{GSPO}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \min(s_i(\theta)\hat{A}_{i}, clip(s_i(\theta), 1-\epsilon_{low}, 1+\epsilon_{high}))$ Note there is no token level summation as $s_i$ already accounts for the entire response. Teh gradient is $\nabla J_{GSPO}(\theta) = \frac{1}{G} \sum_i \hat{A}_i \nabla_{\theta}\log s_i = \frac{1}{G} \sum_i \frac{s_i\hat{A}_i}{T_i} \sum_t \nabla_{\theta} \log \pi_{\theta}(y_{i,t} \mid x, y_{i, <t})$ . So the difference with GRPO is that $J_{GSPO}(\theta) = \frac{s_i\hat{A}_i}{T_i} \sum_i \nabla \log \pi_{\theta}(y_{i,t} \mid x, y_{t,<t})$ and $J_{GRPO}(\theta) = \frac{\hat{A}_i}{T_i}\sum_i \rho_{i,t} \nabla \log \pi_{\theta}(y_{i,t} \mid x, y_{t,<t})$ . So GRPO assigns different state-data weights to individual tokens while GSPI gives entire response one common state-data weight. In experiments with 48-layer Qwen3-30B-A3B model, authors found that after every RL update, about 10% of the experts activated for the same rollout differ between the old and new policies -- which makes the individual token probabilities fluctuate signicantly. They also propose a variant where each token gets different advantage, e.g. using a process reward model but do no experiments with it.	Main limitation is the experiments are only done with one model, Qwen3 which is MoE, no experiment done with dense model, no experiment done with process reward models. Geometric mean is less sensitive to large positive outliers but more senstive to values close to 0, so a single very small ratio can suppress the update for the entire rollout.
2025	Geometric-Mean Policy Optimization (GMPO)	This method is very similar to GSPO, i.e. it also proposes to use the geometric mean of the token-level importance ratios as the entire sequence's aggegated "importance ratio". They say that GRPO takes the arithmetic mean of the importance ratios, which can be swayed by a single high number whereas geometric mean is more robust to change by a single large number. There is one difference from GSPO is when the aggregation happens: in GSPO you first compute the geometric mean of the importance ratios of the entire sequence and then apply the PPO-style clipping, while in GMPO you first apply the PPO-style clipping to the importance ratios of each token and then take the geometric mean. So $J_{i,GSPO}(\theta) = \hat{A}_i \exp(\frac{1}{T_i}\sum_{t=1}^{T} \log \rho_{i,t})$ , and $J_{i,GMPO}(\theta) = \hat{A}_i \exp(\frac{1}{T_i}\sum_{t=1}^{T} \log \bar{\rho}_{i,t})$ , where $\bar{\rho}_{i,t} = \min(\rho_{i,t}, \epsilon_{high}) \text{ if } \hat{A}_i > 0, \text{ else } max(\rho_{i,t}, \epsilon_{low}) \text{ if } \hat{A}_i < 0$ . Suppose a sequence has importance ratios $(\rho_1,\rho_2,\rho_3)=(1,2,5)$ . GSPO would give $\sqrt[3]{125}=2.15$ . GMPO would first clip, so $\sqrt[3]{11.51.5}=\sqrt[3]{2.25}=1.31$ and GRPO would also first clip so $\frac{1+1.5+1.5}{3}=1.33$ . This shows that GMPO is much more stable and in experiments they show they can have a much larger clipping range (which can allow them to explore more -- which DAPO said was a good thing) while having a lot of stablility (unlike GRPO). Another advantage compared to GSPO is that in GSPO if $s_i(\theta)$ gets clipped, then no token gets any gradient, whereas in GMPO even if some tokens get clipped, other tokens might not be so some gradients can still flow, causing learning	Limitation of the paper is lack of theoretical connection from importance ratios (as geometric mean is not the same as true trajectory correction which is the product of importance ratios). Same as GSPO: geometric mean is less sensitive to large positive outliers but more senstive to values close to 0, so a single very small ratio can suppress the update for the entire rollout (and clipping doesn't handle this as PPO-style clipping is directional so for a good action the ratio can still be very small as clipping might not apply). Empirically the work does not compare others well.
2025	On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence (TIC-GRPO)	This paper targets the main theoretical limitation of GRPO where the importance sampling ratios are not true, because the true IS ratios would be the product of all IS ratios, which GRPO ignores as its product can be unstable. First thing the paper shows is under limited number of updates the GRPO objctive is a rough approximation to the true gradient and therefore it still useful as it is. TIC-GRPO makes three changes: (a) replace the token-level IS ratios with exact IS ratios for the entire trajectory, (b) only-upper clip the ratio, remove the low clip, (c) replace the length-dependent normalization with a constant. $L_{TIC}(\theta) = \frac{1}{T}E_{\tau \ \sim \ \pi_{old}}[\min(\rho_{trac}, 1 + \epsilon)\hat{A}_\tau]$ . Therefore every token gets the same sequence-level importance ratio. The approach is critic-free and uses group-level advantage. As mentioned before the trajectory ratios can lead to high variance because of the product and the upper cap prevents the high accumulation, but absence of lower cap means a trajectory can potentially contribute zero gradient. They also shows asymptotic convergence of GRPO vs TIC-GRPO and shows that GRPO has quadractic dependence on trajectory length whereas TIC has linear (the results assume globally Lipschitz score function, not sure how much large llms satisfy that)	Limited experimental validation with 2 models on dapo-math dataset. Raw trajectory ratios are questionable for long completions, but the paper provides no details on that. Won't recommend this approach.
2025	Soft Adaptive Policy Optimization (SAPO)	This is another adaptive clipping approach, but is different from DCPO and BAPO in the sense that it does not impose a hard-cutoff, the cut-off is smoother based in the importance sampling ratio, i.e., if the ratio is close to 1, full gradient is passed. If the ratio is smaller, reduced gradient is passed and as the ratio becomes even smaller, almost 0 gradient is passed but at no point a hard cutoff is imposed, its always soft. In GRPO, two tokens which are similarly off-token one on this side of the clipping threshold receives gradient and one on the other side does not, straight away clipped (GSPO aggravates this issue by clipping away the gradient for the entire sequence) -- in SAPO that problem does not arise as the clipping is softer. Here is the modified gradient $\nabla_{\theta} J_{SAPO} = \frac{1}{G}\sum_{i}\frac{1}{T_i}\sum_{t} w_{\tau}(\rho_{i,t})\rho_{i,t}\hat{A}_{i,t} \nabla_{\theta} \log \pi_{\theta}(y_{i,t} \mid x, y_{i,<t})$ Therefore SAPO takes the usual importance and advantage weighted gradient and multiplies it with $w_{tau}(r_{i,t}) \in (0, 1]$ which changes as a token moves more off-policy. The decay of this function depends on the temperature $\tau$ . SAPO uses two temperatures, one for positive side and other for negative side which are different, their $\tau_{neg}$ is slightly larger as they want to attenuate the negative-advantage tokens more aggresively (they justify that by saying that when negative advantage tokens are suppressed, unrelated tokens might get more weight which is not good, so we want to stop supressing them earlier). They perform experiments with two qwen models and show that having different $\tau$ for positives and negative tokens is important.	The main weakness is lack of experimental details in the paper and not explaining why should the gate be based on $r_t - 1$ instead of $\log r_t$ -- the first one makes the gate asymmetric wrt $\rho_{i,t}$ , so $\rho=2$ is not symmetric to $\rho=0.5$ $(w(2) \approx 0.786, w(0.5) \approx 0.940.)$ . I think this approach can be mixed with GSPO for sequence level.
2025	MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (CISPO)	CISPO is the most different from all the previous trust-region approaches, because it actually does not impose a trust region. It uses the normal importance-sampling ratio REINFORCE gradient except that is caps the ratio with which an update can happen, it does not cap the gradient, only its multiplier. Overall the gradient of the objective becomes $\nabla_{\theta} J_{CISPO} = \frac{1}{\sum_i T_i} \sum_i \sum_t \hat{\rho}_{i,t} \hat{A}_{i} \nabla_{\theta} \log \pi_{\theta}(y_{i,t} \mid x, y_{i,<t} )$ , where $\hat{\rho}_{i,t} = clip(\rho_{i,t}, 1 - \epsilon_{low}^{IS}, 1 + \epsilon_{high}^{IS} )$ , so the importance ratio with which the gradient is multiplied is clipped, but the gradient never dies. Note that this makes the multiple epochs on the same rollouts not follow trust region as the updated policy can go arbitrarily far from the original policy. They also have a trick to set $\epsilon_{low}^{IS}$ to be pretty small, this makes basically no lower bound -- this is important in order to stop suppressing the bad tokens. Consider an example where $\rho_t = 0.1, \hat{A}_t = 1$ , for this token, $\nabla L_t^{CISPO} = -0.1\nabla \log \pi_{\theta}$ , so the token still contributes but weakly. As its probability continues to decrease $\rho_t \rightarrow 0$ , the gradient coefficient automatically approaches zero. If we were to set $\epsilon_{low}^{IS} = 0.8$ , then it would keep aggressively suppressing a token that was already extremely unlikely, we don't want that, so having no lower bound is important. On the positive side, there is cap on the gradient coefficient, but not on the policy update itself. So overall CISPO reinforces positive discoveries, but gradually stops suppressing already-suppressed tokens. CISPO liek DAPO averages over all tokens in a batch and adopts dynamic sampling, length penalty, and removes KL penalty as well. It reaches DAPO's performance at half the number of tokens.	Since trust region is not imposed, rare tokens are not discarded but spurious tokens can be repeatedly reinforced as well. Other limitation of the paper is the lack of wide empirical validation.
2025	The Art of Scaling Reinforcement Learning Compute for LLMs	This paper fits a scaling law for RL, it does not propose a new algorithm. It found that a sigmoid scaling law is better than power scaling law for RL and uses that. They compare GRPO/DAPO/PPO/CISPO/GSPO and found CISPO to be the best, closely followed by GSPO and choose CISPO for the scaling law experiment. They compare 3 kinds of loss-aggregation: completion-level, token-level, and prompt-level and found prompt-level to be the best. This avoid every completion receiving equal weight regardless of length. For the advantage computation, they use prompt-level centering so $\mu = \bar{r}_{prompt}$ , but for division they compare prompt-level (as in GRPO), batch-level (as in REINFORCE++), and no standard deviation (as in Dr. GRPO). All three performed similarly, they choose batch-level normalization as it performed slightly better. They filter any prompt whose all grouped samples (16) have same reward (and therefore cannot contribute gradient signal as $r_i == \bar{r}_{prompt}$ , this is both similar to DAPO which is adaptive to such prompts, but unlike DAPO which samples more responses, this paper just drops such prompts) or its passrate has become $\ge 90\%$ , which assumes that once a prompt has become easy, it usually remains easy. It also introduces forced interruption of long completions instead of overlong penalty (unlike DAPO), so at any point after 10K tokens, the generation system can insert "Okay, time is up. Let me stop thinking and formulate a final answer now." and then after some budget comes hard truncation. It also uses Pipeline-RL with upto 8 reuses of the same rollout. PipelineRL generates batches continuously instead of waiting for 8 reuse of a rollout. This means trainers update as soon as batch is available and they can be used to continue partial respones from previous trainers using existing KV cache -- this makes training much faster. Overall, they found larger MoE, longer generation budget, larger prompt batch, upto 16 generations per prompt produces a better RL scaling model. To ensure that generator and trainer have $\rho = 1$ , they use FP32 computations at LM head (differences can arise due to different kernels.)	Can't think of major limitations for this paper -- maybe more comparisons of different algorithms would have been cool.
2025	Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards (A3PO)	This paper makes a very simple edit to the DAPO objective, it says that it should use a different advantage for tokens in a completion even with just a terminal reward. So instead of all tokens in a completion having the same advantage, what this does is to have an advantage multiplier for two cases (a) low probability tokens for correct completions ( $\hat{A}_{t} > 0$ ) and (b) high probability tokens for incorrect completions ( $\hat{A}_t < 0$ ). In experiments it considers bottom-20% tokens in positive rollouts and top-20% tokens in negative rollouts for this update. The advantage such tokens is $\hat{A}_t = A_tmax(2 - 0.005training\_step, 1)$ , so this just multiplies the positive or negative advantage for a factor of 2 (at step = 0) and a factor > 1 upto 200 training steps (after which its just the normal advantage with no multiplier). The intuition for this change comes from their earlier experiments where doing so increases entropy and therefore encourages exploration in the earlier part of training.	Limited empirical validation and lack of explanation of why the multiplication factor should be 2 and decay over 200 steps, why not make that prompt-dependent.
2026	Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR (DHPO)	This paper interpolates the GRPO based token-level importance sampling ratio and GSPO based sequence-level importance sampling ratio (which is the geometric mean of the token-level IS ratios). GRPO uses $\rho_{i,t}$ for each token, whereas GSPO uses sequence-level ratio $s_i = \big(\frac{\pi_{\theta}(y_i \ mid x_i)}{\pi_{old}(y_i \ mid x_i)}\big)^2$ , which it gives to each token in the sequnce. DHPO mixes the two, so the multiplier for each token becomes $w_{i,t}\rho_{i,t} + (1-w_{i,t})s_i$ . It describes two strategies: averaged mixing, i.e., $w_{i,t} = 0.5$ (hardcoded) and entropy-guided mixing which uses the entropy of each token to determine $w_{i,t}$ (high entropy tokens get larger token-level weighting, i.e., larger $w_{i,t}$ as the authors say that for uncertain policies one should focus on the individual differences in the token ratios, and low entropy tokens get smaller $w_{i,t}$ .) Final objective is $L_{DHPO} = E[\frac{1}{G}\sum_{i=1}^G\frac{1}{T_i}\sum_{t=1}^{T_i} \min(m_{i,t}A_i, \widetilde{m_{i,t}}A_i)]$ , where $\widetilde{m_{i,t}} = w_{i,t}clip(\rho_{i,t}, 1-\epsilon_{low}, 1+\epsilon_{high}) + (1-w_{i,t})clip(s_i, 1-\epsilon_{low}^{seq}, 1+\epsilon_{high}^{seq})$ (which means both the token-level IS ratio and the sequence-level ratio are individually clipped, in order to detect independent deviations). Gradients are: $\nabla_{\theta}\rho_{i,t} = \rho_{i,t}\nabla_{\theta}\log\pi_{\theta}(y_{i,t} \mid x, y_{i, <t})$ and $\nabla_{\theta}s_{i} = s_i \nabla_{\theta}\log\pi_{\theta}(y_{i,t} \mid x_i, y_{i, <t})$ , therefore $\nabla_{\theta}L_{DHPO} = E[\frac{1}{G}\sum_{i}\frac{1}{T_i}\sum_{t}m_{i,t}\nabla_{\theta}\log\pi_{\theta}(y_{i,t} \mid x_i, y_{i,<t})]$ . So DHPO is a group-relative REINFORCE gradient with a novel multiplier. Paper performs experiments with 3 models and 7 math benchmarks, shows better performance than GRPO and GSPO and higher late-training policy entropy.	(a) They use asymmetric clipping like DAPO, but do not compare to it. (b) The paper assumes high-entropy tokens should receive more token-level weighting, but this can also introduce noise. (c) DHPO still assigns the same sequence-level advantage to every token, it uses novel multiplier to the advantage. (d) This approch kind of has GMPO vibes as even it sequence level clipping happens, each token can receive gradient which can mean more training signal and therefore the performance improvement can be because of that instead of mixing.
2026	Soft Sequence Policy Optimization: Bridging GMPO and SAPO (SSPO)	As the paper's title suggests, this paper takes GMPO's geometric aggregation over a response, and replaces each hard-clipped token ratio with a smooth bounded transformation of its IS ratio as proposed in SAPO. SSPO proposes a differentiable gate $W_i^{SSPO} = \big[\Pi_{t=1}^{T_i} f(\rho_{i,t} A_i) \big]^{1/T_i}$ and the objective becomes $J_{SSPO} = E[\frac{1}{G}\sum_{i=1}^{G}W_i^{SSPO}A_i]$ . This notations acts as a useful unification, if $f(\rho) = \rho$ , then $W_i = \big(\Pi_{t} \rho_{i,t} \big)^{1/T_i}$ -- which is the GSPO's geometric sequence ratio. If $f(\rho) = f_{clip}(\rho)$ , then it becomes GMPO and if that is clipping is changed to sigmoid function that becomes like SAPO. The paper investigates 5 possible gates and chooses the exp arc tan. This gives SSPO a soft sequence-level trust region without a discontinuous clip, GMPO-like response-level multiplier, and GRPO like token-level attention (so like DHPO it mixes token-level and sequence-level attenuation as well). So overall the gradient is $F_i w_{i,t}A_i \nabla_{\theta}\log\pi_{\theta}(y_{i,t} \mid x_i, y_{i,<t})$ , where $F_i$ is the sequence-level information, $w_{i,t}$ is the token-level information, and $A_i$ is the sequence-level advantage.	(a) The approach still gives each token in a sequence the same advantage. (b) Experimental results show SSPO performs better, but the maximum response length is just 512 so scalability is not tested.
2026	GIPO: Gaussian Importance Sampling Policy Optimization	This paper is just like SAPO, replace the hard clipping in PPO with a softer function to retain some gradients. GIPO replaces the sigmoid attenuation in raw IS ratio space with gaussian attenuation in log-ratio space. $g_{SAPO}(\rho) \propto sech^2(c(\rho - 1))$ and GIPO uses $g_{GIPO}(\rho) = \exp\big[-\frac{(log \rho)^2}{2\sigma^2}\big]$ . Because of this, it has a few differences to GIPO: (a) ratio symmetry: GIPO's gate satisfies $g_{GIPO}(\rho) = g_{GIPO}(1/\rho)$ , where SAPO's does not. (b) For large $\rho$ , SAPO's sigmoid gate decays more much quickly than GIPO's square of log ratio (I guess this is fine because if CISPO which basically has no trust region is working means that clipping is much less important than PPO thinks). (c) They use $\sigma$ to decide which stale trajectories contribute to the gradient update and find that suppresing stale negative-advantage samples more than stale positive-advantage samples is better (I think this should be dependent on problem hardness, for harder problems finding a correct solution is rare therefore using them for more epoch is fine compared to for easier problems)	The paper does not perform any experiment with LLM RLVR benchmarks, instead it uses small Gridworld, Meta-world, classic control, and Mujoco tasks.
2026	CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR	This paper alters the binary rewarding scheme by proposing that not all successful trajectories are the same, one might be because of accident or luck and therefore the successful trajectories that look like others should be rewarded higher. So they train a embedding model with it, so for a group of responses they first score them using the verifier, and then take the average token embedding, pass it thorough an embedding model and train that using contrastive learning (pair of positives and all negatives). The overall reward is a mix of verifier reward and (negative of) contrastive loss, so similar looking successful trajectories receive higher reward than dissimilar pairs.	(a) I don't agree with the hypothesis, there might be multiple ways of reasoning to solve a problem, and this approach can cause strategy collapse. (b) Cannot work for a prompt that only got 1 correct completion, you need a minimum of two, (c) empirical evidence of improvement is shallow. Won't recommend this approach.
2026	FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization	The goal of this paper is to introduce token-level advantages without training a PPO like critic model (other papers that want to do this are PRIME, A3PO, VinePPO). It hypothesizes that if increasing a token's probability increases the probability of the subsequent trajectory, then this token might be an important anchor and therefore its advantage should be amplified. Consider, for a sampled token $y_t$ , $\Delta \log p_t = \log \pi_{\theta}(y_t \mid x, y_{<t}) - \log \pi_{old}(y_t \mid x, y_{<t}) = \log \rho_t$ . Therefore if $\Delta \log p_t > 0$ : the updated policy has increased this sampled token's probability (and vice-versa), this does not tell if the token is objectively good, just that the updated policy has moved relative to the sampler. The "Future-KL" part accumulates the log-probability shifts of all future sampled tokens (discounted), $F_t = \sum_{k=t}^{T} M_k \gamma^{k-t} \Delta \log p_k$ , where $M_k = 1[\rho_k \le c]$ , filters extreme ratios ( $c = 10$ in experiments). The $\gamma$ factors discounts the next full response and only considers the next few hundred tokens for this hypothesis. This computed $F_t$ after clipping acts as the extra advantage multiplier specific to this token, i.e., $f_t = clip(e^{F_t}, 1- \epsilon_{low}, 1+\epsilon_{high})$ . Without discounting or clipping $e^{F_t} = \frac{\pi_{\theta}(y_{t:T} \mid x, y_{<t})}{\pi_{old}(y_{t:T} \mid x, y_{<t})}$ , which is the full suffix importance ratio, with discounting it becomes $e^{F_t} = \Pi_{k=t}^{T}\rho_k^{\gamma^{k-t}}$ , this is the geometrically discounted product of future importance ratios, and then clips it to make it usable. Final objective of FIPO becomes: $J_{FIPO} = \frac{1}{\sum_i T_i}\sum_{i}\sum_{t} \min(\rho_{i,t}f_{i,t}\hat{A}_i, clip(\rho_{i,t}, 1-\epsilon_{low}, 1+\epsilon_{high})f_{i,t}\hat{A}_i )$ In the implementation $f_{i,t}$ is detached, so it just acts as a scalar multiplier (no gradient flowing from future tokens) and the gradient is $f_{i,t}\rho_{i,t}\hat{A}_i\nabla_{\theta}\log\pi_{\theta}(y_t \mid x, y_{<t})$ . In the experiments, $f_t \in [1, 1.2]$ , $f_t$ is never less than 1, therefore when $F_t < 0$ , then $f_t = 1$ , when $F_t > 0$ , the update is amplified upto 20%. Exepriments finetuning a Qwen model on Dapo-17K math data with max response length of 20K, and find slight improvements in AIME results.	(a) Empirical validation is shallow, (b) comparison with baselines has a confound of the number of samples in a group (FiPO 64 vs DAPO 32) (c) The logic that if future tokens have become more likely then this token would have influenced it, is not causal and can be even debatable.
2026	ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models	This paper wants to also get token-level advantages and uses the entropy at any state ( $W_{i,t}$ ) as an estimate of "critical decision pivots" and reinforces them more strongly. First in a small experiment they take the top-5% and bottom-5% of the tokens (wrt to entropy) and perturb them, and show that perturbing high entropy tokens changes the results more. For a prompt, it samples G rollouts and divides each rollout into K buckets where each bucket is supposed to refer to the "same stage of reasoning". Then it calculates a measure of "progress" within the same bucket across all G rollouts, lets call these buckets $b_k$ . For each token in $b_k$ , they define "progress" as $s_{i,t} = \log \frac{\pi_{\theta}(y_{i,t} \mid s_{i,t})}{\pi_{ref}(y_{i,t} \mid s_{i,t})}$ . Note that this is not IS ratio, as this is wrt the reference model, not sampling model. This measures long-term drift from the reference model. Now, each token in $b_k$ is normalized wrt to all tokens in $b_k$ , $\widetilde{s_{i,t}} = \frac{s_{i,t} - \mu_k}{\sigma_k}$ . A positive $\widetilde{s_{i,t}}$ means that the current policy has increased this token relative to the reference more than it has increase typical tokens approximately at the "same stage of reasoning". Now they construct $B_{i,t} = W_{i,t}sgn(\hat{A}_i)\widetilde{s_{i,t}}$ , so for a successful completion $sgn(\hat{A}_i) = 1$ and $B_{i,t} = W_{i,t}\widetilde{s_{i,t}}$ . Therefore a high entropy token with $\widetilde{s_{i,t}} > 0$ gets higher reinforcement and low entropy token gets small adjustment. O.t.o.h, if a high entropy tokens occurs in a response that failed, that should be aggresively suppressed. The way this is done is by (a) scaling $B_{i,t}$ so that it has a target std. deviation, called $\Psi_{i,t}$ , (b) adding it to the sequence-level advantage to convert it into token-level advantage $C_{i,t} = \hat{A}_i + \eta\Psi_{i,t}$ , and (c) normalizing across all tokens in the G rollouts, i.e, $\hat{A}_{i,t}^{final} = Norm_{G}(C_{i,t})$ . Final objective becomes: $J_{ERPO} = \frac{1}{\sum_i T_i} \sum_{i}\sum_{t} \min(\rho_{i,t}\hat{A}_{i,t}^{final}, clip(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_{i,t} - \beta D_{KL}$ So EPO retains the GRPO clipping (same on both sides), IS ratios, and even the long forgotton KL penalty to reference.	(a) Empirical validation is performed with max response length of 2048 which is quite small. (b) ERPO has small margins over GRPO in its experiments. (c) High entropy token does not necessarily meaningful reasoning fork, it could indicate confusion, linguistic ambiguity, tokenization effects -- so identifying such tokens is still open.
2026	Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective (CTPO)	This paper wants to fix the issue with the incorrect IS ratio that is used in GRPO which does not account for the probability of sampling the prefix itself (similar to TIC-GRPO). GRPO/PPO uses the IS ratio of just the specific token, however there is a prefix $s_t = (x, y_1, y_2, \dots, y_{t-1})$ and the probability of observing that state is differente under the sampling and the current policy, its $\Pi_{j=1}^{t}\rho_{i,j}$ , this is cummulative token IS ratios. (Note that TIC-GRPO, GSPO, and GMPO uses the full sequence ratio, i.e, TIC-GRPO uses $\Pi_{j=1}^{T}\rho_{i,j}$ and GSPO/GMPO use $(\Pi_{j=1}^{t}\rho_{i,j})^{1/T_i}$ -- this is mathematically incorrect as gradient of a current token depends on a future token.) The paper also shows that variance of the cummulative token ratio is lower than the full sequence. They find an issue if a single clipping is applied to the full cummulative IS ratio as early cummulative ratios remain close to 1 and are rarely clipped, while late cummulative ratios have accumulated many changes and are clipped frequently. Therefore CTPO uses a position-adaptive clipping, so the bounds grow with position, i.e., $\epsilon_{high}(t) = \epsilon_{high}t^p$ and $\epsilon_{low}(t) = \epsilon_{low}t^p$ , where $t$ is the token position. They use $p = 0.5, \epsilon_{low}=0.025, \epsilon_{high}=0.05$ . This leads to the following clipping bounds: t=1, lower bound = 0.97, upper bound = 1.05; t=100, lower = 0.78, upper = 1.6; t=5000, lower = 0.17, higher = 34.3. Although this looks very permissive for late tokens, those bounds are on product containing thousands of small IS ratios and the asymmetric values also permit more upward cumulative movement than downward. CTPO uses GRPO's group-level advantage and the final objective is: $J_{CTPO} = \frac{1}{G}\sum_{i=1}^G\frac{1}{T_i}\sum_{t=1}^{T_i} \min(\rho_{i,t}^{cum}\hat{A}_{i}, clip(\rho_{i,t}^{cum}, 1 - e^{-\epsilon_{low}\sqrt{t}} , 1 + e^{-\epsilon_{high}\sqrt{t}} )\hat{A}_{i})$ They compare with GRPO and GSPO and show better performance with two qwen model on 40K DeepscaleR problem. They show ablation with clipping, and position-adaptive clipping leads to much better improvement than fixed clipping.	(a) Theoretical results have strong assumptions that might not hold in practice, (b) empirical evidence is shallow as they use only one model-family with one specific task (tool-integrated maths reasoning), (c) no comparisn of cummulative ratio vs token ratio matched clip rates.
2026	Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective (ConSPO)	This paper tries to break a different symmetry that is used in GRPO: each positive rollout for a prompt receives the same score gradient coefficient irrespective of how the current model scores that rollout, similarly each negative rollout for a prompt receives the same score gradient coefficient. Suppose for a prompt (with success rate $p$ ) there are $N^+$ positive rollouts and $N^-$ rollouts, the objective (same as in DisCO paper) becomes $J = \sqrt{p(1-p)}[\frac{1}{N^+}\sum_{i=1}^{N^+}s_i^+ - \frac{1}{N^-}\sum_{j=1}^{N^-}s_j^-]$ , therefore, $\frac{\partial{J}}{\partial{s_i^+}} = \frac{\sqrt{p(1-p)}}{N^+} \text{ and } \frac{\partial{J}}{\partial{s_j^-}} = - \frac{\sqrt{p(1-p)}}{N^-}$ -- this is the same for all positive rollouts and all negative rollouts (separately). Here score is $\frac{1}{T_i}\sum_{k=1}^{T_i} \log \pi_{\theta}(y_k \mid x, y_{<k})$ (same as geometric mean of token-probs as in GSPO). However, some positive rollouts might have a score much lower than others (which means it is less likely) and some negative rollouts might have a high score (which means they are more likely), and we should concentrate on these rollouts. ConSPO proposes to use a saturating contrastive objective for a positive-negative pair $J_{ConSPO} = \tau\log\sigma(\frac{s^+ - s^- - m}{\tau})$ , (m is margin) so this is log-sigmoid which goes to 0 as soon the value inside reaches high enough (therefore saturating). Let $d = s^+ - s^- - m$ , then $\frac{\partial{J}}{\partial{s_i^+}} = \sigma(-\frac{d}{\tau})$ and $\frac{\partial{J}}{\partial{s_j^-}} = -\sigma(-\frac{d}{\tau})$ , so now the scores gradients are dependent on the scores. (Usually the loss consists of one-positive and all negatives, not only one pair and the margin starts at 0 and then is slowly increased during training, and a high temperature $\tau=10$ is used because low temperature heavily focuses on hard-negatives. ) They do extensive experiemnts on deepscaleR dataset and test on 7 match benchmarks. They compare with GRPO, DAPO, DisCO, GMPO, CISPO, SAPO and perform better than all of them. They show ablation experiment in which removing the contrastive objective to have no dependence on rollout score leads to largest drop in final accuracy. So changing GRPO's $\nabla_{\theta}J_{GRPO} = c_+\sum_i\nabla_{\theta}s_i^+ - c_-\sum_j\nabla_{\theta}s_j^-$ to $\nabla_{\theta}J_{ConSPO} = \sum_ic_i^+\nabla_{\theta}s_i^+ - \sum_jc_i^-\nabla_{\theta}s_j^-$ is the main contribution.	(a) They don't do any clipping of IS ratio (they don't even use IS ratios) which can lead to instability, (b) No token-level advantage.
2026	Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards (CIPO)	This paper does not introduce a variation of RLVR loss -- it just says that not all failures are the same, and therefore it feeds the failures again to the same with context of problem and the failed solution and prompts it to try again in hopes of distinguishing failures due to minor arithmetic errors and failures due to massive reasoning issues (which GRPO does not distinguish). This gives this approach two sets of samples (original) and re-tried (here they do a mix of 30% correct and 70% incorrect resposes from the first group) and do independent GRPO on both groups. $J_{CIPO} = J_{group1} + J_{group2}$	Results show improvement, but not directly comparable to previous methods as it uses additional supervision in some sense.
2026	From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning (SCRL)	This is another approach like CIPO to provide partial credit for failures, especially for hard problems where all rollouts can result in failures. To address this, it uses additional supervision -- another LLM is given the <problem, answer, reference solution> and it is asked to break down the original problem into 4 subproblems of increasing difficulty (final problem being the original one) and answers for each of them. Then it gives all 4 subproblems together and asks the model to solve it and verifies the answers. So the reward can be $(1, 1, 0, 1)$ for the 4 subproblems, but the paper only takes the longest consecutively correct prefix, i.i.e., $(1, 1, 0, 0)$ . And then for a group of rollouts, it computes the GRPO's group-level advantage within each subproblem, so the same rollout can have a positive advantage for first two parts and negative advantage for last two parts providing a denser signal even for an overall failed rollout. To correct for train-test mismatch, they generate $G/2$ rollouts without subproblem thing and $G/2$ rollouts with and do separate GRPO on both subgroups.	(a) the llm subproblem generator can produce an incorrect answer for the subpart which can mislead the training. (b) cannot to compared to other methods as it uses additional supervision.
2026	RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning (POPO)	This paper is a replay buffer approach (kinda similar to DCPO) -- if a prompt generates an ineffective samples (i.e, std(samples) = 0), then pick another group from the most recent buffer instead of dropping that prompt (which reduces batch size) and for that replayed sample use a multiplier for the advantage that takes into account the importance ratio correction for 3 models instead of the usual 2.	Not a novel RLVR approach, so no limitations as such.
2026	A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization (WAPO)	This paper studies the effect of updating the model using RLVR and shows that in several situations it can still lead to instability. Consider a case when logits for next token prediction are (-0.35, -1.61, -2.3), after softmax, the probabilities are (0.7, 0.2, 0.1) and say the sampled token is the one with probability 0.1 and assume this token has $A > 0$ . Since $A > 0$ , we'll reinforce the sampled token and that will lead to change in the logits and the probabilities. In this case, token-1's probability goes down to 0.68, token-2's probability goes upto 0.204, and token-3's increases to 0.114. So even with decrease in the logit for token-2, its probability increases. Thus reinforcing a rare successful token can increase the sampled rare token, but also increase other alternatives and the authors claim this can lead to instability. Overall they categorize 4 cases: (a) reinforcing an already-likely successful token -- decreases entropy, stable; (b) reinforcing a rare successful token -- increases entropy, can collapse but also induce discovery; (c) suppressing a likely unsucessful token -- increases entropy, can collapse; (d) suppressing an unlikely unsuccessful token -- decreases entropy, can still collapse due to low entropy. So the authors claim that training on positive-only rollouts is stable and they just reject all negative rollouts. So the objective is: $J_{WAPO} = E[\frac{1}{GT}\sum_{i=1}^G\sum_{t=1}^{T_i} \min(\rho, 1+\epsilon)A_i^+]$ for only the rollouts that have advantage higher than baseline (correct completions in case of binary rewards).	(a) rejects useful information from the negative rollouts and therefore poor empirical resutls, its not best in most experiments in its own paper.
2026	Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (OPEFO)	This paper's goal is to address entropy collapse during RLVR training. Similar to WAPO, they identify 4 cases: (a) $A_t > 0$ and the token is unlikely -- this leads to increase in entropy, (b) $A_t > 0$ and the token is likely -- decrease in entropy (c) $A_t <> 0$ and the token is unlikely -- decrease in entropy, and (d) $A_t < 0$ and the token is likely -- increase entropy. It proposes to partition sampled tokens into tokens that increase entropy and the ones that decrease entropy and introduces a coefficient in the update equation that balances them. Change in entropy for a token is computed using: $\widehat{\Delta H}_t =-\eta A_t(1-p_t)^2(\log p_t+H_t)$ , where $p_t$ is the probability of sampled token. This approach multiplies the positive token's gradients by $1+\lambda$ and negative token's gradients by $1-\lambda$ and choose $\lambda$ such that $(1+\lambda)P = (1-\lambda)N$ , giving $\lambda^* = \frac{N-P}{N+P}$ . Since $\lambda$ exactly balances entropy, therefore when $P > N$ , $\lambda^* < 0$ and the equation with automatically downweight entopry-increasing updates. OPEFO performs strictly on-policy updates, so no need of IS ratios. Here is the final objective: $\nabla_{\theta} J_{OPEFO} = E[(1+\lambda) \sum_{t\in S^+}A_t\nabla_{\theta}\log\pi_{\theta}(y_t \mid s_t) + (1-\lambda) \sum_{t\in S^-}A_t\nabla_{\theta}\log\pi_{\theta}(y_t \mid s_t) ]$ The paper analyzes other ways of addressing entropy collapse issue like increase $\epsilon_{high}$ in DAPO or entropy bonus, but say they either indirectly address the problem or the produce conflicting gradient updates.	(a) limited empirical settings and even the results show only slight improvement over the strict on-policy GRPO (b) derived $\lambda$ is 0.0001 which means that the issue of entropy collapse is overplayed.
2026	Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards (SC-GRPO)	This paper aims to provide token-level advantages without training a critic model (like PRIME, A3PO, VinePPO, ERPO). The way is proposes to do so by using a correct solution to understand which token is important and which one is not. It creates two versions of the same model, one has the verified solution in its context and other does not. Now at any token position of the current prefix that is being generated by the model, they compute the KL-divergence between the token distribution of what the two models, i.e., $D_{i,t} = D_{KL}(\pi_{\theta}(\cdot \mid x, \tau, y_{i,t}) \|\| \pi_{\theta}(\cdot \mid x, y_{i,t}) )$ , where $\tau$ is the verified solution of the problem. Now the hypothesis is that if $D_{i,t}$ is small, which means seeing the correct solution does not change the model's next-token distribution, it means this token must be a routine token. However if $D_{i,t}$ is large, that measn the position may be important. They just use this as a multiplier to the advantage in the GRPO's objective (after transforming it $f(D_{i,t})$ such that it gets bounded between 0 and 1). Final objective of SC-GRPO is: $J_{SC-GRPO} = E[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_i}\sum_{t=1}^{T_i}f(D_{i,t}) \min(\rho_{i,t}\hat{A}_i, clip(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_i) ]$ this just makes $\hat{A}_{i,t}^{eff} = f(D_{i,t})\hat{A}_{i,t}$ .	(a) Requires atleast two correct rollouts as we don't want to condition the prefix on the same rollout (b) requires an additional forward pass of each solution through the model (and that too with longer context as it has the correct solution in context) -- led to it being 51% slower on Livecodebench, 37% slower on DAPO-math.
2026	Reinforcement Learning via Self-Distillation (SDPO)	This paper also aims to provide token-level signal and proposes to use the environment feedback and correct solutions if available. Similar to SC-GRPO, SDPO tries to minimize the KL-divergence between the distribution over the next token from the student model (that is only given the problem in context) and the teacher model (one that is given the probllem, and additional feedback in context). The objective is $L_{SDPO} = \sum_{t=1}^{T} D_{KL}(\pi_{\theta}(\cdot \mid x, y_{<t}) \|\| sg[q_{\theta}(\cdot\mid x, f, y_{<t})])$ . If we take the gradient of this term, it can be written as $\nabla_{\theta}\mathcal{L}_{SDPO} = E[\log\frac{p_t(a)}{q_t(a)}\nabla_{\theta}\log p_t(a)]$ , so if we define a modified advantage as $A_t^{SDPO}(a) = \log\frac{q_t(a)}{p_t(a)}$ , then $\nabla_{\theta}\mathcal{L}_{SDPO} = -\sum_{a}[A_t^{SDPO}(a)\nabla_{\theta}\log p_t(a)]$ , whcih is same as policy-gradient ascent using $A_t^{SDPO}$ . Its interpretation is simple, if $q_t(a) > p_t(a)$ , then $A_t^{SDPO} > 0$ , so the feedback-informed teacher wants token $a$ increased and vice-versa. Very interestingly, unlike other algorithms, SDPO can change the probability of a token that was not even sampled. Since SDPO's update equation includes all tokens $\sum_{a \in \mathcal{V}} p_t(a)A_t^{SDPO}(a)\nabla\log p_t(a)$ , not only can it suggest to reduce the sampled token, it can suggest to increase specific tokens that the teacher thinks are more likely (GRPO can only decrease the sampled token's probabilities which as a result can increase other token's probability but it cannot directly do that). (for memory efficiency purposes they only use the top-20 or 100 tokens from the vocab for the student's distribution.) From the ablation experiments in the paper, they found that providing the environment dense textual feedback along with a correct completion got the best accuracy (even a failed rollout can help, but correct is better and dense feedback is better than binary pass/fail). In order to prevent the student from imitating a mistaken teacher, the teached is a slow-moving EMA weight of the student -- even frozen teacher is fine, just equally fast moving teacher is bad. Similar to GSPO/GMPO, the authors tests sequence-level KL but found that to perform similar to token-level KL. Authors also test SDPO combined with GRPO.	(a) there are additional costs with having another separate teacher model, which is just like having PPO's critic model (although I personally believe this model is more helpful than critic) (b) the approach is dependent on textual feedback from environment (c) results on livecodebench are not comparable to baselines as they train and test on same prompts.
2026	VIMPO: Value-Implicit Policy Optimization for LLMs	This paper wants to have a token-level advantage, but does not want to train an expensive critic model, so they want the policy model itself to encode the value function. They do some derivations to arrive at the desired advantage to be $A^(s_t, a) = \beta[\log\frac{\pi^(a \mid s_t)}{\pi_{ref}(a \mid s_t)} - D_{KL}(\pi^* \|\| \pi_{ref})]$ . Now in autoregressive generation, $Q^(s_t, a_t) = r_t + \gamma V^(s_{t+1})$ since $s_{t+1} = (s_t, a_t)$ . Therefore $\beta[\log\frac{\pi^(a \mid s_t)}{\pi_{ref}(a \mid s_t)} - D_{KL}(\pi^ \|\| \pi_{ref})] = r_t + \gamma V^(s_{t+1}) - V^(s_t)$ . Define $z_t(\theta) = \beta[\log\frac{\pi^(a \mid s_t)}{\pi_{ref}(a \mid s_t)} - D_{KL}(\pi^ \|\| \pi_{ref})]$ . Then with $\gamma=1$ , $V_{\pi_{\theta}}(s_t) = V_0 + \sum_{k=0}^{t-1}+[z_k(\theta) - r_k]$ . Note that $V_0 = \bar{R}$ and for outcome only reward, $r_k = 0$ except for final token, thus before the terminal reward $V_{\pi_{\theta}}(s_t) = \bar{R} + \sum_{k < t} z_k(\theta)$ . At end of completion, no future reward remains, so $V_{\pi_{\theta}}(s_T) = 0$ , subsituting gives $0 = \bar{R} + \sum_{k < t}^{T-1} z_k(\theta) - R_i$ . Therefore VIMPO wants $\sum_{t=0}^{T_i-1} z_{i,t}(\theta) = R_i - \bar{R}$ . Therefore, its loss is: $L_{V} = \frac{1}{2G}\sum_{i=1}^{G}[\sum_{t=0}^{T_i-1} \beta\big(\log \frac{\pi_{\theta}(y_{i,t} \mid s_{i,t})}{\pi_{ref}(y_{i,t} \mid s_{i,t})} - sg[D_{KL, i, t}] \big) - (R_i - \bar{R})^2]$ This is only one part of the full objective, the other term is the usual PPO update term with the advantage $A_t = \beta[\log\frac{\pi_{\theta}(a_t \mid s_t)}{\pi_{ref}(a_t \mid s_t)} - D_{KL}(\pi_{\theta} \|\| \pi_{ref})]$ Overall objective is a weighted sum of these two losses.	(a) there is a jump in theoretical derivation, from $\pi^*$ to $\pi_{\theta}$ . (b) empirical validation is shallow with only one 4B model on math dataset and that too with only slight improvements to the GRPO's number, no comparison to DAPO, GMPO, etc.
2026	KIMI-K3, same RL as Kimi-2.5	This paper uses Multi-teacher on-policy distillation to train the final model. For that it first trains 9 experts (3 domains with 3 reasoning efforts each) using RL and then uses MOPD to distill into the final student model. The RL algorithm used to train the teachers are very similar to GRPO, its $J = E[\frac{1}{\sum T_i}\sum_{j=1}^{K}\sum_{i=1}^{T_i}(clip(\rho_{i,t}, \alpha, \beta)A_j - \tau(\log \rho)^2)]$ . This is critic-free, group-mean baseline, same advantage across all token, aggregation over all tokens, per-token importance ratio approach. Unlike GRPO which has a min term that can allow the gradient to flow, KIMI directly clips the importance ratio, but the second term which is > 0 whenever $\rho_{i,t} \ne 0$ , can take care of those scenarios. There are many efficiency strategies applied in the paper like continuing partial rollouts as soon as trainer is updated, not using a reference model. The way they enforce the different reasoning efforts is interesting. It basically a limit on the reasoning length, which starts very liberally, but with more training that is tightened and if you exceed the length, you get zero reward.	(a) No ablation of this loss with GRPO.
2026	Laguna-Poolside	They use leave one-out group-level baseline (like RLOO), however the advantage function has response token-length weight with CISPO clipping with very loose bounds (upper bound is 5). the likely motivation for length-influenced advantage weight is that the long successful trajectory already contributes many token terms. Giving each of those tokens a smaller advantage can approximately balance its total contribution against the shorter unsuccessful trajectories. $A_i = r_i - b_i$ , where $b_i = \frac{\sum_{j\ne i}w_jr_j}{\sum_{j\ne i}w_j}$ . The binary terminal rewards are additionally shaped by slight changes like malformed tool call with -0.1, tool exceution error with -.05 and so on. They also do curriculum learning	--
2026	Qwen-3.8	No RL algorithm mentioned	--

Discussion

Name is optional. Sign in to edit or delete your own comments.

What did you think?