The Essence of LLM Reasoning and Training: From Surrogates to Reinforcement Learning Geometry

This paper combines recent studies and discusses the question of whether current LLM post-training (e.g. RLHF / RLVR) is creating new reasoning capabilities or is it more effective to use the capabilities already available in pre-training? The articles will be presented from both macro-wide and micro-geometric perspectives.

The questions in this article can also be addressedLLM Enhanced Learning in Alignment: From Incentive Signals to Estimation of Strengths、Post-linguistic training and fine-tuning of practice: from SFT, LoRA to human alignmentHow the concept of a relatively close read together is developed in different contexts.

LLM The Nature of Logic and Training: from Surrogate to Geometry for Enhanced Learning

Wedges: Re-entry to the proxy nature of training

Before exploring large model reasoning, it is necessary to look at a basic fact in model training.

And so many times, we're constantly seeking to bring some of the models, the Los, closer to the limit, as if we could just take it and find the truth. But that is not the case. To quote the phrase "Loss Function is a lot of surrogate, like predicting the next word" Its central purpose is to generate a gradient that allows the expression to go in the right direction, which is the most important logic.”

Los Function is SurrogateI'm sorry. If there were only the right training objectives, there would be no such number of different loss functions that would be effective in their respective areas.

The target function is of course important in math, but it is not the end. Optimizing losses is only a means of finding useful parameters. If we don't care what data structure the model has learned, and just stare at the surface of Los, we're vulnerable to trapping. The most typical example is the 0-1 evaluation mechanism: if we only teach the model "Yes" = 1 point, I don't know.'t know) = 0 points, which seems reasonable, and which actually encourages speculation, naturally gives rise to hallucinating models.

So instead of looking at the ross, we should start from the data structure we want to learn to design training signals that match our needs; rather than asking if loss is showing any indigence, ask whether this sorrogate rewards your desired behaviour (and whether systematic punishments for uncertainty expression). Considers the loss as a surogate, and more naturally focuses back on data, constraints and assessments.

From this perspective, we can revisit SFT and RL (enhanced learning):They should not be simply ranked as being of high and low quality. Both are adjusting large and super-high parameters, except for the data used and the direction of Surrogate. The training methodology affects speed, stability and exploration, but does not necessarily determine the end point separately.RL effectiveness, largely derived from RL data and incentive structures; incentives themselves do not directly introduce new knowledge but can be scaled up to specific capabilities.

Following this agent's line of thought, it's clearer to look at the reasoning. Whether SFT data or RL reward signals, how exactly are these Surrogate influencing modelling capabilities at both macro and micro levels?

The macro-views of the panorama and breakthroughs: from TTS to the edges of power

To understand the macro-wideness of the large language model, we cannot go around the thought chain (Chain-of-Thought, COT).

The current dominant large-language model of reasoning, whether visible COT or implicit, can be broadly classified as one. Test-Time Company (TTS, extension calculation at test)I'm sorry. Its theoretical origins can actually go back a long time ago -- for example, the idea of a daptive comparison Time (from the time of adaptation) and Ling et al. (2017), which was put forward in 2016 by Alex Graves, to lay the groundwork for solving the problem by generating distribution steps.

COT, why is it working?

If you follow it Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning The value of CoT is not mysterious when looking at modelling on a controlled two-hop general task. Author first writes the distribution of the test as a mix of ID and OOD; the ID part has been fully formulated, i.e. $D_{\text{KL}}(P_{\text{test}}^{\text{ID}} \Vert P_{\text{train}}) \to 0$ The system's expectations are largely dominated by OOD components:

$$ \overline{\text{error}} \leq \sqrt{\frac{2R^2\alpha}{N} D_{\text{KL}}(P_{\text{test}}^{\text{OOD}}(Y \mid X) \Vert P_{\text{train}}(Y \mid X))} $$

Here, a close line is needed: this is not a general theorem for all CTT tasks, but rather a model and error that the text gives above the set of controlled combinations. And the core instinct is that if the model is just learning a rough one, $X \to Y$ Map, and once the subject is changed to a new combination mode, OOD dispersion becomes a bottleneck, making the overall KL dispersion completely uncontrollable.

How to break the trap? The idea of this paper is to introduce a middle-level reasoning step. $C_i$。

Inducting $C_i$ Then, the original condition probability. $P(Y \mid X)$ It can be broken down into $\sum_i P(Y \mid X, C_i) \cdot P(C_i \mid X)$Accordingly, the upper boundary of the general error was also broken down into two more detailed parts: First, from $X$ Generate middle reasoning $C$ The second is the middle reasoning. $C$ Towards a result $Y$ The expected dispersion.

$$ \overline{\text{error}}^2 \leq \frac{2R^2\alpha}{N} \left[ D_{\text{KL}}(P_{\text{test}}^{\text{OOD}}(C_i \mid X) \Vert P_{\text{train}}(C_i \mid X)) + \mathbb{E}{C_i \sim P{\text{test}}^{\text{OOD}}(C_i \mid X)} \left[ D_{\text{KL}}(P_{\text{test}}^{\text{OOD}}(Y \mid X, C_i) \Vert P_{\text{train}}(Y \mid X, C_i)) \right] \right] $$

In short:COT intermediate step by visible generation $C$, converts an OOD combination that was hard to push directly into several local sub-issues that are closer to the ID distribution and easier to control dispersion. If you move your perspective further, you can use the term COT as an inspiration: intermediate steps do not necessarily appear as an output token, or may be encoded into the hidden layer.

RL How to extend the edge of the power?

When we know that TTS can break down the difficulty, a final question that has been controversial in recent years: do we invest in mass-based intensive learning (e.g. PPO and GRPO) or do we learn the new reasoning from the AC model (upgrading the ceiling pass@128) or simply train it to learn more in order to find the answers it understands (upping the pass rate of pass@1)? That's...RL Whether or not to expand the reasoning of the underlying model。

Paper by Carnegie Mellon University (CMU) On the Interplay of Pre-Trai The answer given through rigorous control of variables is:There's a dynamic boundary of a phase game.。

RL offers an incentive to be a Surrogate and the effects and models of this Surrogate are relevant to the mission.

For a mastered ID task: Irrespective of the increase in enhanced learning, the main reason for the increase is the first hit rate (pass@1). RL at this time is more of a use of existing capabilities, and even ablaze collapse; it does not necessarily expand absolute capacity boundaries (i.e. pass@128 is not upgraded).
For a completely unknown OOD job: RLs that are trained back to back are still powerless because RLs do not inject new knowledge if they are not exposed at all to the corresponding atomic computing operation (even if only 1% exposure becomes a seed).
Ccapability EdgeRL, this powerful Surrogate It's the most visible at the edge of the power.I'm sorry. The edge of the ability is the hard task of models that are difficult to succeed on pass@1 but that occasionally succeeds on pass@128 in many experiments. Large-scale intensive learning is more likely to be substantially enhanced by the design of training signals for these targets to the mission, as well as by the extension of the pass@128.

This study addresses the contradictions of the previous study: Some studies suggest that RL only causes a collapse of entropy, but others suggest that RL expands the edge of capacity (e.g. RLVR) because of different areas of the training difficulty spectrum. For distributed tasks that the underlying model can solve, RLs can hardly provide advantages as the value k in pass@k increases and the apparent expansive increase is observed when RLs aim at the poorly performing out-of-distribution tasks of the underlying model and RL data are close to the current model 's capacity edge.

Well, let's just say...Design RL data around the power edge of the model;that is, filtering RL data sets, a task where the focus model failed on pass@1 but succeeded on pass@k. This strategy avoids redundancy training on high pass@1 missions while preventing the issue of incentive thinness on zero pass@k missions. The process can also be iterative: the capacity edge task pool is reassessed periodically, and as model capacity increases, the original out-of-distribution tasks will gradually enter the solvency zone, and the re-evaluation of the capacity edge is sufficient.

This RL approach has gradually become the consensus for current training: using layered training data, rather than static distribution data, the ability margins are taken into account when training data are being cleaned.

If needed, the question of whether to answer is included in the incentive design, and the refusal to answer is rewarded to discourage illusions.

However, the above conclusion also has an important premise: the assessment indicators you use are really measuring “the boundaries of reasoning”, rather than “the ease with which answers are guessed many times”. This is particularly critical in the context of pure mathematical questions and answers, as the final answer is often short, in a fixed format, and the probability of multiple sampling is rapidly magnified. So, Pass@K on a math mission, sometimes it's not just measuring capital, but also confusing capital.

When Pass@K is no longer reliable: Cot-Pass@K, LLM-as-Rubric and COT awards

MRRA Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs And it adds this story from another clue: it complements the CMU perspective on the Capability Edge, not the CoT training mechanism. Is it possible that the power-only RLVR is more correct to hide the preference?I'm sorry. The author concludes that enhanced learning with results awards alone can improve modelling capabilities in mathematical reasoning and code tasks, while also exposing the limitations of Pass@K in a purely mathematical scenario.

The author re-examined Pass@K in Pure Math Query, pointing out that the underlying model, in missions like AIMS, which have limited space for answers, could indeed “crash” the right answer by sampling many times; they then suggested that CoT-Pass@KThis sample is a real success only if the final answer is correct and the entire COT is basically right. And with this indicator, the RLVR's ability to expand on a mathematical mission is again visible. In contrast, in the code task, correctness is verified by the program, and the "specify" space is significantly compressed, so the traditional Pass@K is still closer to realising sounding.

To approximate whether a COT is viable, the paper uses a rather light LLM-as-rubric: a more mathematical reasoning model to examine the full solution, rather than just the final answer; and multiple separate scores of CTTs. any-correct、majority-correct and all-correct Three methods of aggregation reduce the noise caused by a single miscalculation. It is certainly not a perfect decision, but it is sufficient to distinguish between “the answer happens to be right” and “the reasoning itself is right”.

There are three different things to be done here: first, just reward. final answer . RLVR; II, a single reward for the whole COT; and III, a process reward model (PRM) for each token / step. The main experiment of Microsoft's paper proved the first thing:Even if you look at the answer, the answer-only RLVR is statistically more statistically correct than the COT.I'm sorry. And this blog is a second-level potential: it is not a traditional token-level process award, but a single reward, but a goal that extends the final answer to the correctness of the entire COT. In pure math questions and answers, this single COT-level award is closer to the behaviour that we really want to optimize than just the final answer.

This also corrects a common misconception: what is really supported here is not that the result reward has been equated to a visible process monitoring, but that RLVR, which is only looking at the answer, has implicitly improved the quality of the COT, given that the model already has a logic project. And if we go further and directly incorporate the CTT correctness itself into the design of the reward, then it's the more advanced sorrogate design that suppresses shortcuts, guesses and rewarding with a signal closer to the quality of the reasoning. The step-wise or token-level incentives are a further way to discourage shortcuts, guesses and rewarding.

Interaction between pre-training, mid- and post-training phases

Now that RL gains are often on the edge of power, how does a big model get to this border? This is dependent on the interface between Pre-Training, Mid-Training and Post-Training phases.

Pre-training: no exposure without generalization While RL has a very high capacity for border extension, it does not in itself instil new knowledge in models, but rather pushes the edges of capacity outside existing knowledge. The blogger says:If the pre-training stage is completely zero for a concept or long-term context, no matter how much follow-up is, RL will be unable to induce the migration of the unknown.I'm sorry. However, even a very small amount of relevant exposure (such as the most basic atomic computational footage) in pre-training can provide critical capacity seeds for subsequent RLs. With this seed, the basic model acquires the necessary initial reasoning language, and subsequent RLs will be able to refine it precisely and achieve a robust cross-linguisticization. Thus, the role of pre-training is not to be closely developed in church models, but to spread the widest possible coverage of cognitive and basic concepts.

Medium-term training: connecting the original language to the edge of the ability Between mass data and precision strikes from RL, which cut across pre-training, the current industry is gradually realizing that the introduction of the medium-term training phase (Mid-Training) is a highly influential hidden lever. The main objective of the mid-term training, which is based on reasoning, is to establish the bottom lines: it can significantly enhance the suitability of the model for subsequent RLs, using a moderate distribution of resources and links, and consolidate the underlying language for stabilizing the underlying language. At this stage, the data structure has begun to aim towards the edge of capacity.For Mid-training, the selection of high-quality, well-targeted field data is generally done through continuous pre-training on next-token predation, paving the way for subsequent back-training.

And this involves a hard accounting game based on constraints. How should the share of resources for medium-term and post-training (RL) be allocated under a fixed calculation budget?

Core competencies guaranteedIf the primary task of current delivery is to ensure high performance of the tasks assigned to the ID, the budget should be calculated in favour of medium-term training, followed by a lighter RL to strengthen the established perception.
Expansion of external capacityIf the intention is to climb the more difficult distribution (OOD) in general terms, then only the reduced budget for medium-term training “just enough to establish the necessary a priori” will be left to the most critical of the calculations in RL exploration.

SFT after training is the same as RL to improve

RLVR further validates the homogeneity of RL and SFT in the surrogate perspective. It gives a very simple sense of speech: if pre-training has given the model basic knowledge / logic prior, then it's not a good idea.Right, COT is easier to direct than wrong CTT to the right answer.I'm sorry. So even if RLVR only looks at the final answer, the gradients tend to reinforce the cleaner reasoning paths more often than expected. In other words, the answer-only RLVR is not just to sample the answers faster, but it will gradually wash the COT in Rollout more like learning data.

The practical significance of this is very straightforward: these COTs, which have been improved by RLVR and are validated, should not remain on-policy rollout, but can also be recycled into new SFT data. It was observed in the paper that using these COTs to do SFTs would allow for a considerable replicability of the training model. This can be seen as a cheaper offline/off-strategy learning loop:Pre-Training The blogger says:Mid-Training Organisational languages,RLVR Sifting better reasoning tracks on the edge of power, and SFT And then you take these high-quality COT sedimenting models.

In conclusion, RL wants to play its irreplaceable role by ensuring that the target mission is not so rigidly recaptured as to remain exploratoryly flexible, and that training data are on the edge of the current model; and that, in the longer run, SFT can continue to recover the quality of RLVR-generated resonating data.

This is also closer to the distribution of water lines behind today ' s top model: pre-training provides exposure as much as possible;Medium-term training (Mid-Training) (b) Build common cognitive bridges and strengthen specific reasoning;SFT. (b) Provide better cold start-up effects;RLVR Better reasoning is sifted on the edge of capacity; eventually, model capabilities are given training for the next stage.

And here, the macros answer mainly what the goal is to reward; the next question is: How does it go through the token fork, the stylium and the geometry of the parameters, even if there is only one seemingly rough reward?

Motivation of micro-views: tacts, tokens and three theories

The macros discuss the data and mission boundaries, while the micros look in two steps: first, see how the COT training is going to put the reasoning steps inside the model, then see how RL changes model behaviour through entropy management, token split fork and parameter geometry.

COT is how to internalize into two-stage circuits.

Look first. Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning Its own most unique structural analysis. The author observed one with logit lens and causal traffic two-stage compositional circuit: Under visible COT training, the model will first decipher the intermediate result of the bridge at a shallower level, and then hand over the intermediate result to the subsequent layer to complete the next jump.

The significance of this matter is not just “explainable”. The intermediate results are read at a more shallow level than training without COT, which suggests that the capacity to process subsequent steps is still retained at a deeper level. So, the one in the paper that was very popular -- Cot -- it was more like teaching models. how to think♪ And not just ♪ what to think- It's a place to put, but the language needs to be tightened: It is a combination of training under a visible step, not an unconditional law of all the reasoning.

The author also linked this set of observations to a limited reality experiment: when COT SFT was on MetaMathQA, and then on GSM8K, CTT was significantly better trained than the answer-only SFT; and when the reasoning steps were mixed with some noise, the model retained a considerable capacity to generalize. So the answer to the question was the internalization mechanism of the COT and limited noise.

The double-edged sword effect and the law of experience of the swarms.

If the perspective is cut to the point where RL for reading, the entropy in the information theory becomes critical.The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models It is this thing that is discussed: when the model tastes sweet for the first time in an action with high expected returns, it quickly increases the probability of this choice and simultaneously lowers the probability of other possibilities. When this feedback cycle is running too fast, we'll meet. Policy Entropy CollapseAnd that is, the model is becoming more confident and less willing to explore.

The policy entropy (average token class) is used to describe uncertainties in the probability distribution of token when model output is performed. Defined as: I'm sorry. =-\mathbb{E}{D, \pi\theta}[\log \pi_\theta(y_t \mid y_{<t})]. $$

Intuitively, the model will further strengthen the choice when a high probability action is rewarded, thereby reducing the overall entropy; on the contrary, when a low probability action is unexpectedly rewarded, the model will raise its probability, thereby increasing the overall entropy.

The paper shows that changes in the policy entropy are closely related to the update of the logit driven by the advantage, both theoretically and empirically; the author also observed a similar relationship:$R = -a \exp(H) + b$I'm sorry. It points to a very realistic conclusion: a lot of RLs are improving, actually, consuming exploration entropy. Without additional interventions - such as the KL-Cov selective constraints discussed in the text, or more traditional entropy normalization - models can easily become obsessed with local high-point shortcuts, leading to short-sighted behaviour of rewarding.

Token and the tokens are split.

So, when the COT grows, every token forecast has the same strategic weight? Apparently not.

A recent job. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning Token level has been fined. The author found that the distribution of entropy in the COT rollout is highly uneven:

Very few "high-pane" Token: They are usually located at the branch fork in the logical route, at the hypothetical inverted point or at the critical operational nodes. Each step of the selection here may lead the model to a completely different branch of reasoning.
Most of the "Low Americ" Token: They are more responsible for the cushioning, completion of the connecting words or the continuation of the existing expression, where the model is often well established.

And that's why all tokens are likely to have a resource mismatch back at the same time. The core of this paper found that if the strategy gradient is concentrated on this small group of high americium token, it can be brought closer or more than full token. It supports the idea that the RLVR validity is mainly from the key fork token, rather than the COT training mechanism itself.

Three-Gate Theory: RLVR Gos to Slender Update

The previous section deals with entropy and fork in token-level /policy-level; this section turns to another category of questions: how geometry of parameters sifts RL updates into thin, localized and undominant changes.

When we look at specific, cost-effective, intensive learning (e.g., RLVR, which is focused on program-based certification incentives in Meta AI), we find an anti-intuitive phenomenon:This RL method causes more variation of parameters than simple SFT and is concentrated in a given area. The question is, why is this updating of the matrix of neurons only for a few?

Meta's research paper "The Path Not Taken: RLVR Provably Learns Off the Principals"♪ With ♪ Three Theory (Thri-Gate Theory) An explanation framework is given.

Gate I (Tactical Containment, KL Anchor): At each round of the updated port on-policy, KL anchors impose a limit on the pace of the update. It ensures that each attempt is pre-trained in the radiation belt of the model.
Gate II (model geometry door, Model Geometry): Large language models have formed a large "Gavid Landscape" parameter matrix when pre-trained. In this space, there are some main directions (i.e. the most curvature, the most dense passages of the original knowledge weight of the model). When the RLVR gradient begins to modify the parameters, the bottom geometry of the model makes it easier for these updates to avoid the main backbone (Off the principal directions) and to shift to subspace margins with a lower curvature and without a significant disruption of the given spectral feature. This explains why particular models favour certain fine-tuned paths and why this preference is associated with pre-trained landscapes.
Gate III (Accuracy mask, Precision): Ultimately, because of filtering restrictions on the storage of low precision parameters, very minor changes that did not otherwise yield geometry were erased at this stage. This leads to the eventual appearance of RL changes at the observation level.Local and thin。

Screechness is not a conclusion, but is the result of the limited precision of the parameters and the selectiveness of their updating. Changes in parameters for those new areas that were not selected were hidden with insufficient precision.

This power mechanism reminds us that, as we move to a post-training era, the direct replica of the SFT-era PEFT (high-impact fine-tuning of parameters, for example, reliance on low-stealing criteria, LoRA) may be unstable. The traditional low-sovereign re-enactment path may not fully match this need for a bias in the direction of the main curvature, the edge of the high americium, and minor adjustments. The new efficient approach to parameters requires consideration of how to protect and use this part of the fluid.

End: Different agents, different data, different targets

So, big model reasoning doesn't have to be understood as magic. Pre-training, mid-term training, SFT and RLVR use data and loss agents at different stages to sustain plastic modelling capabilities.Modelling capacity improvements rely heavily on more appropriate design, Surrogate, to make target signs more accessible for learning.

But the goal of the big model should not be just to reason. Whether the tool is used, role-playing, memory modules or planning capabilities, the available capacity in the parameter space may be gradually transferred, as long as there are more suitable sorrogate coordinates, without necessarily waiting for a completely new algorithm. Algorithms are determined many times by the speed at which we discover and stabilize these capabilities.