Agentic RL: Why the Training Loop Matters More Than the Algorithm

This English version was machine-translated from the Chinese original. Technical terms may require verification.

PreviousThe conclusion was that reward was not just a line of fractions from a model, but a link of interfaces that converted the target into a training signal. This one is about:When this link is caught in the training cycle of angent, what do the training systems need to fill in?

Over the past year, many of the teams that made substantial progress in the direction of the angent have moved along similar lines: not just get RLs to the tools, but reconnect the whole chain from data, environment, feedback to training.The issue of the Agentic RL is not a single algorithm, but a systemic one: can you connect tasking, environmental contracts, feedback counters, cold start-up, online exploration and deployment backflows into a closed training loop that can last in one order?

This article begins with a picture of this closed circle. Next"How Reward and Training Closed in Real Age"This will be done in specific systems such as AMAP, Agent K, PaperSmart.

Why SFT and the traditional RLHF crashing walls in Agent scene

Let me first look at a concrete example.

Suppose you do SFT with high-quality demography, training a code agent. In the assessment set, its first tool is called with a high accuracy -- it's a good choice. read_file、search_code、run_testsI'm sorry. But once the tool returns to an unexpected result (e.g., failed tests, empty searches, documents do not exist), its behaviour becomes rigid: either repeat the same call or jump directly to me to help you sum up and give up the task.

It's not a question of capacity. SFT data does contain samples of recovery after the tool failed, but demontation givesPathshot——"That's what we should do in this case."I'm sorry. Models learn the mapping of a given situation to a particular action, not a specific action."When to change strategy, when to drop the current course and explore other options."This decision-making process itself.SFT can teach imitation, but religion doesn't judge.

And look at RLHF. You trained a reward model with preference data, and then you did a round of optimization with PPOs. The expression of angent has indeed changed — more natural, more standardized, and more user satisfaction. But if you put it in a real task that requires 5-10 steps to interact, the improvement almost disappeared. Why? Because tradition RLHF is closer to one. T = 1 degradation MDP: give a prompt, generate a response, give a preference score at the end. It optimizes the quality of a single response rather than a multi-step strategy. It has no idea how important step 3 tool selection is for the success of step 7 or how important it is for the final success or failure to be distributed back to the middle of the decision-making process — that is the problem of credit agreement.

That's the value of Agenic RL in antect scene. When angent needs to continue to operate in the environment - read the state, choose the tool, revise the plan based on feedback, recover in error, stop when appropriate - the training target is no longer the target."Generate good text"And...You're going to walk in the environment.。

Yeah. DeepSeek-R1 The final complete recipe also illustrates this: it is not a simple RL-only, but a cold-start SFT filtering and SFT & RL multistage loop. SFT alone is difficult to cover the exploration process, and traditional RLHF alone is difficult to handle; models need to be explored in the environment and updated with environmental feedback.

Change in target audience: from text to trajectory

The example above already implies one thing: the difference between angstic RL and traditional post-training, not only the many tools to call, but also the fact that the two of us are not just the same.The target of training has been changed from a single answer to an environmental trajectory.。

Traditional reference-based reinforce fine-twining T = 1 MDP: Give a prompt to generate an answer, end point. And angtic RL is onePartly detectable, time-sequenced, and real-life transfer POMDPI'm sorry. The state is no longer static prompt, the action is no longer just token; the action changes the environment, creates new environment, delays return along the trajectory, and creates an indication from"Did you get the last sentence right?"I'm gonna..."Which step really contributed to the mission."。

Once this is clear, a lot of it would look like it."Functional Enhancement"And things are easier to locate. Planning, tool use, memory, self-improvement, reflection, and even the various forms of antgent in different mission areas are all related to the same target audience: not just looking at it."Single answer quality"I'm looking. Behavioural quality when you walk a full track in the environment。

A more specific example: a code agent first used to fix bug search_code Position the relevant files, and then read_file Confirm context, then edit_file Change, last run_tests Authentication. If the test fails, it needs to be judged — whether the modification itself is wrong, whether the test environment is problematic or whether a completely different rehabilitation strategy is needed. Every decision in this entire trajectory is a subject for policy to learn. The traditional post-training is only a last step in the right direction, and it is difficult to evaluate the quality of the intermediate path.

Agenic RL is not just an upgraded version of RFT or an old RL for tools to call. The question here is not whether the model will call the tool, but...The goal of training has been changed from optimal response to strategic learning in the environment。

Train in the full view of the flying wheel.

If you want to summarize a link to the real world, you can see that the training chain is the shortest form possible:

graph TD
    A["Raw Tasks / Logs"] --> B["Curated Task Pool"]
    B --> C["Environment Contract"]
    C --> D["Verifier / Reward Stack"]
    D --> E["Seed SFT"]
    E --> F["Offline Shaping"]
    F --> G["Online RL Rollout"]
    G --> H["Replay / Curriculum / Environment Scaling"]
    H --> I["Distill / Deploy"]
    I -.->|"经验回流"| B

style A fill:#f9f,stroke:#333
style I fill:#9f9,stroke:#333

The most important thing in this chain is not how fixed the order is in itself, but rather it reminds us:Now, reward, not a single line, RL, not a single line."Let's get back to the training."。 In the system, rewards are often data filters, and then training signals; the environment is not a benchmark companion, but a part of the target; distillation is not a final rehabilitation, but a necessary component of deployment strategies to sink high-value behaviours found in rollout.

The following is the sequence of each layer of the chain.

Data governance: the true step 0 of the training closed circle

Many talk about angent training, and default that there is already a body of training data ahead. But for real anent systems, this is often the most unstable assumption. Because what the world gives you is usually not neat. (prompt, response), but a bunch of messy query, historical logs, tool call records, user fixes, failure tracks and incomplete environmental conditions.The first step in the training closed circle is often not optimization but collation.

The data in the angentic RL is not just a text sample, but a cut in the taskspace. You need to know first which requests belong to the same task, which constraints are the main difficulty of the task, which requests, though high frequency, should not be handed over to the current angent, which failures are low-value noises, and which failures are the most worthwhile restoration samples to retain. Without this step, the back of the reward, curriculum and online RL can easily just spin on the noise.

At least four things must be done first.

First, cut the mission pool first. You need some kind of taxonony to distinguish between type of mission, binding density and capacity boundaries. Not because the classification itself is elegant, but because you have to discuss the coverage, long tails, the distribution of difficulty and the curriculum, all of which are premised on the fact that the mission space is first cut into a manageable shape.

Second, it's difficult to write a job. The difficulty is not subsidiary labels, but the foundation of the back of the curriculum. A system that does not know what is simple, what is long-range dependency, what is multi-tool coordination, what is high-barrel, low-intensity error is often difficult to understand what to learn first, what to consider failures as normal explorations and what to consider failures as breakdowns of agreements.

Third, negative samples and border samples must be retained. A lot of angents don't do things, but don't say."I shouldn't have done it.", will not admit that information is inadequate and will not stop when it goes beyond the instrumental boundaries. It is also for this reason that the un-of-scope request, the unsolved request, the hazardous request and the high-value failed sample should not be simply cleaned out. They are part of the training foundation behind refusal, recovery and trustworthy.

Fourth, to harmonize the projectorry scheme as soon as possible. What does a trajectory cut by, either utterance, tool call, code exception, environment transfer, or more abstract step? If it doesn't come to that, the back of the question will float. A lot of systems get more and more messy back there, not as an optimizer bad, but rather as a removable, verifiable, learning track.

Data governance is not a subsidiary cleansing process, but is the 0th step of the underlying RL. The problem is not the sample is enough, but...Did you make the world a training pool and a track language that can be replayed back and forth?

Environmental contracts and feedback counter: the infrastructure level of Agent training

When the task pool was organized, the next step was not to go to SFT immediately, but to write the environmental contract clearly. An important difference between angstic RL and normal post-training is that:The environment is no longer merely a stage assessment, but a part of the target audience. From agentic RL document for verl Go to home and train repé, this has been repeatedly verified.

Environmental contracts are the smallest but rigid training interface: what the model can observe, what actions are text, what actions change the outside world, how to move output, how to go into context, how to reset, what fail code, which side effect allows to emerge, and which must be restrained by Sandbox.

There's a piece of engineering that can be easily underestimated.angent loop document for verl Repeatedly, async rollout, snicky session, message vision and token-level discussion are all telling the same thing:Once angent starts to rotate more than one tool, rollout no longer equals the simultaneous generation of a text. If the trajectory boundaries, the message returns and the token alignment is unstable, the training will be de-factory at the very basic point.

The feedback should also not be aggregated into a total score, but rather into a layered stock:

• verifier Part responsible for procedural validation, such as whether the test is passed, whether the answer is met, and whether the format is in compliance;
• process signal Responsible for whether intermediate steps advance tasks such as tool selection, completion of sub-targets, effectiveness of partial restoration;
• judge Responsible for the open part of the verifier that is temporarily not covered but which has to be assessed;
• trace / audit It's for you to know after training what the model is."I won't."、"What a mess."Or is it?"They're rewarded in the wrong place."。

Many feedbacks should be used to define environmental boundaries and data gates, and then to access the optimizer.

Trustworth is not the last security clause to be filled. The attack face of angent was much larger than the normal LLM from the beginning - tools, memory, external API, web pages, databases, cross-ent communications, all becoming part of the state shift. Sandbox, permission boundaries, auditable track and refusal policy should be written in from the environmental contract stage instead of waiting for RL to learn the dangerous strategy before patching it.

Cold-start SFT and Offline Shaping

It is only when the environmental contract and feedback bond are steady that the cold-start SFT becomes meaningful. It's not about repeating what the model already does, but...The first step is to write a priori, basic rhythm and interactive protocols into the policy.

And many times, you write SFT and RL into a relationship that is more than good, as if the first demontations were irrelevant as long as the line RL was strong enough. But the more you look at the real system, the more you think it is a single-wheeled task imagination. For angent, SFT is not brought in abstractive powers, but specific behavioral habits: when to read the observation, when to call the tool, when to ask, when to try again when the tool fails, and when to stop exploring and delivering the results.

DeepSeek-R1 The final complete recipe also illustrates this -- it's not a simple RL-only legend, but it goes back to the close of the cold-start, filtering, SFT, and RL. This is particularly true of systems in the tool environment: if models cannot even be legally called, online RL tends to only make it waste a large amount of rollout budget in the thin noise.

It's worth taking out the SFT alone. Many systems require not a long-distance search, but a more manageable distribution of strategies: format revision, partial agreement compliance, language stabilization, factual confirmation, short-range recovery. The DPO, reward-guided reporting, rejection Samling, and verifier-based reflow data all belong to this layer.

If only one sentence is retained:SFT is responsible for sending models into learningable areas, offline shaping to clean up this area. Many"Back there, RL. It's steady."The system, not suddenly learning to explore, has cleared unnecessary noise and low-level errors.

Online RL: Exploration, Recovery and Timing

It's not worth it until the mission pool, the environmental contract, the feedback counter and the cold-start project are in place. It also has a different function in the angent than a single-wheel preference: not simply to make answers more like popular texts, but to make policy learn.When, when, when, when to switch strategies, when to stop, and how to recover mistakes in long-range interaction.

The most easily underestimated layer is not the algorithm name, but the rollout itself. Not a long text that's so simple for an angent rollout, but a series of interactive history of continuous and environmental exchange of events, action, feedback. Without reliable stept execution, information recovery, state tracking, masking and track, the so-called agentic RL can easily be left with only the acoustic ross, without a training system available.

I'm getting less and less understanding the problem of line RL. Done."Should I use the PPO or the GRPO?"And I'm asking three questions that are lower:

• Which token or action should really be trained?
• Where should a Credit track go?
• Which intermediate behaviours are worthy of direct process rewards and which should be settled at the end?

It's also more difficult for angtic RL than a single RFT. In single rounds, 0/1 rewards are often barely sufficient; but in multiple rounds, success is likely to be the result of several local decisions. A right choice of tools, a timely waiver, a valid recovery of errors is often more critical than a beautiful answer to the last sentence. Yeah. Agent Lightning rewrite the track-to-transaction, or GiGPO To break down a more detailed group-based advantage, all answer the same question:How exactly does a long track credit get back to critical local action?

At the same time, stabilization training is not just numerical stability. For angent, it also includes stability of rewardbackback, stability of tools to be used to absorb, stability of delayed environmental response, and stability of strategy to explore not to push systems into dangerous areas. From the beginning of the online rollout, training stability and secure borders are no longer optional topics -- they will decide directly whether or not the training will be sustainable.

If only one more sharp sentence is retained:Online RL is scarce, not giving the model a higher total score, but rather allowing it to learn to explore, recover and time in the environment. Only when your bottlenecks really fall in these places will the online RL be worthy of its expensive system.

The latter part of the ship: Replay, Curriculum, Envirronment Scaling and Distil Back

Once the model begins to learn in the environment, the second half of the chain emerges: experience goes back, the difficulty goes, the environment changes with the model, and finally how to sink rollout policy back into deployment. And it's here that it's only been the process of Agenic RL that has been transformed from a training to a sustainable iterative process.

replay The value is not just the saving of samples, but the determination of whether success stories, border failures and high-value recovery paths will remain in the system.curriculum And the value is not just from the easy to the hard, but from keeping angent close to the learning boundary, rather than being given a zero-sum direct reward for long-distance, thin, and too loud.

One more step, yes. environment scalingI'm sorry. I am now increasingly convinced that the cap of anent is often limited not only by the model size and training steps, but also by the training world in which it operates. The model will soon be fully developed if the environment is too poor, too static, too unsafe, too difficult; and an expanding, synthesizeable, processable environment will turn data and training into a more sustainable and iterative source. Yeah. VeriEnv This kind of environment effort is trying to open this bottleneck.

But the second half is still easy to underestimate. distill backI'm sorry. Many rollout policies are strong in training but not suitable for direct deployment: too costly, too long context, too heavy reliance on Sandbox and track, too much style to explore the middle, or not enough secure borders. So the more common and realistic route is to re-formulate the high-value trajectory, recovery strategy, stop conditions and tool call times from the line RL to more stable data, and retort the cheaper and more obscurantistic deloyment policy. It's not a last step in the training loop, but rather a final step in the training; without it, the high-value behaviors discovered in the training will remain in expensive rollout policy, which is difficult to serve.

The latter part of the Agenic RL is managed not only by continuing training, but by four things tied together:

• The experience is not sustainable.
• Whether or not the difficulty movement is attached to the modelled capability boundary;
• (b) Whether the environment will expand with models;
• The training taught high-value behaviours that could be used to sink deployable tactics.

It's written at the end.

If only this short sentence were retained:The difficulty of Agenic RL is not who first connects LLM to RL, but who can combine a training pool, an implementable environment, a layerable feedback, a re-orbitable trajectory, a sustainable environmental expansion, and a deployable retrenchment stream.

Looking forward, I think the two biggest bottlenecks that follow this link will fall. environment generation and scalable verificationI'm sorry. The former decides how much anent can learn, and the latter decides how much he can be trusted. When the environment can be programmed to synthesize and expand, when the coverage of verifier increases with the difficulty of the task, it is easier for Agenic RL to form a stable backstream of training.

This article is a map of the landscape. Next"How Reward and Training Closed in Real Age"See how this chain runs from the first line of code in specific systems like AMAP, Agent K, PaperSmart.