Synthetic Data as Engineering: Starting from Terminal-Corpus

Before the text

Recently read NVIDIA's 《On Data Engineering for Scaling LLM Terminal Capabilities》I'm sorry. It's about training data for the terminal anent, but it's not about the terminal itself that really attracts me, it's about the syntheses that are very engineering. Synthetic data are not generating more text, but rather a world where behaviour can be trained. Think first about what the world is going to offer, and then reverse tasks, the environment, trajectory and filtering.

The paper was based on a system called Terminal-Task-Gen, and the output data were called Terminal-Corpus, which was finally used to train Nemotron-Terminal with the SFT tracks. It brings together task generation, environment, teacher rollout, filtering, course learning, context and scale experiments. Reads comfortably because it does not give one result, but separates the training data lines from the one that was adapted from the old data set, which tasks are synthesized by skills, how the team runs, how the failed samples are processed, how the training is mixed.

This note only talks about SFT. RL will certainly appear at the second half of the training at the same time, but I'd like to think about the monitoring data. In many cases, the model has not reached the stage where complex online optimization is required and has been blocked by dirty missions, weak tests, chaotic tracks and hand-held data.

If you need to complete the SFT, LoRA, the apparent estimate and the preferences, you can get it fromPost-Language Training and Micro-Adjusting PracticeStart. This paper continues only along the line of synthetic data.

Define mission world, generate verifiable tasks, collect learning tracks and maintain traceable data lines. Prompt is still important, but it's just the entrance. The effectiveness of training depends on the stability of the world in generating, filtering and reflowing data that change model behaviour. There is also a set of assessments to prove that these efforts are effective.

graph TD
    A["能力地图"] --> B["任务世界"]
    B --> C["种子 / taxonomy / 知识结构"]
    C --> D["可验证任务"]
    D --> E["环境 / verifier / schema"]
    E --> F["Teacher rollout"]
    F --> G["过滤 / 去污染 / provenance"]
    G --> H["SFT 混合与调度"]
    H --> I["评估与错误分析"]
    I --> A

Start with the mission world.

I used to think of synthetic data as a collection. (instruction, response)I'm sorry. It was barely enough, but not enough from 2026's perspective. Once the real scenes of code, tools, terminals, data analysis, etc. are entered, the data is no longer just a paragraph input and an answer. Models are about how to do things in one environment. The environment itself has become part of the mandate.

The idea of this paper, NVIDIA, is to break down the training data into three objects. The first is the question, what the model is to do, what the input output is, where the boundary conditions are, and how the conditions for success are written. The second is environment, where the model can see the files, what dependability, where the commands are running, and how to start. And the third is, the projectory, a teacher who would do this, how to work in the environment.

The three objects are put together, and the synthesis of data is less like the prompt project. It's more like designing a small training world. The most obvious thing about this is that action and operation are in the command line. The traditional QA anent, RAG, role conversations and even entertainment chat data have their own world, except for the environment that has been transformed from Docker to a knowledge base, user portraits, role-setting, dialogue history and style boundaries.

From command to teaching process

Before looking at the projects that NVIDIA did, look back on the lines that lie ahead: how to activate the task pool, how to control complexity, how to make the data more learning, how to break the generation process into a detectable stream of water.

Self-starter: Self-starter

Self-Instruct The problem of starting points is addressed. When there is not enough manual infusion, it starts from 175 manual seed tasks to expand the model itself to the task pool. When each round is generated, it draws eight entries as context, six of which are from artificial seeds and two from data generated before the model. The model Mr. is new, then determines whether it meets the demand, and eventually generates input and output.

There's a little detail here. Common tasks are used input-first, output-first,First give the label and then generate the input, avoiding the general bias of the classified data towards common labelsI'm sorry. Then you can filter the unsupported pattern, bad format and repeated tasks with the rules and weigh them with the ROUGE-L similarity. It finally gets about 52K examples of regulation and 82K.

It is not important to let models synthesize data, but rather to do it. The minimum closed rings, seedpol, promotion loop, dedup/filter, are a good starting point. Its limitations are also evident: the complexity of the task, the skill mix, the state of the environment and the verifier are not well managed. Self-Instract is a good start, but it's not too short away from the latest engineering practice.

Complexity Editor: WizardLM / WizardCoder / Textbooks

WizardLM Turns complexity into an object that can be edited. Evol-Instract has evolved from the existing instructions: in-depth allows for deeper tasks, such as adding constraints, concrete scenarios, requiring multi-step reasoning and making input more complex; in-breadth generates new rare tasks in the same field. It also discards samples without information gain, model refusal, change of point or leak of prompt with input eliminator.

WizardCoder Move this idea to the code scene. It does not reproduce generic complex templates, but rather adds code tasks to their own evolutionary algorithms: time and spatial complexity requirements, boundary input, error reference codes, debugging and repair, performance constraints. This migration is important. Complexity is not an abstract word, but must be in the field.

Textbooks Are All You Need It's not always in the SFT Synthetic Data discussion, but it always reminds me that data can learn. Phi-1 training is not a simple stack of codes, but rather uses GPT-4 to mark educational values, filter the Stack and StackOverflow, synthesize textbook-style data and CodeExercises. It emphasizes clarity, inclusion, sequencing and balanced coverage. This is particularly true for small models, where loose large data are not necessarily more useful than small, structured data.

Diversity and complexity can be broken down into sampleable dimensions. For QA, it may be the location of evidence, the type of question, the multiple jump relationship, the non-respondable sample; for role conversations, it may be the user relationship, emotional state, context span and human boundaries; for triminating anent, it is the file status, the tool call, the test failure and environmental restoration. Complexity is not automatically the same as being of value, and complexity is seen in combination with validation, coverage and training.

Waterlineation: AgentInstract / LAB

AgentInstruct The most interesting place is that it didn't start with artificial infusion, but with Raw seems. Seed can be an article, a chapter of a teaching material, a web page, a code clip, or an API description. Often, general researchers do not have systematic data on them, but there is a large amount of raw material. What Argentina Instract is trying to solve is how to turn this material into a modeled mission.

The first step is the Grant Transport Flow. It does not rush into the issue, but instead changes the original material to a more appropriate intermediate representation. In reading understanding, a regular page may be recast as an argument passage, debate, review, meeting transit or long passage; in the tool/API scene, code segments can be organized as API description or API list. This middle layer is important. Many of the failed data are not poor-written, but material itself is not suitable for direct questions.

Step two is the "Sed Investment General Flow." It holds the middle expression, which is based on skills that generate a variety of tasks. Tuxonomy here is not a post-post label, but a sampler. Read understanding allows for the adoption of topics such as as asmption, flaw, reference; text modification allows for the use of paraphrasing, simplification, reaction, style transfer; tool/API uses searchable, callable, combined, multi-wheel interactive. In other words, Taxonony decides where the model will be. Lee."Practice"。

The third step is to develop a new approach to the issue. After the initial assignment, you can give it to a character like Suggester-editor. The suggester suggested how to make the task more difficult, more reductive, less answerable or how the question might require more steps; the editos then put the recommendations to specific tasks. The re-election here is not simple tweaking, but continues to explore in the mission's neighbourhood. It makes the data more than just overwhelm."It's a standard question.", also covers boundary, trap, tool and multi-problems.

LAB The order is more like an alignment data factory. It goes on to take the texonony and divides the data into three branches: knowledge, foundational skills and general skills. Knowledge is oriented towards documentation, manuals, teaching materials and field knowledge; foundational skills include mathematics, code, language competence and reasoning; and computer skills combine knowledge and basic skills to complete complex user requests. A small number of manual examples are placed at each leaf node to serve as anchors for generation. This design is for"What can I do?"It becomes a table that adds or deletes nodes, not a vague wish.

After having taxonomy, LAB did two more types of SDG. Skills-SDG takes four steps: Mr. enters into evaluation, then produces response, final evaluation-response level. Knowledge-SDGs then move the model to documents, manuals, books, and ask questions from outside material generation areas, rather than simply learning about the parameters of the teacher. This is more stable for professional QAs, the business knowledge base and any area that is easy to hallucinate.

LAB has a warning in the order of training. It first does knowledge training, divides knowledge and foundational questions into two stages of short and long-response training; then does skills training, training more complex capabilities with generic questions and using replayers to bring back data from the previous phase to reduce forgetting. I'm just thinking of it as a SFT data-motion idea: first, to get models to know their knowledge and basic actions, then to train their skills, and not to flush out what they have learned.

When you do your own data, you can write the water lines. 素材池 -> 中间表示 -> 任务生成 -> 任务改写 -> 回答生成 -> 质量门 -> 训练混合I'm sorry. Every step leaves metadata and the reason for failure. Or you'll see it in the end."It's not working."The blogger says that the government is not aware of the evils of the sighted content, mission description, teacher response, critić, or mixed training.

From overlay control to data exhibition

The second group is closer to the current data mapping exercise: how to get data to cover the target, how to achieve dynamic difficulty control and course learning, and how to understand the value of wrong samples and data scales. The answers given in different studies are not entirely consistent.

Structured sampling: GraphGen / CONDOOR

GraphGen Faced with a closed-book knowledge-inputative SFT. The random generation of QA is easy to misdirect, long tails, and shallow. It draws entities and relationships from source languages, builds knowledge graphs, then makes interpretation assessments for true, paraphrased, Negated states, etc., to estimate the degree of knowledge ownership of the knowledge margin. Then the border sampling around the model is unstable or under-covered k-hop subgraph, generating atomic, aggregated, multi-hop QA.

GraphGen did a very solid job of digesting. It has been melted in four groups, each of which answers a clear assumption: which parts of KG are really valuable (entity-only vs relation-only vs full KG), whether training data should be aimed at the unstable areas of the station or whether they should be fully sampled (high-loss data are better), whether the output length is confusing with variables (not related to length) and the impact of the edge selection strategy (less variation between the three groups, honest reporting of nululsult). The digestion is not about the number of experiments, but about letting the reviewer see which components of his method are working and which are just set.

CONDOR With World Knowledge Tree. It expands on a large number of topics, then combines themes and common chat scenes, such as Daily Chat, creation, roll-playing, and controls the difficulty. Generate and then do Self-Reflection Refeightment, which allows the model to evaluate the strengths and weaknesses of the answers and then rewrite the answers.

These two articles are particularly enlightening for QA angent. When QA data are synthesized, the problem is often taken from where the problem is written earlier than the problem is. Random document clips bring random QA; knowledge maps, theme trees, student panels and user scenes are combined to be more like training programs. The limitations are also here: the maps and the trees themselves are biased, and the poor construction will systematize the deviations.

SWE-smith / OpenCodeRessoning / CHIMERA

SWE-bench usually looks for PR/issue, then goes back to the historical version.SWE-smith In turn, the real Python warehouse is being built to operate the environment, and a lot of bugs are being built in this environment. It creates a patch that breaks the existing test in four ways: LM Modife, LM Rewrite, Procedural Modification, PR Mirror, and only retains the candidate that triggers the Fail-to-Pass test, and then uses LM to write a question description of GitHub issue.

This is a similar thing to the Docker design of Terminal-Corpus: fix the world, then build missions in the world. When the environment is stabilized, failure is more likely to be attributed and storage and maintenance costs are low. SWE-smith also warned against exposing the verifier too bluntly. The provision of the Fail-to-Pass test will make it easier for the subject to be solved, but students may learn to skip the recurrence process and meet the test directly.

OpenCodeReasoning The value of the wrong sample is being reconsidered. It collects 28,904 unique topics from TACO, APPS, CodeContests and CodeForces, and uses DeepSeek-R1 to produce 736,712 Python researching samples. However, the results were somewhat surprising: the model performance declined after filtering out samples that had not passed the unit tests. This does not directly indicate the usefulness of the wrong sample itself, and is more likely to indicate that filtering changes the difficulty of the subject, the coverage or the trajectory distribution. One possible explanation is that the failure sample covers more difficult questions, the result and the NV ' s projectory response: correctness is an important signal, but not the whole value of training.

CHIMERA It's a counter-alarm on a scale. It only constructed about 9,225 articles of computer science research data for cross-science reasoning. The process is to expand from the rough discipline to eight disciplines, 1,179 topics, to generate self-contained and verifiable questions, cross-validate with multiple models, and finally keep the correct trajectory for SFTs using thinking model synthetic solution tracks. It indicates that small data can also be valuable, provided that the density of coverage and quality of validation is sufficiently clear. The fact that the government is not a party to the law is not a matter of the right opinion. The former illustrates the efficacy of organized and validated small-scale data, while the latter reminds us that radical correct filters may be removed along with difficult samples and useful coverage. Filtering and data reorganization itself would introduce the bias, so each cleansing rule would need to be justified and checked how it changed the distribution of data.

Back to Terminal-Corpus: Data Synthesis Project

Now go back to the Nvidia article. Terminal-Task-Gen does not generate the subject by a large prompt mass, but instead puts data sources, task formats, operating environments, teacher behavior and subsequent filtering on the same line. The environment is closely related, but not only to the environment. Role-Play, QA, RG-Sys, Chatbot have their own environment, hidden in chat records and a priori world knowledge.

Wide and deep, too.

The paper can be presented in two broad layers. The first level is the dataset operation, which is connected to the various types of task data available of high quality, rapidly spreading the basic capabilities of mathematics, code, software engineering. The second level is the task of synthesizing the task closer to the mission with the sighted and the pre-mititic skills. The former addresses size and breadth, while the latter addresses skill mix and depth.

Directly synthesize all tasks from zero, which is costly and difficult to know if coverage is balanced; only old data are modified and the format and depth of the original data set are limited. The NVIDIA approach is to spread the foundations first and then fill the depths where skills are lacking, tools are lacking to interact, and environmental conditions are lacking. Many of the judgements in the data project are in this order: what is available for cheaper reuse and what must be redesigned. This is the part that we have to consider when we're thinking about building a data set.

Especially when we use this idea of data synthesis, we must maintain Schema in a sufficiently complete way, and task_id, source, difficulty, generation of the task information. Different sources need to be absorbed, and different difficulties need to be addressed, so metadata should be recorded as fully as possible during the synthesis phase. Save Only (instruction, response) And a simple form, early savings, later check-ups can be painful.

Dataset tasks: connect task

Let's talk about the dataset adapters. Thesis selects the three categories of math, code, SWE prompt. Math, from Nemotron-Cascade, Math responding SFT Stage-2 policy set, approximately 163 K unique issues; this data removes easier questions from the front processing, such as DeepSeek-R1 response of 2K tokens. Code from Nemotron-Cascade code researching SFT data, original collection 79K prompts, further filtering and re-resulting to get 35K. SWE data from Nemotron-Cascade SWE code repair SFT, which includes SWE-Bench-Train, SWE-reBench, SWE-smith, SWE-Fixer-Train, and filtered and obtained 32K unique prompts. The main filter-de-regulating technique is the filter that is attached to the inspection and field of vector similarity at n-gram (NV = 9).

The daapter itself does not need to be involved with LLM. It fills in the existing Prompt 2 {instruction} placeholder, and a subfix by task type. The math question requires that the final answer be written. /app/solution.txtCode number requires Python solver to be saved to /app/solution.py,SWE Question requests to generate SEARCH/REPLACE Edits and save diff to /app/solution.patch。

The benefits of this route are cheap, stable and large. It converts existing data into behavioral data in a model to practice"Read tasks, operations files, deliver results by path"I'm sorry. The shortboard is also clear: Nemotron-Cascade raw data mainly provides prompt, and these dapser missions usually do not have the appropriate tests, and the real environment is not as rich as the synthetic tasks that follow. It is basic training, not the most representative part of the capacity, but it is necessary.

Seed-based promotion: changing the title to a task

The second route is ted-based synthetic task promotion. This is not an old case shell, but an inspiration for an existing problem, and an implementationable terminus. Every seen entry has at least a problem determination, domain label and reference solution. Reference solution is only used when generating test expectations, without exposure to ant, avoiding ant learning the answer directly.

LLM plays task here. It is intended to complement the full project task with abstract topics: where the input file is, where the output is written, what dependencies are needed, what are the boundary conditions, how the float error is judged, how the format is checked, how the pytest test covers normal samples and edge cases. I'm just saying,"Accomplish an algorithm"And the terminus is clear."Read /app/input.csv, Generate /app/output.json, the field must meet this schema, the test will check the behavior."I'm sorry. As indicated below.

你正在把一个 seed problem 改造成 terminal agent 的 SFT 任务。

输入：

problem_description
domain_label，可为空
reference_solution，只能用于设计测试期望，不能泄漏给 agent

生成：

task_prompt：自包含，说明输入路径、输出路径、约束、完成条件
files：任务启动时需要存在的文件
tests：pytest 检查，覆盖格式、数值容差、边界条件
metadata：domain、skills、difficulty、source、generation_version

要求：

不在 task_prompt 中透露算法或参考实现
任务要难解，但测试要能自动判断
如果题目太大，拆成可验证的小目标

This type of program is only the first step, which really makes it possible to control, is the isolation of the reference file, the structure of the test file, the recording of the metadata, and whether the tracker rollout can be run behind.

Skill-based promotion: grouping tasks from productive skills

The third route is skill-based promotion. It no longer relies on existing topics, but instead uses skills to synthesize new tasks. Nine papers were defined: data processing, data querying, data science, debugging, decendency management, file operations, scientific communication, security, software engagement. Each domain maintains a set of dimensions covering the agorology, systems, data processes, mathammatic, testing, web/security, etc.

Here's the taxonomy not a post-label, but a sampler. When generated, LLM is asked to combine 3 to 5 prime cases, write a new scene and output prompt, pytest testes, waters, metadata, files and test references. For example, the data science task may mix file reading, groupby aggregation, statistical testing and chart validation; the security task may mix encoding, authentication logic, payload construction and test judgement; and the software engineering task may combine file modification, reliance on resolution, test restoration. See the following tips, and specifically the original text of the template.

你是 [domain] 任务生成器。

可用 primitive skills：

[skill_1]
[skill_2]
[skill_3]
…

请组合 3 到 5 个 skills，生成一个真实、具体、可自动验证的 terminal task。
输出：
<prompt>给 agent 看的任务说明</prompt>
<files>初始文件、数据、配置</files>
<tests>pytest 测试与权重</tests>
<metadata>domain、skills、difficulty、expected_tools</metadata>
约束：

场景要自然，不要机械拼技能
不要泄漏解法、测试细节或参考代码
任务应当需要多步操作，而不是一次函数调用

Complexity can be broken down into sampleable dimensions, not general requirements for more difficult tasks, but rather decisions on which lower layers of capacity to combine, which domain to combine, and what tests to observe whether they actually occur. This is more like a training map than a bunch of random puzzles.

Task format and environment: Ensure validation first

Both synthetic submissions end up in the same task format: natural language: task programt, pytest-based testes, weighted test entries and partial scores, supplementary input files, and a domain-special Docker environment. The paper did not generate standard answers for each mission, and instead pursued a more engineering objective: the task could be difficult to solve, but the validation should be as clear as possible.

In the angent scene, first have a verifier more practical than standard answers. If the test is stable, the test is successful, the tester fails, the trajectory is interrupted, the output format is wrong, it is recorded and then the analysis is followed. Without a verifier, it would depend on another model to score, and many problems would become style judgements.

Environmental design also serves this objective. NVIDIA does not allow LLM to generate Dockerfile for each task, but rather to maintain 9 pre-built domain-specific Docker images. Every image pre-assessed that domain is often dependent on, for example, the pacandas, scikit-learn, security cryptography libraries in the data science. This saves each mission Dockerfile validation and multiple-wheel restoration, reduces mirror construction costs and decorates the environment and tasks.

Tacheer rollout: Leave the work to the end.

After the mission was created, there was a need to go in and do it again. The paper selected DeepSeek-V3.2 because it had achieved good results on the various data sets of Terminus 2 angent africawork. Teacher can't just pick a strong model to see if it works in the target interactive protocol.

Trajory is generated by Terminus 2. Models see the mission statement and current mission state, and then output JSON, including analysis、plan、commands and task_complete。commands - Yes. keystrokes The system is sent as it is to the terminal, waiting for the specified time, and then fills out the latest terminal output back to the next round. The training sample is..."Status, reflection, planning, action, observation."- The cycle. And not just the final answer. Each step of analsis, plan, action, operation, success, number of tokens, number of turn, time-consuming and failure stages should be recorded. The tracks don't necessarily go into the training set, but they can get you back in the back."Why did the model learn to behave like this?"There's evidence to follow. Teacher rollout can also record difficult information and is a common experiment in training that mixs difficulty and sequences with a certain weight.

The compressed objectory scheme is probably this way.

{
  "analysis": "当前状态、已经完成什么、还缺什么",
  "plan": "下一步为什么要这样做",
  "commands": [
    {
      "keystrokes": "pytest -q\n",
      "duration": 1.0
    }
  ],
  "task_complete": false
}

This type of projectory teaches work habits. Model to read directories, read README or tests, run commands, see errors and go back to the error stack, adjust files, and finally check. Many models write codes, but the ability to work in the terminal environment is weak (the format symbol required for the terminal is difficult to output). The SFT trajectory does not necessarily suddenly give the model new intelligence, but it can place the capacity on a more appropriate behavioral path. Only half of the data synthesis is complete; the other half occurs in filtering, mixing and dispatching before and after training.

Return to Terminal-Corpus: Training dispatch strategy

The benefit of this NVIDIA paper is the integrity of the project, which did not stop at data generation. It continues to test these tracks in SFT to see what filters really work, whether course learning is useful, whether the context is better, and whether data size continues to bring benefits. By this chapter, data engineering has evolved from how samples are made to how samples can enter training.

The paper uses Qwen3-8B as the main ablation, and the size is verified with Qwen3-14B and Qwen3-32B. Training is conducted using veRL, learning rhe-5, course decay 1e-4, 2 epochs, max security left 32,768, global watch size 128, Adam W, cosine schemer and 10% warmup.

First, we clear the contaminated stuff.

Base cleaning first to pollute. Thesis deletes and Terminal-Bench 2.0 test samples have 14-gram overlap prompt, removes entries and drops the samples containing Chinese characters as the most basic filter. These rules look trivial, but it's common in the data of ant. Teacher may reveal his identity in the trajectory, or bring strange formats into training; assessing leaks is more problematic - once mixed, the latter scores are difficult to explain. We need to preserve the credibility of the experiment, and naturally we need to keep the data clean enough.

Filter experiments: Failure tracks may not be waste.

Change of intuition is projectory filing. Intuitively, it would be cleaner to keep only the full track or the track through testing. It's not like that. On dataset adapters, no filler is best combined: 226,313 samples, TB2.0 9.66; complete-only 196,940 and TB2.0 8.09. The difference is even greater in synthetic questions: no filler retains 264,207, TB2.0 is 12.4; complete-only only has only 104,603, TB2.0 is 6.74; access-only only only 83,448 and TB2.0 is 5.06.

It's not that the more failures the better. JSON is broken, identity leaks, long air turns, obvious nonsense, and these remain to contaminate the model. But some failures are not waste. The error reading after the test failed, the fixation of the path after the error, and the change of the program by relying on failure to fit, are all part of the technical event. Strict filtering removes dirty data and removes difficult tasks, restores behaviour and real errors.

Filters should be reorganized into two layers, and the Hard project is responsible for throwing out the obvious bad sample, soft label is responsible for recording the sample's state and ultimately for determining the weight in the training mix. The failure code sample in OpenCodeRessoning covers more difficult questions, and the failure here of NVIDIA is likely to carry recovery mode. Together, the two articles remind me that correctness is a strong signal, but not a complete definition of the value of training.

Data mixing and course learning

Data source ablation gives a more steady signal. In the Dataset adapters, Math, Code, SWE have separate returns, which are better combined: Math 5.39, Code 6.29, SWE 7.02, All 9.66. Different sources provide complementary behaviour - mathematically brings together reasoning formats, code brings capacity, SWE brings habits of reading files, changing codes, generating patch.

In synthetic tools, skill-based is the main source of gain, with article 139,841 reaching 12.4; and seed-based 124,366 with only 6.18; combined, both are still 12.4 but the difference is even lower. The task of seeingd-based comes from existing problems and is more stable, but ultimately it is seen as space-limited; the task-based is organized directly by the permanent skills, which can more easily cover the real action missing. The combination of the two did not continue to increase, but made the results more stable, which is also a recurring gain in the data engineering.

Course learning experiments are also not intuitive. Thesis compares two-stage curriculum and single-stage mixed training. The former trained first, Dataset adapters, then synthetic questions; the latter mixed all data. Qwen3-8B, misted training gets 13.03, curriculum gets 10.39.

This result does not indicate that the course learning is not useful, but rather that it is not course learning by data source. It's not necessarily easy, it's not even harder. The real difficulty may come from the number of files, the number of tools to be used, the length of context, hidden tests, polylysis, the success rate of the teacher, the base model initial capability. In the future, if the course is to be followed, the difficulty label is best obtained from the trajectory and from the data source.

Long Context Training and Scaling Experiments

The trend is observed in the ratio of 1%, 5%, 10%, and 100%, where a preliminary assessment can be obtained using a smaller ratio. If 10% of the data are close to 100%, it may be that coverage under the current assessment is close to saturation and that there are limited marginal benefits of continuing to increase the same distribution. This discovery is of engineering value in itself. This is a reduction in the size of the data, a classic training digestion technique, but not all.

In addition, the paper was based on the 8B model to perform the main digestion, and the 14B and 32B models to validate the scale trends, reducing the likelihood that conclusions would be formed only at the size of a model. And we can also see if similar scaling trends will emerge in our own scene.

Different context lengths are also part of the digestion experiment. In some scenarios, the length of the text generated varies considerably. There may be a small part of the trajectory that goes beyond the context limit and is thus cut off by the SFT algorithm. If you want to do so, you can consider not using the YARN2 model for SFT; using the YARN2 model for SFT; and using only the YARN2 baseline model for the assessment. Many scenarios have longer context lengths already contained in standard context windows, with long tails potentially confusing and information-intensive.

In the paper, the negative digestion is not hidden. You just have to explain."Why does this result have to be a counter-inputive, but reasonable."— For example, the course failed because the ranking of data sources was not the same as the order of difficulty, and the strict filtering was harmful because the recovery of difficult tasks was also eliminated together. The digestion experiment is not necessarily positive, and negative results are valuable. Under this type of budget and experimental setting, the main digestion can be done using the 8B model, followed by other model sizes of scaling exercises; this is a resource allocation strategy, not a rule that is established for all studies.

First, establish assessment benchmarks

After reading these jobs, I will not start by looking for 100,000 data. A more comfortable starting point is to put together the assessment and validation shelf, to know how to judge what is right and how to make the data. This remains part of the data engineering backbone: without a credible assessment, the data generation pipeline will not receive stable feedback. The verification of their work and data should not have been separated in real papers.

This step is the most easily bypassed and determines the credibility of all the experiments that follow. Without benchmark, training begins, even without saying anything about the optimization.

Benchmark and Datset are two completely different things.

Before you do your research, think about what you're building.

Benchmark's purpose is to...Measurement of capacity boundariesI'm sorry. It requires small and sophisticated, and each subject must be detached and verified. Its audience is the reader and the community, and the version should be frozen - once the benchmark topic is mixed into training data, all the results of the assessment are no longer credible. Terminal-Bench 2.0 has 89 missions, each of which cost about 3 quality-hours for quality assurance. SWE-bench starts from about 90K PRs, filters and executes, and eventually leaves only 2,294 entries.

The purpose of Dataset is to...Change model behaviorI'm sorry. It requires coverage that tolerates noise and failure samples. There are retrogressive behaviours and miscorrection patterns in the trajectory of failure, which are valuable for training. The audience of Dataset is the model itself, which should be continuously iterative and updated as the model's capabilities evolve. Terminal-Corpus eventually used about 490 K samples of SFT, but filtering experiments during the generation show no filters are much better than access-only.

One sentence summed up the difference:Benchmark asked you to get better, Dataset to make you better. The quality of Benchmark depends on the accuracy of the validation of each subject, and the quality of Dataset depends on the breadth of the coverage and the learningability of the trajectory. If you mix them together — for example, with the subject in the training data — it's a circular argument, and the reviewer can see it at first sight.

How do you start?

First, one or two weeks to do 10 or 20 verifiable tasks. Each mission has at least three components:

Independent environment: Each mission has its own clean operating environment (Docker or equivalent isolation), so that angent cannot be reconciled with the memory of the mission.
Artificial Oracle solution: Proof of the task is real. It does not need to be complicated, but it is necessary to ensure that a human being (not LLM) confirms the solution that can be adopted. Terminal-Bench requests confirmation from each contributer"My solution.sh was written by a human"。
Automation testing: Let the score be repeated. The key design principles are:Final status rather than intermediate process— Inspection"Output file matches schema", Not Check"Agent, did you execute a specific order?"I'm sorry. Give angent creative space and avoid being too vulnerable to testing.

The quality door process for Terminal-Bench can be used as a reference for scaling: automated check (oracle running, dummy anent cannot pass through) LLM auxiliary check (typo, test/description alignment) manual check and counter-test (specially running a cheating anent to see if it can be bypassed). You don't have to do 3 criteria for each task, but the first three steps are the minimum that every homemade benchmark should take.

Validate the quality of Benchmark itself

After you've built Benchmark, you have to prove to yourself (and to future readers) that it is a good assessment tool:

Distinction check: Run at least three types of models with your benchmark - the strongest current closed source model (GPT-5/ Claude Opus), the strongest open source model, and the base model you will use later. Scores must be distributed meaningfully. If all models are 0%, benchmark is too difficult and may be problematic. If all models are 100%, benchmark is saturated and cannot measure progress.
Human performance: If conditions permit, find one or two humans who behave with you do it again, report the score as upper base. This figure is much more convincing to the reviewers than any comparison between models.
Pollution control: N-gram overlap check immediately after the baseline is established to ensure that there is no overlap with any open dataset that may be used for training. This is the bottom line - CHIMERA paper clearly reported about 8-gram and 13-gram overlap, which should not be sub-sub-sections, but rather a symmetry.

When Benchmark was not perfect

Frankly, most self-built benchmarks cannot get the input from Terminal-Bench. You don't have 93 constribators, no ~3 missions per assignment, no specialized adversarial audits. And then you need to use additional means to make conclusions more credible, not pretend that benchmark is beyond criticism.

Case Study

Select 5-10 representative cases, with in-depth comparisons of behavioural differences between base model and trained model. The presentation of both success stories and failures — just picking good examples would put the reader on the spot. Each case must at least be clear: what the mission is, what the base model does, what the traded model does, why the difference makes sense. Case story cannot replace quantitative assessment, but it tells readers your method."Changed behavior patterns."♪ And not just ♪"The score's up."。

LLM-as-Judge calibration

Automatic scoring with a strong model (GPT-5 / Claude Opus) can significantly reduce assessment costs, but only if calibrated to be credible. MT-Bench shows a high degree of consistency between GPT-4 justice and human preferences, but this result relies on specific tasks, samples and evaluation protocols, and cannot be directly migrated to new research scenarios. If it is not possible to report on its own data the consistency of the judge with human beings, there is good reason for the reviewers to question the reliability of LLM-as-judge.

Calibration: first, using 3 human experts to score 50-100 samples independently, calculate inter-annotator agreement (Cohen)'s 100), then test the same sample for the consistency of the judge with the human race. Only when consistency is acceptable can the judge be used for large-scale assessments.

And at the same time, the LLM-as-Judge has three known errors to address:

Position bias:Judge may prefer a fixed-position answer. A/B could be evaluated at each of the two points of exchange and a separate report on stability before and after the exchange.
Verbosity biasJudge may misjudge a longer and more complete presentation as having higher quality, and the tendency to match the sample with length or manually labeled.
Self-enhancement biasJudge may prefer output close to its own model family, so it is preferable to use multiple Judges or to add cross-models and manual calibration.

Multiple Signal Triangular Validation

When you have small benchmarks, case studies, LLM-as-judge calibrations, a single signal cannot support any conclusion. But if the benchmark score is raised + case story shows behavioral change + LLM-as-judge gives a trend in the same direction, the three weak signals are more convincing together. What really repulsed the reviewer is the excessive assertion of an unreliable indicator — matching the strength of the claim with the intensity of the evidence — the principle most often overlooked in the publication strategy.

Evaluation and training iterative

The evaluation wasn't the end of your run once after the experiment.Make an emror anaclysis part of the pipe.: Train a version of the → running benchmark → analysis failure model → identification model remains weak in which areas of capability return to the data synthesis tube and target matching training samples → retrain.

This closed circle is simple conceptually, but it takes a premise: your benchmark is classified by area of capability or type of error, and error analsis is able to trace the data gap. If Benchmark is just a whole pass rate, Error anallysis can only tell you."There's another question."But you can't tell you what to add — that's why you need to put a capability tag right when you define mission space.

From data engineering to publication: how to convince the reviewer

The validity of the data project will ultimately need to be explained through experimental design and the chain of evidence. Here we continue to discuss how to move the whole process from an experiment to a paper: Industry can be online based on A/B Test and long-term performance indicators, and papers need to be convinced of other perspectives with replicable experiments, baselines and accommodations.

Build benchmark as a contribution

If you're not in the right direction, benchmark, you don't have to be ashamed of it. One of the core contributions of a large number of high-citation papers is the definition of confrontational authenticity by benchmark: TruthfulQA, the definition of software engineering angent by SWE-Bench, and the definition of terminal angent by Terminal-Bench. Each section is a question of definition, re-proven questions, and then prove its method on benchmark.

Thesis structure can naturally be cut in two: first half to define problem space and build benchmark (with quality certification, human performance, differentiation proof), and the second part to present methods of data synthesis and training and validate them on benchmark (with digestion analysis). And it's a complete contribution -- not as bad as you find a ready-made benchmark to the top.

The baseline is fair.

Can't just base model. If your method is to synthesize SFT data, then baseline must include the same methods of synthesis of data that already exist -- as in the case of the Gramgen, and at the same time, there are many methods like WRAP, Genee, LongForm, EntiGraph. One of the types of papers that the reviewer most often rejects is that"A is better than untrained base model, so A is effective"No, A, it's probably just a minor improvement."Best available method for data synthesis"。

At the same time, ensure that the baseline approach is treated fairly in the context of your mission. If you test data generated by the baseline method in the terminator, but the baseline method never optimizes the environment of the terminological environment, this comparison is unfair. The solution is to harmonize the suitability conditions (e.g., to give one identical adapter) or to establish two or more different assessments to cross-check.

Pollution control is the bottom line of trust.

All the results of the article are no longer credible once the reviewers suspect that benchmark has been contaminated. CHIMERA paper demonstrated this problem: 8-gram and 13-gram overlap were clearly reported near 0 and scripts were provided for the examination. Terminal-Corpus uses 14-gram overlap to remove a training concentration that overlaps with benchmark. The prevention of pollution from training and testing is the bottom line of an article like SFT Training.

We'll do a solid digestion experiment.

The true digestion is characterized by a single variable at a time, a pre-supposition for each experiment, not always positive results, and an interpretation of the results of the COUNTER-intuitive. Characteristics of false digestion: multiple variables at once, positive results picked out to be magnified, none of which were negative.

Your digestion does not have to be exhaustive -- GraphGen made four groups, LIMA made three groups, Terminal-Corpus made six groups -- but each group answers one clear question. The drafters do not add points because of the amount of digestion, but because each group melts will make it clear why this group does it.

It's a small data paper.

CHIMERA has only 9,225 training data, LAB has less than 1M, LIMA even validated it with 1,000 pieces > Quite. The reader is not concerned with the amount of data, but whether you have made it clear why the 9,225 data are sufficiently clear and what they cover and what they don't cover.

The advantage of small data is that you can clearly explain each data profile and quality judgement. And the big paper reviewer is more likely to wonder if you've ever seen that much.

Open source lowers the concerns of the drafters

The Terminal-Corpus open source contains data sets and training configurations. This reduces the core concerns of the reviewer -"Your conclusion is not that it happened to be on the data you didn't disclose."I'm sorry. If you can open the source, then open the source; if you have constraints that cannot open the source (data copyright, privacy, etc.), at least open data to generate scripts, training in configuration and evaluation codes, so that the methodology itself is replicable.

It's written at the end.

The article is specific: re-use existing data, draw skill maps, generate missionable environments, get strong teachers, record tracks, do decontamination and filtering, train with SFT, and finally use ablation to see which options are really useful.

This line could be written with me. How Reward and Training Closed in Real Age: From Data Governance to Online RL Put it together and read. The article discussed the more RL-oriented angent training closed-ring, where the environment was changed from a triminal tool, log, verifier and reward to a real interactive system, but the idea of taxonomy, weight, difficulty label, negative sample, projectory filtering and curriculum is linked to the SFT data governance discussed here.

Synthetic data is a set of maintenance work to train the world. The blog, "Propt", "Teacher", "Verifier" and "schema" are important. It is becoming a more systematic project, and it is good news for ordinary researchers: once the problem is understood sufficiently, real engineering experience can be built up from small-scale data.

References

Renjie Pi et al., On Data Engineering for Scaling LLM Terminal Capabilities
NVIDIA, Nemotron-Terminal-Corpus
Terminal-Bench Team, Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Yizhong Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions
Can Xu et al., WizardLM: Empowering Large Language Models to Follow Complex Instructions
Ziyang Luo et al., WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Gunasekar et al., Textbooks Are All You Need
Zhangchen Xu et al., Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Arindam Mitra et al., AgentInstruct: Toward Generative Teaching with Agentic Flows
Shivchander Sudalairaj et al., LAB: Large-Scale Alignment for ChatBots
Yiran Liu et al., GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
Subhajit Dutta et al., CONDOR: Enhancing LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
John Yang et al., SWE-smith: Scaling Data for Software Engineering Agents
Wasi Uddin Ahmad et al., OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
Xinyu Zhu et al., CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
Chunting Zhou et al., LIMA: Less Is More for Alignment
Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Anthropic Engineering Blog, Demystifying evals for AI agents
Soham Bhattacharjee et al., CuratorKIT: Data Curation and Synthetic Data Generation for LLM Post-Training
Hyacehila, How Reward and Training close the loop in real Agent: from data governance to online RL
AMAP Team, AMAP Agentic Planning Technical Report