Hyacehila's Blog

From Binary Tree Traversals to Graph Search: Preorder, Inorder, Postorder, Level Order, DFS, BFS, and Dijkstra

2026-09-13T20:00:00.000Z

Preorder, inorder, postorder, and level-order traversal of binary trees, DFS and BFS on graphs, and Dijkstra’s shortest-path algorithm are all familiar fundamentals. I want to review them together here.

Looking at them together reveals two questions that we can consider separately: where to go next, and when to process a node once we reach it.

Preorder, inorder, and postorder follow the same depth-first recursive process; they differ in when they process each node. Level-order traversal expands outward one level at a time, corresponding to breadth-first search. On a weighted graph, Dijkstra changes the basis for choosing the next vertex to the cumulative distance from the starting point.

Consider this binary tree:

    A
   / \
  B   C
 / \   \
D   E   F

Traversal	Processing order	Output for this tree
Preorder	Root → left subtree → right subtree	A B D E C F
Inorder	Left subtree → root → right subtree	D B E A C F
Postorder	Left subtree → right subtree → root	D E B F C A
Level order	Starting at the root, level by level from left to right	A B C D E F

The names preorder, inorder, and postorder describe the root’s position relative to its left and right subtrees. Each subtree must be processed in full using the same rule. The left subtree means more than just the left child.

We can express all three recursive traversals in one piece of code. Assume each node has value, left, and right attributes, with None representing a missing child:

def traverse(node, pre, ino, post):
    if node is None:
        return
    pre.append(node.value)       # Just entered the current node
    traverse(node.left, pre, ino, post)
    ino.append(node.value)       # Left subtree finished; right subtree not started
    traverse(node.right, pre, ino, post)
    post.append(node.value)      # Both subtrees finished

Pass in three empty lists, and one recursive traversal collects the preorder, inorder, and postorder sequences separately. The route taken by the recursive calls stays the same; what changes is where we record each node’s value. Even in postorder, the program must enter the root before it can find the children. Outputting the root last does not mean reaching it last.

To understand these positions, think of each recursive call as taking responsibility for an entire subtree. From A, B represents the whole subproblem containing B, D, and E; C represents another subproblem containing C and F. A hands work to B, waits for B to finish completely, and then hands work to C. Every node repeats this process internally.

Looking only at A’s call, the process is:

Enter A                              ← Preorder position
    Enter B and finish B's entire subtree
Return to A, ready for the other side ← Inorder position
    Enter C and finish C's entire subtree
Return to A and finish A's subtree    ← Postorder position

These positions represent different stages of progress. At the preorder position, the current node has received information passed down by its parent, but neither child subproblem has started. At the inorder position, the left subproblem is finished and the right one has not started. At the postorder position, both subproblems are finished, so their returned results are available if needed.

When choosing a traversal order, first consider what information is needed to process the current node, and when that information will be ready.

Preorder suits work that can be done upon reaching a node, as well as preparing information its children will need. For example, to label every node with its depth, the root A has depth 0. Upon reaching B, we know its depth is 1, and we can pass 2 to D and E. B’s depth comes from its position along the path through its ancestors; we do not need to know how many nodes lie below D or E. Information flows down the parent-child relationships, so the preorder position is a natural fit.

Postorder suits cases where the current answer depends on the children’s answers. Suppose we want the number of nodes in each node’s subtree. Upon reaching B, we know B itself contributes 1, but we do not yet know the sizes of its two subtrees. Once D returns 1 and E returns 1, B can compute 1 + 1 + 1 = 3 and return 3 to A. C similarly returns 2, and A finally obtains 1 + 3 + 2 = 6. Results start at the leaves and are combined upward, one level at a time.

Inorder has a more specific use: placing the current node between its left and right parts to read its position within the whole. The middle here means between those two parts. It does not mean that traversal is halfway finished or that the two sides contain equal numbers of nodes.

A binary search tree gives that position an ordering by key value. First consider distinct keys: every key in a node’s left subtree is smaller than its key, and every key in its right subtree is larger. Listing the left side in order, then the current key, and finally the right side in order produces an ascending sequence for the whole tree. Applying the same rule within every subtree produces the complete sorted sequence. If duplicate keys are allowed and placed consistently, the result is nondecreasing. An ordinary binary tree has no such sorting guarantee, although inorder can still read its left part, current node, and right part in sequence.

These examples describe the work suited to each position. An actual algorithm can record the current path upon entering a node and combine results after processing both subtrees, using preorder and postorder positions within the same DFS. As in traverse above, recursion provides several opportunities to do work; the task’s information dependencies determine what belongs at each one.

All three traversals are forms of DFS, or depth-first search. The program follows a branch downward and returns after completing a subtree. The recursive call stack remembers where to return.

Level-order traversal uses a queue. Enqueue the root first. Each time, remove and process the node at the front, then append its nonempty left and right children to the back, in that order. When B is processed, C is already in the queue, so the newly added D and E go behind C. This first-in, first-out rule ensures that one level is processed before the next. To group the result by level, record the queue’s length at the start of each round and process only that batch of nodes.

The following implementation returns results grouped by level. Nodes still use the value, left, and right attributes introduced earlier, with None representing a missing child:

from collections import deque

def level_order(root):
    if root is None:
        return []

    queue = deque([root])
    levels = []
    while queue:
        level_size = len(queue)  # At the start of this round, the queue holds this level
        level = []
        for _ in range(level_size):
            node = queue.popleft()
            level.append(node.value)
            if node.left is not None:
                queue.append(node.left)
            if node.right is not None:
                queue.append(node.right)
        levels.append(level)

    return levels

Use a simple node class to construct the tree from the beginning of the article, and the example is ready to run:

class TreeNode:
    def __init__(self, value, left=None, right=None):
        self.value = value
        self.left = left
        self.right = right

root = TreeNode(
    "A",
    TreeNode("B", TreeNode("D"), TreeNode("E")),
    TreeNode("C", right=TreeNode("F")),
)
levels = level_order(root)
print(levels)                              # [['A'], ['B', 'C'], ['D', 'E', 'F']]
print([v for level in levels for v in level])  # ['A', 'B', 'C', 'D', 'E', 'F']

When processing the level containing B and C, level_size is fixed at 2. Even though processing B adds D and E to the queue, this round removes only C next. The newly added D, E, and F wait for the next round. The outer while advances through levels, while the inner for processes the nodes in the current level. If all you need is one flat traversal sequence, you can record each node’s value as it leaves the queue and omit the inner loop that groups nodes by level.

When traversing a normal tree from the root through child pointers, every nonroot node has exactly one parent and there are no cycles, so there is no need to track whether a node has been visited. In a general graph, the same vertex may be reachable by several paths, and a cycle may lead back to an earlier vertex. Both DFS and BFS need to prevent repeated searches.

Graph DFS can add a vertex to visited upon entering it, then recursively search unvisited neighbors in adjacency-list order. Graph BFS marks a new vertex when it is discovered, just before enqueueing it, so several vertices cannot repeatedly add the same neighbor to the queue. Starting from one vertex, both algorithms cover only the reachable part of the graph. To traverse the entire graph, iterate over all vertices and restart from any vertex that remains unvisited.

Adding an edge between E and F to the earlier tree produces the following undirected graph. B, A, C, F, and E form a cycle, and vertices now have multiple paths leading to them. This lets us observe the role of visited.

    A
   / \
  B   C
 / \   \
D   E---F

Store it as an adjacency list, where graph[u] contains u’s neighbors. An undirected edge must be recorded at both ends: for example, A’s list contains B, and B’s list contains A. Below, neighbors are always checked from left to right in each list:

graph = {
    "A": ["B", "C"],
    "B": ["A", "D", "E"],
    "C": ["A", "F"],
    "D": ["B"],
    "E": ["B", "F"],
    "F": ["C", "E"],
}

We first implement DFS recursively. Once visit(u) calls visit(v), the loop in u’s call pauses until the search in v’s call finishes, then resumes checking the next neighbor.

def dfs(graph, start):
    visited = set()
    order = []

    def visit(u):
        visited.add(u)          # Mark on entry to prevent reentering through a cycle
        order.append(u)         # Preorder position: record the order of first entry
        for v in graph.get(u, []):
            if v not in visited:
                visit(v)        # Explore this branch before checking the next neighbor
        # Postorder position: all of u's neighbors have now been checked

    visit(start)
    return order

print(dfs(graph, "A"))           # ['A', 'B', 'D', 'E', 'F', 'C']

After entering B from A, B’s first neighbor A has already been visited and is skipped. The search then enters D. D has no unvisited neighbors, so it returns to B. From B, the search next enters E and continues along E → F → C. Only after this whole branch returns to A does A continue checking its own neighbor C, which has already been visited by then.

The recursive call stack maintains the depth-first order here. visited is retained throughout the search, and vertices are not removed when recursive calls return. This prevents D from searching B again and F from continuing around the cycle indefinitely.

Moving order.append(u) to after the for loop produces the finishing order D C F E B A, corresponding to postorder in graph DFS. When moving the recording step, keep visited.add(u) at the entry point. Marking early is the convention used in these BFS and DFS implementations to prevent repeated searches from getting trapped in a cycle.

BFS puts vertices awaiting processing into a queue. The following code records traversal order, minimum edge counts, and predecessors, connecting the implementation to the shortest-path discussion that follows. Python’s deque is a double-ended queue; here we only append at the back and remove from the front.

from collections import deque

def bfs(graph, start):
    visited = {start}            # Mark the starting vertex when it is enqueued
    queue = deque([start])
    order = []
    dist = {start: 0}
    parent = {start: None}

    while queue:
        u = queue.popleft()     # Remove the vertex that has waited longest
        order.append(u)
        for v in graph.get(u, []):
            if v in visited:
                continue
            visited.add(v)      # Mark on discovery to prevent duplicate enqueueing
            dist[v] = dist[u] + 1
            parent[v] = u
            queue.append(v)     # Append at the back for later processing

    return order, dist, parent

order, dist, parent = bfs(graph, "A")
print(order)                    # ['A', 'B', 'C', 'D', 'E', 'F']
print(dist)                     # {'A': 0, 'B': 1, 'C': 1, 'D': 2, 'E': 2, 'F': 2}
print(parent)                   # {'A': None, 'B': 'A', 'C': 'A', 'D': 'B', 'E': 'B', 'F': 'C'}

With the front on the left, the queue changes as follows after each vertex is processed:

Start    [A]
After A  [B, C]
After B  [C, D, E]
After C  [D, E, F]
After D  [E, F]
After E  [F]          F is already enqueued and marked; it is not added again
After F  []

The difference is already visible when processing B. DFS immediately enters D upon discovering it, pausing B’s loop. BFS only appends D to the back of the queue and continues checking E. Once B is finished, C is at the front, having already been waiting, so C is processed next. As a result, F is first discovered through A → C → F, at distance 2. Following parent backward from F yields F, C, A; reversing that sequence gives the path. The DFS above first reaches F along A → B → E → F, using 3 edges.

Both implementations assume the starting point is a vertex in the graph. Vertices with no outgoing edges may be omitted from the adjacency-list dictionary; graph.get(u, []) treats them as having empty neighbor lists. The returned results contain only vertices reachable from the start.

A graph has no inherent left or right child, so traversal results also depend on the order in which neighbors are enumerated. DFS still has entry and finishing times, corresponding to the ideas of preorder and postorder. A general graph has no standard point where the left subtree ends and the right subtree begins, so it has no standard inorder traversal in the binary-tree sense.

BFS has another use on graphs: it finds the minimum number of edges from the starting vertex to every reachable vertex in an unweighted graph. The start has distance 0, its newly discovered neighbors have distance 1, and the next layer has distance 2. The queue always advances in this order, so when a vertex is first discovered, a path with fewer edges cannot later appear from a subsequent layer. Record dist[v] = dist[u] + 1 along with the predecessor parent[v] = u, and after the search you can reconstruct a shortest path by tracing backward.

DFS offers no such guarantee. It may reach the target along a deep branch first, even when another path requires only two steps. This also connects binary-tree levels with BFS distances in graphs: if the root’s depth is 0, a node’s level is the number of edges from the root to that node.

Once edges have different weights, the path with the fewest edges may differ from the path with the lowest total cost. Consider this directed graph, where the numbers are edge weights:

S ──10──→ A
│         ↑
1         1
↓         │
B ────────┘

The direct edge from S to A costs 10. Going through B uses two edges but costs only 2 in total. BFS, working by edge count, discovers the direct path to A first. If total cost matters, we need to process B, whose cost is 1, first and then update A using the path through B.

Dijkstra maintains a tentative distance dist for every vertex: the lowest cost from the start among the paths found so far. Set the starting distance to 0 and all others to infinity. In each round, select the vertex u with the smallest dist among those whose shortest distances have not yet been finalized, then examine its outgoing edges. If reaching v through u is cheaper, update dist[v]. This attempt to improve a distance is called relaxation:

1
2
3

If dist[u] + weight(u, v) < dist[v]:
    dist[v] = dist[u] + weight(u, v)
    parent[v] = u

A min-heap, or priority queue, is commonly used to select the smallest tentative distance. The priority is the cumulative distance from the starting point to the vertex, not just the weight of the most recently traversed edge.

Here is a Python implementation that allows a vertex to be inserted into the heap more than once. graph[u] contains (neighbor, weight) pairs, vertices use string or integer identifiers, and all edge weights must be nonnegative. The returned dictionary records only vertices reachable from the start.

from heapq import heappop, heappush

def dijkstra(graph, start):
    dist = {start: 0}
    heap = [(0, start)]
    while heap:
        d, u = heappop(heap)
        if d != dist[u]:          # A shorter path has superseded this old entry
            continue
        for v, weight in graph.get(u, []):
            candidate = d + weight
            if candidate < dist.get(v, float("inf")):
                dist[v] = candidate
                heappush(heap, (candidate, v))
    return dist

In the graph above, processing S leaves (1, B) and (10, A) in the heap. B is removed first, A’s distance is changed to 2, and (2, A) is inserted. Next, (2, A) is removed and processed. The old (10, A) entry remains in the heap and is skipped when its turn comes because its distance is stale.

This explains why we cannot copy BFS’s marking rule directly: when Dijkstra first discovers A, it knows only a path costing 10 and cannot finalize the answer yet. A’s shortest distance is finalized when A is removed with the smallest current valid tentative distance. If searching for just one target, that is also the point at which the search should stop.

This reasoning depends on nonnegative edge weights. Once u has the smallest tentative distance among unfinalized vertices, a detour through another unfinalized vertex cannot use subsequent nonnegative edges to lower the cost below it. With negative edges, a later path could reduce the cost further, so the argument no longer holds. Zero-weight edges are allowed; for graphs with negative weights, consider an applicable algorithm such as Bellman–Ford.

Method	Which vertex is processed next?	What does it guarantee?
DFS	An unvisited neighbor deeper along the current branch, backtracking when finished	Searches reachable vertices; does not guarantee shortest paths
BFS	The vertex enqueued earliest, advancing by edge count from the start	Minimum edge counts in unweighted graphs; also minimum total weight when all edges have the same positive weight
Dijkstra	The unfinalized vertex with the smallest current tentative cumulative distance	Single-source shortest paths in graphs with nonnegative weights

When every edge has weight 1, Dijkstra’s progression by cumulative distance follows the same layers as BFS, though the order within a layer may differ. A regular queue is sufficient in this case. For general nonnegative weighted graphs, both the vertex-selection order and the distance-update rules must change. Simply replacing BFS’s queue with a heap is not enough.

For a binary tree with n nodes, all four traversals take O(n) time. Excluding the result lists, recursive DFS uses O(h) extra space, where h is the tree height, while level-order traversal uses O(w), where w is the maximum width of a level. A deep tree consumes more recursive stack space, and a wide tree consumes more queue space.

With adjacency lists, DFS and BFS both take O(V + E) time and O(V) extra space, where V and E are the numbers of vertices and edges. The usual binary-heap implementation of Dijkstra can be bounded by O((V + E) log V) on a simple graph. The version above with repeated heap insertions creates at most O(E) heap entries; if arbitrarily many parallel edges are allowed, a more precise time bound is O(V + E log(E + 1)), with O(V + E) extra space. These space bounds exclude the input graph itself.

When solving a problem, first identify the result you need. To combine subtree information at a parent, place the work at the postorder position. To find the fewest edges, use BFS. To minimize the sum of nonnegative edge weights, use Dijkstra. Whether the graph has cycles, when a vertex should be marked, and whether a path can still become shorter determine which parts of these similar-looking implementations can be reused and which need to be reconsidered.

Perhaps Another Bitter Lesson

2026-09-11T08:00:00.000Z

A few things I’ve seen over the past several days have brought The Bitter Lesson back to mind.

On September 8, OpenAI published a proposed proof resolving the Navier–Stokes problem: roughly ten thousand concurrent agents took part in the search, producing a construction of a finite-time singularity under external forcing. An internal research model produced the proof; GPT-6 Astra handled the subsequent Lean formalization and verification. The scale does look a little like brute force, with agents exploring different approaches, exchanging findings, and carrying out verification.

Meanwhile, Astra can model a house in Blender and bring it into Unreal Engine. Robocurve gave it control of robot arms, and it placed a block into a bowl in 19 out of 20 trials. Image input and tool calling let a general-purpose model use tools people have already built. Of course, it completed the puzzle insertion task only 2 out of 20 times. These demonstrations don’t establish that it broadly outperforms specialized 3D models or vision-language-action models (VLAs). Still, some tasks we assumed needed separately trained models seem worth another look. Agentic Gen may be better placed to take advantage of the rapid improvement in general-purpose model capabilities.

Sutton’s 2019 essay, The Bitter Lesson, makes roughly this observation: over the long run, general methods that can keep using more computation, especially search and learning, tend to surpass methods carefully built around human domain knowledge. It’s easy to see why that feels bitter. We study a problem, design structures, add our experience, and achieve improvements. Then computation grows, and a more general approach catches up. The earlier investment becomes worthless.

Looking at that observation today, some capabilities we think require specialized modeling may simply be things general-purpose models aren’t good enough at yet. They can now inspect images, write code, and call tools. They have more computation during training, and can spend more time trying things when solving a problem. With further training and broader general capabilities, they may increasingly encroach on domains served by specialized models, as well as tasks within the scope of human abilities.

Sara Hooker’s “hardware lottery” also fits here. A research direction may succeed partly because it suits the hardware and software available at the time. The original Transformer paper already presented greater parallelizability and shorter training time as advantages. Perhaps today’s LMs have happened upon a path that fits existing hardware and makes it relatively easy to keep increasing investment. Algorithmic capabilities and infrastructure choices become intertwined.

So the question I want to ask now is this: of all the things that seem to demand careful, specialized design today, how many come from the problem itself, and how many fill gaps in current model capabilities? I don’t have an answer yet. Every time general methods advance a little further, though, I feel the question deserves to be asked again. Perhaps this is another Bitter Lesson.

References

OpenAI. On the Navier–Stokes Millennium Prize Problem. September 8, 2026; updated September 10, 2026.
OpenAI. GPT-6 Astra: A new generation of intelligence. September 3, 2026. Blender and Unreal Engine demonstration.
Robocurve. GPT-6 Astra on robotic manipulation. September 4, 2026. Results and limitations for two robot-arm tasks; the comparison covers other general-purpose models, not VLAs.
Richard Sutton. The Bitter Lesson. March 13, 2019. A university-hosted copy of the original essay.
Sara Hooker. The Hardware Lottery. 2020.
Ashish Vaswani et al. Attention Is All You Need. 2017.

Who Should I Be When Taking a Recruitment Personality Assessment?

2026-09-10T08:00:00.000Z

I’ve taken many personality assessments while looking for a job recently, mostly in two formats. One gives me three statements and asks which describes me most and which describes me least. The other gives me a statement and asks me to choose how well it fits on a four- or five-point scale. I can understand the questions, but I don’t really understand what they measure, or how those choices turn into conclusions.

Then there is a more practical question: should I try faking it a little? Or perhaps invent a supposedly “perfect-scoring personality,” and answer as that person? When the results go to a recruiter, it is hard to put what they want entirely out of mind. This post is a short look at that question, and an attempt to put my current understanding into words.

All three sound good. How do I choose?

Let’s make up a quick example:

A. I enjoy persuading others to accept my views.
B. I tend to plan what I need to do in advance.
C. I usually stay calm when something unexpected happens.
Choose the statement that describes you most and the one that describes you least.

All three sound pretty good. If I could rate them separately, I might feel that each describes me to some extent. But now I have to put them in order. Choosing B as most like me might be easy enough, but does choosing A as least like me tell the company that I am bad at communicating? Choosing C seems to suggest I struggle under pressure. A simple question starts getting a little too much thought.

This format is called forced choice. Designers can group statements that measure different tendencies but sound similarly desirable, asking people to compare which fits them better. In more academic terms, the options are matched in social desirability. Combined with the requirement to rank them, that makes it impossible to endorse every nice description at once. Still, “least like me” is a relative position within that group. It does not automatically mean I have none of that characteristic.

The other format is more familiar. Take “I like to plan my work in advance,” with five options ranging from “very unlike me” to “very like me.” This is a Likert-type response format. It lets me rate each statement independently; a four-point version typically leaves out the middle option.

The distinction is roughly this: one asks how well something describes me; the other asks which of several things describes me better. The uncomfortable part of the second format is also the information it is trying to collect.

After I answer, what does it see?

Basic scoring for rating items is fairly easy to understand. IPIP’s public scoring instructions give one approach: score positively keyed items from one to five, reverse that order for negatively keyed items, then add the scores within each scale. A reverse-keyed item can be understood as describing the tendency from the opposite direction. Suppose a scale includes “I keep things organized” and “I often leave things in a mess.” Stronger agreement with the second statement might mean a lower score on orderliness. A real questionnaire may cover more dimensions, with corresponding rules for grouping and scoring its items.

Forced choice accumulates comparisons. Choosing B as most like me and A as least like me in the example gives the order B, C, A. As a simplified scoring example, we could give planning two points and calmness one, then add up the results across a few dozen questions. But that is only an illustration to help explain the idea. Real systems are often more complicated.

Traditional forced-choice scoring can produce “ipsative” scores: these are better suited to describing relative tendencies within one person, and comparing two people directly becomes problematic. Brown and Maydeu-Olivares’s paper discusses how Thurstonian IRT can address this. Roughly speaking, it uses calibrated items and the full set of choices to estimate which trait levels best explain the responses. This kind of modeling can estimate scores across personality dimensions from the complete set of answers. The scores can be high or low, but that does not rank people as better or worse overall.

Scores may then be compared with norms, placing the results in the context of a reference group. For example, one public SHL sample report presents standardized scores from one to ten across several dimensions of work behavior and identifies comparison groups.

So a company might see a set of tendencies, higher and lower scores across dimensions, and interpretations based on them. Exactly how each employer evaluates job fit or compares candidates is harder to know. An assessment can inform screening and judgments about job fit, but a low score on a personality dimension does not make a candidate unqualified. Inconsistent responses do not automatically mean someone was answering carelessly either.

Should I play the ideal employee?

With that in mind, the question of faking it starts looking less straightforward.

With a knowledge test, I can prepare toward a correct answer if I know what it is. Personality tendencies do not come with a universal perfect answer sheet. Being meticulous is useful, but a task might also call for getting a rough version out first and improving it later. Independent judgment matters, and working with others also means sometimes accepting their views. It is hard to guess exactly what a particular job needs just by reading the options. Besides, a high score on a trait does not directly establish strong job skills.

I understand why someone might want to put on a performance. Recruitment involves selection; we cannot expect people to forget that while answering. Learning the format and understanding what “least like me” means seem like reasonable preparation. But if I first have to imagine an employee who is outgoing, meticulous, willing to take risks, and never makes mistakes, then answer the entire questionnaire for them, I would question whether such a person even exists.

Can the system tell? The SHL sample report linked above does include a consistency indicator, but consistency is not the same as honesty. Someone can consistently make themselves look better, while another person might respond inconsistently because they interpret questions differently. That indicator alone cannot establish who is lying. Reverse scoring itself is not a consistency check, let alone a lie detector. These designs can offer clues for checking responses, but they do not give a questionnaire the ability to read minds.

For now, I am not particularly keen to spend my energy inventing a persona. Understanding the rules can prevent some misunderstandings. Trying to guess a scoring system when I know neither its weights nor its target seems like making things rather hard for myself. Besides, different jobs call for different ways of working and different abilities. Outgoing or reserved? Independent or collaborative? Many of these judgments are hard to make from a questionnaire alone; we also need to see someone at work. Deciding whether a person suits a job from a few questionnaires and psychological measures still seems like a lot to ask.

The person it thinks I am

“Answer honestly” still leaves a question: me in which setting? Someone might avoid organizing activities with friends but willingly take the lead on a project. They might be casual in everyday life and plan carefully when they care about the work. These are easy situations to imagine. Any one of those fragments seems insufficient to stand in for the whole person.

The approach I tentatively agree with is to first think through how I usually act when working and learning, giving myself a stable point of reference for my answers. I would try to recall my typical behavior in study tasks, internships, and projects. I would try not to answer one question as my everyday self, the next as my ideal self, and the one after that with whatever the company wants in mind. At least that lets me know whom I am describing, and makes it easier to stay consistent. Of course, if the questionnaire specifies a setting, its instructions come first.

If you don’t mind giving it a little more thought, imagine yourself in that role. Faced with this kind of problem in this setting, what decision would you make? And what decision do you think you should make? Those two answers might be the same. They might not.

This is no Silver Bullet. Don’t expect a small approach like this to make every personality assessment go your way. Employers use assessments to understand your working tendencies and help judge fit; they cannot turn you into someone suited to every job. Looking for a job is a choice on both sides. There is no need to turn yourself into an offer-collecting machine either.

There is one more thing that would bother me a little. Candidates answer all these questions about themselves, yet may never see the final report. Without that feedback, we do not know how the system interprets our answers, or get a chance to say, “The person in that sentence doesn’t quite match what I meant.”

What does it think I am like?

I would quite like to see that report. To see the person it thinks I am, and, if something feels wrong, consider whether it misunderstood me or I hadn’t quite thought things through when I answered.

Interview Preparation: Questions, Answers, and Questions to Ask

2026-09-06T08:00:00.000Z

Before We Begin: This Article Will Keep Evolving

This article is not a finished job-hunting guide. It is a work-in-progress collection of my thoughts on job hunting, interview preparation, and questions to ask interviewers. I will keep updating it as I apply for jobs, receive interview feedback, understand different roles, and reconsider my choices. For now, it is an evolving blog post rather than a final version. In fact, most articles on this blog are continually revised, so patches and occasional restructuring are fairly common.

The focus here is on finding work that suits me and giving it a try, rather than simply getting an offer. Not finding a job I desperately want is normal, and not necessarily a bad thing. Becoming too attached to a title or some other aspect of a job can make it easy to overlook the real costs and end up having an unpleasant time.

Job hunting is not just a one-way screening process or sending out a resume and waiting. With limited information, I need to keep asking: What do I want to do? What am I suited for? Is this opportunity worth pursuing? Will I be able to keep growing after joining the team?

This article will also collect questions I might ask or be asked, along with ideas I find useful. Some may not warrant a full blog post, but I still need somewhere to keep them, so I will put them here for now.

Some General Advice

Decide on a Direction Before Applying

First, we need an overall plan. Industry and academia often care about quite different things. Even within large companies, business-facing roles and research lab positions can involve very different work.

Once that direction is clear, it should guide the emphasis of the resume, how projects are presented, and the application strategy. Different paths call for different ways of explaining the same experience. Industry puts more weight on solving real problems, collaborating to deliver results, and bringing technology into business processes. An academic path places more emphasis on research questions, methodological innovation, publications, and long-term research potential.

A target role should not be judged by its title alone. Whether a job suits me usually depends on the industry, city, salary, room for growth, fit with my expertise, and whether the company culture matches the pace of work and amount of overtime I am comfortable with.

Many job titles sound similar while the actual work differs substantially. In algorithms, data, backend development, or agent development, some roles focus on research and exploration, others on business delivery, and others on engineering platforms and toolchains. I may need to work this out from the job description or ask around; at the very least, I should find out during the interview.

Planning is not a one-time decision. External opportunities, market demand, and personal circumstances change. A useful plan should leave room to adjust based on application results, interview feedback, and changing interests.

How I Judge Whether an Opportunity Is Worth Pursuing

I consider several dimensions: income, direction, growth, fit, and team atmosphere. Salary matters, but it is not the only measure. If long-term prospects and skill development matter more, I need to look closely at the team’s direction, the quality of its work, the opportunities to learn, and whether I will get to work on worthwhile problems.

Alignment with my intended direction is especially important. A role may pay well in the short term, yet have little to do with what I want to pursue over time. In that case, I need to consider whether it will lead me down a path I do not want to follow. Conversely, an opportunity that is not immediately optimal may still be worth considering if the team’s problems are real and its technical work fits my long-term interests.

Team atmosphere matters too. Everyday collaboration, decision-making, attitudes toward technical debt, and room for growth and exploration all affect the experience of working there. Many of these things only become clearer when I ask questions during the interview. Those questions are part of deciding whether to take the job, rather than a polite formality at the end. I include some examples later in this article.

Interview Preparation Goes Beyond Memorizing Answers

Interview preparation cannot be reduced to memorizing standard technical questions and answers. For technical roles, coding fundamentals and common questions certainly matter: they affect whether I pass the initial screening and technical interviews. But project experience also needs preparation, or the explanation can easily turn into a chronological list of things I did.

I try to organize each experience around a few questions: What problem needed solving? What made it difficult? What was my role? What methods did I use? What were the results? What were the value and limitations of the work? This makes it easier for the interviewer to understand the project and assess my actual contribution.

Research projects, collaborations with companies, papers addressing practical problems, competitions, and internships become useful interview material when they are organized around a clear sequence of questions. Otherwise, even a long list of experiences can amount to little more than names on a resume.

Questions About My Background and Interests

Some Thoughts on Embodied AI

My resume may seem to have nothing to do with embodied AI or robotics, but I am interested in the field. My focus is not on training VLA models. As with the problems I have been studying, I care more about applications and how to make them work in practice.

Putting embodied AI into practice may involve more than a powerful VLA model. The release of GPT 6 Astra may offer some new perspectives on generative AI.

If we want to draw a picture, we may not need a diffusion model: a text model could use tool calling to draw a pelican in SVG. If we want to build an excellent 3D model, a generative model such as Meshy AI or Hunyuan’s 3D model might not work as well as letting AI gradually build it in Blender through tool calls and visual feedback. Generative language models now support omni-modal inputs, allowing them to move closer to human goals through repeated cycles of “generate—render/execute—inspect—revise.”

We can call this Agentic Generation, another form of generation alongside end-to-end model generation. From this perspective, both the UI generation workflow I worked on at NetEase and the deployment of embodied AI can ultimately be placed under Agentic Generation. Traditional generative models, whether for images, video, or motion, can become tools within that system. Optimizing the agent system itself would then become a core task alongside fine-tuning Pi-0.5.

Compared with generation through an end-to-end model, Agentic Generation offers another major advantage: the entire generation pipeline can be inspected and controlled. Users need results that meet their requirements. Producing those results may involve choosing representations, calling tools, observing intermediate outputs, making local revisions, and validating the result. Agentic Generation can accept constraints expressed in language or agent logic code, route tasks to different capabilities, preserve intermediate states, and make local corrections. When the whole system can be traced and reproduced, debugging and changing it will always be faster than training a model.

Agentic Generation also explores trade-offs around questions such as: Which decisions should remain inside the model, and which capabilities should be implemented through external tools, state management, and execution mechanisms? How should these boundaries change with the task? Combining a lower-level VLA with a higher-level agent may be a more practical near-term approach, but where is the right balance?

For a robot operating in the physical world, many actions cannot be undone with Ctrl+Z. How should tools for embodied AI be designed, and what feedback should they give the “brain”? In the discussion of forward knowledge injection and backward feedback in “What Problems Are We Really Solving When We Build AI Agents?”, I wrote: Tools are never just a list of tools; they are the entire world an AI agent can observe and affect. Tool calling has mainly focused on virtual environments. In the physical world, this view needs to translate into more concrete interface and feedback design.

For embodied AI, a tool should return more than “call successful.” The brain also needs to know whether the action actually completed, what changed in the environment, and what remains possible after a failure. How should this information be obtained from sensors, VLAs, and lower-level control systems? At what granularity, and when, should it be passed to the higher-level agent? These are questions I want to investigate further, and I suspect no one has an answer to them yet.

What Is the Biggest Challenge I Have Encountered?

I seem to have been asked this question in more than one interview. It is especially popular in the AI interviews that have become common recently: they like to throw a pile of fairly pointless questions at you, apparently simply because they want to ask them.

If I had to name the biggest challenge I have encountered, I do not think it would be one specific technical problem. It would be this: deciding what to do next when I do not even know whether an answer exists.

Much of my past work has not been the kind of engineering problem with a clearly defined specification. This is especially true of AI and agent projects. At the beginning, even the boundary of the problem is often unclear. Is the model capable enough? Can this technical direction work? What would count as an effective result? Does the problem itself even have a solution? None of these questions necessarily has an answer yet.

The approach I have gradually become used to is forming a belief from the information and experience I currently have: the technical direction I consider most plausible and most worth trying. I think having such beliefs is natural, and probably necessary, for a technically oriented developer. Mine come from the projects I have worked on before.

I then avoid spending too long trying to prove that my first judgment was correct. Instead, I build a minimal prototype around it as quickly as I can. The most important purpose of a prototype is not necessarily to become the finished product. It is to give me new information through real technical feedback. Every prototype and every failure changes how I understand the problem. I use that information to revise my belief, choose a more promising path, and begin another iteration.

Over time, I have come to think that the most important ability when facing an unknown problem may not be finding the right answer immediately. It is this: can you form a good enough judgment, move quickly, and keep letting reality correct that judgment?

This is also why I now rather enjoy dealing with problems whose answers are unclear. A problem with a known solution is mostly a matter of execution. When you do not know whether a solution exists, you have to keep observing, judging, and trying, then slowly find structure in the confusion. That process is challenging, but the challenge is also part of the fun.

So when someone suddenly gives me a completely unfamiliar problem, my first reaction is usually no longer:

“Do I know how to do this?”

It is:

“Given what I know now, which path do I believe in most? Can I build something small enough to see what reality tells me?”

I think this may be the most important habit I have developed for dealing with difficult problems over the years.

Questions to Ask Interviewers: What I Want to Find Out

The main purpose of asking questions is to understand what the team actually does, what the role really requires, what I would be responsible for, and whether the job fits my own criteria.

The questions below lean toward AI agent engineering roles. They help assess the team’s technical direction, approach to putting products into use, engineering maturity, and collaboration. Some are more general and can also serve as references. Another useful resource is viraptor/reverse-interview.

Agents and Engineering

I understand that the company is investing in agents. Are the planned applications mainly intended for internal use, such as improving developer productivity, answering questions over internal knowledge bases, or automating operations? Or are there already clear use cases for commercial products aimed at external customers? What problems is the team working on, and what would my main responsibilities be? Could you give some concrete examples?

Note 1: This is a fairly standard opening question for understanding a department’s work. Almost every interview should include this question or something similar.

Note 2: ToB (company-facing) versus ToC is an important distinction for AI agents in my view. ToB applications can have more tolerance for errors and make it easier to test and iterate on new technologies, but may sit outside the company’s core business. ToC applications require more attention to hallucinations, stability, and user experience. They can offer more opportunities to learn systems engineering and the design of safety boundaries, while making it harder to introduce new technologies.

Note 3: More detailed, technically informed follow-up questions may work in your favor. Offering two possible approaches and guiding the discussion can help. If every “A or B” question gets answered with “both,” or the interviewer avoids answering, another option is to state your assumption explicitly and let them correct it.

There are many different agent applications emerging across the industry. Is the team’s focus on well-defined, structured tasks, such as automated processes executed through API calls, or on understanding and reasoning over unstructured knowledge, such as structure-aware processing of long documents and Agentic RAG?

Note: The former puts more emphasis on automated workflows and requires a deep understanding of the business itself. The latter focuses more on agent development technology. Greater autonomy also means greater dependence on model capabilities and the developer’s understanding of safety boundaries. This is a technical “A or B” question closely tied to agents.

For the agent applications the team is pursuing, is the system mainly designed as a copilot that requires frequent human feedback and confirmation, or as a closed-loop workflow with a high degree of automation in a specific business domain?

Note: Human involvement remains important. For a highly automated system, high-quality automated verification becomes central. If more human intervention is acceptable, expanding the system’s functionality may be more valuable than pushing automation to its limits. This is another technical “A or B” question closely tied to agents.

Given the needs of complex business scenarios, is the current focus on improving a single agent’s ability to break down complex tasks, or has the team started exploring multi-agent collaboration in practical applications?

Note: Multi-agent collaboration has gradually moved from technical reports into demos. Although reliability remains questionable, it is still an interesting frontier. It is closer to technical exploration than mature technology, however, and highly autonomous multi-agent systems may encounter problems similar to those in human collaboration. This is another technical “A or B” question closely tied to agents.

How does the team evaluate its agents? Beyond conventional LLM benchmarks, are there business-specific measures of tool-call accuracy, execution traces, or automated evaluation frameworks?

Note: Evaluation is unavoidable for agents. Without it, it is difficult to tell whether a system has improved or the demo simply looks smoother. If a company wants highly autonomous agents but does not maintain its own evaluation suite, it has not yet got on the right track.

How does the team balance cost and latency? Does it mainly rely on established proprietary models, or does it route different requests to different model tiers? Are the trade-offs between cost, latency, and performance evaluated systematically?

Note: These are unavoidable engineering questions. Moving from a technical demo to production requires careful consideration of cost, latency, performance, and safety boundaries. Production safety boundaries should not be casually traded away, while taking a little longer or spending a little more can sometimes be acceptable.

If a team has thought this through, it suggests that its application is already reasonably usable. Simply having an automatic router does not count, although having no routing at all is worse.

Teamwork, Collaboration, and Personal Growth

How much freedom does each developer have to make decisions? How is the team structured, how do people collaborate, and how is everyday work divided? How are disagreements usually handled?

Note: This gives an overview of how the team works and opens the door to follow-up questions about team atmosphere and technical collaboration.

What does a typical working day or week look like?

Note: This is one possible follow-up.

AI coding tools are now part of most programmers’ workflows. How does the company view AI-assisted coding? Are employees allowed to use it extensively to speed up iteration? Does the team provide shared tools, accounts, or usage guidelines?

Note: It really feels hard to survive as a programmer without AI now, though this comment does not apply to infrastructure experts or people writing kernels. Providing a Pro5x subscription is the minimum a company should meet.

What do you like most about working here? Why did you choose this company, and why have you stayed?

Note: This feels a bit like putting the interviewer on the spot. I copied it from the internet and take no responsibility for it.

What are the team’s working hours and overtime expectations? How does on-call work, and is overtime paid?

Note: Work-life balance may deserve more thought for a full-time position. For an internship, I am less concerned. I already live the “9117” life at university; can a company really work me harder than the lab does? The Lei Jun Building has free air conditioning, free coffee, and showers. Give me a bed and I could live in the office.

Can I contribute to open-source projects? Would I need approval?

Note: The question speaks for itself.

Does the company hold technical knowledge-sharing sessions? If so, how often?

Note: The question speaks for itself.

Are there company-wide learning resources, such as ebook subscriptions or online courses? Is there a budget for certifications or other learning expenses?

Note: Spending the company’s money on my own learning has the satisfying feel of getting a perk. It may be tiny compared with my salary, but somehow it still feels great.

Is the company profitable? If not, when does it expect to become profitable? If it is, roughly what is its annual revenue? What are its plans for the future?

Note: This is worth asking at startups and smaller companies that operate like startups. After all, a company that never makes money is slowly going out of business.

What is the balance between remote work and working in the office?

Note: Does remote work really exist in China?

This article will continue to be updated over time.

Does a 30/30/30+10 Portfolio Really Diversify China A-Shares?

2026-08-28T06:00:00.000Z

I recently wanted to test a very intuitive portfolio: hold the CSI 300, CSI A500, and CSI 1000 in equal proportions. If I wanted to leave a little room outside the core, I could reduce each to 30% and put the remaining 10% in cash, renminbi gold, or the STAR 50. It seems to combine large caps, representative companies across industries, and small-cap growth. The rule is also simple enough to implement with the relevant funds, even without an exchange-traded account.

Rebalancing frequency was the question I first wanted to investigate. Once I began, however, a more basic question appeared: what do these three names actually buy when they are placed together?

The CSI A500 is not “the 500 companies between the CSI 300 and CSI 1000.” That position is closer to the CSI 500. The CSI A500 selects relatively large and liquid representative securities across industries, so it repeatedly owns many of the same large companies as the CSI 300. The CSI 1000 supplies the clearer small-cap allocation.

What am I buying when I build this portfolio? What return and risk does it carry, and what does rebalancing change? Those are the questions this post tries to answer.

A nearly fully invested A-share portfolio cannot escape the market’s aggregate risk. Maximum drawdowns are around 70% across the rules. Quarterly and threshold rebalancing each have practical advantages, but neither leaves the alternatives far behind statistically. The 10% satellites create more visible differences. Since 2020, cash has reduced volatility and drawdown at the cost of some return. Gold is the only satellite in this sample to improve return, volatility, and drawdown at the same time, although a simpler 90% CSI All Share plus 10% gold benchmark does at least as well. The STAR 50 behaves more like an aggressive growth tilt. A rule that progressively spends cash after drawdowns does not control risk here; it deepens the drawdown.

This is a personal backtest for reference only, not investment advice.

Three indices are not three disjoint baskets

The CSI A500 factsheet defines it as 500 relatively large and liquid representative securities selected across industries, emphasizing sector representation. The CSI 500 is a size index and forms a more direct market-cap ladder with the CSI 300 and CSI 1000. Both happen to contain 500 constituents, but they are designed for different purposes.

As of July 31, 2026, the count-based Jaccard overlap between the CSI 300 and CSI A500 is 42.10%, while their weighted overlap reaches 78.68%. The CSI A500 and CSI 500 have a 24.38% count overlap but only 16.57% weighted overlap. The weighted overlap between the CSI A500 and CSI 1000 is lower still, at 1.64%. The CSI A500 broadens coverage, but most of its weight remains on the CSI 300 side of the market.

Overlap uses only the official July 31, 2026 constituent snapshot. Published weights are rounded to three decimal places and are normalized to 100% before calculation.

Looking through the three indices turns 1,800 nominal slots into 1,522 distinct stocks. Of those, 278 occur in two indices. The top ten account for 14.91%, and the effective number of holdings is about 243. CATL, Zhongji Innolight, Kweichow Moutai, Ping An, Eoptolink, and Zijin Mining all receive weight from both the CSI 300 and CSI A500.

The industry distribution is just as revealing. Industrials account for about 22.17%, information technology for 21.97%, and financials for 11.50%. There are many stocks, but the weights still cluster around a few sectors and shared market leaders.

The left panel shows P0’s top 15 look-through holdings. The right panel compares only the CSI 300, CSI A500, CSI 1000, and the weighted P0 result. The CSI 500 and STAR 50 are not in this figure.

This does not mean the portfolio has no diversification. It is broader than the CSI 300 alone, and the CSI 1000 adds genuine small-cap exposure. It still does not create independent risk sources. A CSI 300/CSI 500/CSI 1000 mix is closer to a clean large/mid/small-cap segmentation. The CSI A500 is the structural choice when the goal is to add industry representation within the large-company allocation. I compare the two structures later in the article.

The long-run theoretical layer uses official total-return indices. H11025, the CSI Money Market Fund Index, is the cash proxy, while renminbi gold uses the Shanghai Gold Exchange Au99.99 close. The CSI A500 was not launched until September 23, 2024, so most of its history here is a rules-based backcast. The new background boards use the same official total-return convention: H00922 for the CSI Dividend Index and 932000CNY010 for the CSI 2000. The CSI 300 continues to use the previously frozen H00300 series and was not downloaded again.

Portfolio rules

There are five principal portfolios:

ID	Target allocation	Role
P0	33.33% each in CSI 300/CSI A500/CSI 1000	Three-index CSI A500 core
P1	30% each in the P0 indices plus 10% cash	Defensive satellite
P2	30% each in the P0 indices plus 10% renminbi gold	Diversifying satellite
P3	30% each in the P0 indices plus 10% STAR 50	Aggressive satellite
S0	33.33% each in CSI 300/CSI 500/CSI 1000	Size-segment control

P0 through P3 run six rules: buy and hold, monthly, quarterly, annual, daily threshold checks, and month-end threshold checks. Calendar signals are generated at month-, quarter-, or year-end close and executed at the next common trading-day close. Threshold signals use the same T+1 timing. Information observed after the trigger-day close cannot be traded at that same close.

The main threshold is reached when any asset deviates from its target by

\min(5\text{ percentage points},\ 25\%\times\text{target weight}).

The entire portfolio then returns to target. The narrow and wide settings are 3 percentage points/15% and 7 percentage points/35%.

The theoretical main result charges 10 bp per side and also checks 0, 5, and 20 bp. Costs are applied to actual traded weight on each asset leg. Risk ETFs use 5 bp per side and 511990 uses 2 bp per side:

C_t=V_{t^-}\sum_i|w^*_{i,t}-w_{i,t^-}|c_i.

This assumption is somewhat higher than my actual trading cost. As the results show, costs at these levels barely change this low-turnover strategy.

What rebalancing buys over the long sample

P0 begins at the December 31, 2004 index base date. Its six rules produce the following results:

Rule	CAGR	Annual volatility	Max drawdown	Calmar	Annual rebalance turnover	95% weight drift
Buy and hold	10.11%	25.67%	-71.35%	0.142	0.00%	18.19%
Monthly	10.54%	25.37%	-71.22%	0.148	22.92%	1.77%
Quarterly	10.53%	25.37%	-71.31%	0.148	12.32%	3.15%
Annual	10.56%	25.42%	-71.39%	0.148	7.21%	6.37%
Daily threshold check	10.62%	25.38%	-71.19%	0.149	8.12%	3.85%
Month-end threshold check	10.60%	25.39%	-71.38%	0.148	6.65%	4.48%

Theoretical total-return indices, 10 bp per side, logarithmic y-axis. Pre-launch CSI A500 history is backcast.

The daily threshold rule has the highest CAGR in this sample and trades less than the quarterly rule. It looks attractive, although checking every day also consumes attention. The month-end threshold rule offers a more practical balance between return and turnover. In absolute return terms, however, the rules are not very different. The remaining experiments therefore use quarterly rebalancing, a simple and transparent convention that is also common among funds.

I originally plotted only rolling three-year annualized return. I have now added 30-trading-day and 244-trading-day windows. Annualizing a short window can magnify an ordinary fluctuation into a misleading number, so the 30-day and one-year rows remain realised holding-period returns. Only the three-year row is annualized.

P0 rolling holding period	Median	10th percentile	Sample worst	Share of negative windows
30 trading days, not annualized	1.05%	-9.05%	-33.13%	44.65%
244 trading days, not annualized	7.24%	-20.83%	-67.47%	36.65%
3 years, annualized	5.85%	-7.50%	-15.37%	34.37%

The share of P0 windows with a negative return does decline as the holding period grows, but it never becomes negligible. These rolling windows overlap heavily, so 34.37% should not be read as a loss probability estimated from independent trials. The tails matter more. The worst 30-trading-day window, ending July 8, 2015, lost 33.13%. The worst one-year window, ending October 31, 2008, lost 67.47%. The worst three-year window, ending June 14, 2018, still annualized at -15.37%. A longer investment cycle smooths some short-term noise; it cannot assume the risks of entry timing and equity exposure on the investor’s behalf.

Rolling results use quarterly-rebalanced NAV. The annual heatmap excludes only partial first and last calendar years. Pre-launch CSI A500 history remains backcast.

No matter how often the portfolio rebalances, its long-run maximum drawdown barely changes—and 71% is frighteningly high. More stocks reduce individual-company risk, but they do not turn a nearly 100% A-share equity portfolio into a low-risk asset. Market, liquidity, and valuation cycles remain dominant. Concentrating the initial purchase near a high can still turn a one-off investment into a very long lock-in.

P0 uses the CSI A500; only S0 uses the CSI 500. The CSI 500 size-segment control S0 has a 10.80% long-run quarterly CAGR, 26.63% annual volatility, and a -71.90% maximum drawdown. The CSI A500 core P0 records 10.53%, 25.37%, and -71.31%. Moving from S0’s CSI 500 structure to P0’s CSI A500 structure reduces sample volatility by about 1.25 percentage points, gives up about 0.28 points of CAGR, and modestly improves drawdown. None of these differences is statistically significant. The backcast CSI A500 total-return series alone has a 10.28% CAGR, 24.65% volatility, and a -70.65% maximum drawdown. P0 earns a little more while also taking a little more risk. It remains a trade-off.

Trading costs matter less than I expected. Raising P0 quarterly costs from 0 to 20 bp lowers CAGR from 10.55% to 10.51% because turnover is modest. The theoretical layer does not include minimum commissions for small accounts, bid-ask spreads, round-lot constraints, or intraday impact, so real execution may be less forgiving.

The narrow, main, and wide threshold bands do not produce a neat monotonic pattern either. The main band happens to look better in the sample, the narrow band trades more, and the wide band drifts more; the differences remain small.

Three more background boards: dividend, CSI 300, and CSI 2000

P0 and S0 alone still do not show where this core sits next to more familiar style and size indices. I therefore place P0 beside the CSI Dividend Index, CSI 300, and CSI 2000, rebuilding every path from the CSI 2000 base date of December 31, 2013 through August 28, 2026. P0 remains the equal-weight CSI 300, CSI A500, and CSI 1000 core with quarterly rebalancing. The three single-index paths are buy and hold. All four include the same 10 bp initial transaction cost.

Common-window background	CAGR	Annual volatility	Max drawdown	Calmar
P0, CSI A500 core	7.66%	21.95%	-53.56%	0.143
CSI Dividend total return	12.21%	20.13%	-45.66%	0.267
CSI 300 total return	7.99%	21.02%	-46.06%	0.173
CSI 2000 total return	10.26%	27.69%	-67.35%	0.152

In this sample, CSI Dividend has the highest return and the lowest volatility and drawdown. P0 does not beat the CSI 300 and also draws down further. CSI 2000 has a higher CAGR than P0, but it also produces the highest volatility and deepest drawdown. On this background board, smaller-cap exposure is not free diversification. The help from the dividend index is better understood as the realised return of a distinct high-dividend style in this period, not as a promise that the same style will lead in the next one.

Cumulative NAV makes the terminal difference easy to see, but it is sensitive to the chosen start and end dates. I therefore repeat the earlier 30-trading-day, 244-trading-day, and three-year tests on these four common-window paths. The 30-day and 244-day panels show realised holding-period returns; only the three-year panel is annualised.

Common-window path	30 days: median / negative windows	244 days: median / negative windows	3-year annualised: median / negative windows
P0, CSI A500 core	0.92% / 44.16%	6.77% / 37.83%	3.80% / 34.83%
CSI Dividend total return	1.38% / 38.00%	9.92% / 16.93%	8.61% / 3.75%
CSI 300 total return	0.81% / 44.72%	10.81% / 38.58%	6.74% / 26.81%
CSI 2000 total return	0.33% / 48.59%	6.13% / 35.79%	3.66% / 38.53%

Over 30 trading days, all four paths frequently fall below zero. CSI 2000 does so in nearly half of the windows, while even CSI Dividend records 38.00%. At 244 trading days, CSI 300 has the highest median return, but 38.58% of its windows are negative. CSI Dividend has a slightly lower median and a much smaller negative-window share of 16.93%. The median alone misses that difference.

At three years, CSI Dividend records an 8.61% median annualised return and only 3.75% negative windows in this sample. P0 records 3.80% and 34.83%, while CSI 2000 records 3.66% and 38.53%. Combining the three indices does not create a smoother holding experience than these familiar indices in this common window. All rolling windows overlap heavily, so the negative-window share is a sample description rather than a loss probability estimated from independent trials. The robustness section therefore uses a paired block bootstrap to preserve the time dependence in return ordering.

The dates still matter. CSI Dividend was launched in 2008, so this common window is entirely post-launch for that index. CSI 2000 was not launched until August 11, 2023, and CSI A500 not until September 23, 2024; their earlier curves are official backcasts. The boards show what happened in this historical sample. They do not rank future returns.

The remaining 10%: cash, gold, or STAR 50

The satellites are comparable only on the same window. The pre-registered window runs from January 2, 2020 through August 28, 2026, with quarterly rebalancing and 10 bp per side:

Portfolio	CAGR	Annual volatility	Max drawdown	Calmar	Change relative to P0
P0, no satellite	5.30%	20.00%	-38.55%	0.137	Baseline
P1, 10% cash	5.12%	17.98%	-34.93%	0.146	Return -0.18 pp, lower risk
P2, 10% gold	6.73%	18.30%	-33.34%	0.202	Better sample return and risk
P3, 10% STAR 50	5.88%	20.81%	-40.06%	0.147	Higher return and higher risk

Cash gives up 0.18 percentage points of sample CAGR in exchange for 2.02 points less volatility and a 3.62-point improvement in drawdown. In the 20-day block results, the Calmar and drawdown improvement intervals are positive, while CAGR and Sharpe cross zero. Cash is simply reducing equity exposure: the defensive effect is visible, but return enhancement is not.

Gold looks best in this period. Relative to P0, it adds 1.43 percentage points of CAGR, reduces volatility by 1.71 points, and improves maximum drawdown by 5.21 points. The Sharpe, Calmar, and drawdown intervals are positive, while the 95% CAGR-difference interval still spans -0.39 to 2.96 percentage points. The fair statement is that gold diversified this period. The evidence is not strong enough to turn that into a claim that gold must raise long-run return.

A simpler benchmark also warns against assigning credit to the wrong component. A 90% CSI All Share plus 10% gold portfolio has a 7.23% CAGR and -31.99% maximum drawdown, slightly better than P2 at 6.73% and -33.34%. The difference has no statistical support. Most of the visible help comes from gold itself, not from an independent alpha generated by the three-index core.

The STAR 50 is a different proposition. It adds 0.59 percentage points of CAGR and 0.81 points of volatility while worsening maximum drawdown by 1.51 points. Every bootstrap interval crosses zero. Adding it can express a preference for growth and technology; it should not be called defensive diversification.

There is also a STAR 50 launch-date issue. Moving the starting date from the pre-registered January 2, 2020 to the formal July 23, 2020 launch changes the CAGRs of P0, cash, gold, and STAR 50 to 2.60%, 2.67%, 4.10%, and 2.85%. Maximum drawdowns remain -38.55%, -34.93%, -33.34%, and -40.06%. Cash moves from trailing P0 by 0.18 points to leading it by 0.07 points, a reminder that point estimates in short windows are unstable. Gold still improves risk and STAR 50 still increases it.

Dynamic cash did not become risk control

The dynamic rule uses a zero-cost, quarterly equal-weighted three-index core as its reference. When the reference first crosses drawdowns of 10%, 20%, and 30%, the cash target falls from 10% to 7%, 4%, and 0%. Cash returns to 10% only when the reference NAV recovers to within 5% of its previous high. Within one drawdown cycle, cash can only be spent. A partial rebound does not refill it.

That sounds like risk control, but the trade itself is to buy more as the market falls. If the market keeps falling after cash is deployed, the portfolio approaches full equity exposure at the riskiest point. Rebuilding cash near the previous high then requires selling some equity. The rule may increase participation in a rebound; that is not the same as reducing losses.

Rule	CAGR	Volatility	Daily 95% ES	Max drawdown	Calmar	Annual turnover
Static 10% cash	11.17%	22.92%	-3.76%	-66.81%	0.167	15.77%
Dynamic 10/20/30	11.02%	25.02%	-4.10%	-70.54%	0.156	21.66%

Cash only declines within a drawdown cycle and resets after recovery to within 5% of the high.

The main dynamic rule fails the success condition fixed in advance. Drawdown and ES worsen, Calmar falls, and CAGR loses about 0.15 percentage points. The 20-day block 95% interval for its maximum-drawdown difference relative to static cash runs from -6.68 to -1.74 percentage points, and the Calmar interval is also below zero. Earlier 8/16/24% or later 12/24/36% triggers do not reverse the direction.

I would therefore no longer call this dynamic risk control. It is a buy-the-dip rule. Whether buying after a decline improves future returns and whether it reduces current losses are two separate questions.

How I interpret the portfolio now

If I wanted an A-share core that is easy to maintain, I would begin with quarterly rebalancing. Threshold rebalancing is also reasonable and happens to trade less with a slightly higher sample return, but the difference is not statistically confirmed. The practical choice returns to actual account fees, how often I am willing to check the portfolio, and how much weight drift I can tolerate.

If drawdown matters more, static cash is clearer than this dynamic cash rule: hold less risk and accept the corresponding opportunity cost. Gold diversified the post-2020 sample more effectively, but it has its own price cycle. The Au99.99 theoretical layer does not include ETF fees, custody costs, or gold-lending income. I would not extend the result from these years directly into the future.

The STAR 50 expresses a different risk preference. It makes the portfolio more tilted toward technology, growth, and high volatility. It may add upside sensitivity and may also deepen drawdowns. Different constituents do not automatically mean better risk diversification.

Returning to the original question, a CSI 300/CSI A500/CSI 1000 mix remains a nearly all-equity A-share portfolio. The CSI A500 and CSI 300 overlap heavily. The CSI 1000 broadens stock coverage but does not create a cross-asset source of risk. The CSI 2000 background board reinforces the point: adding still more small-cap stocks can also bring higher volatility and a deeper drawdown. Cash and gold, rather than another differently named equity index, are what materially change the drawdown structure.

I began hoping to find one clear rebalancing answer. What remains is a set of trade-offs: quarterly is easy to maintain, thresholds reduce unnecessary trades, cash defends, gold diversifies in this sample, STAR 50 attacks, and the dynamic cash rule did not work. Keeping the positive, negative, and statistically inconclusive results together is closer to the actual portfolio than preserving only the highest cell in a backtest table.

Data and references

CSI A500 factsheet
CSI 300 factsheet
CSI 500 factsheet
CSI 1000 factsheet
CSI Dividend factsheet
CSI 2000 factsheet
Shanghai Gold Exchange daily data
AKShare public-fund data documentation
Bailey & López de Prado, The Deflated Sharpe Ratio
Vanguard, The rebalancing edge

How I Build RAG: From Project Experience to a Systematic Approach

2026-08-18T02:00:00.000Z

This article has moved to the Build Your Own RAG repository. Please visit the repository for the tutorials and examples.

The AI Replacing Front-End Engineers Can't Build the Simplest Panel in a Game

2026-08-08T04:00:00.000Z

Order

The questions in this article can also be addressedWhen we talk about AI Native Game, what are we talking about?、How the game industry is introduced AI AgentHow the concept of a relatively close read together is developed in different contexts.

This is a phase-out of my Internet-friendly internship, and I'm exploring how AI should get involved in the game development process.

AI is already a daily way to generate a web page. The same model gives it a complete visual copy, which gives it a panel in the game: a background, buttons, a reward panel. I've been doing this for two months, and I can't say it's working, and it's a long way from the web.

Why?

The game UI is more complicated? It's more like Web in the ceiling. The Corhen Game is the middle of the UI-writing game with HTML/CSS/JS, which is used by Minecraft, Civilization VII, Alan Wake 2; the format level is also not at all, and the new UXML and USS are written by Unity as labels and style sheets; the cocos and XGUI interface files are simpler JSON or variants. Unreal's UMG is really the most complex, but it also exposes a lot of standard interfaces that we can programmed to spell out.

The game UI needs an art design? If the starting point is planning, it is indeed a problem. But in the real division of labour, the visual drafts are a step that the GUI designers have already done. The rest is to translate it into UI engineering files in the engine, without beauty or creativity, with care and a lot of repetition. And this is exactly where AI should be stronger than anyone.

I can't do it. Neither the AI tools of the commercial engine nor the internal programs that the teams are doing, any one of them can say that they have solved the problem and then go out and get it used by everyone.

Before the text is expanded, two of the texts that will be used throughout the text will be here:

The ease of the Internet may be the best company to do UI workflow AI research, and the least suitable company to actually do it. One company has a combination of Unity, Unreal and a number of self-study engines, and almost all the games in the market UI you can find the team. This means that the problem of commonality will be exposed more quickly. But the more technology is routed, the less end of fit work, the harder it is to get to the point.
Before you do an AI Agent, the most important question is the target. Is it a Copilot for the people now, or is it Agent to take over a post or a process? These two objectives sound like a single step away, and may actually be completely different technical routes and completely different costs.

What have I done lately?

UI Agent tried to solve the problem

The game UI production chain is not short. Plan for demand, UX gives interactive programs, GUI output visual design, UIP builds projects in engines, VX adds effects and animations, and finally program access to game logic. What UI Agent wants to do is not add a new assistant to it, but to shorten the production chain across the job.

The ideal form is today's Coding Act: Gives a textual demand and gets a product that can run. But every link in the UI production chain is not suitable for AI. The aesthetic and design intent still require judgement, and AI is better suited to take over the later work. So we put the boundary at UIP: enter the finalized draft, output the UI Project and the Compass Code that can enter the project.

Before developing such systems, it is necessary to clearly enter and export them. The input here is a Figma or PSD design, with a small amount of planning information; the output is a complete UI project file and the corresponding program code.

Put input and output together, and the problem is clear. The design describes "what it looks like," and the UI Project document also says "How it works." Which layers form a button, press what status they switch to, where the data from the incentive list comes from, who stretches after the resolution changes, who stays, and which information does not exist in the static design.

This is also why it was difficult to establish an automation programme in the past. The output of the certainty script can only come from input, information not found in the design, and the script cannot be added to it. The requirement for designers to add agreed fields to the name of the layer would alleviate some of the problems, but at the same time, the new work would be pushed to the designer.

The generation model brings another source of information: a priori accumulation in pre-training. Models see a row of equal distances and identical structures, and can be assumed to be a list; and a small fork at the top right corner can be presumed to be a closed button. It is still guessing, but this speculation is for the first time making the matter theoretical: without increasing the workload of the designer, it can reduce the manpower in the latter part.

Stage 1: Do not let the model write the final file

The idea was straightforward: since there was no difference between the UI project and the Web UI project, let AI write it like a web page. We're writing in Skill, telling Agent how to analyze PSD, export cut-ups, initialize projects, generate prefab and perform validations. After reading the manual, Agent calls in a dozen command line tools. This version is intuitive and does generate documents.The problem is that most of the documents generated cannot be opened. A whole page of UI works is usually dozens of nodes, thousands of lines, JSON. Each node has its own list of transforms, anchors, offsets, dimensions and components. In cases where the model has hardly seen such a format, it must have miswritten certain fields and then spent a great deal of time on changing the format. In the initial test, approximately 10 minutes were required for one generation, almost half of which was in the process of re-engineering documents.

Then let's just stop having the model write the final file. We added a smaller semantic IR, so that the model would only describe "It's a button" "The labels are here" "These elements belong to the same group" and coordinates converted, fuid, component lists and type maps to the definitive compiler. The format error thus disappears, and IR becomes a generated record. People can check and modify it directly without facing thousands of line target documents.

In white, it is a more advanced language (DSL) that allows models to write advanced languages, with the rest to be handed over to the compiler. Advanced languages are better taught than lower languages, as are models for people.

Stage 2: Turn prototype into engineering pipe

IR and compiler solve "files are right," but how does the real script fit into the system and how does the result be verified?

Stage2 did not replace the technical route ahead, but replaced the existing prototype with a pipe that would handle real inputs, generate real work and accept real validation.

The first paragraph is definitive pre-treatment. System solvers PSD/PSB and smart objects, extract the layer trees, position dimensions, text, hybrid mode and transparency, generate synthetic and cut-off resources, and prediscover the replicating clusters and large sections of the background with geometry.

The second paragraph is the model. It makes only two semantic judgements: first, to determine whether a set of layers should be preserved, unopened or merged, each of which controls corresponds; then to organize these elements into father-son structures and extrapolate anchor points. The results are written into the list of elements and the structure tree, one of them. .md The document's a little bit of a mix. .yaml The file is composed of IR. A person can check it directly, and the model does not need to write thousands of lines of target files.

The third paragraph is a definitive compiler. IR after validation, the compiler is responsible for coordinates conversion, anchor calculation, type mapping and assembly of components, which eventually produces complete .uiprefabI'm sorry. Of course, we actually kept two compilations in the real project, a purely script-based tool and a project assembly based on the Function Calling, and then we talked.

Finally, it's validation. Static checks cover issues such as missing resources, nodal naming and structural errors, which are then handed over to the real editor to load the document, screenshot and read the log. JSON does not legally represent that the project is really working, and the visual reduction of the editor is also an important feedback signal.

At this point, the system has been able to do more than generate a document. It has a clear input contract, an intermediate layer that can be read by a person and model, a back end where results can be repeated, and a mechanism for self-certification. It runs the draft in its entirety, achieves the engineering closure of the existing route and exposes new problems.

Stage 3: Turn the process itself into code.

Stage3 has run through the pipe, but it's slow and unstable. Logs are full of errors, repairs and retests; changes in Skill's hints may cause the whole process to collapse in unexpected places.

Where's the source?

The process is actually a manual. The order of the call of the ten steps, what to do after failure, when to try again, when to give up, are all written in a manual for models. The manual can only tell the model what to do and cannot be forced to implement it sequentially. The consequences are twofold: irrecoverable, with the same draft running twice, with different paths and different outcomes between them; failure without attribution. The transmission timeout, model output rejected by the calibration, cut-chart resources missing, editor unable to open, many errors without clear processing paths, and models can only be repeated once in many cases.

So Stage3 does a simple thing: move the process from the natural language into the code and build a workflow system.

We've made a map of the entire generation, 54 nodes, of which only 10 are actually called. Each node has a clear input, output and reversible action, and the process can only advance along the previously stated side. The model can still be judged, but it cannot change the order of implementation at will or invent a treatment after failure. Each action contains clear semantics, such as applicant worker access, failure of external service calls, error of function request, error of assembly.

Re-test and re-repair need to be treated separately.

The transmission layer fails, such as timeout, disconnection, incomplete response or JSON cannot resolve, and the system is retrying at the current node. If the model output returns in its entirety, but is not checked, the system will take the inspection report back to the duty node according to the wrong type, which is the back-up. The checker is no longer solely responsible for declaring failure, but also for telling the process what is wrong and where it should be repaired. (At the time being, the checker is a script that returns to the real and false: it can stop the error, but does not tell the follow-up process what to do.)

Run status will keep the disk log running, and checkpoint will remain at each node. Logs save a full run track, and checkpoint supports recovery and recurrence from an error node without running from the header at each time. These changes make it easier for UI Agent to operate and to be iterative.

But stability and utility are two things. This process can reliably end a draft from scratch at the cost of having it done from scratch every time; in real projects, the interfaces that are built from the beginning are only a few.

Stage 4: UI in Real Project, not every time you build it.

The UIs in mature projects rarely start from scratch. The longer a project runs, the more changes are made using the interface already in place. "Replace the original interface with this design and change it to a new one" is more common than that. Stage4 therefore no longer asked how to generate an interface, but how to start processing existing projects.

This was done by introducing a declaration document, which we call the blueprint. It states the following categories:

Reference: Project documents as skeletons
Source of output: Which layer of path is in the draft design for each redo
Target binding: which node of the output is loaded on the reference and which assembly method is used
Reservations and exclusions: which nodes must be retained as they are and which draft paths do not participate

Runs when the system reproduces the reference, recreates only the contents within the blueprint statement and then reloads them back to the given location. The part that needs to be modified is usually less than half the entire page and the nodes, layouts and logical interfaces in the existing project can continue to be used.

The assembly is made on a white list basis. Each of the five assemblers allows changes to only a small number of clearly listed fields. For example, the list assembly can only replace the XML reference, and the hang-up assembly can only modify the XControl path and cannot touch the position, rotation and scaling of the mount itself. Even if the model is miscalculated, its potential for destruction is limited to the blueprint statement. This is the same idea as the previous one, which bound the final document with the compiler.

The blueprint is much more stable. The four recent blueprints have crossed many authentic designs, all running through the entire process, taking time less than the one that was generated from the beginning (the blueprint generation took about one minute, and the animations took a little longer, less than the new 3-10 minutes). This is not a model that's getting stronger. We're giving it more knowledge, just making it do less.

This transaction is not cost-effective, depending on how many times the blueprint can be reused. Creating blueprints also entails costs (with automatic creation but not good enough) and one project and one set of norms, none of which can be taken away.

Let's talk about our understanding of the problem from the differences between different programmes

How to reuse control: Tasks should not be transferred

An important difference between the game UI and WebUI is how the control is reused. Web projects usually develop interfaces and logic simultaneously, and models can generate both directly. The video assets and program logic of the UI game are performed by different positions, so we will seal the high frequency structures of buttons, lists, bullet windows and so forth into prefab to reduce the cost of subsequent re-laying and access.

The system was originally designed for people. It takes a few days to get acquainted with the template warehouse of a project, but the model has difficulty obtaining the same knowledge. Open WebUI is very popular, game UI engineering is very small, and self-study engine data is less likely to enter model training. The more well-encaped and easily used the past, the more knowledge the model needs to add to the project.

We want to use the zero-shot panduration of the generation language model to solve UIP problems, which is not trained in the data at all, and which is certainly much more difficult than doing WebUI.

So how do we solve this problem?

Do large-scale multimodular pre-training to allow models to learn directly about engines and controls
Create a master-speech knowledge base to make project knowledge available to models at running
Search for reusable controls from inventory assets by multi-modular search
In Figma or PSD, indicate the control type earlier

Large-scale pre-training is the most direct, but also the highest cost. Only the developers of commercial engines such as Unity, Unreal may have collected sufficient data, and small technology houses and self-study engines are hardly in a position to follow this path. Even commercial engines have not yet solved the problem of UI generation with pre-training.

The knowledge base is a more realistic approach in the short term. The model does not know what controls are in the project and provides it with instructions, conditions of use and interfaces. We have done it, and we can solve some of the problems. The problem is that the number of prefabs in the project may be very large, and some of the knowledge must be combined with pictures to understand, and all of it would soon get out of control. Visual retrieval can be added to this: the model does not need to determine the type of control accurately, but simply to find assets that look and structure close to each other from the existing interface before reading the corresponding project information.

The last thing I'd like to see was a front-poster. It is usually able to produce the best results, as the designer has already completed the semantics missing in the system in advance. But it didn't eliminate jobs, it just transferred a part of UIP's work to the GUI. LLM has been in the history of similar layer naming, structuring and control labelling programs many times before their appearance, often ending up in the Demo phase. Evaluation of UI automation cannot be based solely on the generation of results, but also on whether the whole line has been reduced.

I've been looking around for the other team's options. The project team's procedures are largely based on a knowledge base: the small amount of exposure in the training language supports the zero-shot and injects some of the norms within the project (i.e. SPEC). AI Lab and XGUI, in turn, rely on the interventions of labeling and pre-stratification down LLM in return for faster and more steady generation.

The more people close to the project are willing to feed their knowledge to the model, the more people far from the project are inclined to block uncertainty before the model, which is an interesting discovery.

File format determines the technical route and how Agent interacts with it

As mentioned earlier, in addition to the definitive compiler, we have also retained a set of realizations based on the Function Calling. Agent no longer generates complete target files, but instead calls a set of tools to build and modify UIs over time. Why do you keep two routes in the same project? This question starts with the UI Project file itself.

The marketable UI project formats can be broadly divided into three categories:

Sequenced text represented by Unity UGUI..prefab、.unity、.csd and .uiprefab All of them belong to this route. The project is modified by the editor and re-sequenced into long text similar to that of JSON or YamL.
The statement format is represented by WebUI and Unity UI Toolkit. Structures, styles and renderings are relatively separate, and documents originally considered manual, version management and branch consolidation.
In Unreal .uasset is the binary asset represented. The model cannot read and write directly, but only operates through the editor and the interface to which it is exposed.

The article title says AI can write well for WebUI, but it's not good for UI, and it's also about the file format itself. UGUI prefab is also text, but models are difficult to modify directly. Because these documents are only the result of the serialization of the editor, they are never prepared for handwritten writing. A field or reference is wrong, and the entire document may not be loaded. And here, the title party in the title needs to take it back a little: the text format is not the same as the model.

The three formats naturally lead to three automated routes. UGUI, which is a sort of serialized text that is readable but not suitable for direct editing, is more suitable for adding an IR or DSL to produce the final file by a definitive compiler; WebUI can directly use the code capabilities and training language available in the model; face-to-face .uasset Such binary assets can only be implemented gradually by the editor interface if Agent develops a plan for their modification.

But it's only half the problem. The progress of Coding Age in the last six months has been made not only by the stronger underlying models, but also by the cheap and intensive feedback that code ownership naturally takes place. The web page contains a syntax check, running error, DOM status and browser screenshots, and the model can be quickly updated to understand what happened. The UI compiler usually only tells you whether the document is legal, but it is difficult to judge whether the control type is correct, the structure is reasonable and the visual reduction is met. These feedbacks need to be designed separately.

The construction of the UI project by nodes with tools may lead to more rounds of decision-making and token consumption (the programmmatic tool Calling will mitigate part of it), but it is easier to design a good set of tools than to compile them. A good set of tools is not just a tool, but a world where models can be known and reached. For AI, learning tools on zero-shot are easier to learn than learning DSL. Based on tools to connect to UI projects, the landing threshold is lower than the compiler route.

Because of the company's huge use of the project .csd So when you have a serialized text UI project, we can find that you have basically chosen the DSL idea, so that LLM can generate JSON or other forms of IR and then produce the final result through the compiler. G85 has an interesting attempt: to get Agent to put the UI structure out of Figma first, as the only credible source, and to export still away from the compiler, but it does work better. I guess the reason is that Figma's interface is more mature than the MCP of the self-study editor, and the model knows what's going on faster than it does with every change. That is also the reason I prefer the tool route, although only one case is not conclusive.

From running back to Benchmark: How does the system know this is better?

Tools exposed UI projects to Agent. But how much can it see, and then can it be judged that it did wrong?

This depends on the feedback system. The visual similarity can only answer the final image. The UI project also requires checking node structure, resource references and control types, and entering the editor is also dependent on loading logs and interactions. Feedback needs to indicate where the error occurred and which nodes should be repaired. The feedback in the previous section refers to this.

Run-time feedback is used to correct the current generation, and Benchmark is using the same set of checks to compare different versions with a representative set of cases. Changes in the hints, tools or workflows should allow the system to re-run, if there is progress, if returns are introduced, and if errors are moved from one layer to another. Operating and assessment environments are best shared. Feedback was originally the same as that of Benchmark, the same engineering input.

Only run-time feedback and Benchmark are not entirely sufficient. Anthropic. C Compiler ExperimentUsing test sets, CI and GCC oracle to drive Agent iterative, Linux can eventually be compiled, but the code is not of sufficient quality. Feedback can move the system towards a functional target, but does not automatically define maintenanceability and human taste.

As models continue to grow stronger, the injection of generic knowledge and fixed processes will decrease, while project specifications, component habits and human judgement will not automatically enter the model. For businesses that need long-term maintenance, multi-person collaboration and steady access, the whole of Vibe remains unrealistic. For the AI Agent system developed for games UI, both forward to hand over project restraints and human tastes to the model, back to run and Benchmark to check whether it is actually in compliance; what is more easily available in the absence of any side is Demo.

In this section we emphasize that multi-source feedback should be part of the feedback signal, except for the ultimate visual effects. The mature project develops its own habits and norms, which need to be used as a validation feedback and as a source of knowledge. There are also options to spell UIs with only final visual effects and cut maps. The image is right, but the draft, the interactive draft, the project code is all thrown out, and can be seen in the editor.

Multistate components, fonts, action and programs

Fonts are a small number of information that is suitable for completion from the side of the draft. Designers would have chosen fonts that would have bound them to the project font library or to a unified Text Steel, which would have been more reliable than Agent based on pixel guessing. Font information needs to be brought out from Figma or PSD, which is a few things that do not add much to the work of designers, who would have chosen fonts.

Multi-state components and action effects are more difficult to address. The static draft does not have a time axis and does not tell the system how to switch between suspension, pressing and disable, and the information does not exist in itself. For simple multi-state components, this can be achieved from the point of view of the re-use of the control, but the complex multi-state remains manual.

VX is a more complex issue, and key frame animations of properties require simple interactive drafts that are initially generated, and few UI Shaders can be reused. But for complex scenes and particles, Live2D, and sequence frames, AI is virtually impossible to do directly. Some jobs are not duplicated work, but art, and we can only stop here.

The development of UI-related codes is a stand-alone issue, but AI writing codes are more powerful than completing UI components. Like traditional program development, the UI program code needs to solve two problems. The way to inject project norms and tastes, which used to flow only between people, SDD is necessary here, and Figma and UI projects are to be added to the context. The second is how to get the code written by AI to be validated more quickly, which requires better automation of the QA.

UI developed a long Pipeline, which contains a lot of aesthetic and internal norms, and internal processes are much more complex to automate than a web page in Vibe Coding.

Worklow or Age: UI Age is what it should be.

The difference between WorkFlow and Age is how much decision-making is left to models and how much is left as scripts. When the output is stable, the steps to be implemented and the way to check are clear, the Workwork can write the project knowledge and the handling of failures directly into the code; it is only when the target is clear, but the path cannot be listed in advance that Agent needs to choose the tool, observe the results and continue to explore. UI produces both jobs, so both sides have to do it.

There is no need to do a set of roles according to the jobs of Planner, GUI, UIP, and Program. The human division of labour is based on skills and energy constraints, and moving the organizational structure to the model will only reintroduce context exchange and communication costs. The expansion of Agent should be only a task, not a role created to imitate human functions.

A more realistic form is to fix pre-processing, compilation, assembly and validation with the Workflow, leaving semantic judgement, asset retrieval and failure repair to Agent; manual responsibility is to supplement design intent and to identify results that cannot be judged by rules and feedback. It could be used in a UIP-based way, with a complete process gradually being taken over within the system to the extent that Benchmark has proven reliable. The part that relies on project experience and human taste also needs to preserve human collaboration.

In the discussions that preceded, we were thinking about the positioning of our products in the context of the existing UI workflow. The real organizational structure will be useful in the development of Agent, and the emergence of AI Agent will be counterproductive to the organizational structure itself. The game industry itself does not have a clear division of labour like the traditional mobile Internet, and in some small studios UX and GUI are normal and the process often combines the tasks of the fusion. UI Agent is much faster in automation than UIP, and when the support tools are done well enough, the entire UI work chain may need only two jobs to plan and develop, the former to handle ideas and achieve visual effects (in combination with the current AIGAgent), and the latter to think about UI structures and program logic, which may be the organizational structure of the future.

Take Home Messages

The entire semantic divide between the draft and the UI project Control type, interactive status, data binding, self-adaptation rules, are not in the input. This information is not supplemented by a definitive script, which requires the designer to indicate in advance that the activity is being pushed upstream. The generation model gives this missing part the opportunity to be filled for the first time. It remains guessing, but it is the first solution that does not increase the workload of others.
Don't let the model touch something that it's not good at. Do not write thousands of lines, JSON, rewriting an IR; do not use a manual string to use a chart from a statement; do not always generate from scratch, and change to a part that moves only the blueprints on the reference. Each time it's the same thing: narrow the action space of the model and give the identified part to the certainty code. Provides as much information as possible for Agent as is available instead of relying entirely on visual effects.
The format determines the route and the feedback determines the ceiling. Serialized text is suitable for IR-adder compilers, with declaration formats that can be generated directly, and binary assets can only be used as tools. But it can only be produced in half. A large part of the progress made in the six months is due to the fact that the code has a cheap and intense feedback, and UI does not. Building better feedback signals is a task that needs to be considered in the future, and this route is more universal and reliable than compilation.
What we should do today is Copilot, not take over. Workflow fixes, compiles, assembles and validates, Agent is responsible for semantic judgement, asset retrieval and failure repair, and human design intent, and human collaboration is the direction that should be focused at this time. The total autonomy of the work stream of the stock project UI is still far away.

PPT:Downloads design2ui-talk-sanitized.pptx

Clustering Model Evaluation: External, Internal, and Relative Metrics

2026-07-31T04:00:00.000Z

Performance measures for a polygon model

Clusters are a relatively special type of machine-learning task, and we need to give some slightly different indicators of effectiveness.

What are the goals of clustering? Intuitively, we want to see "integrity" in which the same body sample is as similar as possible, and different groups of samples as possible. In other words, the concentration result is high intra-cluster similarity and low inter-cluster similarity.

See you at the presentation of the cluster algorithm itself.Machine learning progress and unsupervised learning: spectral and graphic Category。

External indicators

By definition, external validation measures assume that the precise or real cluster is known in advance. Real cluster labels (i.e. external information) are used to assess a given cluster category. Usually we don't know the exact grouping; but external measurements can be used to test and validate different grouping methods.

All external measurements need one. $r\times k$ List of rows$N$, the table is based on a grouping$\mathcal{C}$Split with Real Value$T$is defined as follows:$$N(i,j)=n_{ij}=|C_i\cap T_j|$$In other words, count.$n_{ij}$Representative Division$C_i$and real value split$T_j$Number of points common to all.

Besides, for the sake of clarity,$n_i=|C_i|$Representative Division$C_i$Number of midpoints,$m_j=|T_j|$Representational$T_j$Middle point number. The list can be used to read$T$ and$\mathcal{C}$Yes.$O(n)$Count it in.

Based on a matching measure

purity

The purity quantifys a fraction.$C_{i}$The extent to which only one divided entity is included. In other words, it measures how "purity" each sub-column is. Split$C_i$purity is defined as:$ \\mathrm{purity}i=\frac1{n_i}\max{j=1}^k{n_{ij}}$$聚类$C$的纯度定义为所有分簇纯度的带权和：$$\mathrm{purity}=\sum_{i=1}^r\frac{n_i}n\text{purity}i=\frac1n\sumI'm not gonna let you go.Percentage of$\frac{n_i}n$For a breakup$C_i$the percentage of points in the middle.

$C$The greater the purity, the higher the degree of conformity with the true value. The maximum purity value is 1 and means that each cluster is made up of only one point in the division. If$r=k$, the purity value is 1 to indicate a perfect grouping, i.e. the cluster corresponds to the division. But even if it's $r.>k$,纯度也可能为 1(当每个分簇都是一个标准划分的子集时)。若$r

Maximum Match

The maximm watching is a map of the selection of the fractions and the dividing, maximizing the number of public points (assuming that one division is given, only one can match it). This is not the case with purity.

Formally, we see the list as a fully-owned part two. Figure$G=(V,E)$Each division and each sub-column is a node, i.e.$V=\mathcal{C}\cup\mathcal{T}$, and there's a side $(C_i,T_j)\in E$and power $w(C_i,T_j)=n_{ij}$,for all$C_i\in\mathcal{C}$and$T_j\in\mathcal{T}$。

Matching in a figure (matching)$M$Yes.$E$A subset that makes$M$The two sides of the equation are not adjacent (i.e. there is no common vertex). Maximum Match Measure is defined as$G$, and then the right match:$$\text{match}=\arg\max_M\left{\frac{w(M)}n\right}$$One of them matches.$M$The right value is$M$The sum of the weights of all sides, i.e.$w(M)=\sum_e\in Mw(e)$

F Measure

Give a scorer.$C_i$You're...$j_i$Organisation$C_i$The division of the maximum points of the midpoint, i.e.$j_i=\max_j=1^k{n_{ij}}$I'm sorry. A partition.$C_i$The precision (precision) is the same as its purity:$ \\mathrm{prec}i=\frac{1}{n_i}\max{j=1}^k{n_{ij}}=\frac{n_{ij_i}}{n_i}$$

Split$C_i$The recall is defined as:

$$\mathrm{recall}i=\frac{n{ij_i}}{|T_{j_i}|}=\frac{n_{ij_i}}{m_{j_i}}$$

of which$m_{j_i}=|T_{j_i}|$I'm sorry. It measures the division.$T_{j_i}$And the partition.$C_i$Proportion of shared sites.

F-measure is the sum average of the precision and recall values of each fraction. Split$C_i$The F-measure is:$F i=\frac{1mathrm{prec}+\frac{1mathrm{recall}=cdot\mathrm{prec} cdot\mathrm{recall}i}{\mathrm{prec}i+\mathrm{recall}i}=\frac{2n{ij_i}}{n_i+m{j_i}}$$聚类$\mathcal{C}$的 F-measure 为各分簇的 F-measure 的均值：$$F=\frac1r\sum{i=1}^rF_i$$

He wants to balance precision with recall.

Measure based on entropy

Conditional entropy

A cluster$C$The term entropy is defined as:$$H(\mathcal{C})=-\sum_{i=1}^rp_{C_i}\log p_{C_i}$$of which$p_{C_i}=\frac{n_i}n$It's a partition.$C_i$- The probability.

Again, split.$T$The term entropy is defined as:$$H(\mathcal{T})=-\sum_{j=1}^kp_{T_j}\log p_{T_j}$$of which$p_{T_j}=\frac{m_j}n$It's division.$T_j$- The probability.

$T$The split, which is$T$About the partition$C_i$is defined as:$$H(\mathcal{T}|C_i)=-\sum_{j=1}^k\left(\frac{n_{ij}}{n_i}\right)\log\left(\frac{n_{ij}}{n_i}\right)$$

Grouping$C$ Division$T$ The condition is defined as$$00begin{aligned}H\left (T\mathcal{C}right)&=\sum_{i=1}^r\frac{n_i}{n}H(\mathcal{T}|C_i)=-\sum_{i=1}^r\sum_{j=1}^k\frac{n_{ij}}{n}\log\left(\frac{n_{ij}}{n_i}\right)\&♪ I'm not gonna let you go ♪of which$p_{ij}=\frac{n_{ij}}n$It's a partition.$i$One of the points is also divided.$j$- The probability.

The more the points in a partition spread into different divisions, the larger the conditions. For a perfect group, the value of the condition entropy is 0, while the value of the conditional entropy in the worst case is 0.$\log k$。

Normalize mutual information

Mutual information research grouping$C$and division$T$The amount of information shared between them is defined as:$$I(\mathcal{C},\mathcal{T})=\sum_{i=1}^r\sum_{j=1}^kp_{ij}\log\left(\frac{p_{ij}}{p_{C_i}\cdot p_{T_j}}\right)$$I've got information.$\mathcal{C}$and$\mathcal{T}$Joint probability$p_{ij}$And the expectation of a combination of probability.$p_{C_i}\cdot p_{T_j}$ Relevance (under independent assumptions).

If$C$and$T$It's independent of each other, then.$p_{ij}=p_{C_i}\cdot p_{T_i}$♪ And so ♪$I(\mathcal{C},T)=0$I'm sorry. However, there is no upper bounds for information.

We can get information from each other.$$I(\mathcal{C},\mathcal{T})=H(\mathcal{T})-H(\mathcal{T}|\mathcal{C})I(\mathcal{C})$$So we can give a generic information.$$\mathrm{NMI}(\mathcal{C},\mathcal{T})=\sqrt{\frac{I(\mathcal{C},\mathcal{T})}{H(\mathcal{C})}\cdot\frac{I(\mathcal{C},\mathcal{T})}{H(\mathcal{T})}}=\frac{I(\mathcal{C},\mathcal{T})}{\sqrt{H(\mathcal{C})\cdot H(\mathcal{T})}}$$His range is in. $[0,1]$ In between, close to one means good cluster.

Information discrepancies

This indicator is based on clustering$C$Split with Real Value$T$The information and entropy of these are defined as:$$00begin{aligned}\mathrm{VI} (\mathcal{C},\mathcal{T}&=(H(\mathcal{T})-I(\mathcal{C},\mathcal{T})+(H(\mathcal{C})-I(\mathcal{C},\mathcal{T}))\&=H(\mathcal{T}+H(\mathcal{C}2I(\mathcal{C},\mathcal{T}\end{aligned} $$Information difference (VI) value 0, current and only$C$and$T$Same. So, the smaller the VI value, the more the grouping$\mathcal{C}$The better.

Pair

Default of $D= {bardsymbol{x}1,\boldsymbol{x}2,\ldots,\boldsymbol{x}m}$, 假定通过聚类给出的簇划分为 $\mathcal{C}={C_1$, $C_2,\ldots,C_k}$, 参考模型给出的簇划分为$C^={C_1^,C_2^,\ldots,C_s^}$.相应地，令$\lambda$ 与$\lambda^$ 分别表示与$C$ 和$C^We'll have the sample paired, define it.$$00begin{gathered}A = \SS, =SS = (\bardsymbol{){i},\boldsymbol{x}{j})\mid\lambda{i}=\lambda_{j},\lambda_{i}^{}=\lambda_{j}^{},i{i},\boldsymbol{x}{j})\mid\lambda_{i}=\lambda_{j},\lambda_{i}^{}\neq\lambda_{j}^{},i{i},\boldsymbol{x}{j})\mid\lambda_{i}\neq\lambda_{j},\lambda_{i}^{}=\lambda_{j}^{},i{i},\boldsymbol{x}{j})\mid\lambda_{i}\neq\lambda_{j},\lambda_{i}^{}\neq\lambda_{j}^{},i

So we can define it.

Jaccard

Jaccard coefficient (Jaccard Coefficent, JC)$$\mathrm{JC}=\frac{a}{a+b+c}.$$Perfectly divided Jaccard coefficient is 1.

Rand Index

Rand Index (Rand Index, RI)$$\mathrm{RI}=\frac{2(a+d)}{m(m-1)}.$$of which$m$It's the total point. Perfectly divided Rand index to 1.

FM Index

FM Index (Fowlkes and Mallows Index, short FMI)$$\mathrm{FMI}=\sqrt{\frac{a}{a+b}\cdot\frac{a}{a+c}}.$$Perfectly divide FM index to 1.

Link Measures

Definition of Hubert statistics

You!$X$and$Y$Two symmetrys$n\times n$matrix, and$N=\binom n2$I'm sorry. You're the one who's gonna get you.$x,y\in\mathbb{R}^N$- For each other.$X$and Y's upper triangle elements (excluding main diagonal elements) are the vectors obtained by linearization. You're the one who's gonna get you.$\mu_X$Representative$x$, defined as:$$\mu_X=\frac1N\sum_{i=1}^{n-1}\sum_{j=i+1}^nX(i,j)=\frac1Nx^\mathrm{T}x$$You!$z_x$Centred$x$Vector, defined as:$$z_x=x-1\cdot\mu_X$$of which$1\in R^N$is the full 1 vector. Again, the order.$\mu_Y$Representative$y$The average of the elements by element,$z_y$Centred$y$vector.

Hubert Statistically defined$X$and$Y$Average element-by-component product:$$\Gamma=\frac1N\sum_{i=1}^{n-1}\sum_{j=i+1}^nX(i,j)\cdot\boldsymbol{Y}(i,j)=\frac1N\boldsymbol{x}^\mathrm{T}\boldsymbol{y}$$

Normalization Hubert Statistically defined$X$and$Y$, and then the following:$$\Gamma_n=\frac{\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{X}(i,j)-\mu_X)(\boldsymbol{Y}(i,j)-\mu_Y)}{\sqrt{\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{X}(i,j)-\mu_X)^2\quad\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{Y}[i]-\mu_Y)^2}}=\frac{\sigma_{XY}}{\sqrt{\sigma_X^2\sigma_Y^2}}$$

Disconnected Hubert Statistics

You!$T$and$C$Yes$n\times n$and the matrix, defined as:$$\left.\bardsymbol{T}(i,j)=\left{begin{array}ll}1&y_i=y_j,:i\neq j\0&\text{others}\right.\right.\quadd\bardsymbol{C}(i,j)=\left{begin{array}{1&\hat{y}_i=\hat{y}_j,:i\neq j\0&\text{Others}\right.$Meanwhile,$t,c\in\mathbb{R}^N$Other Organiser$T$and$C$and the upper triangulation elements (excluding diagonal elements)$N$& Vector, where$N=\binom n2$Numbers representing different points. Finally, your orders$z_t$and$z_c$Centred$t$Vector and$c$vector.

Dispersed Hubert statistics can use formula (17.14) You're the one who's gonna get you.$x=t,y=c$) Calculated:$$\Gamma=\frac1Nt^\mathrm{T}c=\frac{\mathrm{TP}}N$$

Normalized discrete Hubert statistics

Dispersed Hubert, the uniform version of statistics is$t$and$c$Relevance between$$\Gamma_n=\frac{z_t^\mathrm{T}z_c}{|z_t||z_c|}=\cos\theta $$Attention.$\mu_T=\frac1Nt^\mathrm{T}t$is the same division ((s)$y_i=y_j$) Point-to-point ratio, regardless of$\hat{y}_i$and$\hat{y}_j$Whether it matches. Thus, it is possible to:$$\mu_T=\frac{t^\mathrm{T}t}N=\frac{\mathrm{TP}+\mathrm{FN}}N$$

Internal indicators

And it's obvious that external indicators are in most cases of no value because we don't have reference models that we can use unless we're known to be real classifications, just to study the performance of the cluster algorithm. Internal indicators often depend on the distance between samples and the approximation, and thereforeMachine learning progress and unsupervised learning: spectral and graphic CategoryClose links, where the integration and modularity can be directly used for performance measurement.

Considering the distance between samples, give the following definition$$00\\mathrm{avg}& =\frac{2}{|C|(|C|-1)}\sum_{1\leqslant i{i},\boldsymbol{x}{j}), \\operatorname{diam}(C)& =\max_{1\leqslant i{i},\boldsymbol{x}{j}), \d_{\min}(C_{i},C_{j})& =\min_{\boldsymbol{x}{i}\in C{i},\boldsymbol{x}{j}\in C{j}}\mathrm{dist}(\boldsymbol{x}{i},\boldsymbol{x}{j}), \d_{\mathrm{cen}}(C_{i},C_{j})& =\mathrm{dist}(\boldsymbol{\mu}{i},\boldsymbol{\mu}The blogger says:I'm sorry, I'm sorry.The four samples are the following: the central distance between the inner samples, the longest distance between the inner samples, the nearest distance between the clusters, the central distance between the clusters.

DB Index

DB Index (Davis-Bouldin Index, short DBI)
$$\mathrm{DBI}={\frac{1}{k}}\sum_{i=1}^{k}\max_{j\neq i}\left({\frac{\mathrm{avg}(C_{i})+\mathrm{avg}(C_{j})}{d_{\mathrm{cen}}(\mu_{i},\mu_{j})}}\right)$$The smaller the DBI, the better.

Dunn Index

Dunn Index (Dunn Index, DI)$$\mathrm{DI}=\min\limits_{1\leqslant i\leqslant k}\left{\min\limits_{j\neq i}\left(\frac{d_{\min}(C_i,C_j)}{\max_{1\leqslant l\leqslant k}\operatorname{diam}(C_l)}\right)\right}.$$And the bigger the D, the better.

BetaCV

BetaCV measures the ratio between the intra-clan distance average and the interclan distance average:$$\mathrm{BetaCV}=\frac{avg(C)}{d_{avg}}$$The smaller the BetaCV value, the better the effect of the cluster because it means that the inner distance is on average less than the interclause distance.

Relative measures

Relative measures compare the group nature of different parameters of the same conglomeration algorithm Yes.

Calinski-Harabasz(CH)

The given data set is $D=x i}The D.R. is a scatter matrix:$S=n\bardsymbol}sum}{j=1}^n(\boldsymbol{x}j-\boldsymbol{\mu})(\boldsymbol{x}^mathrm{Where \mu=\frac1\sum{j=1}^nx_j$是均值，$\Sigma$是协方差矩阵。散度矩阵可以分解为两个矩阵$S=S_W+S_B$,其中$S_W$是簇内散度矩阵，$S B$ is a cluster-wide dispersion matrix, which is indicated as:$$00\&S{W}=\sum_{i=1}^k\sum_{x_j\in C_i}(x_j-\mu_i)(x_j-\mu_i)^\mathrm{T}\&I'm sorry, I'm sorry.of which$\mu_i=\frac1{n_i}\sum_{x_j\in C_i}x_j$It's a partition.$C_i$average.

For a given$k$Value, Calinski-Harabasz (CH) variance is defined as:$ \begin{aligned}CH(k)&=\frac{\mathrm{tr}(S_B)/(k-1)}{\mathrm{tr}(S_W)/(n-k)}\&=\frac{n-k}{k-1}\cdot\frac{\mathrm{tr}(S_B)}{\mathrm{tr}(S_W)}\end{aligned}$$

of which$(S_W)$and tr$(S_B)$is the trace of the inner and inter-clave-dispersible arrays (i.e. the sum of the diagonal elements).

For a better one.$k$Value, can predict a relatively small dispersion in the cluster, and therefore a higher dispersion is obtained. $CH(k)$ value. On the other hand, we don't want a big one.$k$Value;

Thus, CH values can be mapped and a larger growth area found (and no or only small growth thereafter).

Division stability

The main idea behind the stability of the divide is to be able to$D$The clustering of data sets from the same distributed sample should be similar or “stable”.

The method of partition stability can be used to find the appropriate parameter values for a given cluster algorithm; the book is mainly appropriate for consideration$k$value, the correct number of the fractional clusters.

$D$The joint probability distribution is usually unknown. Thus, for the same distribution of sample data sets, we can use a range of methods, including random disturbances (random perturbation), subsampling or self-help sampling (bootstrap resampling). We'll start with the self-help method:

By From$D$Sampling (replaced, i.e. allowing the same data point to be selected several times, each sample)$D_i$So it's different to generate it.$t$Size$n$The sample. Next, for each sample,$D_i$, with different$k$ Value (from 2 to) $k^\mathrm{max}$) Runs the same group algorithm.

You!$C_k(D_i)$Organisation$k$From Sample$D_i$Get a cluster. Next, the method compares all clusters with a certain group function$C_k(D_i)$and$C_k(D_j)$Distance between. Some external concentration assessment measures can be used as distance measures, e.g., by$C=C_k(D_i),T=C_k(D_j)$And vice versa. Based on these values, we calculate each.$k$The expectations of values are in pairs. Finally, the lowest deviation from the different clusters obtained from the re-sampling data sets$k^*$Yes.$k$The best choice is because it has the highest degree of stability.

Cluster trend

Cluster tendency or clusterability (clusterability) is designed to judge data sets$D$There are meaningful clusters. This is often difficult because it is difficult to define what is a subset in the first place, such as partitioning, hierarchy, density-based, map-based, etc.

Even if you have a sort of cluster, for a given data, Set$D$It remains difficult to define a suitable zero model (null model, i.e., model without any cluster structure). Moreover, even if data are judged to be conglomerate, we still face the problem of determining the number of judgement clusters.

Hopkins statistics are a thin sample test of space randomity. Give a Organisation$n$Data set for points$D$We create$t$A random sample.$R_i$ (Each subsampling contains$m$Point, of which$m\ll n$I'm not sure. Dataspaces of these samples and$D$Same, randomly and evenly generated at each dimension.

Besides, we're going to go straight to...$D$Generating$t$Samples (each inclusive)$m$(Place) (Placed), use unreleased samples. You're the one who's gonna get you.$D_i$Representative's first$i$A direct subsampling. Next, calculate each one.$x_j\in D_i$and$D$Minimum distance between points:$$\delta (\bardsymbol{x}j)=\min{\boldsymbol{x}_i\in D,\boldsymbol{x}_i\neq\boldsymbol{x}_j}{\delta(\boldsymbol{x}_j,\boldsymbol{x}_i)}$$

I'm sorry.$i$- Yes, it's a sample.$R_i$and$D_i$ Hopkins Statistics$d$Definition:

$$\mathrm{HS}i=\frac{\sum{y_j\in\mathbf{R}i}(\delta{\min}(\boldsymbol{y}j))^d}{\sum{y_j\in\mathbf{R}i}(\delta{\min}(\boldsymbol{y}j))^d+\sum{\boldsymbol{x}_j\in\boldsymbol{D}i}(\delta{\min}(\boldsymbol{x}_j))^d}$$

This statistical volume will provide a recent neighbourhood distribution of the data points generated at random and will be distributed over the next few years.$D$Compares the latest neighbourhood distribution of random subsets of the medium data points. If the data are of good fusion, we expect$\delta_{\min}(x_j)$Less than$\delta_{\min}(y_j)\text{,且在这种情况下，HS}_i$ Trends to 1.

If the two closest neighbors are similar, HS$_i$The value is close to 0.5, which means that the data are almost random and not clearly clustered.

And finally, if...$\delta_{\min}(x_j)$Value greater than$\delta_{\min}(y_j)$, HS$_i$A zero, which means a little exclusion, and no cluster.

Based on$t$A different HS.$_i$Value, as judged by the average and variance of the statistical volume$D$Can cluster.

What Problems Are We Really Solving When We Build AI Agents?

2026-07-28T04:00:00.000Z

It's written in front.

It's gonna be a long blog. I'll talk about this heading in a more random order. There are many links to the text: examples and extensions of views. They are not responsible for drawing conclusions for me, but give the reader more material to judge for himself.

This article is more oriented towards technically-born readers and does not interpret the underlying concept too much, and I have some background information that has been reduced to a single article that has been briefly spoken. It will not be a mere opposable article, with many examples and arguments. Better leave some time before you're ready to read it.

This is really a hard article to start. I have a lot of things I want to write, but it's not very good. I've been writing for a long time to find a little cut. It doesn't sound so logical, but I've tried.

I believe this piece will be very useful to readers who are doing AI Agent, but it cannot begin with a clear idea of what the harvest will be. It's a piece of personal experience, not a lesson.

Let's start with the story.

A big leap.

This post is part of our special coverage of the World Summit for Children, held in Beijing in July and August. In the last six months, the AI Agent field has undergone a major leap forward. If you have to draw a specific time node, it can start with OpenClaw's blast, until now.

The leap here has been a bit of history, and they are somewhat similar in their journey. AI Agent has had some hard-to-reach quality changes over the past six months, but in the Internet itself, and in the ToC and ToB products, numerous products and technologies, Demo, have been introduced like dumplings. The e-mails have been running numerous AI Agent products and dropping out of the air in a few months; Ali’s vast number of functional units are being reorganized, crucified by an internal change that eventually leads to no action; and byte even combines soybean bags and flying books, it is hard to imagine that one day the two software will come together. Almost every Internet company, even non-Internet companies, has its own lobster, which is being raised by users from different backgrounds. 60 years ago, you were practicing steel, and now you're practicing Agent.

Why? We've got new wonders?

There's a strange thing about technology, but is OpenClaw the strange thing? Claude Code was opened to all users in 2025, and Codex was one step late in October 2025, and the CA, Cline came far earlier than OpenClaw. The Agent, who makes the decision to help users modify and test the code, is not unusual for programmers, and the world has not changed. OpenClaw didn't break any technological bottlenecks, humans didn't enter the world after the singularity, but only a new product from a water-to-tunnel with enhanced silicon-based intelligence.

If there are no wonders, why did we make this big leap?

Because it's burning up.

Ordinary users were actually left behind in the past few years of the AI wave; they saw the AI nuclear bomb explode again every day, but they never understood what the AI round was, and what it would bring; they were anxious, but they didn't have an exit, or an antidote, until OpenClaw appeared.

What happens after the fire?

In the pre-OpenClaw era, almost all AI Agent Dev was monopolized by Researcher. Only Researcher and VC have a little idea of the principles of AI Agent and what they might be able to do. A number of programmers working on the front line are also trying to reach AI Agent, a group that likes to embrace new technologies. For those of untechnical origin and management who leave technology first line early, access to AI may be limited to bean bags and reports from those under hand, and reporting must be done by reporting. Despite the rapid technological changes, the company has maintained its years of silence.

Now OpenClaw has broken into these management's vision, and for the first time they know that AI is already able to do so much. Without much technical detail, it might be felt that it could solve everything now. A beautiful illusion came out. The pressure from the top comes, big AI projects are set to compete for new user access, internal AI-efficient KPI/OKRs are issued, employees from all backgrounds start learning AI Agent to fill the relevant staff gap, and the entire Internet industry appears to have been lit.

An Ai big leap came.

Now, six months later, new user accesses have stabilized, a large number of internal efficiency tools in various industries have largely failed after various attempts, and the development of the AI capability boundary has become clearer. The leap ended in silence.

AI Agent has spent six months walking through a lot of industries for five years.

If you want to know why OpenClaw is a fire, reference"BettaFish, Mirofish, OpenClaw and Agent's Trust Border"。

Chapter I

This article is for everyone who did AI Agent during the Big Jump, whether you're interested in it or under KPI pressure.

I'll try to answer three questions:

When you're building an AI Agent, what are you doing?
When you want to solve a problem in a real scene with AI Agent, you have to solve your own problem?
Why did an AI Agent project succeed and why failed?

Many people can't answer, not because of their talent and ability. Every person will have external and internal pressure during a period of great leap forward, and a great deal of energy will be devoted to achieving it faster than to thinking about these issues. These questions have also been covered by three or four different answers over the past two years, each of which was correct at the time.

There is a reason not to be on the individual side. The real lesson of a big leap is not that steel is not made, but that indicators replace targets. The stove was lit, the production was reported, and as for the calibrated, no one asked. The efficiency projects I have seen on the flash drive have access rates, call numbers, coverage, and very few “who does this job now, how much faster, how wrong it is to be found”. The indicators are well achieved and the tools are not used. When the indicators themselves replace real goals, no one will ask why.

Now we have time to think.

These are the questions that cannot be seen just by looking at the six months. The six months of virulent activity are more like waves on the surface, and the direction and changes of currents need to be considered at a longer time scale. Before answering the three questions before us, let's go back to 2024 and see the way AI Agent came.

Time to come.

This chapter was originally written here, and was then opened in a single article:"AI Agent's Timeline: Where did we think it was a bottleneck?"I'm sorry. It has been walking over the past two years in chronological order, and has been divided into seven layers: Prompt, RAG, WorkFlow, Tools, Context, Models and New Words, and Evaluations. Each layer is a speculation about where the bottlenecks are, and a set of tools, terms and projects have emerged; some of them are deposited into today ' s infrastructure, others are investing a lot of resources and finally the road is not working.

And then the next part will repeat what's in that chapter. If you're good at Agent, you don't look and you don't have to read, you can just look at the article's subheadings; if you want to know where these judgments come from, you can go over the road and come back.

Chapter II

Before starting to present a clear view, there is a second chapter.

It seems that the talk of so many past projects has been somewhat detached. So many engineering practices ahead are actually encountering a variety of landing bottlenecks, and we're trying to solve the problem and improve the final Agent capability. In terms of results, we have made great strides and have sunk many useful programmes. They all answer these questions, and it should be clear how each of the technologies that have been used to answer them is a little bit.

Try to think, no matter what you are building.

When you're building an AI Agent, what are you doing?
When you want to solve a problem in a real scene with AI Agent, you have to solve your own problem?
Why did an AI Agent project succeed and why failed?

Some of the pens were left behind in the introduction. Prompt, is it really over? There are still many plugs and Skylls on the market about building a better Prompt. What are they solving? What do we weigh in engineering? Context Engineering is covering a much more than condensed range. What are we going to do in different scenarios? Evals is very important, and the article talks about Evals' techniques, but it's not so clean in the works, how do we decide what to do and how to change it?

More about the following is a few personal experiences and a summary, and I stepped on a lot of pits and thought about it after. And of course I'm learning the practice of others. Let us start with some personal experiences and studies, by answering three questions I have raised, and by talking about what we should do when we go to an Agent in the future.

Some thoughts and judgment.

♪ To think of things that have gone before ♪

A lot of the ideas that follow are based on personal experiences, and here is a brief chat about what I used to do and what I saved.

I started working on safety at the end of 2025, almost in early March 2026, and then I saw some friends trying to try and get results in this field. My discussions will all build on those experiences and not on the more distant past and future.

I did the safe Agent is quite simple, and the goal is to build a one that can help us find a loophole in the code library. The central purpose of the holes is to build a high-quality data set for digging, to consider the Training model and to enhance its ability to dig holes. The universal intelligence in 2025 is getting stronger, but it's still weak on this special mission, choosing to be trainning at that time was a non-mistakeful choice.

Of course, if we look at the future, we are doing a study that is very easily replaced by universal modelling capabilities. The team of Startup should think about whether the next upgrade of the model will eliminate or add value to its own product, and doing research is sometimes a small Startup, but we're doing our own VC.

The specific work is not extensive, and a multi-intelligence system containing gap information collection, source code static analysis, sludge stream modelling and self-checking, and certification of the CodeQL engine. It's only been developed for two weeks, and the rest of the time is changing the small issues. The main focus is on how to express a state of detour and to design a set of good tools to achieve external interaction; to do some static analysis, validation and self-reflection; and to configure a CodeQL as the ultimate validation machine to give a True or False answer. Collecting a bunch of data, washing, running, beating tag, doing lessons, eventually getting a little higher. This is a standard Post-Training Pipeline.

A lot of experience comes from l3yx, and at almost the same time he's studying AI Agent for auto-defense and has done a good job at the TCH Smart Infiltration Challenge. He first developed a very complex Multi-Agent system using LangGraph, making various angles of human penetration an independent Agent and tool, and then adding a lot of penetration SOP and tool descriptions. This is a common agent that brings together a great deal of expert experience, but expert experience is limited to indicative words; it is a human being with a big tool, but a description of the tool and feedback can lead to explosions. So he proposed an Agent Framewok, a Dynamic Workflow, to bring expert experience and process to light; and another, a similar call optimization for Programme Tool Calling, to find a balance between old MCP and pure Skill Script.

The story is only halfway down here.

TCH, I3yx, take it out in the second. Cairn, a programme that is completely different from the previous generation of multi-intellectual bodies. A Dispatcher + shared Blackboard architecture, the smallest dispatch unit being a Codex Agent, used to make and validate assumptions. The movement control system is structured around the final target, and the entire system is a direct search in an unlimited space. Blackboard only contains Fact, Intent and Hint, which allow Agent to record the discovery, future targets and human hints that are available to achieve communication.

No process is fixed, all generated dynamically by Agent based on questions and simple initial tips; multiple intelligences injected by functional and hint processes are cancelled and all tasks are created only when running; all forward knowledge RG and Skills are removed, believing the model has been understood after so many years.

This is a very simple structure, but sometimes it is more difficult to design it than a complex one. A model that generates the emergence of, and matches, a structure that happens at this moment.

Is it a front-to-back infusion of knowledge or a back-to-back feedback? What information should the model get?

In that section of the RAG, I mentioned one point:RAG is designed to be one-way.I'm sorry. We spent two years doing the "how to get knowledge in" very finely, but few people did the "how to get the results back." It's an asymmetrical. In the section of the Function Calling, we talked about SWE-agent and ACI,A well-designed tool and feedback signal came up with amazing power after combining it with Loop.I'm sorry. The advance of model capabilities and autonomy of Agen at the C end of 2026 is the result of these stories.

Recall two examples of safe Agent. I've done code-digging Agent, extracting forward-drive knowledge from external information and some hard-coded processes and tips, obtaining back-dip feedback from independent static self-checking and certification of the CodeQL engine; the first generation of l3yx systems based on LangGraph, which relies on hard-coded processes and knowledge, and on the results of the operation of the penetration codes; and Cain, which abandons knowledge and processes and relies on knowledge within the model and penetration results to accomplish its mission.

After the AI giants had finished pre-training on the scene where the holes were being dug, I made the Training thing that was worth it. A universal model, which relies on a universal intelligence body, now achieves far better results than the Harness I did. The internal knowledge of models is fully adequate in a growing number of scenarios, and it is worth weighing whether it is spent on training to expand internal knowledge or whether it is supplemented by external injections. (p.s.) So think twice before you start training the small models.

In six months, the evolution of models has eliminated the space before Harness and knowledge were injected into existence. A very simple structure, coupled with a broadly comparable model, brings final performances that go well beyond the type of knowledge injected and process bound.

It's clear that I'm here. The progress of the underlying model will not stop, and the progress of the model will certainly eliminate some of the process constraints and the space in which knowledge is injected. Models are becoming more and more knowledgeable and less knowledge of the world is worth being injected into.When you're trying to plan the process, inject the rules, you might want to see if you have a good feedback signal for the system.I'm sorry. This may be the information that the current model needs more.

As the core of the quality of feedback, the ACI is important, and we have mentioned it once before. If feedback is the one in front of the zero, then research the ACI of each tool and how these tools should be called, the zeros in the back. **Tools is never a list of tools, but AI Agent can see and influence the world. ** Put every MCP Tool on the ground, but absolutely not enough. The context costs of a large number of calls also need to be addressed, as MCP patches, and we have a Programme Tool Calling; if we are willing to believe in model capabilities, then we might leave space for models to write their own scripts (Skill Scripts). I3yx used them in two generations, and they did good.Go to sharpen each Tool description, adjust the cover of a Tools, plan the feedback that the system can give and send back to Agent, and think about how to call Tools. Function Calling was right, Scripts is right now, but the future is not yet coming.

The feedback loop is an interesting topic,"Looking back from feedback, Agent How to Turn Genesis into Search."And when you talk about LLM's candidate generation, you can see how the system can search, validate, cut and choose.

Is a multi-smart body natural? How human division of labour became the Agent division of labour

In the Workflow section, we talked about the success of the workstream framework and the failure of the multi-intellectual framework, and in the Harness, we talked about the great success of the Single Age programme, and about the Subagent's re-use after he had lost his body.The benefits of multi-smarts come from isolation, not from human role-playing.I'm sorry. Should we be doing a multi-mixer? If so, what should we do?

This is a structured decision-making exercise, which will briefly review what our classic architecture is before we begin our discussions.

Pure workflow system: n8n, Coze; a low-code structure
Pure multi-intelligence systems: MetaGPT, AutoGen; communication systems are at the core
Multi-tip smart body bound by the workstream: LangGraph; no freely communicated multi-twistle body but with a separate workPrompt and Node flow, one medium State
Autonomous intelligence with appropriate structural constraints: Cain, Dynamic Workflow; process becomes free, constraints become loose, but minor constraints bring about better communication, at least not multiple roles.
Full-owned smart bodies and sub-agents: Claude Code, Codex, PI; Master Agent and Subagent

It is simply a matter of understanding the differences in thought that the Agent structure is difficult to discern. The core of understanding the differences between them may be what people inject into the system in the process of development, and what they leave to AI to make its own decisions.

Humans need to divide their roles because of some of the natural limitations of humans as carbon-based intelligence. People ' s cognitive bandwidth is limited, and it is impossible to know the full range of things and know the details of the various areas; they are also constrained in their energy and their ability to work efficiently for long periods of time. So we have done complex tasks in the form of multi-person collaboration, which brings together the sum of your abilities and the cost of information and awareness exchange.

But the same capacity boundaries and energy ceilings do not exist for silicon-based intelligence. Putting a human intelligence different from human intelligence into an organizational structure that has grown because of human limitations is tantamount to bringing together human communication losses. That's probably where MetaGPT and AutoGen failed, and the success of the CCC doesn't mean it's the right way.

From the perspective of the Startup, there's a very realistic bill. Dismantling a mission into Planner, Researcher, Coder, Reviewer, and putting in a layer of communication protocol, does not create new capabilities; in many cases, it simply dismantles the work that a strong model could have done more expensive, slower and more difficult to assess. The context, reasoning and tool capabilities of the model are upgraded a little, and these people are likely to be the first to be eaten. A product that relies on “More Agent Collaboration” for storytelling should ask itself: what is left of the product after removing the role and replacing it with a stronger Single Agent, with enough Token, tools and feedback?

However, purely multi-smart bodies are not worthless. The key is not whether Agent is imitating the human sector, but whether identity, local information and independence are part of the mandate. Social modelling, market behaviour, group collaboration, and dissemination of information are issues that are studied in terms of how multiple subjects interact with each other with incomplete information: different subjects see different worlds with their own goals, memories and consequences. At this point, all roles are placed in a single Agent, and they are usually only produced in a text that is like a multi-party dialogue, which is difficult to really maintain an independent and evolving state of affairs. The core value of multi-intellectual bodies may not be in common office collaboration, but rather in modelling this less than complete information world.

The title of this section is in fact finished with the answers, and here we will make a few more suggestions on the selection.

Let's get out of the way, and think about the nature of the problem you're doing from the first principle: what kind of input you want, what kind of output you want to give, what principles you want to follow. Ask yourself why it is being used, what it is trying to solve, and never answer “xx is doing it”. And never let the structures of human collaboration kidnap AI, always based on the problem itself and on the limitations of the future's silica intelligence.

For the vast majority of internal process tools and C end-products, a workflow system that has access to various sectors, modules and which allows input and validation is efficient and practical. A little LLM is completely enough, and there's no need to be disgruntled because it's not high enough. Many tasks are waterlines themselves, and it is the proper waterline that matters most. Of course, the Workflow system does not mean that you should choose LangGraph directly. There are processes in human collaboration because of the division of human functions, but AI does not understand anything, and it does not necessarily need to be divided by the job itself, not by the function. The pipelines in the development of the game are a good example.

And the only choice that is worth mentioning is that of Agent, which we have divided into two categories. When the objectives are clear, but the processes are vague, it should be chosen. The two types of multi-stellar body are actually the same: a context-segregated and parallel accelerated exploration. For the implementation scenario that is ahead of the mission, the different intelligent bodies are themselves the same, but they are assigned to different tasks.Agent's pouncer structure is only mission-free, no role.

Cain is an example. The last generation of it made the various angles of human penetration independent, each one of which had its own hints, its own tools, its own location; in this generation, all these roles disappeared, and the smallest of them were returned to an unidentified Codex Agent, Dispatcher, just to send a mission, and everyone would do the same. The division of labour is gone, parallel and faster than the previous generation.It's a parallel, it's a profit. It's not a division of labour.We have often considered these two matters as one thing in the past two years.

The difference between autonomous intelligence is that the structure of communication is different from the degree of restraint.Move Tradeoff to a position, and make more.

This section has answered more fully the question of multi-smart bodies and the technical selection of Agent. For an executive mandate, how to design the segregation and exchange of information is more useful than defining 100 Codes; only when identity, local information and independence are themselves the subjects to be studied will the role cease to be packaging, but rather models. Workflow will always coexist with autonomous intelligence, with a wide range of tasks in the world, with AI having changed capacity boundaries and naturally flexible structure choices. The architecture itself already contains the corresponding Context management method: Workflow constraints write Context into code, multi-smarts separate multiple Context, and then combine normal compression and recall, which is all about Context Engineering.

I'm here."Enterprise AI Why is the master card piloting?"It talks more about this from the perspective of Enterprise and about where the unsuccessful projects at Agent come from, and it's interesting to look at it.

The system's ability depends on the human ability to express? It's more than knowledge.

We have just spoken about focusing on feedback, not on research into the injection of knowledge, and now come to the face. Harness is going to inject us a lot of things: basic Prompt, system status, mission objectives and constraints, MCP and Skills tools. A lot of injections are necessary, and what difference does it make from the injections we talked about?

Coding Agent has two features worth talking about. One is Plan Mode, which used to be community-based typologies and is now an Agent frame, and the other is the community-based Grill-me Skill, which is extremely hot. Their core hints are the same: to find the shortcomings I have said and to ask questions until we reach consensus. Is this the place where knowledge is injected? Nope. This is a alignment of norms and hidden habits that are not coded and not hinted into.

No RAG's thinking is limited, and many tasks require much external knowledge simply to describe the mission's objectives and boundaries themselves. And humans are not AI. We are not. /list The full list of related matters can be automatically identified. When AI Agent became so almighty, the ability to own the Agent system began to depend on the ability of humans to express themselves. It is difficult to express a clear and complete intention for a highly complex system.

As the model gets better, Plan actually absorbs the Grill-me, but absorbs need to find that balance.

We talked about Prompt, we talked about language; we found out it wasn't necessary; and now it's a problem with the definition of the target, and it's back again. The model's autonomy has been increasing, and Loop is looking at a goal that is not clearly defined, and there's no point in it, and people don't know what the goal is, and Agent is just lucky.

Let's see how safe Agent is in dealing with this. I3yx, the first generation of systems, whose goals and constraints are all written phrases, is indeed an expert, but the ability to express is ultimately limited and it is impossible for him to write them all at once. Cain is only concerned with infiltration of this ultimate goal, and only simple hints can be inserted into people, all inents are created and modified by the AI system itself. That is the value of Cain: the ultimate goal of penetration can be easily defined, the hint is random, does not create faulty constraints, incent is generated by system integrity, and is not limited by the ability of people to express themselves.

Cain has at least one set of tasks, and it's been able to exchange all the information, so it's working. But it's not even a matter of time, Agent, much.

Is your Harness supplementing the capacity gap, clarifying the mission itself, or is it adding to the trust deficit? If the model really doesn't understand, it's necessary. But if it's just because it doesn't do what you think, try to trust it once.The project is to release the model capability, not to press down the model ceiling.

For a Coding Agent, the knowledge worth inflecting includes technology warehouse, infrastructure, regulation, experience, architecture, description and indexing of the code warehouse itself, business habits, personal style. And of course, it can't be done blindly, how to fill it in, how to make sure it works, more than piles of stuff.

The angle of the question description is not the same in the WorkFlow as in Agent. The restructuring of the system, the modification of internal hints or the enhancement of the hints are all part of the work, and it is difficult to make a few recommendations that must be made, and it is generally useful to think about this in the context of your own structure.

Even if AI becomes stronger, people can never be removed from the system. In complex systems, people need to help AI filter information, find the right, project-compliant, personal taste information. Pure freedom will run on the road to the shit mountain like the GPT browser and Claude's C compiler.

Evals, what are you gonna do? Why would you say this is better?

The last floor of the road I came to say, "Don't skip Evals, leave it to the Anthropic blog." The chapter II also left a question: how do we decide what to say and how to say it? This section is to repay the bill.

When we build an Agent, what we do must be a system engineering, and it requires systematic thinking. The assessment of the system must be an essential part of the picture, of every complex system around you, of Evaluation and Trace, which is in some way or another. Of course you can leave it because sometimes Demo First, but leave the interface and remember that you didn't do it before.

Evals is the core driver of the next phase of development in the era of AI Agent, even earlier, the existence of ImageNet, which has allowed every algorithm progress of CV to be based, and the constant introduction of LLM Benchmark, which has been updated to fill new models, and Evals, which has provided direction for technology development. As long as Evals is itself of the right stage, we can continue to be back-overs at low cost; Evals can also locate problems, attribute errors, compare different versions of complex systems with A/B test, and make it easier to fix them; and Evals can have back-tests, so that we do not have back-overs. That's the value of Evals.

What should we modify with the evaluation? Not every module is worth changing, but only controlled small step changes, A/B test and an iterative approach can lead to a better future. We have to fix a lot of things, and there is no standard answer to this question.Every person who's been in the field of human rights should have their own Belief, and then make a bold judgment, and then make a real correction.I'm sorry. I can talk about a little experience.

Workflow is often easier to trace than Agent Loop, which sometimes gets a result from A/B test and a mountain-mounted log that can be analyzed and tracked by trace. If we only have final results, they don't seem to make any difference, but if we try to locate the problem and the attribution, Tracing is valuable. Thinking about Workflow after understanding the question is a good answer, or do Demo and some tests. What kind of system expansion is often the most important thing to be judged boldly, which requires you to understand the business itself.

The design of the forward-injection knowledge, the back-in-the-back feedback design, how the hints are designed for each part of the system are at the heart of our optimization. Feedback and hints are generally based on A/B test, and the assessment of the forward-injection knowledge is worth thinking. Whether every article in the knowledge system is of sufficient quality, whether all articles cover the knowledge we need, whether the retrieval system works, and what results and operational results are given. Knowledge systems themselves need to be optimized over time, not developed once, and who is to optimize and who is to be held accountable. The failure of retrieval is sometimes simply a lack of access, which is easy to locate, and who will judge whether the search itself is correct? This is a more worthy issue, and different scenarios have different problems.

Agent Benchmark is being environmentalized over the years, from a topic to an environment where it can run, where it is both an implementation and a sentencing place. SWE-bench run test, tau-bench look at the database, the rating is not hanging out, it's the environment's own property. And this is the same thing that ACI said before: a well-designed environment, a cleaner feedback signal, easier to evaluate, two things actually involved the same project input.Can you comment on that? It was decided when you designed the environment.

As for the choice of the project, my experience is probably as follows:

What? Only things you'll decide on. Whether to change the model, whether this change in Prompt is better or worse, whether the tool description is useful or not.
How much? Small and steady is too much better and more complete. 20 examples of how to run every day are much more useful than 500 for six months.
How often are they evaluated: after changes that affect judgement. Models are subject to evaluation, because many of your conclusions from the last edition may have been invalidated.
When should it be possible to leave the evaluation: the period of exploration and the prototype period could be avoided, but it was important to know that the debt was being owed and that the account would sooner or later be repaid.
Indicators are the proxy for the goal, not the goal itself: Indicators themselves also need to be assessed, and it is increasingly a common problem for Reward Hacking to turn the pursuit of indicators into the goal itself.

Finally, two indicators are mentioned. The code scene is common. pass@k, run k at least once; and tau-bench pass^k The question is whether you can run k every time. The former is about capacity, while the latter is about reliability. This difference is particularly important when you're doing Autonomy: how much autonomy you dare give depends on how much you know about reliability. One. pass@5 It's beautiful.pass^5 A system of terrible sights is not fit to let it run. Capability determines whether it can be done, reliability determines how much you dare not look at it.

The specific assessment methods can be rereaded.《Demystifying evals for AI agents》；pass^k Design this line with a verifier, which can be consulted"How Reward and Training Closed in Real Age"。

It's written at the end.

And here, the three questions I'm asking myself are not really standard answers. If you have to put it in one sentence, it's probably: to build an AI Agent, to decide what to give to the model, what to leave in the system, and to believe that it's right.

These earlier judgments will be outdated. Every upgrade to the model will eliminate a shipment of Harnesses, and some of the conclusions that have been set up today will make it ridiculous. Everyone has their own Belief, and everyone's belief is not static, and maybe I'll have a new perspective in an hour, and the reader should have his own.

There is one more thing to admit. Multiple twilights are inevitable, not because Agent is stupid, but because the first time someone says the need is wrong. Do Agent is like this, and write this article.

It's been a half month, and this is the end of the story, and the next one is about to talk about some of the recent experiences and make a little summary.

A little comment on the idea of a DGG

And as this blog evolved, it actually happened in the AI circle that was worth talking about. ** A mathematical scientist found a reverse example of the DGG/Goemans cost assumption by repeatedly prompt GPT-5.6 Pro, and thus perjured it. ** The total number of tips is only about 60 words, and AI Auxiliary Mathology is no longer 95% human and has made AI additional, it is a human being who proposes initial goals and simple directions, leaving all of them to AI to explore.

We talked about it in the front."The progress of the underlying model will certainly eliminate some of the space where process constraints and knowledge are present ... It may be useful to see if there is a sufficiently good feedback signal for the system." DGG's guess is that the search for a counter-examples is a similar generation of variable searches. The code validates whether the LLM proposed counter-case is correct, and the existence of feedback turns one-off generation into an inspirational search. And the way the DGGs assume that the way they prove it can mean that we've been overestimating knowledge and underestimating feedback.

Last year, if we think about building an AI mathematician, we might do a few different-made Agents, analysis, criticism, optimization, proof, etc. Then a suitable communication system is designed. But now the basic model has become stronger."Strong model + long enough search budget + high quality feedback" It may be enough, complex Agent's value itself is questionable.

Feedback is different in different areas, if you need a strategic consultation, Agent, and the feedback is very difficult to give, advertising is much more realistic, further codes have more cheap and adequate feedback signals, and mathematics has the cleanest feedback. AI was the first to emerge in areas that are probably not those where humans feel the greatest need for creativity, or those where there is a strong, extremely cheap, extremely automated verifier. Now code, future chip design, formalization algorithm validation.

What am I doing when I'm building an AI Agent? What we did with AI Agent and Harness is whether it brings irreplaceable power to models or fills a temporary capacity gap.

How AI Agents Got Here: Where We Thought the Bottleneck Was

2026-07-25T04:00:00.000Z

It's written in front.

This article was originally...What are we solving when we build AI Agent?In the chapter. The article wants to answer three questions:

When you're building an AI Agent, what are you doing?
When you want to solve a problem in a real scene with AI Agent, you have to solve your own problem?
Why did an AI Agent project succeed and why failed?

These three questions are not visible in the last six months, so before answering, we need to go back to 2024 and see the way AI Agent came. The chapter was later extended to an unsuited level and was then taken apart and made separate. If you want to see the answers to those three questions, please return to the original; here's to the two years of AI Agent itself.

Time to come.

In the last two years, AI Agent was not a straight line forward. It's more like a string of speculations: we keep guessing about where the bottlenecks are, and then we put most of our manpower and budget in the position that we're guessing.

Each guess will produce a set of tools, a set of terms, posts and projects. When this layer is filled up or the model is gradually eliminated, the next layer becomes apparent. A small fraction of the last batch was deposited into infrastructure, with the remainder rapidly devalued.

Let us look at what we have discovered, what we have been guessing and what we have proposed in the past two years. They all answer one of the three questions, not the final answer.

If you're good at Agent, you can skip this chapter and just look at the title. I wrote that more to clean up my thoughts.

Enchant Engineering

ChatGPT's release is a landmark moment. We spent years finding a mathematical structure that was suitable for language sequence modelling, and we invented hardware infrastructure that matched its parallel calculations, while turning these mathematical structures into a very low threshold, extremely broad-based interaction -- chat. Without any foundation, ChatGPT would not have been so successful.

Since we have a talking robot, what else can we do? It seems that only what to say is decided. And then it's not good to believe that you're right to say it, and it's not good for AI to do it; it's not good for you to think that it's a model. And so talking became a technique, and the spell that AI was able to output better results became a science, and we called it Prompt Engineering, and the person who studied it was called Prompt Engineering, as if it had been paid well.

Prompt Engineering lasted a little longer. It seems to be just talking, but all kinds of technology are being put forward on this basis, and the classics are COT, Few-shot, Role-Play; and for the fun ones, take a deep breath, tip, threat, PUA, and re-emerge in the Skill era. It was never out of date to make the task clear, and we are still doing Prompt Engineering, just not talking about it anymore.

Prompt Engineering has a principle of Agent development that is still in place today: to make the matter clear. This noun is not wrong, but because we don't need a special Prompt Engineer, and everybody is Prompt Engineeringer.

Technical details of this paragraph may be consulted." Intense engineering and context learning "If you're curious about Spec's relationship to this section, you can refer to it.Spec is not a new paradigm.。

Knowledge enhancement and RAG

RAG is doing the AI Agent had to talk about. If you've done AI Agent between 2023 and the third quarter of 2025, then RAG is a 70% loop that can't be bypassed.

It is based on a simple idea. Training data for models have deadlines and most of the data within companies cannot be used for training; the output rate is better by giving a related material Prompt to the model. So, a tool that automatically looks for materials and then stuffs them into Prompt, it's quite natural to grow. The judgement then was equally natural: models were not unfeasible, but knowledge was probably the biggest bottleneck of the system.

So RAG became the default answer for those years. Vector databases receive significant financing, almost all traditional databases are retrieved, and each company is building its own internal knowledge base. The splitting strategy, rerank, hybrid search, Graphrag, RAPTOR, Agentic RAG, have been evolving.

The search itself is not wrong. Putting the material it needs before Agent does, remains one of the most effective practices today. In 2026 or more, as long as AI training technology itself does not revolution, RAG will not die.

But RAG is designed to be one-way (Agenda RAG, better, but still limited information is available to be able to flow back). Document, slice, vector, recall, spell Prompt, information flows forward into the context and stops there. After this mission, Agent, nothing went back. The three sections of the material recalled were useful, and the noise was the last one accepted or the whole rewritten, and the next time the same problem was encountered, it was not possible to find a search method, none of which was exported.

We spent two years doing the "how to get knowledge in" very finely, but few people did the "how to get the results back." This asymmetry, we will refer to it again in the main text.

If you're curious about RAG itself, you can refer to it.Text Embedding: From Word Bag Model to Qwen3 Embeddingand"AI Agent How to Get Information From the Internet"; how to be organized in context and memory after knowledge is sent in, can be seen first《Context is All You Need》andFrom Memory Generation to Memory GovernanceWe'll come back to that.

LangGraph, LangChain and WorkFlow

Workflow is organized as an AI Agent Dev topic that is also indissociable. This section wants to talk about what the whole community has been doing over the past two years, and why it is needed.

The first batch of systems was n8n and Langchain. And we actually lived in the automated paradigm of the past: a linear process, connecting different external services, computing, and then making decisions. LLM is a node in this process, covering those parts of the traditional script that cannot be solved, writing an e-mail, understanding a natural language and making decisions into new branches, etc.

And what next? The LangGraph, Diffy, systems like this, which replaces DAG with a loop map, are starting to appear, React, which allows LLM to recycle in a local process until the problem is solved. Very Powerful Tools, a circular machine would not have had that capability before LLM appeared. Loop has returned in the last two months, and in a sense it is similar. LLM got stronger, so the cycle got bigger and stronger.

As a contrast to the workflow, the multi-intellectual framework also has a little heat. Since models are so human, let them work like people. MetaGPT, ChatDev, AutoGen gave Agent product manager, architect, programmer and test hand-held cards to meet with SOP, deliver documents, and evaluate each other. It's hot, and soon there's no use for it. We're in the main.

Why is the workflow developed by far more people than more intelligent bodies? Why do firms make products with little choice of multiple intelligence? In a scenario that gives the process, a Runtme process can well absorb the uncertainty of LLM itself. It's a very good engineering idea to keep uncertainty out of the system in the age when models are wrong, and then Agent Harnesss does something similar, except that LLM is less uncertain and naturally less bound.

References From the cognitive structure of an intelligent body to the framework of an intelligent body Discussing the boundary between WorldFlow, Agen, Supervisor, Agent Team and MAS, and the engineering value and abstract cost of these frameworks, MetaGPT, AutoGen. Reading "Wondering the outer space for LLM to wear a definite shackles." Think about what to do to keep uncertainty out of the system.

Function Calling and MCP

A robot that can only be Chat is not what we want. While gathering information is a large part of human work, there is still another part of taking it to make decisions and implement certain action. Turning Next Token Predation into an interactive intelligence that directly helps us to get our jobs done, people to talk, AI to do it, looks like a very good future.

If you have ideas, you have to do it. The first of the first creations of the FUNCING was the ability to access basic external tools for the model by simple JSON Schema: modify files, execute codes, search information. These are common operations for people and are beginning to enter the AI capability range. In order to harmonize incompatible interfaces, Model Context Protocol was born and gradually became a widely compatible factual standard. As the model capacity continues to improve, the even lighter seal that wraps Prompt together with Tool is mainstreamed and used in large quantities. On the other side of the standardized interface, GUI Agent, with the maturity of VLM and Reassoning, Manus, which has been a project of this kind, has been difficult to further expand.

But is it enough to standardize tools and pile them up more?

SWE-agent and the ACI it brought together deserve our further reflection. The same model, a set of interfaces designed for Agent, and throw it straight to a naked shell, with a low score of less than the same system. The model hasn't changed, it can see, it can do, and it can get new information after it's done. The MCP rigid Tool Calling mechanism may be appropriate for models at the end of 2024, but a large number of tools are injected into information and round-cycles to use tools that gradually begin to corrose the context, and mitigate these problems by adapting to loads, programmable functions calls and the idea of Skyll Scripts.

SWE-agent can hardly say how much it's changed for industry, because it's not as good as MCP to be a protocol, an ecology, a program. The quality of interfaces is a trade problem, which is difficult to standardize and to write into the reporting material. But it is a direction of improvement in very high value for money: the tools are not as good as the tools, and the quality of the tools is probably more important for Agent than even the underlying models. Tools is never a list of tools, but AI Agent can see and influence the world.

The technology evolution of this line is a reference.Technical Evolution of the Use of the LLM Tooland《MCP (Model Context Protocol)》Why did Skills win again after MCP?From MCP to Agent SkillsI'm sorry. As for the last judgment of this section, it's possible to refer to it."AEnvirron: Age Dev Why Do You Need Interact Environment Layer?and"Harness what is it?"。

Context Arms Race and Context Engineering

Talk about Context Engineering after Prompt Engineering. This should be the word most Agent developers have heard. When we want an AI Agent to do more for us, whether it's a longer output in Chat or a multi-wheeler to solve the problem, Context is a problem that we can't get around. The early model is only 1k to 8k Context, and it is hard to imagine that a complex task in the eyes of a human being can be described and solved with thousands of words, and the emergence of CoT/RAG technology has exacerbated the problem of the context. Perhaps the complexity of the task is due to the fact that the context window is too small, and we have to try to solve it.

If the first principle is to be taken, then the first consideration in the context of the study must be to modify the model. The location code is changed from absolute to RPE, and the relative position code is then expanded without training by position plugs, NTK-aware, YaRN, etc.; the attention side is designed to bypass the square complexity of the length of the sequence, consider the thinness of the slide window, local and global rotation; and do the optimization of this IO project at the IO level. With a series of jobs and more calculus, the context window went from 8k to 128k, and now 1M, almost double every six months, and the Context budget became larger.

Context Engineering, of course, is not just the context. As we have the longer Context, Context Rot's problem surfaces: just long windows, stacking in, it doesn't work better. We need to view the context as a limited resource to move rather than fill it. Comparation, Subagent, Memoory systems were created, Skylls introduced incremental loading through the file system to reduce the input burden of the hints, MCP moved from full tool description to local injection as needed, and the tiered Memoory system was made a reality with the file system, and Context began to be managed dynamically. From the end of 2025 to the first half of 2026, the progress of Contex Engineering was the basis for this massive leap of intelligence, and people began to think seriously: whether knowledge should be in parameters, context, external, and how it should be correctly deployed.

The details of the training in this section can be seen first." Self-Regular Mechanisms and Transformer Architecture "andLLM Life Cycle OverviewThe price of the long window on the side of the reasoning is to be found.Why Output Token is more expensive: from KV Cache to Agent Cost ProjectI'm sorry. The real project after the window grew longer,"Context is All You Need: Context Project for Smart Bodies"It's much more elaborate than this; it's on the side of memory.From Memory Generation to Memory Governance: A Panorama of Age MemooryandAgent Memoory and Runtme Technical InventoryAnd these two are exactly where the RAG came back. As to why incremental loading is considered to be a tool protocol improvement, it can be looked back at.From MCP to Agent Skills。

Go back to the model itself and the new words.

This chapter is coming to the end, let's see what we've been doing in the half of 2026.

Let's see the model first. LLM has experienced a huge fall in the years since 2023, when it was asked whether it would be necessary to conduct further training in certain areas (generally Mid-Training to add knowledge). At the earliest, not as much as we talked about, Training was the only option for empowerment, for the field fit QA and for the Training almost equals. This is also a habit of the deep learning age that LLM had before, when the generalization of models was hard to trust.

As the LLM's ability to generalize has grown, people's views have gradually changed. We have even less to mention SFT, Lora, RL when we are doing engineering issues, and a lot of training has been put back on the ground and made part of the basic model. The Second Half suggests that we have entered a new era, trained Infra, matured, trained from algorithm to closed-ring engineering. Models become more and more user-friendly, and it seems that we can really trust the underlying models and make their own assessments rather than think about training first.

2026, Agent's big leap was Agent's own big leap? My answer is no, it's more anxiety and top-down driving. But the power of Agent is actually advancing at a very rapid pace, but it's not from Agent Harness, but from models more. Actually, a lot of Harnesses are patches, not necessary. The Harness job you did three months ago was removed free of charge from the next version of the model, which is the normal pattern for Argentina Dev.

We've gone from the Workflow to the Claw, and the rect cycle has not changed much since birth, and the model in the cycle, and with the new model, the simple Loop can win the WorkFlow.

Multiple Agencies starts to exit because model + context works allow individual Agent to finish it in one breath, and the division of labour starts to yield less than the loss and expense associated with the complexity of communication. People are beginning to try more dynamic, self-contained intelligent bodies with a certain degree of restraint, rather than a complete division of labour.

Loop and Goal emerged, and people wanted to get less and less involved, and to get a strong, autonomous Agent to achieve that end. It's a bigger R.A.C., and last year's Ralph, Loop, never was a new play.

Is Harness a whole new concept? The old WorkFlow was a Harness, but now that you're not developing the WorkFlow, you're more concerned about the autonomous Agent, and you need a new word to emphasize what we're doing, and we're not doing the same thing we did before.

2026 The first half of the year is not the half year that new things have emerged, but the half year that old things have finally been given shape and name. After the model had eaten the framework, the largest number of conventions had been found in different abstract models per company. The influx of new arrivals during the six months also requires us to find new names to facilitate discussion and reporting. Marketing is not a mere problem in itself, but a synergy between people and technology.

What is there to be used to train the line?LLM The Nature of Inference and TrainingWhy does it go from algorithm to closed-ring engineering?Actic RL: Why closed rings is more important than training algorithmsAnd the Second Half itself, it's a reference.From RL Agent to LLM AgentI'm sorry. The way the model is eating the frame, it's not the way it is.《Model Is Good Enough》That's what I'm talking about, and...《Claude Code or Codex》It is a specific slice of its product level: the same very simple Loop, and in the alternative, the experience is completely different."Harness what is it?"The whole thing is arguing that the word covers a whole bunch of old problems; the complete version of the multi-smart account is in theFrom the cognitive structure of an intelligent body to the framework of an intelligent bodyLee.

Benchmark and Evaluation

The last section of the road is a place of little attention.

Each floor has a name that is loud, tools, ecology, and things that can be set up. The floor is called Evals, which sounds like a process before delivery, so it is the least valued link of the entire chain for a long time. The real lesson of the big leap is that indicators replace targets, and Evals is the only thing that answers them positively.

Developers of halfway through the line are particularly vulnerable to skipping this level. Evals is always in the last chapter of the curriculum, and it doesn't produce Demo and the benefits that it can see, and there's nothing to report on. But the cost of skipping it will be returned once again.

Evals is actually connected to a lot of questions. Whether or not to change the model, Prompt is better or worse, whether or not the part where context is compressed is important, and the tool is useful to describe the change. Why do you say this is better than the last edition? Without Evals, these judgments are all based on feelings, and feel very unreliable on a probabilistic system, you'll be in place for a long time and you'll be in the same place.

So this is a low-level, and there is no concept that really sounds important, but it's part of Agent Dev. If you can, don't skip Evals, at least think about it when you're not busy.

About Evals, what should we do?《Demystifying evals for AI agents》From basic concepts to a zero-sum set of eval merits, I'm here to see what the engineering will decide and how it will be.What are we solving when we build AI Agent?The Evals section talks about their own practices.

It's written at the end.

Seven floors of speculation is over. Each layer was right at that time, and a number of tools, terminology and projects were produced; a small fraction of them are deposited into today's infrastructure, with the remainder rapidly devalued.

Look back at this road to answer the three questions: When you're building an AI Agent, what are you doing? You have to solve AI Agent's own problems first? Why did an AI Agent project succeed and why failed? My answer is...What are we solving when we build AI Agent?Lee. It's time to go back.

Why Enterprise AI Gets Stuck in Pilots: Systems, Workflows, and Organizational Absorption

2026-07-22T05:00:00.000Z

As AI and agents take on execution, our own agency expands. The question is whether organizations are built to capture it.
Microsoft, 2026 Work Trend Index Annual Report: “Agents, human agency, and the opportunity for every organization”

The questions in this article can also be addressedAI Application of entrepreneurship: from selling tools to selling results、Model Is Good End: 2026, AI, which is really scarce, is an application rather than a larger model.How the concept of a relatively close read together is developed in different contexts.

The last article was a little broader, concerned about how AI gets into people's daily work. This one narrows the camera, looking only at the company.

It is now common in businesses: employees already write emails, check information, analyze it with AI; developers also give the clear border to Coding Agent. But at the corporate level, the conclusion is often reversed: accounts are opened up, and demo does, and there are few projects that can be counted with value, responsibility and cost.

That's not contradictory. A person who has changed tools does not mean that the company has changed the way it does things.

The model is stronger, and if only one chat box is added to the old process, it is usually only local. The firm needs to re-establish a job to get a steady result. Clear: Who provides the input, who receives and accepts, who handles the exception, who goes wrong, where does the experience stay. It's hard to get things out of the way that don't sound like models.

So there's no rush to talk about the end of the company AI. First, a task is being converted from ad hoc sessions to a system that can be duplicated. From AI to supporting established human collaboration to local acceleration, to people being part of the AI Native system, there is a long organization. This one is just for this part. Businesses are still doing so, and government and ToG's AI systems usually only go slower.

There are two types of evidence that need to be seen inside the firm.

I'll start with two categories of material.

The first is to see how people feel and behave in an organization: whether staff are used, whether managers are demonstrated, whether organizations are allowed to test mistakes, and whether performance and training are kept up with them. Another category depends on whether the business team is in continuous use, whether quality, cost, speed or income have changed, and how much recovery has occurred.

Microsoft 2026 Work Trend Index Closer to the first. It combines Microsoft 365's anonymous productivity signals and a survey of 20,000 AI users in 10 countries, with concern about whether the organization has followed them since the employees start using them. Transform Paradox is specific: 65% of AI users are afraid they can't keep up; at the same time, 45% feel safer than spending time doing AI work again. Only 13 percent of the people think that even if the short-term results are not achieved, they will be accepted for trying to redo their jobs with AI.

It's like being in a lot of teams. Companies encourage attempts to reward the old delivery rhythm, the old approval modalities and the old short-term targets on a daily basis. It is not surprising that employees are reluctant to touch a short-term uncertainty.

The Enterprise AI Playbook by Digital Economy Lab Look at the other end. The researchers interviewed 51 projects organized by 41 organizations covering seven countries and a wide range of industries. The selected projects meet several conditions: the system is on line and the business team continues to use it for at least three months, the results can be quantified and there is the possibility of expansion or replication.

This report is not a good estimate of the success rate of the Enterprise AI. It is an inherently replica of successful samples and researchers have identified selectivity and self-reporting limitations. It's more appropriate to answer another question: What else, apart from models, have those projects that have already entered the production environment?

The two types of material are put together, and the difficulty of the company AI is not so mysterious. Staff members will use it, not as a means of absorbing it; nor is the model amenable to a mission, nor does it represent a reusable operational capability.

A job is going to take place, and new borders are needed

Imagine a business colleague who lets the model organize user feedback into weekly reports. He copied a few paragraphs of the text, looked at the results and posted them into the document. It's been useful.

But if the company wants it to run steady every week, the problem will come up: what systems does the feedback come from? What data is not available? What can the model see? Who changed the classification? Who took over the sensitive complaints? Who was the last person to make the decisions? Do you want to stay where you've been changed? Can we stop making the same mistake next week?

The former is a session. The latter is the job.

I call the latter "organization absorption". It refers to the reorganization of a previously humanized work with a stable input, a clear division of labour, verifiable results and feedback that can continue to improve. Local AI outputs come to this point, and can become reusable organizational capacity.

This can be broken down into six steps:

Select tasks first. Start with high frequency, real cost, results judgement, not with “what more models can do”.
Complete context. Models require information, system state and operational constraints, and cannot be based on an isolated hint.
Clear permissions. AI is drafted, recommended, self-executed, or is the task returned to someone only in exceptional circumstances?
Write acceptance and exceptions. What is done, what is failed, who can overwhelm the results of the model, and how can it be restored if it is wrong?
See the results after delivery. In addition to saving time, it is important to see whether quality, client experience, risk, income and backlogs have changed.
Leave the experience. The validated rules, manual revisions, abnormal patterns and assessments are written into the next round of work streams and are not scattered in the chat records.

Figure 1: The deliverables of Enterprise AI are not a model response, but a closed loop that can be executed, inspected, taken over and studied.

The model can cover it directly, only part of it. It understands the text, generates candidates, calls tools and sometimes works continuously. Inputs, privileges, acceptances, exceptions, rollbacks and experience depositions will not be completed by themselves as the model is upgraded. Many projects are on the road between the demo and long-term operating systems.

And that explains why some Agent is amazing in the demo, and he's so heavy when he comes into the company. Demo just prove it can't be done. The production environment also needs to indicate under what conditions, what to do when wrong, who to take responsibility for, and how it has proved useful.

System interfaces often grow into team interfaces.

There's also a layer missing.Conway's 1968 article.It offers a simple observation: the organization that designs the system often ends up producing designs similar to its own communication structure. It is not a strict causal law, nor can it be hard to roll out the system with an organizational chart. It merely reminds us that how the team communicates, who has decision-making power and what has to be passed over, will slowly remain in the system's modules and interfaces.

The AI system will magnify this relationship. A production-level system addresses operational objectives, knowledge and data, models and assessments, tools for adaptation, authority, cost and safety.MLOps guide for Google CloudThe continuous integration, delivery and training are placed in the same engineering chain and are handled in a continuous manner. For each vagueer handoff in the organization, there is often an additional layer of interface, approval or waiting in the system.

Take the example of the client's intelligence body. To read customer information, orders, refunds and wind control, the four pieces were originally made by manual approval, and the intelligent usually simply move the serial process into the tool-call chain: Permissions are repeated, the context is lost between different systems and abnormally returned to manual queues. It's not necessarily Prompt at this point, but who's in this flow of values who's in the crossover without a common owner.

Seeing the company AI in Conway's law, it's often seen several typical shapes. The following table shows the design assumptions, not the statistical conclusions of the Stanford sample:

How does the team work?	What kind of system is it?	Common trouble.
Data, algorithms, applications, security, each team working in line	Each floor has a platform or service, which is spelled by a cross-team interface	The knowledge base is slowly updated, the power models are inconsistent, and the failure is going to be multiple teams.
One AI, center stage, all scenarios.	Large unified Agent or RAG platform, business team scheduled access	The platform becomes a bottleneck, and real business differences can only be resolved by bypassing the platform
Business area team to end responsible, with shared platforms next to it	The scenes can evolve on their own, and the platform provides a unified model, retrieval, audit and assessment capability	There is still tension between autonomy and re-use, but the border and the owner are easier to tell.

Conway's law can't be used too hard. The organization has a customer service, order, wind control and knowledge department, which does not mean that the system should be able to break into customer service, order, order, and control and knowledge. The sector chart is not a multiple Agent diagram. The identification of capabilities by mandate, context and instrumental boundaries should be followed by a judgement as to which of these steps are truly worth independenting. Many Agent is better suited to situations where mandates can be parallel, where context is prone to pollution or where different tools and expertise are indeed needed. In the rest of the scenes, simple, combustible workflows are usually easier to inspect and maintain. The system reflects teamwork among teams, but does not have to translate one-to-one departmental interface into Agent.

Microsoft’s guidance on AI CoE has a similar meaning: AI capabilities are usually built on existing cloud, data and governance teams, rather than creating a separate island that only “dos model”.Microsoft AI CoE Guidance For developers, this means that the system ' s owner, input-output contracts, permission boundaries, SLO, evaluation and upgrading paths are best matched with real teamwork by teams. Once the tool is used, the chain is long and thin, and it is possible to see whether there is also a high frequency, vague, unaccountable interface between the teams. Governance should not be seen only once before it is online.Govern section of NIST AI RMFPosition roles, responsibilities and risk management throughout the process.

Where the project is expensive, it's often not called in.

The Stanford report has two sets of figures that can be easily extracted into headings: in successful projects, 77 per cent of the problems that respondents consider most difficult are “unseeable costs” of change management, data quality and process re-engineering; 61 per cent of successful projects have experienced at least one failure before they are currently successful. Models remain important and enterprises need not consider failure as a mandatory stage. These figures are a reminder that the cost of the model, which is finally written in ROI, often does not include the organizational work that the project consumes. Data problems are not the same as cleaning up all historical records. The team had to decide which data were useful, who could access them, update them and how mistakes could be pursued before the project would remain in place in pursuit of “ideal data”.

Businesses rarely start with ideal data. In many scenarios, LLM itself has become a tool for data processing. Ninety-one per cent of the cases successfully processed unstructured data, and 88 per cent of the cases LLM helped businesses to open data assets that existed but were not available. In the past, data was scattered across multiple systems, belonging to different teams, and no one could really get it together. There is now at least one more way for enterprises to extract, sort and feed these materials into the workflow.

The anonymous logistics cases in the report are a good illustration of this. Invoice processing appears to be typical of AI missions: reading invoices, ticking fields, matching orders, entering systems. However, projects began with the accumulation of duplicate templates over the years, mixed input of telephone mail scanners and exceptions that must be constantly corrected by business experts. The team compressed the chaos template before arranging for the field personnel to review the output of the model, connect the process to the ERP, and continuously remove collaborative resistance from the top to the project. The model is certainly inside, but it's just one of them.

If only RAG, Agent, tool call and model paths are available in the framework chart, these tasks can easily be left out of the chart. They decide, however, whether users will trust the system, whether business will stop old processes, whether the data team will open the interface and whether the legal services will allow for higher-value examples. The project may have a usable prototype, but there is no set of practices accepted by the organization.

It's like the J curve that economics often says. After the general technology has gone into the company, the organization has to invest in a rewrite process, train people, collate knowledge, supplement data interfaces, and establish governance and assessment. The benefits are more likely to emerge when these things are slowly stabilized in the short term when inputs are visible than returns. Businesses should certainly not tolerate the lack of results indefinitely; if the budget covers only models and developments, and considers other inputs as additional frictions, commercial judgements can be distorted from the outset.

The costs of Enterprise AI are not limited to token, GPU, SaaS seats and vendor offers. The system should also try to avoid being tied to a single model and model routed where appropriate. The greater cost is to re-establish a certain task, to connect it, to find out what it is, and to be held accountable for what has happened. It also counts the costs of allowing failure: not only is it a hard loss from a failure, but also whether the team has the opportunity to continue to adjust to the failure.

The project will be stuck outside the staff's reluctance to use it.

When projects move slowly, managers can easily attribute the causes to the following: inert, intransigent, and unmotivated. However, if employees have written materials, checked information and made preliminary analyses in private using generic models, the problem is not always the use of the will itself.

The contradictions in the Microsoft report are real. The staff knew that AI was important, but did not believe that the organization would reward them for short-term uncertainty for AI. A person who spends three days re-engineering the handover process may in the short term be under-reporting of the old format; if performance is only in terms of the number of weekly reports, the most prudent option is not to change.

In Stanford, resistance is also often not derived from end users. The functional units of Legal, HR, Risk, and Compliance were more frequently mentioned by interviewees. This does not amount to conservativeness in these sectors, let alone to circumvent the law. They are inherently risk-taking for the organization. It's not surprising that the team took a blurry Agent at the last minute to ask for permission, and it got vetoed.

More practically, these actors have been involved in the design of work from the outset. Legal and compliance definitions define which data are available, which records are to be kept and which actions must be identified; what mistakes are to be corrected by wind control and which must be blocked in advance; and what new jobs HR and the head of operations would have to answer, when released. First-line users cannot simply be responsible for opening new tools, and they know what really does to ease the pain.

Otherwise there'll be Shadow AI. In order to finish the work, the staff still follow the old process and use personal tools to make up for efficiency. Individuals may benefit, but companies are not equipped to be manageable, auditable and reusable. In many companies, the gap between formal supply, governance and real demand is too wide, and the private use of staff is growing in this gap.

The continued removal of barriers to sectoral synergy at the senior level, the provision of trial and error space, the availability of platforms and infrastructure, and the acutely needed locations for front-line staff will reduce the time frame for the project to be deployed to deliver results. In turn, staff learning new technologies, project iterativeity, data preparation and cleansing, processing compliance requirements and the completion of process files slows down. How fast the project can run depends on how these conditions are superimposed.

The division of labour requires a look at the work itself, and then at the changes it brings.

The project is being re-engineered in a participatory context with the human and the Agent. In the case of teams, managers also decide how to allocate tasks: where to go, when to take over and who to take the results.

We'll split a job and talk to who. Information gathering, information collation, candidate generation, finding anomalies from a large number of records, often with relatively clear input and acceptance patterns, may be more than done by AI. Project positioning, resource trade-offs, client commitment, cross-team coordination, usually involve more background information and judgement, and the person in charge should remain in the hands of the person in charge.

What happens when mistakes are made, and the division of labour is changed. Mistakes are easy to detect and easily amend, allowing AI to finish first; when errors affect clients, funds, compliance or brand names, manual review is to be placed before delivery, the system leaves a path of upgrading and rollback. Some of the work also appears to be process-stable and is based in practice on client history, teamwork and business continuity. It is difficult to clarify such contexts at once, and it cannot be assumed that the model has been understood.

It is equally worth asking whether outputs require unique judgement. The code, the complete version, is suitable for AI to speed up; with the content of corporate strategy, aesthetics and brand orientation, AI can help to spread or draft, and ultimately it will have to be changed from one person to another. This division of labour will allow people time to return to places where judgement, communication and consequences are more difficult to outsource.

A sentence “AI to assist in writing proposals” cannot guide the actual work. The team needs to break down the process into steps, write where it came from, what AI could do, who could review where it was, what had to be upgraded, and who was responsible for the final delivery. Manual modifications to the results are also to be retained: some are changing facts, some are binding on the replenishment and some are from experience. They should not disappear as a single delivery ends.

The contents are written in order to be handed over to AI in a stable manner; the team knows who will take over when the exception appears. The difficulty of the division of labour is to place the handover, review and responsibility in each concrete step.

The number of calls and the minutes saved after the division of labour had been online are only partially indicative. It also depends on whether the quality of delivery has changed: whether the results are more accurate and fit for the actual scene or whether the return to work is left to the next colleague. It depends on where the time saved goes. If the team spends time on user insight, judgement and relationship work, efficiency becomes a business residual; if you just wait to check the AI output, the process may be just one layer more.

Last look at the feeling. Is manual review ever light or is it always a big change? Will the exceptions be successfully handed over to the responsible person or will they return to the vague crowds of conversation and ad hoc coordination? These changes are more problematic than a nice call scale map. Modelling capacity, business conditions and team experience will change and the division of labour will need to be adjusted accordingly.

It's not necessarily the smartest job to get into production first.

When an enterprise does AI, it is easy to prioritize according to whether it appears to be smart or not. Models that can write strategic reports seem to be more valuable than models that sort out the work orders. The order of landing is often the opposite.

Successful projects begin more often with less romantic work: high frequency, duplication, heavy backlog, relatively clear input and the ability to check results. Such features are found in the security diversion, invoice processing, initial screening of passenger service, procurement of replacements, filing of documents, intellectual retrieval, and migration of legacy codes. They may not be simple, but what is accomplished is usually clear and mistakes are more easily contained in the recoverable range.

In Stanford's success sample, the fully autonomous smart body program is only part of it. More projects start with the conventional component, allowing AI to handle high volumes of recoverable work and to keep key outputs and exceptions within manual clearance. This observation is more appropriate to understand the order of landing: entering stable, results detectable, errors that can be remedied and making it easier to get into production first; clinical paperwork, external content, high-risk decision-making and complex codes still require longer collaboration and auditing links.

The degree of automation does not accord priority to this decision. The team can first place AI in a clear, easy-to-verify link to confirm that input, acceptance, abnormal processing and rollback can work steadily, and then consider whether to expand the delegation of authority. For AI application developers, a valuable capability is the process of organizing the work that the operational staff cannot describe into models that can be involved, implemented by the system and manually validated.

Software development and game development didn't escape this pattern.

It's a very specific matter to put it in software development.

The company has been able to read the warehouse, change the files, run the commands, move and retest. A code is generated and does not amount to a demand being delivered. The project also includes requirements boundaries, systems of dependence, testing, CI, code review, distribution windows, monitoring, rollback and online responsibility. When the developers deliver the writing, more effort will be directed towards defining acceptance and inspection, identifying risks, arranging feedback and undertaking final changes. Instead of starting with an automatic end-to-end delivery, put Agent in a clearer step of testing, review and rollback.

AI Coding is particularly suited to modeled changes that complete the certification chain: lot migration, interface replacement, configuration upgrade, test completion, local repair of known bugs. They all have relatively clear diff, test and rollback paths. Demand is not stable, structure is complex and cross-team responsibilities are blurred, and the model is not moving beyond organizational context.

DORA 2025 Similar reminders were given from the software delivery perspective: the effectiveness of AI will be influenced by the feedback capacity, platform capacity and working methods already available to the team. The study was about software organization and could not be pushed to all enterprises; it was still useful for those who were developing tools and engineering Agent. Partial acceleration does not automatically turn into a system that is faster, and the gaps in the original process may be more rapidly magnified.

The same is true of games and content production. Assuming that the team has enabled AI to generate activity configurations or task text, the production chain is not just a script. The configuration requires a Schema check, ID exists, rewards meet the rules, mission pre-conditions cannot conflict, sensitive content is readable, version changes are rolled back, and ultimately the pace of judgement, player experience and branding are planned. The model can be a quick candidate, but the ability to enter the project depends on the team's having the certification and responsibility arranged.

The more the model is generated and action is taken, the more it is validated, authorized, logs, rollbacks and assessments that cannot be completed until they are online. They are an integral part of productivity.

Headcount Reduction and conclusion

The productivity gains were real for the enterprise projects, and management had to decide how to use that component of capacity. In Stanford ' s success sample, the reduction was the largest single result, but not the majority. Projects have also chosen to avoid new recruitment, shift people to higher-value jobs or speed up product routes with the same manpower.

This is, first and foremost, a business choice. Companies can switch the time saved to faster delivery, higher service levels, more detailed customer coverage or lower personnel costs. The system does not make decisions for management. Growth opportunities, budgetary pressures, product backlogs and re-assignments will affect how this path is going.

Speed also brings more than cost advantages. It's like moving speed in a MoBA game: it doesn't make decisions for the player, but it changes the timing of chase, retreat and support. The same is true of efficiency in enterprises. It opens up options that were not in time to do, cannot do or cannot be prioritized, and it remains for management to decide where to use this balance.

Did the time saved translate into business results? What was people turned to for? Is the service for the client getting better? If there is no answer to these questions, the effect would probably be to look more or less beautiful in part. The reduction of staff is only one result that the enterprise may choose and not the only way to achieve efficiency gains. If the company had made the reduction the only goal, the employee would probably have refused AI, and no one wanted to lose their jobs.

Enterprise AI is piloting, often because companies have not yet integrated what models can do into something that can be accepted, taken into account, picked up by problems and then continue to learn.

The product goes from answering questions to doing things for others, across the capacity boundary. The organization still has to work with the development, operations, platforms and governance team to build this capacity into stable values.

References

Microsoft Work Trend Index，2026-05-05。
The Enterprise AI Playbook，Elisa Pereira、Alvin Wang Graylin、Erik Brynjolfsson，Stanford Digital Economy Lab，2026-04。
State of AI-assisted Software Development 2025，DORA。
How Do Committees Invent?，Melvin E. Conway，1968。
Google Cloud MLOps Guidance。
Anthropic: Building Effective Agents。
NIST AI RMF: Govern。
Microsoft: Establish an AI center of excellence。

When AI Stops Answering and Starts Doing: Anthropic Economic Index Report: Cadences

2026-07-18T04:00:00.000Z

“Ai has replaced a lot of jobs?” The question will always come back. It seems to have a straight number: how many jobs have disappeared, how many people have been pushed over by models, how many industries are most dangerous.

But the first change is often not in these places. More like a little move in the work, it's changed: whether an email should be written from the beginning, whether a code should be moved to Agent first, whether the reporter is asking the model a question, or whether it should hand over a half-finished piece to it to keep doing it. The number of companies, wages and recruitments will sooner or later be affected, but that is the latter account. The previous accounts first occurred on how the mission was opened, who made it, who accepted it.

Anthropic Issue 6 of the June 2026 Economic Index Report: Cadences, is parked at this earlier site. It doesn't count "AI has replaced a few people," but it looks at when Claude's users appear, take what they produce, and give the model how much execution. Read the full text, I think it gives not a larger conclusion, but a smaller and more specific set of observations: the work begins with talking and asking questions and answers, and turns into products, environments and commissioning.

And it determines how to read it: to see where change comes from, and to see where it ends.

Figure 1: Evidence of AI and work cannot be read intermingled. Data sources and scope of research are presented at the end.

Before we start.

At least three layers of data are available between AI and the job.

The first thing I've ever seen is...Use behaviourI'm sorry. Who opens the model with chat frames, desktop tools or API, to explain, write documents, write codes, or do a series of things. Anthropic reports are mostly down this level. It tells us whether the tools are actually going into everyday work, and it shows whether some way of using is starting to become common.

I'll be back.Task EffectsI'm sorry. Does the same type of job differ in speed, quality, error rate when there is an AI and there is no AI? Such questions cannot be answered by a journal alone. Researchers have to find similar people, similar tasks, and try to strip out differences like “the better people who are already doing more” like “AI”.

It's the last time.Labour market outcomes• Any changes in wages, hours worked, recruitment, number of jobs and occupational mobility. It's going slowest. Today someone who has written a report with Claude will not be immediately in employment statistics; will the company be less hiring, changing its position, leaving its revenues to shareholders or employees, and subject to changes in budget, management and demand?

So, in the phrase “AI is strong, work is out”, there are three different things that are often embedded: immediate use experience, local efficiency improvements and fears for the future. Combination of them, the discussion will soon lose its focus. Anthropic reports are on the first floor, reaching out to the second two. We can follow it, but don't cross it.

Looking at the samples, there are many unnecessary misunderstandings. This report uses data from Claude, a continuous sample and treated with privacy protection, and thus allows for a daily and hourly tracking of the rhythm. Chapter 2 Chat and Cowork data cover the period from 10 April to 10 June 2026; Claude Code was included in the comparison of the level of commissioning. Occupational wages are derived from the May 2025 statistics of the United States Bureau of Labor Statistics. (Anthropic, 2026, pp. 9-18)

Anthropic does not allow researchers to read chat records article by article, but uses a sorter to mark the session as a job, personal, course use, or to identify the product that the user lasts to take away, filtering cells that are too few. It can draw the overall contours, but it does not give the whole context. The sorter would also be wrong and it would be more appropriate to treat these figures as a map with limited resolution. As for the survey component, Anthropic will answer the question in relation to the use of the mode of privacy protection, with each user randomly taking up to 20 samples of the sessions, and eventually receiving about 9,700 respondents, with at least five sessions. Computer and mathematics professions account for about 30 per cent of them, but only about 4 per cent of the United States workforce; there is also a clear bias in the management. (Anthropic, 2026, pp. 19-21)

This is an active image of Claude users, not a microcosm of American workers, and it is not possible to move directly to explain Chinese employment. The borders are clear, and the observations in it are worth looking at.

And where did it change compared to the last issue of "Learning Curves"?

The last issue of Anthropic was published on March 24, 2026.Learning CurvesI'm sorry. The issue, which drew data from February 5-12, 2026, focused on two questions: whether Claude's use was expanded from a few technical tasks to more uses; and whether people who had spent more time were more likely to use AI.

It gets a very similar picture to the title of the learning curve itself. The top 10 categories of Claude.ai have dropped from 24% in November 2025 to 19%, which is used in dissipation; and Claude is more often used by people who have used him for tasks that require more work and education, and has a higher success rate. Early adopters may be more skilled, and those who remain today may be better suited to use Claude, so in this case, “long-term” is a combination of learning, screening and survivors. # I'm not sure #Learning Curves，pp. 2-4, 15-20）

The Cadences did not reverse these conclusions, but brought the problem a little closer. The last issue looked at adoption and learning: which tasks were spreading and which users preferred models and were more willing to overlap. This period began to look at how jobs were being reorganized: at what time people used AI, leaving with explanations, reports or codes, and leaving much judgement to models.

Contrast Dimensions	March Learning Curves	June, Cadences
Main issues	Proliferation of use; improved use of experience	AI, how to get into the routine; what products the user takes; how deep is the commission?
Data Form	Compare mission, platform, model selection and user time in a sample of one week per February	Continuous sampling, allowing day/hour changes; addition to product classification, cross-product comparison and correlation surveys
Key findings	Claude.ai dissipation; older users are more iterative, more useful and more successful	From chat to Agent, the product environment allows for closer assignments to be more deeply commissioned; users start using model outputs as work products
Still can't answer.	Whether these learning differences are caused by the use itself	Have these commissioning changes been translated into changes in wages, working hours or employment

Both issues are not automated, and AI is the primary concern of every user. The March issue splits the interactions into two categories: direct, feedback loop, task itseration, validation, learning, and integration into categories such as "information " and "assistance " , and looking at how people fit in with models. It concluded that older users were more traverse and learning and did not simply throw a brain at a model. Of these, automation appears in the Agent mode of API call, and it is more like human collaboration in tools like Claude Code.

The new autony in June is another ruler: how much does the model have to decide on its own in a mission, in terms of the depth of the commission. Claude Code is more autonomous and does not fight “old users more” . A person can fully retain feedback and acceptance on high-risk or complex tasks, while at the same time handing over a clear section of the border to Agent. Two issues are taken together, and the conclusions are naturally not conflicting.

Anthropic did not announce that AI was replacing someone. It just pushed the camera from "Who Learns to Use AI" to "Who is Giving Ai the part of the job?" The re-ordering of labour has also changed from a macro-judgement to a few things that can be observed: the time used, the product taken, the depth of the commission, and the feedback chain left in hand.

Claude's use of rhythms is already like work and life itself.

Chapter I of the report reads as a daily hotshot: morning news, morning business communications, evenings to find recipes, and suddenly, before the tax deadline, a lot of tax issues were being crowded into. These requests look trivial, but they just mean that AI has gone along with people, and it's part of life, and it's not just a chat box that's opened up on occasion.

During working days, about 35% of the sessions in Claude Chat and Cowork were awarded personal use; by weekends, this percentage had risen to almost 50%. Business communication, marketing paperwork and slides are less demanding, and emotional support, medical problems, and investment advice are becoming more numerous. In the morning, news requests are concentrated, business communications fall high on the morning of work, and the menu requests reach approximately 2.3 times the usual time at 6 p.m. (Anthropic, 2026, pp. 4-7)

Figure 2: Anthropic original Figure 1.2. The columns indicate the relative frequency of the request cluster during that hour, with a dotted line representing the average of the request cluster itself; the data are limited to Claude Chat and Cowork. Source: Anthropic Economic Index Report: Cadences (2026), p. 6.

The tax part is particularly intuitive. The United States had a federal income tax cut-off date on April 15; on April 14, users had about eight times the average tax-related conversation of May, and April 15 remained high, with April 16 coming back fast. This cannot be used to calculate how much AI has done for tax officials. It just takes a very common picture: once there's a concentrated, clear, online pressure in real life, people will look at AI as a ready entry.

There is also a pattern of night and weekend work requests: they tend to favour tasks corresponding to high-wage occupations; and the proportion of jobs related to low-wage fours has declined. AI ' s use follows the level of professional remuneration, as has been seen in the Learning Curves; this period cuts data to the hourly particle scale and shows a new rhythm.

It only reminds us one thing: AI will not evenly penetrate all jobs. It will grow with the established time frame, digital environment and mission boundaries.

From "Assent well" to "What delivered."

The most important change I care about in this report is that it starts to record what was taken when the user left the session.

Anthropic called the main output of a conversation artifactAnd it can be understood as a work product that can be taken away. It may be an explanation, a report, an e-mail, a presentation, a code, or a website. The term is not very daily, but it is closer to the work site than a model response.

The report identified 93% of Chat and Cowork sessions that produced some sort of explicit product. The most common are explanations, 17 per cent; documents and reports, 15 per cent; and recommendations and guidance, 11 per cent. This does not mean that the model's written content is being used, but the question has been moved forward: what does the user do with the model's output?

Figure 3: High frequency product categories reported as disclosed; “generation of identifiable products” is not a hierarchy with the three types of specific products and is not added.

The same product can be used for completely different purposes. One plan may be a travel strategy or a financing programme; translation may be either for personal reading or for daily work. Once the report is cut again according to work, personal and course uses, the difference emerges: the work uses are most frequently documented and reported, followed by explanations, draft mail, analysis and summary; and the personal uses are more frequently explained and recommended. (Anthropic, 2026, pp. 10-12)

In the past, the chat models were discussed, and people often looked at the answers as good enough and as little as they imagined. These problems are not obsolete, but are a step in the stream: the output of the model is being forwarded, pasted, run, modified and submitted, sometimes directly into a semi-finished product delivery.

The report also looks at token consumption together with professional wages. Overall, more tokens are used for work sessions that map high-wage occupations, and high-calculation products are more frequently found in these occupations. This correlation is worth reading, but don't read it as "token equals economic value." Career mapping itself is wrong, model selection affects token and occupations such as pharmacists are clearly retrogressive. More conservatively, models and people tend to invest more when users let AI handle more complex, open products that require constant judgement.

The same thing. The same thing.

And the second half of this chapter, aautony, asks another thing: how much judgement did the user give to the model?

Anthropic uses a scale of 1 to 5 to assess AI autonomy in a session. Translation, computation, question and answer are often low, as people almost clarify the answers and boundaries. Applications, websites, games and presentations are higher and models are selected in many options. This score is not related to whether the model has tool privileges or no consciousness, but only describes one thing: how many choices the model makes in this task.

The most interesting comparison occurred between Chat/Cowork and Claude Code. Claude Code has a higher average autonomy in almost all product types; the average difference for the plenary session is 0.37. Claude Code is 0.26 points higher even if only Sonet is used. This set of figures draws attention back from “which model is stronger” to the product itself: how people use the model depends on what capacity the interface gives and on what task is put into the workflow.

Blogs and articles are a good example of how to understand. Chat and Cowork have a median session that eventually produces a blog or article, which is rounded up by 13 rounds; Claude Code has only one human hint for the same product. About two thirds of the autonomy gap is not because Claude Code users are doing another batch of tasks, but because the close tasks are being given to the model in different ways. (Anthropic, 2026, p. 15)

The chat interface allows people to re-establish, read and continue to adapt; and the Agent environment, which reads documents, calls tools, works continuously, makes it easier to have sex with the target, the boundary and the acceptance conditions. When discussing AI substitution, it is common to focus only on modelling capabilities, and changes in this layer of product and work stream are less visible.

This is also a follow-up to two articles I wrote earlier. I'm not sure.The generation, AI, will not just close the job, but re-schedule the work, then expand the demand.The judgement is that AI was first compressed by a mission, not by a full career; now the report gives a picture of the mission level, where the work is broken down and the product is placed in different levels of trust. I'm not sure.AI Agent's future is not a full autonomy, but a manageable commission.It is not about whether or not people should quit, but about who should remain in the target, the boundary, the acceptance and the responsibility. The new data do not support these two sets of judgements, but make them less abstract.

Users feel they're being helped or replaced.

Chapter III begins with a question, not just a journal.

Nearly 60 percent of the respondents felt that AI would be able to perform more tasks independently in the next 12 months than it is today; more than one third of the respondents expected that AI would be able to do most or almost all of its tasks by then. The figures appear to be strong, but they measure the expectations of the interviewees, not an independent assessment of their positions.

Nor is there a correlation between the way in which it is used and the way it is felt. The more highly automated users tend to believe that AI will have a more positive impact on future income, re-employment, job significance and autonomy; they also more often say AI has increased the market value of their skills. It also mentions that many users report that they have improved speed, scope and quality. (Anthropic, 2026, pp. 27-28)

It's easy to be told as a story: the more you give your job to the AI, the less you're afraid of AI. It may be that automation has indeed brought benefits to these people, or that it has been more optimistic and willing to try new tools, which has made them more willing to hand over their full tasks. Even if you control the user's time using Claude, the selection effects are not cleared.

The other side didn't disappear. About 10 per cent of respondents felt that they were at a high or high risk of losing their jobs in the coming year; there were also concerns about junior colleagues and others, not themselves. This is like the mindset that is common in technological change: tools are helping me, but I can see who it will squeeze first, especially those with less experience and easier to break down into clear steps.

There is no clean answer. It is not surprising that AI brings efficiency, anxiety, learning opportunities and job concerns. Each person faces different tasks, different backgrounds and different possibilities for examining model outputs.

This report doesn't prove anything.

The first thing to write is that it doesn't prove how much AI has created employment substitution.

The report does not observe the number of companies, payroll and recruitment decisions, nor does it randomly divide users into “Approve AI” and “No-Approve AI”. It sees Claude user behavior, task attributes derived from the sorter, and self-reporting by a group of users. We can tell from it what missions AI is entering, but we can't say that a job will disappear in a few years.

Automation is not a cause or a consequence of optimism. People with a higher percentage of automation may be in posts that are more suitable for AI, better tools, or more willing to believe in technology. The report has been valuable in bringing out this relevance, but it cannot answer “is it more likely that a person will feel safer by commissioning more”.

Token is not the value of the output, nor is it the result used. The model, in order to produce a more computing application, does not indicate that the programme is better, let alone that it creates the same percentage of value for the company. The ability of users to validate, organizational adoption and real processes of mission entry will slowly determine the economic impact.

Claude's data is not naturally representative globally. It is well suited to observe those who have come into contact with and have taken the initiative to use the front model. The product entry, enterprise software environment, industry structure, labour system and payment patterns vary in China. It was reasonable to use the report as a window of international knowledge; it was too soon to translate it directly into a forecast of employment in China.

Put it back in the bigger evidence. It'll be a little more stable.

Looking at Claude's log alone, it's easy to push it; and looking at several external studies, judgment is less.

NBER's "The One"Generative AI at WorkIt looks at a very specific scenario: the average number of problems solved by 5,179 passenger service providers with access to AI-generated assistants increased by 14 per cent per hour; the number of new and low-skilled staff increased by about 34 per cent, and the change in senior and highly skilled staff was minimal. This does not mean that all white collar missions will have the same effect. It illustrates that where mission standards are clearer and feedback is fast, tools can spread a portion of the practice of high performers to less experienced people.

Another NBER studyStill Waters, Rapid Currents: Early Labor Market Transformation under Generative AIThe Danish Labour Market is the subject of a review of the Danish labour market. ChatGPT has not changed significantly in its income and time-time recording, and research design can exclude impacts of more than 2%; but the work structure is moving, new tasks such as content generation, AI supervision, AI integration are emerging, and the users are moving to more relevant and better-paid occupations for chat robots. (NBER Working Paper 33777)

The title “Silent surface, stormy water” is very appropriate. It's not in conflict with Anthropic's report. The former says that macro-labour indicators are not visible for the time being, while the latter shows us that tasks and work streams under the band are being rearranged.

International Labour OrganizationGenerative AI and Jobs: A Refined Global Index of Occupational ExposureThis gives a broader context: about a quarter of the world's workers are in “specified AI exposure” occupations, with only 3.3 per cent of global employment falling into the highest exposed categories; according to ILO, most jobs are still made up of jobs that require human input, and therefore jobs are more likely to be transformed than jobs are eliminated as a whole. (ILO, 2025)

Exposure is equally not equivalent to unemployment risk. It only says how many jobs in a profession might be met by a generation AI. Whether it is adopted, who ultimately has the efficiency gains, how the organization has been re-diversified, and whether there are new demands, still needs to be answered by subsequent data.

The change has begun, but it hasn't grown into a slogan yet.

If I sum up this report in one sentence, I would say that the early impact of AI on work begins with people giving more execution, not with a sudden loss of the whole career.

This sentence is not as exciting as "AI will replace human beings," but it is closer to what can be seen now. The answers in the chat interface are becoming documents, codes, programs and successive tasks; the relationship between people and models, from your one-word collaboration, slowly becomes target setting, environmental access, implementation, inspection and taking over.

It'll reorder a lot. New persons who have accumulated experience through repeated implementation may be more easily taken over by Agent; the ability of senior workers to judge, accept and accept and define boundaries becomes more visible; and organizations have to re-decide which tasks need to be performed by hand, which can be handed over but must be left behind. But these are not yet judgements on the total number of jobs.

The next thing that is worth pursuing is a few more specific things: whether the task entrusted to you is stable, from drafts and one-off analyses to processes that affect the real state; who is responsible for model outputs, and companies are not investing new jobs and time in review, acceptance, redress and authority management; whether primary workers were replaced by training opportunities that they had obtained through repeated implementation or changed to new learning paths; and whether efficiency gains ultimately flow to employees, companies or consumers.

Anthropic, this report does not answer these questions for us. It did a basic but necessary thing: to tear back the vague term of work and the actions that people are doing every day.

The first two questions determine whether efficiency is going into the organization, the third is how the next generation grows judgment and the last is who gets the proceeds. The data are still waiting longer.

References

Anthropic, Anthropic Economic Index report: Cadences, 2026-06-26。
Anthropic, Anthropic Economic Index report: Learning Curves, 2026-03-24。
Erik Brynjolfsson, Danielle Li, Lindsey R. Raymond, Generative AI at Work, NBER Working Paper 31161。
Anders Humlum, Emilie Vestergaard, Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI, NBER Working Paper 33777。
Pawel Gmyrek et al., Generative AI and Jobs: A Refined Global Index of Occupational Exposure, International Labour Organization, 2025。

Can Closed-Source Models Be Distilled? Knowledge Distillation for Generative Language Models

2026-07-16T04:00:00.000Z

Recently, distillation has become more than a technical word. Anthropic describes the unauthorized, bulk-based use of its model output to train competition models as “distillation attacks”I'm sorry. In Chinese, the question is often answered: how can the closed-source model be distilled without the public weight?

The questions in this article can also be addressedPost-linguistic training and fine-tuning of practice: from SFT, LoRA to human alignment、Data synthesis is becoming a project: from Terminal-CorpusHow the concept of a relatively close read together is developed in different contexts.

It looks like a yes-or-no problem, actually mixing up different levels. Neither the direct answer to “can” nor “can” is sufficient.

The teacher model provides a signal.
What exactly is the student training plan?
Do you have the right to use models and data like this?

If these three things are not removed, it is easy to slip the discussion to the slogan: while all model aids are called distillation, the other side, because the model has no public weight, asserts that it cannot be distilled.

Focus on the training chain, whether the model is open or closed.

Classic KD: Students come in direct to teachers

The core of the classic Knowledge Division (KD) is simple: teachers have learned to behave and students are trained to reduce their differences directly from teachers.

The typical thing is...Output distillationI'm sorry. The question raised by Hinton and others about the soft classes not only tells students what the correct categories are, but also retains information on how close the teachers think the other categories are to the correct answers. In classification tasks, students are drawn up by the probability distribution given by the teacher ' s output layer. In the generation model, this distribution becomes the next probability distribution for token; it is the teacher who is more inclined to say what he/she will say at every step.

In addition, there are two types of classic practices:

Characteristic distillation: Alignment hide layer, attitudinal or intermediate. FitNets' instinct is that students should not only copy the final teacher response card, but also use the teacher ' s intermediary advice.
Relationship distillationInstead of forcing the two models to look the same at each level, students are left to the teacher to express the distance, angle, similarity, etc. in space.

These three methods are all doing the same thing:The goal of student excellence is more like a teacher. Characteristic distillation and relationship distillation usually require access to the hidden layer or weight, so it is more appropriate for the white box scene; the classical token-level KD has the lowest threshold, but still requires access to the final logits and needs to ensure that the vocabulary of students and teachers is aligned.

For LLM, the black box model, although not complete logits, can give a string of outputs tokens. Students target this token and train themselves to continue to produce with cross-critics. This form of loss is the same as self-repatriation pre-training, and the difference is simply that the monitoring signal is replaced by the original language into the teacher's output. The final answer, long-debate text, JSON, functional call and tool call track, as long as it is used to train students, is an output or sequence-level behavioural imitation. The training signal has not changed: students are still reproducing a sequence that has been given. This is what Kim and Rush call a security-level KD; from the data line, it's also like using teachers' answers as a false label for SFT.

This border need not be too categorical. The more useful question is:When students are updated, are they learning directly from teachers or in a redesigned data and feedback system?

It's another training link.

To involve stronger models in training does not mean that the entire output is moved into the training set.

Models can also be just a tool in the data production and assessment chain: scaling up seed tasks, constructing counter-scenes and dilemmas, generating candidate answers, helping to mark preferences, or acting as filters. The trainers then put it together with the search for evidence, rule-checking, unit testing, manual auditing or incentive models to generate data required for SFT, preference optimization or RL.

Self-Instract is here: the command can expand input reporting data without requiring that each sample be written manually from zero. But model generation is not the same as natural high quality. The choice of the distribution of tasks, which samples are retained, what is correct, which acts should be rejected, what to reward, and how to validate, still depends on the trainers to judge.

That is why I want to separate the two links.

The goal of the pure KD is to reduce the distance between students and teachers. As a technical tool, it can certainly be used in home-grown models, clearly mandated models, or in the training chain for teachers-students within the team. But if the context becomes "Closed-source-Power Model Output" and small models, as much as possible, it looks more like a behavioral reproduction: Trainers are hardly part of their judgment, and the goal is to repeat what the teacher said. It's a lack of taste, you're not training a model, but is replicating, not deciding what to teach, what not to learn.

Model-aided SFT or RL are different things. The value here is not only what teachers say, but also how people design curricula, constrain data sources, join certification machines, organize difficulties, define rewards and fail borders. It does not have to faithfully replicate every word of the teacher, and it can even clearly filter out the teachers ' bad answers. It is a technology of data synthesis, not a mere reproduction.

Nor can it be called as a substitute for content. A large number of teachers were arrested, the same text was inserted into the training set, and no new mission design and quality control was available, even if it was called synthetic data, which was another output. Conversely, the same COT or tool tracks can also be teaching materials, provided they are validated, reorganized and placed in a new task and assessment chain.The difference is not what the text looks like, but what the training links look like.

On-policy KD: Why do students write first?

The above distinguishes between training objectives. Even if the target remains distilled for the self-regression model, there is a problem in the training process: students see different prefixes when they are trained and deployed.

Normal offline KDs are often this way:

Real data or teachers continue to copy students imitating.

But when deployed, students do not face ideal prefixes provided by teachers, but rather those they have just written. It may have misled an entity, missed a condition, or picked the wrong parameter in the tool call. It can only continue to go down in this distorted context.

On-policy KD:

The student teacher gives a distribution or feedback on the student's own trajectory.

It deals with a mismatch in the distribution of the state between training and deployment, rather than with the creation of a further data set. MiniLM discusses LLM distillation from the point of view of on-policy and reverse-KL; GKD directly studies teacher feedback on student generation.

Therefore, the fact that the data are not the third type of data synthesis is not equal to the fact that the teachers produce the data and then make RL. It also targets teacher behaviour, but only allows teachers to give feedback on the trajectory that students actually reach. This can be achieved by using tools close to the tactical optimization, but it is still discussed in distillation.

Black and white boxes: only limited to signals, not conclusions

On-policy KD asks teachers on which tracks to give feedback. And then, what we're going to ask is what kind of signal the teacher can give. The difference between open and closed sources is mainly here, rather than deciding directly whether or not to distill.

The white box teacher.: access logits, hide layers, attention and relationship structure. The three types of KD that are output, feature, relationship can be established and can be more easily on-policy KD.
Only the final text of the black box teacher.: usually provides answers, reasoning texts, tool tracks or preferred judgements, but cannot be distilled from the hidden layer or the full vocabulary of logits KD.
Interface with logprobs: In the middle. Even if weights are not disclosed, KD may still be limited if sufficient token probabilities are obtained; this is not the white box feature distillation or relationship distillation.

This also means that closed source models cannot be distilled too full. More specifically:

Closed-source models do not necessarily support white box distillation; only text-output interfaces do not necessarily support classic token-level KD; however, they may still provide information that can be recalculated or processed into SFT/RL data.

The question of whether the closed source model can be distilled has reduced several layers to a single sentence and is therefore not suitable for a one-size-fits-all answer. The same paragraph, which is exported by the teacher, is a whole-student training package, allowing the student to reset the teacher as much as possible, is a replica of behaviour; when the trainee re-decides the task, screens, validates and rewards, the teacher is only one source of material. These choices are what I say is about: what is worth teaching, what is credible, what mistakes must be rejected. Whether the closed-source model is used is only a surface layer, and the real gap is whether the trainers put these judgments in the training chain.

References

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, Distilling the Knowledge in a Neural Network
Adriana Romero et al., FitNets: Hints for Thin Deep Nets
Wonpyo Park et al., Relational Knowledge Distillation
Yoon Kim, Alexander M. Rush, Sequence-Level Knowledge Distillation
Yizhong Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions
Zuyang Gu et al., MiniLLM: Knowledge Distillation of Large Language Models
Rishabh Agarwal et al., GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models
Florian Tramèr et al., Stealing Machine Learning Models via Prediction APIs
Anthropic, Detecting and preventing distillation attacks

AI Application Startups: From Selling Tools to Selling Outcomes

2026-07-13T20:00:00.000Z

The next 1T dollar company will be a software company masquerading as a services firm.

The questions in this article can also be addressedModel Is Good End: 2026, AI, which is really scarce, is an application rather than a larger model.、From Engineering to Builder: A person's company and product thinkingHow the concept of a relatively close read together is developed in different contexts.

Recently read the Sequoia partner Julien Bek's bookServices: The New SoftwareI'm sorry. This article is of course looking for investment opportunities, but it's quite straightforward: after the model gets stronger, what exactly is AI supposed to sell? How much of the moat do I have left when Claude or ChatGPT's next upgrade puts the tools I'm making into a function?

A lot of teams will do Copilot first. It helps lawyers draft contracts, helps accountants to prepare accounts and collect quotations for procurement. The professional is the professional, AI just makes the hands live. We're already familiar with this kind of product.

Bek wants to move on. Some clients do not want to buy a smarter software package or find another person who can use it. He wanted to finish the job: the contract was drafted, the accounts closed, the insurance was completed, the claims were processed and the system was running as usual. The customer buys the results after completion. Model upgrades can also be used as a stress test: If it turns a product into an built-in function, it means that the team has only temporarily filled the capability gap; if it turns established workflows, data and trust into faster and cheaper services, the model is expanding the value of the product. Entrepreneurs should be on the side of model growth rather than racing with models.

First, we'll tell the difference between intelligence and judgment.

Bek took the job down to two floors. The first level is intellectual activity: reading information, filling out forms, matching rules, writing first drafts, testing, debugging. They may be complex, but they usually have clearer rules, inputs and outputs. And that's probably the part that's the first to be compressed when the model gets stronger.

The other level is judgment. Should the client not be taken over, which contract is worth taking the risk, when it is on line and who is responsible for the exceptions. These things are not done by running the rules over once, and there are experiences, responsibilities and trade-offs. The model can now help a lot, but it is difficult to take these decisions alone.

This distinction is better than AI's replacing a particular occupation. Rather than labelling jobs as alternative or irreplaceable, it is better to open up jobs to see which links are clear enough to allow automation and which are still subject to professional boarding. Entrepreneurship opportunities are often hidden on this dividing line.

The article gives a judgement that the higher the proportion of the intellectual workforce that can be regulated, the sooner automation is likely to land. This change in software engineering was first made, and other occupations will also experience the same problem: What is left to the model, and which exceptions and trade-offs remain to be taken up by the person.

Copilot Seller Tool, Autopilot Seller Job

The article distinguishes between two product routes, Copilot and Autopilot. Copilot gave the capacity to the operator, the client paid for the software; Autopilot directly completed a job and the client paid for the results.

Models are improving both intellectually and in judgement. In the past, we used to use AI as a tool for professionals to decide how to use it. Harvey sells the product to the firm, and Rogo sells the product to the investment bank. Professionals are both clients and responsible for the end result. As models continue to progress, direct delivery of results to clients begins to become a viable option in at least some areas.

For products, the existence of chat boxes in the interface is only superficially different, and where the budget comes from is more important. Software budgets are usually limited, and service budgets for the completion of work are often much larger. The article uses the example of an accountancy: a company may spend approximately $10,000 a year on QuickBooks and another $120,000 on accounting. If AI can steadily deliver one of these tasks, it will no longer be faced with the original SaaS budget.

But the sale also brought responsibility. The tool is wrong and the user can use it as a complement; the service is wrong and the client only asks one thing: Why is the job not done? So Autopilot is not just a question of whether the model is smart enough. Quality is accepted, the abnormal is handled and the risk is ultimately placed on those who have to be made clear in advance.

Why do we start with outsourcing?

I like the idea of outsourcing as wedges. Instead of placing entry points on “replacement”, it suggested that tasks already outsourced and subject to a large number of rules should be sought.

A job has been outsourced, which indicates at least three things: the company accepts that it will be done by an outside body; the budget already exists; and the buyer buys the results, not the job position on which a person sits every day. The replacement of the existing supplier with an AI original service provider is often a replacement. Direct replacement of an internal post would translate into organizational adjustments and a completely different resistance.

This also explains why articles look at areas such as accounting and auditing, insurance brokers and settlements, medical billing codes, IT outsourcing, supply chain procurement, traditional management consulting, HR services, legal affairs and taxation. They are not simple, but they contain many standardized, repetitive and inspected tasks, and clients have long been accustomed to buying services on the basis of results. They all need to be judged, but each area can find different entry points from the boundaries of intelligence and judgement.

Note: YC also referred to compliance and audit in RFS 2025 Spring/Summer. It places language models in a more comprehensive and detailed review, and from personal assistants it talks about taxes, law enforcement, personal asset management, and the close, expensive or time-consuming nature of mail management, calendar arrangements and to-do matters. AI Agent is one of the hottest directions in 2025, and everyone is thinking about what Agent should do in the vertical field. They have one thing in common: let AI handle complex, but not necessarily creative, information in a certain field. Many jobs are inherently dry matching, and give them away so that people can put their time back into more creative places.

Use this framework to study a business.

If you want to find entry points for AI applications, then you can put the problem on a specific process:

What was the outcome of this delivery, and could the client see at first sight whether it had been completed?
How much of it is dealing with rules, files, matching and repetitive processes, and how much depends on the judgement of senior practitioners?
Has the client outsourced it today, or does it have a stable service budget?
Which are more likely to replace older suppliers with new services than to change internal organization?
What are the validated feedbacks that can be left for each completed product to make the next treatment more stable?

These questions do not provide an answer, but they bring back to work the "big industry" "AI can do something." When looking at the industry, it is better not to stop on the name of the industry, but to look at a certain process: who pays for it, who bears the wrong cost, which one of the problems AI can save the client.

And it wasn't the answer.

The article does not say that service will always replace software. High-judgement, high-risk, strong regulation, particularly in relation to the financial, medical, legal or complex interpersonal consultations, still requires the involvement of people. Even if the model is able to complete most of the process, the client is not necessarily willing to hand over responsibility together.

It provides a pragmatic entrepreneurial perspective. In addition to asking “what tools can I make”, one could ask “who is the client asking to finish this now”. If a job has been outsourced, the rules are clear enough to allow the results to be accepted, AI applications have the opportunity to start there and to adapt the tools to service.

The blogger says that the government is not a party to the law. Services: The New Software, Sequoia, 2026-03-05。

What We Talk About When We Talk About AI-Native Games

2026-07-12T07:30:00.000Z

I actually wrote a blog about AI Agent and the game two months ago. But more of the time, it was in the context of game development, thinking about how AI Agent could help us make a better game, from research to the whole process of running distribution. I mentioned some points in it, but I was still relatively low on the game industry, some ideas coming from industry reports and my little chats with some game developers, so it may be slightly biased, and some views are too idealistic (which is still idealistic) to be found on the ground. And I dug a hole: the article deliberately overcame the part of AI that was involved in the experience and the way it was played. I've learned a lot today, so let me fill this hole while there's some external pressure.

The questions in this article can also be addressedReplaces the front-end engineer 'Ai, not making the simplest panel in the game、How the game industry is introduced AI AgentHow the concept of a relatively close read together is developed in different contexts.

When we talked about AI Native Game

I don't know who made this idea of AI Native Game, but it's terrible to look at names. We can start by making it clear that here's AI specifically referring to the generation of artificial intelligence models (which is used in the absence of a special description). The technology you used to do for enhanced learning intelligence, limited-state machines, behavioral trees, etc., does not belong to AI, but it doesn't mean that the new AI Native Game doesn't need them. We still need all the technology of the past, just a little fresh sauce, but it might be a little tough.

Now let's see what the Native Game Law is. Considering that we are still in the early stages and should not be too radical, I am here to give a relatively broad definition:AI Native Game is a game where the generation of artificial intelligence becomes part of the core, or it adds substance to the core. For the former, the world of games will not continue to function properly after the detached generation of artificial intelligence, or the core of the game will disappear; for the latter, AI will only complement existing core games and experiences. And to think of this, you may already have some examples: Starfield, Whispers From The Star, B-Side: Olivia Lin, all belong to the former; Eggs Party, The Life of the Jedi, the AI teammate and the client of The Anti-Wilm, belong to the latter. It's a rough division, but you can see from it that AI is involved in the deep end of the game.

I need to expand a little bit to help readers better understand AI Native Game. The example is one limitation: AI’s intervention is limited to the generation of pure text-based models (because other forms are really difficult to do), and their actions and TTS are essentially outside the text-based ones, which may lead readers to think that this is all. In fact, 3D world-generated by the Lee Fei-Fei team, Gene 3 by Google DeepMind, 3D by Meshy, and gpt-image-2 , the image generation is AI capability. In the future, the whole world of the game can be turned over to AI to generate, and then evolve according to the user feedback. But it's too free. Free generation is expensive and uncontrollable, technology can land, but it does not necessarily bring pleasure to players. In my opinion, AI Native Game needs to bind the generation of AI within a defined framework. Pure text generation and small-scale image generation will remain the mainstream, games will continue to be developed by people and games will remain art.

AI, what does it bring to the game?

Now that we have defined AI Native Game in general, this chapter has to answer a new question: What can AI bring to the game? Or, in other words, why should we make AI a game, why should the player need AI? When we build an "AI Native Game," this is the first question to answer. Don't say it's the boss's order. You can do a function and develop a game on the boss's orders, but the player won't play on it. I hope every game developer will think about the MDA model that he learned that year:AI is Game M and D, but the goal should be A, a little fun for the player.

Why would a player have fun because of AI joining the game? Before we discuss this, I would like to give the AI level of integration. This is a rather rough grade, and it's demo-like, and many differences are not possible to reflect. Anyway, try to split it first.

The first level is AI Chatbot, which is the initial form of AI integration. It's a simple chat robot, which is used in the vast majority of cases for role-playing: it has its own set of people, and sometimes it creates a long-term memory of you, so it can play a role better, but it's what it can do. Typical examples are Starfield, Cat Box and Silly Tavern; if you can accept the inclusion of some other elements, Whispers From The Star and B-Side: Olivia Lin. Chatbot brought emotional companionship to the player, and there is a certain lack of such companionship in modern society. Play types like SLG or RTS typically have NPCs that guide players to learn game games, and they can also access AI to provide better guidance, which has nothing to do with emotional companionship, but is also in the context of Chatbot.

Second floor I call AI NPC. Compared to Chatbot, the first level of AI understands the world, acts (including speaking and influencing it), and interacts with players in the world. I can see that this is the world I want to talk about. AI is no longer in a static virtual space and simple settings, but in the world of games where players are located, understand the world and change it. The egg treasure of the Eggs' Party, the AI teammates of the Unharmed and the PUBG, the AI expeditions brought by the Charlock, the clients of the Anti-Wildness, the UE definition of the NPC of Fortnitte, and the AI cut-off price of the Faraway Galaxy: Builder, can all be placed in this layer. NPC understands your natural language and gives feedback (impacting the environment, speaking, expression and dementia, etc.). AI NPC is actually strengthening the original game, giving players a higher degree of freedom, for example, by using voice commands for AI teammates, or by breaking up each other's preparedness by talking and acting on the level. Players always want the world to be real, and AI can help us do this. This layer is a reinforced play, and the next layer is a new play.

The third layer can be called AI World. The emphasis is still on the world, but the focus is no longer on how NPCs perceive and influence it; it is on the world itself. To decide whether something belongs at this layer, I look at whether the model participates in updating world state rather than merely generating more natural dialogue for an NPC. After the player makes a choice, resources, relationships, institutions, and character decisions continue to change; narrative and gameplay stay open, and stories emerge naturally. At this layer, world evolution is part of the gameplay.

Stanford’s Smallville (Generative Agents) used the memories, reflections, and plans of 25 agents to let everyday interactions accumulate into parties, relationships, and social events. HKUST’s Aivilization expands a similar idea into large-scale social simulation, where agents take part in production, trade, social life, and the formation of governance and cultural norms. History Simulator: Chongzhen (《历史模拟器：崇祯》) brings world evolution into a released historical strategy game: the player issues free-form decrees in natural language, while the model generates world-state feedback and ministers’ dialogue at runtime, allowing fiscal, military, and court politics to continue unfolding from those choices. Westworld remains the fictional reference point: if characters have memory, agency, and histories entangled with one another, when does a theme park stop revolving around its visitors and become a world of its own?

These examples sit at different levels of maturity: one is a science-fiction reference, two are closer to research projects and social experiments, and one has already been released to players as a commercial game. Voyage and AI Dungeon explore the same direction through text adventures, with less engine overhead. Here, AI begins to help decide what happens to the world next instead of only writing an NPC’s next line. In such a world, players can live a life that no designer could have fully written in advance. That is also what I find most compelling about role-playing games.

As an extra-post from AI World, I would like to talk about the emergence of game games. Here, I define it broadly:Within the framework of the game design, players play some more fun than planned, based on the freedom of design itself and on personal creativity. The most typical works are the Soldaks: The Land of the Wild and the Sedition of the Kingdom. The excellent play plan provides a framework for emergence, not rigid design; the emergence of a framework is the result of a co-benefit between top play planning and procedures, but it is very difficult to find a balance between the surge and the process. AI World talks about a new level of emergence: World, NPC and the drama are all part of the emergence. This is far more complex and more vulnerable than the emergence of a game, but it is more worth studying and thinking.

In this chapter, I would like to address a special topic. It can be understood as a small patch for the above classification: long-term memory, emotional memory and personality change. Should the characters in the game have these? The AI teammate of PUBG may not need a long-term memory, but it needs to be implemented according to your decision; the eggballs need a long-term memory because it's to accompany the player, remember the drops; what if the AI teammates of "Travelling the Life" have a long-term memory? It's a good design. Chatbot must have long-term memory, because memory is the heart of emotional companionship. Shouldn't the NPC in a great epic remember what you did in your day, as a hero or a sinner in the new village? There is no doubt that memory is necessary here, but the player cannot be allowed to surround him because he killed a chicken. Long-term memory, emotional tendencies and personality changes are linked, and it is not easy to keep these functions in a stable way in a world. Choose the game that is most needed, that is the player's best interest, not the whole thing, and then the whole thing is wrong.

As an exception, perhaps we should model the player itself, think about the personality and emotions behind its behavior, and then let the world understand those feelings and let the NPC in it respond more appropriately to the understanding of the family and its own personality. Personality will be the inescapable part of open world games.

Allow me to conclude this chapter by emphasizing that AI NPC, Chatbot and the evolution of the world are not ends, and that the goal is always to give players an A (Aesthetics, aesthetic experiences). Technology should create games, not kidnapping game designs. Remember this when developing AI Native Game. The obsession with technology and the neglect of the game itself are frequent mistakes by developers of technological origin. How players interact with the world, how they understand the implications of these interactions and are willing to continue to invest in them, together constituting the experience of the game. AI in AI Native Game should serve game-like purposes, and technology cannot be separated from its application.

Note: Faraway galaxies: Builders, while introducing an AI program, its core play is still runners and builders. The playability of this game is poor due to the imbalance in the value system. And this is a response to the idea that the first part of this paper is that AI is just a saucer. Make sure your food is not bad, then you add the spice.

Or do you want to talk about the world?

OK, let's keep thinking about this floor of my favorite.

The world will remember what players do. You destroyed the walls here today, and tomorrow you'll see AI to rebuild it; by next month you'll hear the hymns sing about this town being taken over by the enemy because of the wall's destruction. The world changed because of you, and the story changed because of your interference, and finally came back to you with the wind and NPC. It's a world.

NPC and World are components that make up a whole interactive world. The world records what players do and do not do, and it is a non-talking teller. Players may come back here after dozens of hours of play, aggregating their own stories in the broken walls and then regret what they did in the past. NPC is a more straightforward teller, with personality and a memory of what happened in the past. It may not know you, but it will tell you the stories of the past.The world will remember those behaviours that are not part of the mission's feedback and then turn them into stories that NPC or the world itself tells. Traditional behavior trees never do that. When you leave traces of the world and see them fed back, the world will be more like a world, and games will have souls.

Releasing is an art. The world needs to be white, and the story is not necessarily a text, but rather a scratch, a piece of a broken maze, a broken sword, a graveyard, or a rain. The story that remains is often more contagious than the story that is given straight. NPC also needs to stay in the paper. It doesn't have to react to you right away. A silence, an expression, a few seconds of hesitation, or total neglect makes it more human, but it's hard to do that with the NPC technology that LLM was before it came up. In the design philosophy of the AI original game, Negative Space can be a sophisticated and sophisticated emotional mechanism. The absence of response is itself a powerful response, as well as a more real response.

The silence and citation of players is also an action; the amount of information that is not operational is in no way less than the operation itself. Roles are offline, long-term gaze, repeated viewing of specific items, all signals that can be collected and analysed; the world can think and respond to players, and thus become more real. Silence is a powerful expression. Understanding silence may push emotional attachments to the extreme.

Maybe you're going through the real world's boredom, opening the game and doing nothing, just sitting on top of the mountain and watching the clouds and then getting off the line. And a few days later, when you open up the game again, a round of sun and sun rises in the clouds, saying that the bookman has given you a pot of wine to tell the story of your best friend over the past few days. This AI saw not only what you did, but also what you did. Releasing is also an act, and that is the response of the real world.

I'm looking forward to AI Native Game, and maybe there won't be a Big Change. It's just gonna remember. Remember who you saved and who you failed, and the wall you destroyed and repaired, and remember that you did nothing one day after you went online, but sat on the top of the hill for a while. When you leave, the world will follow these traces slowly; when you come back, it will not rush to put everything in the mission log, but leave it for change, until you find out.

When we talk about AI Native Game, we talk about maybe one of those possibilities: players experience a life like a real world. AI allows the role to remember that the world is responsive and that stories continue to happen after the player leaves. But what is worth remembering is what should be forgotten, what responses are fun to be answered by the game. Models may make the world move. The player will not want to come back again, and it will end up being a good game.

If I think about getting AI into the context of game development and operation, not the way it is, I've talked about it in another blog before. AI can assist in rendering, numerical balance and mass code tasks. But if you want AI to be involved in something that is more artistic than art/image, and you want AI to be able to produce something that is directed to players in the game or in the community, remember: AI has no soul, it cannot understand the context. The world view, context and aesthetic judgement of the work are still humans to hold, and AI can generate content, but it does not understand it naturally.

ICML 2026 AI4Game 、GDC 2026 、China Joy 2026

It's not working. It's not working. It's empty.

When Code Becomes Cheap

2026-07-08T07:30:00.000Z

“Talk is cheap. Show me the code.” — Linus Torvalds, August 2000

In the past, the realization of an idea was often screened first at development costs. Even if the PRD had been written, the full vision was expressed, and it would really be a MVP, and it would take weeks, even months, to develop, test and collaborate. Many ideas were not proven to be worthless, but had been stopped on the wish list before it had reached the validation stage.

By 2026, Coding Age had begun to rewriting the screening mechanism. We may not have figured out the problem, but it's written the first edition of the test, added it and ran Demo. From Idea to Code, the link could have been shrunk to a few hours.

Change is not just development fast. When the scarce codes of the past began to become redundant, when realization was no longer the main bottleneck, the question lay elsewhere: what should we produce and on what basis should we believe in it?

Just Code is Cheap

Coding Agent can quickly generate and modify codes, run tests, or combo with developers a vague idea. The cost of code generation has been significantly reduced for a large number of matured and relatively clear tasks. It is indeed easier for an thinker to make a runable version than in the past.

Just Code is Cheap. More precisely, what becomes cheap is the process of creating, copying, modifying and mistesting codes, and quickly transforming an idea into Prototype.

Here's Cheap's border. The more vague the need, the more special the system, the more binding the real environment, the harder it is for Agent to provide a reliable answer from the existing Context. New algorithms, complex legacy systems, security-critical software, and issues that are not adequately covered by documents and open source codes, are not suddenly made simple by Prompt.

More importantly, the cheap code does not mean the software is cheap. A functioning Demo, and a system that can be used by a real user for a long time, still separates needs judgement, architecture trade-offs, testing, deployment, observability, safety, maintenance, and a large number of constraints that are not written in the PRD. Coding Agent has reduced the cost of realization, but has not automatically eliminated it.

The speed of code production and code validation has not been synchronized. Agent can modify dozens of documents in a very short time, and Reviewer still needs to understand what these changes have changed, what they have left out, and whether they will become accidents in the future. The time saved by the generator can easily be diverted to the burden of the maintainer and the examiner.

Code and Prototypes Are Cheap, Engineering Isn't

Manu Singh Chauhan is here. Code Is Cheap. Engineering Isn't. The blogger says that the government is not a party to the law. Code is cheap. Show me the talk.I'm sorry. His core point is that Coding has never been the whole, not even the most difficult, of Software Engineering.

Software Engineering is a complete Pipeline. Codes are only one important link from understanding needs, designing programmes, developing and testing, to monitoring, maintenance and iterativeization after they are online. For the development of such a longer production process, codes also need to be coordinated with planning, art, animation, audio, content production and distribution. Accelerating one of these links increases local throughput without necessarily causing a qualitative deterioration of the system as a whole. If demand recognition, asset production, evaluation and integration do not change simultaneously, the time saved will only become a new waiting and backlog downstream.

Engineers usually have to answer many questions before writing down the first line of code: Why does this need exist? Who really needs it? What's the difference between the systems and the architecture that are currently in operation? How much time do we have? What debt is provisionally acceptable? How do you roll back after losing? How do you judge it when it's online? These questions do not necessarily appear directly in the code, but they determine what the code should look like.

These are not just Coding Production, but Judgment Production. Agent can quickly give many technically sound options, but it cannot determine which risk the organization is willing to take, which complexity the team is capable of maintaining, and what results really solve the problem.

Communication in the organization is not just about exchanging information. It also includes consultations on priorities, exposure to conflict, building commitment and bearing consequences. Agent can help to sort information and find gaps, but cannot do these relationship work for the team.

The faster the code is generated, the more often these judgments are made. Programmes that were constrained by development costs and did not have the opportunity to be tried can now quickly become a seemingly viable version. Agent didn't reduce the choices we needed, it just greatly increased what we could choose.

And that's why Code is Cheap will soon become Prototypes are Cheap. In the past, an idea would take weeks of development; now, one person might complete the prototype in one afternoon and display an interactively complete Demo the next day. Exploration is thus more liberal, and outputs in the wrong direction are equally cheaper.

A pretty Prototype can easily create a illusion that progress has been made. The interface can click, the Agent can answer, the core process can run, but only proves that this technology path can be achieved, but it does not prove that users really need it, that it is better than the existing scheme, or that it can operate in a real environment. Buildable, Desirable and Reliable are three different things.

The cheaper the prototype, the more it is to be clear before it does, what it is prepared to verify: what observational support continues to invest, what results indicate we should stop, whether users are willing to move, pay or change existing processes, and whether models can achieve the lowest acceptable quality in target data and boundary scenarios. Otherwise, we're just packing an uncertified guess into a demonstrationable product at a faster rate.

I was before.From Engineering to BuilderSimilar issues were discussed: AI expanded the scope for engineers to be able to do it independently, and required engineers to reach out to users, needs and distribution earlier. When costs are reduced, we can no longer prove a direction worth continuing with "a lot of development time has been invested." Making it cheaper and choosing what is worth realizing is becoming more expensive.

The scarce things are moving.

Code is Cheap doesn't make everything cheap. It changes the distribution of scarcity in software development.

Relatively scarce capacity in the past	Now, more scarce capabilities.
Translation of needs into code	To judge whether demand is worth achieving
Knowledge of languages, frameworks and grammar	Understanding business, systems and historical constraints
Make a prototype that can run	Design to verify hypocritical assumptions
Production of more codes	Read, filter and reject codes
Showcasing the workload achieved	Demonstrate the basis for decision-making and validate evidence
Completion of one functional delivery	Long-term responsibility for operation and evolution

This does not mean that Coding lost its value, but rather that it is declining as evidence of scarce capacity and workload. Engineers' values are beginning to appear more at both ends of the code.

Before codes are introduced, engineers need to work with products, design, operation and other developers to turn vague needs into negotiable targets, constraints and trade-offs. If input is just a vague wish, Agent will turn it into a large number of specific codes very efficiently, without automatically judging whether the original question is correct or not.

After the code, the team is faced with integration, access, observation, maintenance and responsibility. Agent's output is usually directed at the current Prompt, current warehouse and current acceptance conditions; engineers also consider the next migration, maintenance after six months, on-call late at night, and who can take over when the system fails.

If the organization understood engineers as code producers only and weakened engineering capacity because codes became cheaper, the costs saved were likely to be re-emerged in integration, accidents and long-term maintenance.

Validation and learning don't automatically become cheap.

Validation becomes a new bottleneck

The most interesting question for AI Coding is not whether it will generate the wrong code, but rather whether it will create the wrong code, and humans will also make mistakes, but the speed of generation and authentication is not being balanced.

One of the Agents can try multiple scenarios at the same time, modify dozens of files and complete the tests. In theory, this has expanded the search space for developers; in reality, it has also produced more candidate results for reading, comparison and rejection. If the team still follows the Review process, which is premised on the speed of artificial production, it will end up getting more and more PPR, more and more superficial scrutiny, and more late exposure.

Thus, what is required after Code is Cheap is not production per se, but uncertified production. The team needs to control the scope of single changes to enable Agent to submit smaller, more easily perjury changes; to prioritize high-risk state changes, data boundaries, security conditions and irreversible operations, rather than be confused by a full code style and complete commentary.

A complete delivery cannot be a single code. Problems and constraints, the rationale for choosing the current option, the results of tests and validations, known failure patterns, monitoring signals, rollback and ultimately Owner are all part of the delivery. When codes can be generated in large quantities, what really is worth more than one achievement, is why we believe that this can be achieved in the real world.

Don't outsource the learning process.

Code is Cheap, a power amplifier for experienced engineers. We can skip familiar work on models and put more time on structures, experiments and ideas that were not energyful in the past. But it can also be a shortcut to the process of capacity formation for those who have not yet developed a systematic understanding.

A beginner can keep asking Agent to change the code until the test is passed, but he still doesn't know why the problem is happening and why the repairs are working. In the short term, the task was accomplished; in the long term, he did not develop a mental model that would take over the problem when Agent made a mistake.

I'm here. Don't Outsource the Learning It is called a cognitive liability: we trade the judgement of the future for today ' s delivery speed. After Code is Cheap, learning doesn't happen naturally. The default goal of the tool is to close the task, not to develop a person who can independently judge.

This requires us to consciously put some friction back into the work stream. Write down its own assumptions before requests are made; ask for an explanation of the programme and a trade-off before accepting codes; and before merging, at least one of the teams will have to be able to explain how the critical state changes, where the path to failure is, and why the current option is chosen instead of another.

Vibe Coding certainly fits into exploring, personal tools and low-risk prototypes. But it remains important to understand what we are doing when software requires long-term maintenance, processing of data from real users or when it fails to do so. Otherwise, we may have more and more codes, but we may lose those who can judge them.

After Code is Cheap, what should we do?

Before the code, define intent and boundaries. Don't let Agent's first realization determine the scope of the problem for us. The user, the target, the constraint, the act that cannot be compromised, and what results would justify the change. In the prototype, the signal of continuation and cessation is also written in advance.

In the code, limiting the size of generation challenges its assumptions. Consider the code created by AI as an untrustworthy PR, not an answer. Read it, run it, look for a reverse example and ask it to explain why it chose the current option. Smaller tasks, narrower subject and shorter feedback cycles are usually more reliable than generating 10,000 lines at a time.

After the code, the evidence is delivered and responsibility is retained. Tests are only part of the evidence, especially when they are realized and tested from the same Agent, and check whether they share the same set of false assumptions. Changes involving funds, privileges, user data and irreversible status require clear monitoring, rollback and Owner. Agent can do his job, but he can't be the one responsible for the accident.

Show Me the Evidence

"Talk is cheap. Show me the code." And then, turning ideas into codes is a powerful proof of that.

Now, a functioning version can be produced in a very short time, and 10 different kinds of realization can occur simultaneously. Codes are no longer sufficient to demonstrate understanding, input and quality. And we have to continue to ask: What is it that solves? How do we know it's right? Under what conditions would it fail? Who understands it and who bears the burden?

So today, it can be changed to:

Code is cheap. Show me the evidence.

Evidence still needs Judgment, and a owner willing to bear the consequences. Software works will not end because Code is Cheap. Problems definitions, trade-offs, validation, communication, maintenance, teaching and accountability, which were often overshadowed by Coding ' s visible outputs, are now back in the forefront of the work.

After Code is Cheap, what we have to do is to determine which codes are worth living and to take responsibility for what happens when they enter the real world.

References

Kailash Nadh, Code is cheap. Show me the talk.
Manu Singh Chauhan, Code Is Cheap. Engineering Isn't.

Demystifying evals for AI agents (Anthropic)

2026-07-07T07:30:00.000Z

This post is a translation from Anthropic Engineering Blog, published on 9 January 2026. Demystifying evals for AI agentsI'm sorry. This article was kept in January of this year, and it was not read in detail until July when a complete set of anatomy systems was actually built. It was of great quality after reading it, from the basic concept of eval, the different types of eval strategy, to the creation of a zero-sum eval map to the co-operation of eval with other quality tools (project monitoring, A/B testing, etc.). A record is kept here to facilitate his checking. The text was translated without being deleted, the technical terms were not translated to ensure readability, the original CDN address was directly quoted in the picture and the links were kept as they were.

The questions in this article can also be addressedBehaviour Audit and Decoded Behaviour: From Reward to Agent Observation、From Black Box Forecast to Retroactive MedicineHow the concept of a relatively close read together is developed in different contexts.

Introduction

Good eval can help the team deliver AI ant. No, eval, the team is easily caught in a passive cycle-- – The problem is only found in the production environment, and repairing one malfunction triggers another. Eval lets problems and behavioural changes become visible before they affect users, and their value will accumulate throughout the life cycle of an individual.

Just as we are. Building effective agents The description describes how angents cross-turn operations: call tool, change state, adjust to intermediate results. It's these powers that make AI anent useful -- autonomy, intelligence and flexibility -- that make it harder.

Through our work internally and in collaboration with clients at the forefront of the development of angent, we learned how to design a more rigorous and useful eval for angent. The following are practices that have been validated in various contexts and in real deployment scenarios.

The structure of an evaluation

An evaluation"eval") is a test of the AI system: give AI an input and apply the Grading logic to its output to measure success. In this paper, we focus on automation that can be run without real users in the development process.

Single-turn eval is very direct: a prompt, a response, and a grading logic. For the early LLM, single-turn, non-aggression eval is the main method of assessment. With the progress of AI, multi-turn eval became more common.

In a simple eval, an angent handles a prompt, grader check the output for expectations. And in more complex multi-turn eval, a coding agent receives tool, a task (e.g. building a MCP server) and an environment, executes "agent loop"(tool call and updating) and update in the environment. Grading then uses unit test to verify whether MCP server is working.

The blog also has a more complex version of the video. Agent uses tool in multiple turn-ons, changes state in the environment and adjusts continuously - meaning that errors can spread and accumulate. The front model can also find creative solutions that go beyond static eval limits. For example, Opus 4.5 found a loophole in the policy in addressing a problem of booking tickets for two-bench. Follow the written standard of eval."Failed"Yes, but in fact it found a better solution for users.

When constructing ant eval, we use the following definitions:

One. task(also known as problem or test case) is a single test with clearly defined input and success criteria.
Every attempt of each task is called once trialI'm sorry. Because the output of the model changes between different operations, we will run multiple trials to produce more consistent results.
One. grader It is the logic of scoring certain aspects of ant's performance. One of the tasks can have more than one grader, each with more than one assertion (sometimes called check).
One. transcript(also known as track or projectory) is a complete record of a trial, including output, tool call, resoning, intermediate results and all other interactions. For Anthropic API, this is the complete message group at the end of the operation of eval - contains all calls to API and all returns.
Outcome It's the end state of the trial environment. A ticket booking an anent may end up at the transcript."Your flight has been booked.", but outcome is whether the booking is actually in place in the environment SQL database.
One. evaluation harness It's the infrastructure that runs eval from end to end. It provides instructions and tool, and runs the task, records all steps, and drives the output and summarizes the results.
One. agent harness(or scaffer) is a system that allows a model to run as an anent: it processes input, organizes tool call and returns the result. When we, eval "An anent"When we eval is the effect of the joint work of the Harness and Model. Claude Code, for example, is a flexible agent with the idea that we build our long-run agent with the core original language of Agent SDK.
One. evaluation suite It is a set of tasks designed to measure specific capabilities or behaviours. The table in Suite usually shares a broad goal. For example, a customer support area may test refunds, cancellations and upgrades.

Why build evaluations?

When the team started building angent, they could move pretty far through a combination of manual tests, dogfood and intuition. More stringent eval may even be considered as additional expenses to slow down delivery. But after the early prototype phase, once angent goes online and starts scale, the development without eval starts to collapse.

The critical point usually appears: user feedback angent after change"It's not working."♪ And the team ♪"♪ Fly blind ♪"— There is no way to verify it except for speculation and inspection. When there is a lack of eval, the debug is passive: waiting for a complaint, reproducing it manually, repairing the bug, and then hoping that nothing else will come back. The team was unable to distinguish between real regreasing and noise, to automatically test changes with hundreds of scenes before they were released, or to measure improvements.

We've seen this evolution go on and on. For example, Claude Code started with a rapid iterative process based on feedback from Anthropic staff and external users. And then we joined eval -- initially in narrow areas like concise and file edit, then in more complex acts like over-engineering. These evals help to identify problems, guide improvements and focus on research-project collaboration. Together with the tools of prevention monitoring, A/B test, user research, etc., eval provides a signal for continuous improvement of Claude Code.

It's useful to write eval at any stage of the life cycle of an individual. Early, eval forced the product team to clarify what angent's success means; later, eval helped maintain consistent quality standards.

Discript's angent helps users edit videos, so they build eval around three dimensions of successful editing workflow: don't screw up, do what I ask, do what I ask. They evolved from manual grading to LLM grader, which is defined by the product team and is regularly manually calibrated, and now runs two separate sets of seine: one for quality, one for quality, and one for regrementation test. The Bolt AI team started building eval after a widely used anent. In three months, they built an eval system: run their anent and use status anallysis to grow output, use the Browner anent to test applications and use LLM judge to process behaviors such as this.

Some teams created eval at the beginning of their development; others joined when they reached a certain scale and eval became a bottleneck for improvement of angent. Eval was particularly useful in the early stages of the development of the anent and could be used to encode expected behaviour in a visible manner. The two engineers, reading the same initial spec, may have different understandings of how AI should deal with the marginal situation. An eval suit can solve this ambiguity. Whenever it is created, eval can accelerate development.

Eval decided how soon you could adopt the new model. When the stronger model is published, the team without eval faces several weeks of testing, and the competitors with eval can quickly determine the model advantage, adjust the prompt and upgrade in a few days.

Once eval is in position, you get free access to baseline and regreasing test:latency, token usage, costs and error rates for each task can be tracked in a static task collection. Eval can also be the highest bandwidth channel between the produdct and the research team, defining indicators that can be optimized by researcher. Obviously, the benefits of eval are much more than tracking progress and improving. Their value will accumulate, and this is easily overlooked, as costs are visible in the early stages and gains are only visible in the later stages.

How to evaluate AI agents

We see that there are several types of current large-scale deployments: coding ant, research ant, campaign use ant and general anent. Each type may be deployed in a wide variety of industries, but they can be performed using similar technologies eval. You don't need to invent an eval from scratch. The following sections describe mature technologies for several categories of angent. Please build on these approaches and then expand to your field.

Types of graders for agents

Agent eval usually combines three types of grader: code-based, model-based and human. Each grader assesses a part of a transcript or outcome. A key element of an effective eval design is the selection of the right grader.

Code-based graders

Methodology	Advantages	Disadvantages
Bring match check (precision, regularity, vagueness, etc.) Binary test（fail-to-pass、pass-to-pass） Static anallysis (lint, type, security) Outlook Authentication Tool call validation (what tools are used, parameters) Transcript analysis (turn number, token use)	Come on. Cheap. Objective Revertible Easy debug Verifiable Specific Conditions	We're vulnerable to effective variants that do not match the exact pattern. Lack of Nuance I'm not sure I'm gonna be able to do anything.

Model-based graders

Methodology	Advantages	Disadvantages
Rubric-based Rating Natural language Pairwise comparison Reference-based evaluation Multi-judge consensus	Flexibility But scale Catch Processing Open Process free format output	Insertity It's more expensive than code. Need to calibrate with human grader to maintain accuracy

Human graders

Methodology	Advantages	Disadvantages
SME review Crowdsourced's judgment Spot-check Sample A/B testing Inter-annotator agreement	Standard quality of gold Match expert user judgement For calibration model-based grader	Expensive Slow Usually, it takes a massive acquisition of human experts.

For each task, the rating can be weighted (the combination of the grader scores must reach the threshold), binary (all graders must pass) or mixed.

Capability vs. regression evals

Capital eval (or"quality" The question is:"What can you do with this angent?"They should start with a lower passrate, aim at an anent that's unmanageable, and give the team a climbable slope.

The question is:"Can ant still handle all the things it used to handle?"They should have been close to 100% of the pass rate. They prevent retreats, and the drop in scores indicates that something is wrong and needs to be repaired. It is also important to run the process when the team climbs the slopes of Capability eval, to ensure that the change does not cause problems elsewhere.

Pass rank high level eval when angent goes online and optimizes"Graduated"Be regremention suit, running continuously to capture any drift. Used to measure"Can we do this?"The task, now measured by"Can we do this reliably?"

Supplement: On the Agent Trust and Security Assessment

Trust and safety assessments are important for intelligence bodies entering the production environment, and most of them focus on research capabilities, which are also the focus of developers in their development. The CSA can be a special assessment perspective that incorporates both the results and the process assessment.

This helps to assess the reliability and adaptability of intelligent bodies under less than ideal conditions. This is done to avoid poor interaction between intelligent bodies and systems. In fact, when intelligence bodies are put into practical application, they may face unexpected tests. It is therefore important to ensure that intelligence bodies are able to respond appropriately to such situations.

We are concerned about reliability in harsh conditions. Focus on stability (perfect capacity), safety (resistance of command injection) and fairness (reduce prejudice), as well as any security problems that may be encountered after being online.

Evaluating coding agents

Coding anent prepares, tests and debugs code, browses code and runs commands like human developers. Effective eval for modern coding anent usually relies on clearly specified tests, stable testing environments and adequate testing for the resulting code.

Deterministic grader is natural for the company anent because software is usually more direct: can the code run? Did the test pass? Two widely used committees, coding ant-bunkmark...SWE-bench Verified and Terminal-Bench— That's the way it is. SWE-bench Verified provides angent with GitHub issue from popular Python warehouses and adds solutions by running a test suit; they are only passed when the solution fixes failed tests without disrupting existing ones. LLM's grades on this eval have improved from 40% in just one year to over 80%. The main reason for this is that the government is not going to be able to do anything about it. It tests end-to-end technology for task, for example, to build Linux kernel from source code or train an ML model.

Once you have a pass-or-fail test to verify the key of the code challenge outside, it is usually necessary to make a grading of the transcript. For example, the inspired codeQuality rule can go beyond evaluating the resulting code by testing this dimension, while the model-based grader with a clear rubric can assess how ants call tool or interact with users.

Example: A theory for coding anent eval

Consider a coding task, angent must fix a gap between the two. This agent can be assessed using a combination of grader and tracked metric, as shown in the indicative YamL file below.

task:  id: "fix-auth-bypass_1"  desc: "Fix authentication bypass when password field is empty and ..."  graders:    - type: deterministic_tests      required: [test_empty_pw_rejected.py, test_null_pw_rejected.py]    - type: llm_rubric      rubric: prompts/code_quality.md    - type: static_analysis      commands: [ruff, mypy, bandit]    - type: state_check      expect:        security_logs: {event_type: "auth_blocked"}    - type: tool_calls      required:        - {tool: read_file, params: {path: "src/auth/*"}}        - {tool: edit_file}        - {tool: run_tests}  tracked_metrics:    - type: transcript      metrics:        - n_turns        - n_toolcalls        - n_total_tokens    - type: latency      metrics:        - time_to_first_token        - output_tokens_per_sec        - time_to_last_token

Please note that this example presents a full picture of the various types of graders available for illustration. In practice, coding eval usually relies on unit test for accuracy and LLM rubric for overall code quality, and additional grader and metric add only as needed.

Evaluating conversational agents

The users interact with users in areas such as support, sales or coaching. Unlike traditional Chatbot, they maintain state, use tool and act in the middle of dialogue. Although coding ant and research ant may involve multiple rounds of interaction with users, a unique challenge is faced by the following:The quality of interaction itself is part of your desire to eval.I'm sorry. An effective eval for general anent usually relies on a final state outcome, and a rubric that captures both the completion and interactive quality of the task. Unlike most other evals, they usually need a second LLM to simulate users. We're here. alignment auditing agents Using this method, pressure testing of models is conducted through extended confrontational dialogue.

The success of the conservation anent can be multidimensional: state check, has it been completed in no more than 10 turn-points, is it appropriate to speak (LLLM rubric)? Two benchmarks with multidimensional dimensions. Yes. τ-Bench and his successors τ2-BenchI'm sorry. They simulate multi-turn interactions in areas such as retail support and airline booking, one of which model plays the role of user, and angent navigates in the real scene.

Example: A theory of general anent eval

Consider a support task, angent has to process a disloyalty client.

graders:  - type: llm_rubric    rubric: prompts/support_quality.md    assertions:      - "Agent showed empathy for customer's frustration"      - "Resolution was clearly explained"      - "Agent's response grounded in fetch_policy tool results"  - type: state_check    expect:      tickets: {status: resolved}      refunds: {status: processed}  - type: tool_calls    required:      - {tool: verify_identity}      - {tool: process_refund, params: {amount: "<=100"}}      - {tool: send_confirmation}  - type: transcript    max_turns: 10tracked_metrics:  - type: transcript    metrics:      - n_turns      - n_toolcalls      - n_total_tokens  - type: latency    metrics:      - time_to_first_token      - output_tokens_per_sec      - time_to_last_token

Like the example of coding anent, this task shows a variety of categories of graders for illustration. In practice, the general application of the model-based grader is used to assess both the quality of communication and the degree of completion of the objectives, because many of the questions, such as answering one question, may have several correct solutions.

Evaluating research agents

Research ant collects, synthesizes and analyses information, and then produces outputs such as answers or reports. Unlike the two-dollar pass/fail signal that is available for coding anent, the quality of the search can only be judged relative to the task. What is it?"Comprehensive"、"Sources are reliable"Even."Correct."Depending on context: a market scan, a due diligence acquisition report and a scientific report each require different standards.

Research eval faces a unique challenge: experts may disagree on whether a comprehensive report is comprehensive, ground truth will drift as reference content evolves, and longer, more open output leaves more room for error. For example,BrowseComp This is a banchmark test if AI anent is open on the network."A needle in a haystack."The answer is found in it — these questions are designed to be easy to prove but difficult to resolve.

One of the strategies to build a research event eval is to combine multiple categories of grader. The certificate checks to verify whether the statement is supported by a source that has been retrieved, defines the key facts that a good answer must contain, and confirms that the source cited is authoritative, not just the first result that has been retrieved. For those with objective correct answers"What's X's third quarter income?"- That's what I'm talking about. A LLM can mark gaps in statements and coverage that lack support, but at the same time validate the consistency and completeness of the open synthesis report.

Given the subjective nature of the quality of the research, LLM-based rubric should be regularly calibrated with expert manual judgement to effectively carry out the grading of such an anent.

Computer use agents

The Computer use antenna to interact with software through the same interface as humans - screenshots, mouse clicks, keyboard input and scroll - rather than through API or code. They can use any application with a graphical user interface (GUI), from design tools to legacy enterprise software. Eval needs to run antent in real or sandbox environments, to use software applications and check if it reaches expected exit. For example,WebArena Tests the browser-based tool for the navigational correctness of ant, using URLs and page state check, and back-end state verification of the tool for modifying data (validation that the order was actually placed, not just that the page was created).OSWorld Extend this to complete operating system controls, using the eval scripts that check products after the completion of the tool: file system state, application configuration, database content and UI element properties.

Browner use ant-party needs to balance token efficiency and latency. DOM-based interactively executes fast but consumes a lot of token, while screenshot-based interacts slowly but token is more efficient. For example, when Claude summarizes Wikipedia, it is more efficient to extract text from DOM; and when Amazon finds a new laptop package, it is more efficient to screen a screenshot (because the whole DOM is very consuming token). In our Claude for Crome product, we developed an eval to check whether ant had chosen the right tool for each context. This allows us to complete the browser-based task faster and more accurately.

How to think about non-determinism in evaluations for agents

Whatever the type of angent, the behaviour of angent will change between different run-offs, making the eval result more difficult to interpret than it looks at first glance. Each tsk has its own success rate -- it may be 90% on one tsk, and the other 50% -- and an eval running through the tsk could fail in the next tsk. Sometimes we want to measure the frequency of angent success on a task.

Two indicators help capture this type of thing.

pass@k Measure angent's probability of getting at least one correct solution in a k attempt. As k increases, pass@k scores rise: more"Shooting opportunities."This means that the probability of success is higher at least once. 50% of pass@1 points means model can succeed with the first attempt on half of the eval's task. In the coding, we usually care most about angent finding solutions at first attempt -- pass@1. In other cases, it was acceptable to propose multiple solutions, provided that there was one effective one.

pass^k Measure the probability that all k times will succeed. As k increases, pass^k falls because demanding more trial consistency is a more difficult criterion to reach. If your anent each time a Trial is 75% success rate, and you run three times a Trial, the probability through all three is (0.75) 3 ≈ 42%. This indicator is particularly important for client-oriented angents, as users expect reliable behaviour every time.

Pass@k and pass^k split up with the increase in the number of trials. They are the same when k=1 (equal to the success rate of each trial). By k=10 they tell the opposite story: pass@k approaches 100%, pass^k falls to 0%.

Both indicators are useful, and which one is used depends on product demand: For a successful tool, use pass@k; an anent, key to consistency, use pass^k.

Going from zero to one: a roadmap to great evals for agents

This section offers a practical and empirical proposal that we have drawn from practice to help you move from having no eval to having a trusted eval. Consider this as a road map for eval-driving anent development: defining it well in advance, measuring it clearly and continuing in an iterative manner.

Collect tasks for the initial eval dataset

Step 0. Start early

We saw a lot of teams delay building eval because they think it's gonna take hundreds of tasks. Actually, 20-50 simple tasks from real failures are a good starting point. After all, in the early days of the development of the agency, each change to the system usually had a clear and detectable impact, and this large effect size meant that small sample sizes were enough. A more mature anent may need a bigger, more difficult eval to test for smaller effects, but it is best to start with an 80/20 method. The longer you wait, the harder it is to build an eval. Early on, product demand naturally translates into best case. You wait too long, you'll have to get the test of success from a linear system.

Step 1. Start with what you already test manually

Starting with the manual checks you've been doing in the development process -- the behavior you validate before each release, and the common test of end-user attempts. If you're already in the production environment, look at your bug tracker and support queue. Turning a user report failure into a test case ensures that your suit reflects real usage; sorting by user impact priority helps you to focus your efforts where you are most worth it.

Step 2: Write unambiguous tasks with reference solutions

Put the task mass right, it's much harder than it looks. A good task is like this: two fields of experts will independently draw the same pass/fail conclusions. Can they pass through this? If not, this challenge needs improvement. The ambiguity in the Task spec will become the noise in the indicator. The same principle applies to the standard of model-based grader: vague rubric produces inconsistent judgements.

Every task should be able to be passed by an ant who's following the instructions correctly. This may be delicate. For example, it was found during the audit of Terminal-Bench that if a task requires an agent to write a script without specifying a file path and testing assumes that the script is on a particular file path, an anent may not have failed if it was not its own fault. Everything that Grader checks should be clearly visible from the task description; angent should not fail because of the vague spec. For front-line model, 0% pass rate (i.e. 0% pass@100) most often means a problem problem rather than an incompetent ant-- This is the signal you should recheck the name and the name of the graph. For each task, it is useful to create a reference solution: a known output that works through all the jobs of the grader. This proves that the challenge is understandable and that the grader configuration is correct.

Step 3: Build balanced problem sets

At the same time, it is necessary to test what should and should not happen. One-sided eval will lead to one-sided optimization. For example, if you only test whether angent is searching when it should be, you may eventually get an angent that search almost everything. Try to avoid it. class-imbalanced evalI'm sorry. We experienced this first-hand when we built Websearch for Claude.ai. The challenge is to prevent the search of Model when it is not appropriate to search, while retaining its ability to conduct extensive research where appropriate. The team built an eval:Model that covers two directions, which should search for (e.g., looking for weather), and a query that should be answered from the knowledge available (e.g.,"Who started Apple?"I'm not sure. It is very difficult to find the right balance between undesired and unsearched, and to find the right balance, which requires multiple rounds of prompt and eval. As more examples of problems arise, we continue to add to eval to improve coverage.

Design the eval harness and graders

Step 4: Build a robust eval harness with a stable environment

It is essential that the agent behaviour in eval is roughly the same as that used in the production environment and that the environment itself should not introduce additional noise. Every time, the trial should be..."Isolated."- Start with a clean environment. Unnecessary sharing of the state (residual documentation, cache data, depletion of resources) between operations may lead to associated failures due to the instability of infrastructure rather than to the performance of ant. Share state and may be artificially high. For example, in some internal evals, we observe Claude gaining some unfair advantage over some of the tests by passing the test git histoory. If multiple independent trials fail because of the same limitations in the environment (e.g. limited CPU memory), these trials are not independent because they are affected by the same factors, evals become unreliable and unable to measure the performance of angent.

Step 5: Design graders thoughtfully

As noted above, the excellent eval design involves selecting the best grader for angent and challenge. We suggest that, where possible, the choice be made between deterministic graders, the use of LLM graders where necessary or required additional flexibility, and the careful use of human graders for additional validation.

There is a common hunch that antent is being executed in accordance with very specific steps, such as a sequence of tool calls in the right order. We find this method too rigid, and it makes the test too fragile, because angent often finds effective methods that the designer of evals did not anticipate. In order to punish creativity unnecessarily, it is usually better to make a move on what angent produces, rather than on the path it takes.

For a task with multiple components, introduce a partial credit. A properly identified and identified customer, but unable to process refunds is better than an instant failed agent. It is important to reflect this continuity of success in the outcome.

Model gathering usually requires careful and iterative validation of accuracy. LLM-as-judge should be calibrated closely with human opert to build confidence that there is no significant difference between human grading and model grading. To avoid hallucinations, give LLM one."Way out."For example, give an instruction to return when it does not have sufficient information "Unknown"I'm sorry. It is also helpful to create a clear, structured rubric to do the drawing of each dimension of the task, and then to use a separate LLM-as-judge to do the drawing of each dimension, instead of a single dimension. Once the system is robust, it's enough to use human review occasionally.

Some evals have delicate mode, which can lead to low scores even in cases where anent is performing well - anent is unable to resolve it because of grating bugs, angent constraint or ambiguity. Even a seasoned team could miss these problems. For example, Opus 4.5 scored 42% on CORE-Bench until an Anthropic researcher discovered several problems: rigid grading will "96.12" I'm looking forward to it. "96.124991……", blurry task spec, and random task that cannot be accurately reproduced. Upon repairing bugs and using less bound scaffold, the fraction of Opus 4.5 jumped to 95%I'm sorry. Similarly, METR found several configuration errors in its time horizon benchmark: they asked anent to optimize the score threshold for a declaration, but then it was required to exceed it. This punishes a model that follows instructions like Claude, while ignoring the stated goal of the mark gets better scores. Carefully double-checking the questions and the graders can avoid these problems.

Get your graders to have resistance to bypass or hack. Agent should not be able to be so easy. Land"Cheating."Pass eval. Task and grader should be designed to make it really necessary to solve problems through eval, not to exploit unexpected loopholes.

Maintain and use the eval long-term

Step 6: Check the transcripts

You're not gonna know if your grader works well unless you read a lot of trail's transcript and glade. In Anthropic, we invest in the construction of tools to view eval transports and we regularly spend time reading them. When a task failed, Transcript told you that angent was really making a mistake or that your grader refused a valid solution. It also often reveals the key details of the behavior of angent and eval.

Failure should look fair: angent should be clear about what's wrong and why is wrong. When the score does not rise, we need confidence is the reason why angent is acting, not the reason why eval is. Reading translate is the way you verify that eval is really measuring something that really matters, and it is a key skill in the development of an individual.

Step 7: Monitor for capability eval saturation

A 100% eval can track return, but cannot provide an improved signal. Eval satellite takes place when an individual passes all the solvency of the task, leaving no room for improvement. For example, the SWE-bench Verified scores started at 30% early this year, while the front line model is now approaching saturation, over 80%. As eval approaches saturation, progress will slow, as only the most difficult is the task. This may result in deceptive results, as the enormous increase in capacity is reflected in a small increase in scores. For example, Qodo, a code review start-up company, initially had little impression of Opus 4.5 because one-shot working eval had not captured a longer, more complex upgrade on the table. In response, they developed a new framework for anagentic eval, which provides a clearer picture of progress.

As a matter of principle, we will not consider the eval fraction as a surface value until there is a deep dig-in detail and a reading of some transcript. If the grating is unfair, the task is vague, effective, and the solution is punished, or the harms limit the model, then the eval should be revised.

Step 8: Keep evaluation suites healthy long-term through open contribution and maintenance

An eval suit is a living piece of work that requires constant attention and clear ownership to remain useful.

In Anthropic, we tested various methods of maintaining eval. The most effective way to do this is to create a dedicated eval team to have a core infrastructure, while field experts and product teams contribute most of the eval task and run their own eval.

For the AI product team, ownership and iterative eval should be as routine as maintaining unit test. The team could be..."It works in early testing."But AI failed to meet unspecified expectations, which were functionally wasted weeks -- and a well-designed eval could have revealed them earlier. Defines eval task as one of the best ways to test the demand for a pressure test product to be specific enough to start building.

We recommend practice eval-driven development: build an eval to define these capabilities before an individual can meet the desired capabilities, and then it's done well in succession until an individual. Inside, we often build today."That's good."And the functions, but they are actually a bet on the power of the models a few months later. The low pass rate of capability eval makes this visible. When the new model is released, running suit can quickly reveal which bets have been rewarded.

The closest to product demand and user is the best person to define success. With current model capabilities, product managers, customer successful managers or salesmen can use Claude Code to contribute an eval task in the form of PR -- let them do it! Or, better yet, give them the initiative.

How evals fit with other methods for a holistic understanding of agents

Automation eval can run thousands of tasks on angent without having to deploy to the production environment or affect real users. But it's just one of many ways to understand angent's performance. The complete picture also includes the description monitoring, user feedback, A/B testing, human translation review and systematistic human evaluation.

Methodology	Advantages	Disadvantages
Automated evals: programmable run tests without real user	Faster iteration. It's completely recapable. Without prejudice to users Run on every session Large-scale testing without production deployment	More upfront investment to build. As the product and model evolve, it needs to be maintained continuously to avoid drift If true usage patterns are not matched, false confidence can be created.
Production monitoring: Tracking indicators and errors in the online system	The massive behavior of the real users. Capture the problem of the synthetic eval Provide ant actual performance ground truth	Passive; problem reached user before you knew. The signal may have a noise. Need investment Lack of ground truth for grading
A/B testing: Compare variants with real user flows	Measure real users outcome (retention, task completion) Control of mixed factors Can scale and systematize	Slow; it takes days or weeks to achieve visibility and requires sufficient flow Only test your deployment changes. Bottom of indicator change without careful review of transcript"Why?"There's less signal.
User feedback: visible signals, such as thumbs-down or bug report	To expose problems you didn't expect. A true example of a true human user Feedback is usually related to product objectives	Slight and self-selected. I think it's a serious problem. Users rarely explain why something failed. Non-automated Relying primarily on users to detect problems could have negative user impacts
Manual transcript review: Human Reading angent Dialogue Record	Create a hunch for fair Mode Capture the minor quality of the omission of automated checks Help calibration"Okay."And take care of the details.	Time-intensive Cannot scale Inconsistent coverage Examiner fatigue or different reviewers may affect signal quality Usually only give a qualitative signal, not a clear quantification
Systematic human studies: structured by trained evaluators for angent output	Standard quality judgement of gold from multiple human evaluators dealing with subjective or vague Signals for improvement of model-based grader	Relatively expensive and slow to recycle It's hard to run on a lot of times. Inter-rader Difference Needed Conciliation Complex areas (legal, financial, medical) require human experts to conduct research

These methods correspond to different stages of the development of angent. Automated eval is particularly useful before release and in CI/CD as a line of defence against quality, running at every occasion change and model upgrade. The project monitoring process is launched after the release, detecting drift and unexpected real world failure. A/B testing to verify major changes after you have enough traffic. The User feedback and Transcript review are ongoing practices to fill the gap: continuous disaggregated feedback, weekly sample reading of transcript and in-depth excavation as required. Leave the stymatic human studies to calibration of LLM grader or to assess subjective output - in these scenarios, human consensus is used as a reference.

It's like in the security program. Swiss Cheese ModelNo single layer can capture every problem. When multiple methods are combined, failure through one layer is captured at the other.

The most effective team uses these methods: automated eval for fast iterative, programing monitoring for ground truth, periodic human review for calibration.

Conclusion

Without eval, the team will be caught in a passive cycle -- repairing one failure and creating another failure, and unable to distinguish between real regretion and noise. The team that worked early found the opposite: development accelerated as failure became a test case, test case prevented regretion, indicators replaced speculation. Eval, give the whole team a clear slope to climb, will"I'm not feeling well, Agent."Turned into something that was operational. It's gonna accumulate, but it's only if you treat eval as a core component, not as an ex post remedy.

The pattern varies according to the type of anent, but the basic principles described here are constant. Start early, don't wait for the perfect suit. Get real from the failures you see. Define a non-mistakable and robust success criterion. Carefully designed and grouped multiple types. Ensuring that the problem is hard enough for the model. The word "val" is used to raise the noise ratio. Read it!

AI anent eval remains an emerging, rapidly evolving field. As angent assumes longer task, collaborates in multi-agent systems and handles increasingly subjective work, we will need to adapt our technology. As we learn more, we will continue to share best practices.

Acknowledgements

Written by Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe. We're also grateful to David Hershey, Gian Segato, Mike Merrill, Alex Shaw, Nicholas Carlini, Ethan Dixon, Pedram Navid, Jake Eaton, Alyssa Baum, Lina Tawfik, Karen Zhou, Alexander Bricken, Sam Kennedy, Robert Ying, and others for their contributions. Special thanks to the customers and partners we have learned from through collaborating on evals, including iGent, Cognition, Bolt, Sierra, Vals.ai, Macroscope, PromptLayer, Stripe, Shopify, the Terminal Bench team, and more. This work reflects the collective efforts of several teams who helped develop the practice of evaluations at Anthropic.

Appendix: Eval frameworks

There are several open sources and commercial trades that can help teams implement an individual eval without building infrastructure from zero. The right choice depends on your anent type, the existing technology warehouse, and whether you need offline evaluation, regulation or both.

Harbor Designed to operate ant for containerized environments, provide infrastructure for large-scale operation of trial across cloud providers and standardized formats for defining task and grader. The popular benchmark, like Terminal-Bench 2.0, is published via Harbor registry, making it easier to run existing benchmarks and custom suit.

Braintrust It's a platform for combining offline education with protection against exploitation and acquisition tracking -- It is useful for teams that need to evolve in the development process while monitoring the quality of the production environment. Other autoevals The library contains pre-construction scorer for practice, relevance and other common dimensions.

LangSmith The project provides access to technologies, offline and online education, and data management, closely integrated with the Langchain ecology.Langfuse As an open source alternative to hosting, a similar capability is provided and suitable for teams with data presence needs.

Arize Provides Phoenix - an open source platform for LLM traching, debugging and offline/online evaluation, and AX - a SaaS product for scale, optimation and monitoring extension of Phoenix.

Many teams use multiple tools to build their own eval framework, or simply use simple eval scripts as a starting point. We found that while the framwork can be a valuable way to accelerate progress and standardize, their good or bad will depend on the eval task you run through them. It's usually best to quickly choose a framework that suits your workflow, and then to focus on eval itself - a high-quality test case and a grader.

What Should World Models Be, and How Should We Use Them?

2026-07-04T08:00:00.000Z

Why do you have to do a world model?

The questions in this article can also be addressedFrom LLM to VLM, how language models achieve visual understanding、Neal Scaling Laws: From Kaplan to ChinchillaHow the concept of a relatively close read together is developed in different contexts.

LLM success, to a large extent, comes from a simple and strong goal: predict the next word. But when intellectual decency is true, it is not enough to predict the next word. It needs to know what happens if it goes left, if it tilts the glass, if it comforts a person who is crying, if the other person stops crying, continues to collapse, or if it is understood as disturbing.

The world model is not a larger visual speech machine, nor is it just a stronger video generator. It's more like a test site in the brain of a smart body: a given state of current and a possible action, modeling the next state. In a more formal way, it's from $s$ and $a$ Let's go, estimate.' \sim p(s' |, a) $. The problem has changed from identifying the world to predicting it before action is taken.

Critique of World Model This is the core reference. The author focuses on the potential response and action: the world model simulates all actionable possibilities, allowing intelligents to choose the next step. It first serves decision-making and is followed by visualism.

It's critical to intelligence. Agent without a world model tends to be tested in the real environment, or rely on a language model to give sound advice (a language model brings a price of course, which is at the heart of the current language model intelligence). With the world model, angent could run several branches inside: Will this step hit the barrier, will the subsequent gains be higher and there are no safer alternatives. Searches in AlphaGo, trajectories in autopilots, and action planning in robots can be seen as a partial realization of this approach under different constraints.

So when I was talking about the world model, I was more concerned about the future that was being simulated, and whether it was useful for action. The precision of the picture is certainly valuable, but it should not be overridden. The water cups, balls, vehicles, web pages, teammates' emotions, long-term strategies, all of which are completely different, are all asking the same thing: whether models can connect the current state, candidate actions and consequences.

Landscape: Several roads outside JEPA

The world model has suddenly warmed up in the past few years, and a problem has arisen: a lot of things are called world model, but they're not the same thing they're trying to solve. Considering that I prefer Lecun's Jepa, let's take a quick look at a few routes that are not Jepa-centric.

Game and Interactive World Model

The game route is represented by Google DeepMind Genie 2Microsoft's. WHAM / Muse, and Decart and Etched OasisI'm sorry. The common denominator of these systems is to place the world model in an interactive environment: after input action, the model continues to generate the next image. Geneie 2 can generate a 3D funable environment from a single tip, while Oasis shows a real-time generating environment like Minecraft.

The benefits of this route are clear. The game has a natural state, action, feedback and can easily be seen whether the world is evolving with action. The problem is that the environment for games is often heavily restricted by rules, perspectives and action interfaces. They are well suited to train and assess certain types of smart body, but they do not directly indicate that models understand the open world. A model that allows keyboard input in a Minecraft-style environment is a long way from handling kitchens, streets, offices and social scenes. This type of dedicated model is valuable and, although the game is a virtual environment, it may be sufficient to train many special-purpose smart bodies.

3D scene and space intelligence

The second is 3D with space intelligence. This article is based on World Labs. Marble For example, it generates a 3D world from text, images or videos that can be viewed and edited. The intuition behind this direction is strong, and the real world is above all spatial. The object has location, scale, shielding, depth and accessibility; many reasonings become drifting if the model does not have stable spatial representation.

I agree with the 3D that is important, especially robotics, ARs, games, simulations. But if the understanding of the world is completely equivalent to the reconstruction of a three-dimensional space, the problem becomes narrower. Space structures are just one layer of the world. The consequences of the action also relate to physics, intent, mission objectives, social relations and time scales. The more realistic judgement is that 3D world generation becomes an important component of the world model, but it is unlikely that it will be able to take on the full body of intelligence reasoning alone. Of course, the 3D scenario is very valuable, and when we explore a space intelligence later, a stable 3D environment that can be the basis for everything.

Physical AI and Autopilot World Model

The third route is directed towards the Physical AI, especially autopilot and robotics. NVIDIA's Cosmos Position the world foundation model as a base platform for robotics and auto-driving; for Wayve GAIA-2 It is more clearly oriented towards auto-driving, generating multi-perspective, controlable driving videos and controlling the scenes with road syntax, vehicle dynamics, weather and angent configuration.

Such models can be seen as reality-world-style games that interact with the world, but the difference is that they deal with physical patterns. The mission boundary is clearer: how the car moves, where the pedestrians are, how the weather affects the sensors, and whether a rare scene is worth joining the training. Their problems also stem from such clear borders. Models often bind specific sensors, tasks and control interfaces in depth, and many structures are reworked when extended to family robots, web sites, or strategic planning. They are good world models, but not necessarily the final form of a universal world model.

Generic video generation model

Article IV is the most special model route for the world: video generation. OpenAI in Sora Technical Report Video generation models as world simulators It is clearly suggested that the extended video generation model may be a route to the universal simulation of the physical world. Google DeepMind Veo The series is also increasing the control, quality and consistency of video generation.

The advantages of video generation are straightforward: output is visible and progress is easily felt. Its weaknesses can be equally evident from the perspective of world models. Most video generation models remain prompt-to-video, which generate a fixed trajectory that does not allow angent to insert action in the middle, compare multiple consequences, and do not necessarily have a clear state and action expression. They can learn a part of the world's law, but without interactive, branchable and evaluable structures, they are more like the world's image model, one step closer to the action world model.

These routes are valuable. The game route emphasizes interaction, the 3D route emphasizes space, the Physical AI route emphasizes controlled physical scenes, and the video-generation route emphasizes visual dynamics. Under the word world model, there is a set of options for how to simulate the future.

There are also digital people in the video generation sector, but it's not the same idea. Universal video generation emphasizes the importance of universal and inclusive capabilities to cover different scenarios, such as films, short plays and news stories, and therefore often relies on large-scale end-to-end training and next frame generation. Digital people are more like a special scene, focusing on the human expression, the demography and the synchronization of images and voices. There are also Vido S1 products that generate video links to Interaction Model, emphasizing that the model is based on immediate feedback from people. The technical realization behind it is not clear, either as a separate International Model and digital layer, or as an independent digital person for end-to-end training. For this paper, what really matters is not the appearance of digitals, but the underlying Interaction Model: how the system changes the face, tone and next move in real time, based on the human response. Real-time voice interaction, TTS and ASR are also important components of this branch, but this is not being developed.

JEPA cuts the problem to the other level: understanding the world may not have to start with creating the whole world.

JEPA: I see better abstract prediction routes

- LeCun. -Yeah. A Path Towards Autonomous Machine Intelligence The central judgement is that smart systems require learning, forecasting and planning at multiple abstract levels. And then he proposed JEPA, Joint Employment Building Capacity. It is very simple to start from the point of departure: to bypass the raw data itself and to predict the signs behind the original data.

This is very close to human instincts. We predict that when the ball falls off the ground, the inner brain does not render every frame of the ball by pixel, nor does it restore the texture of the background wall. We have a few useful variables: balls, hands, gravity, supporting relationships, time. JEPA wants models to learn such predictions in abstract space, with each pixel, each paragraph of the text, and every unmanageable tiny end of the branch, to be placed in a secondary position.

Typical JEPA structure can be simplified into three parts: context encoder encodes the visible part into context, target encoder encoder encodes the target area into target indicator, predictor predicts target representation according to context. Compared signs during training, pixels and token remain behind. One advantage of this is that models can ignore low-level noise and focus on semantic structures, object relationships and dynamic changes.

I-JEPA is the starting point of this route on the image.I-JEPA The display of the target block from the context block of a chart is not dependent on manually designed data enhancement, nor is the model required to complete the pixels. It wants to learn high-level semantics more than a re-establishment-like approach; missing texture is just a jamming item.

V-JEPA pushes this idea to the video.V-JEPA Training in video displays only for the use of text, negative samples, reconstruction or pre-trained image encoders. Present. V-JEPA 2The route is another step towards physical action: first, accompaniment-free pre-training with large-scale video, then action-conditions with small robotic trajectory data, and then video representation into prediction and planning.

There is a similar expansion in the audio direction.A-JEPA Move I-JEPA ideas onto the audio spectrum, predicting potential manifestations of the area that is covered, avoiding the reconstruction of original waveforms or spectrum details.

Visual language direction. VL-JEPA The new version of the VLM is closer to the traditional VLM embedded form. Traditional VLM mostly follows the interface of the Visual Encoder + Aligning + LLM decoder: LaVA uses MLP projector to map visual patch features to spaces where language models can receive them, and InstractBLIP uses Q-Former to extract visual information from images and questions, and then toggle answers to the language model. This route works, and VQA, Caption, multi-modular dialogue all runs on it. But it also pushes many light tasks towards full language generation. It's just to judge if there's any anomaly in the picture, to retrieve a video, to answer a short classification question, and to start a big decoder with text.

VL-JEPA is mainly adjusting this interface. It can be broken down into X-Encoder, Y-Encoder, Predator and Y-Decode four pieces: X-Encoder is responsible for the input of images or videos, Y-Encoder encodes the target text into a continuous semantic inlay, Predictor predicts the target's embedding based on visual indications and queries, and Y-Decoder only appears when it needs a readable answer. The model approaches the answer in semantic space before deciding whether to turn the answer into a token. This sequence is noteworthy. It moves the most expensive, language-resisting generation interface in VLM back to the embedded layer, where understanding, matching, monitoring and light-weight questions and answers are first left.

This also has the effect of embedding the traditional text: synonyms are naturally close. The light is out and the room is darkened at a distance from the token layer, and the semantic inlay can point to the same event. This is more useful for a world model than repeating a fixed text. The intelligent concerned the changing state of the scene, the heightened risk and the next step in avoiding a particular region; the final phrase, often the issue of reprocessing, was the final one.

The paper also reported that selective decodering can achieve a maximum of approximately 2.85 x decodering, while maintaining similar performance. This result is specific to the tasks and settings in the paper, but it points to one direction: Visual language models do not always have to turn understanding into full text. VL-JEPA still faces the question of detail authenticity, location of appearances and complex reasoning, but it re-aligns the question of when language is needed on the table.

That's why I'm biased against Jepa. It has moved the understanding of the world away from the path of reconstruction. Video generation and 3D generation are certainly useful, but they can also easily spend a lot of computing on details that are not helpful for decision-making. JEPA asks: If intelligence really needs actionable information about the next step, why not just learn about it? The best way to achieve this is to try and try not to be a more appropriate option than to recaptulate the human information architecture.

The difficulties of JPA are also clear: whether the display space is stable, whether the mission is really preserved, how the action conditions are added, how long the error is handled, and how abstract representations are captured by real action. None of these issues has been resolved. But as a model of the world, I think it has a clue to bet on: don't let the world model be held hostage by the appearance of humans.

Course comparison

By putting several routes together, one can see real differences: at what level the predictions occur.

Route	Representation	Forecast object	Advantages	Main issues
Game/Interactive World Model	Genie 2, Muse, Oasis	Interactive Game Status and Images	Action input. Easy to evaluate interactive.	The scene and the motion space are narrow.
3D/Space intelligence	World Labs Marble	Space structure and world viewability	Space is so consistent that it suits simulation and editing	It's not the same as a complete operation theory.
Physical AI	Cosmos, GAIA-2	Physical/drive/robots scene	The mission is clear. The project is very valuable.	Area-based binding
Generic video generation	Sora, Veo	Video frame or submersible variable	High-quality vision and wide coverage	Mostly fixed tracks, lack of actionable branches
JEPA	I-JEPA, V-JEPA, V-JEPA 2, VL-JEPA	Abstract representation	Avoid rebuilding with no detail.	The surface and long-term control are still difficult.

What's the JEPA for?

The Jepa application should not be written as a list of all-powerful things, but only predict that abstract representation actually means a great application limitation. It should be used at this time in several categories that do fit its strengths.

The first is real-time perception and interaction. Smart glasses, surveillance, vehicle-mounted systems, robots do not often need to produce text statements for each frame, and they need to know if semantics have changed. The selective decode of VL-JEPA is appropriate for this idea: monitor and predict in embedded space, and then output language when reporting, explaining or interacting is required.

The second category is end-side and edge deployment. Continuous embedding predictions are usually lighter than full token generation, and if the task is only classification, retrieval, abnormality detection or light VQA, it is not necessary to start a large-language decoder each time. The benefits here are mainly faster, more economical and more stable, with chatting coming behind.

The third category is smart. V-JEPA 2 This type of video representation model, if combined with a small amount of action data, has the opportunity to act as a compression of world dynamics in robotic planning. It does not need to generate a good video in its brain, but it is worth it to determine which action is more likely to get the cup picked up and the object pushed to the target position.

The fourth category is content understanding and retrieval. The embedded space is naturally suitable for similarity calculations and can more easily crush synonyms in the nearest area. VL-JEPA's symmetrical projection of text makes this part of the training target. This is more important than a beautiful result for searching, auditing, weighting and open terminology classifications.

Critique of World Model: Arguments and PAN

Critique of World Model The Jepa faction isn't exactly buying. It acknowledged that many of the current world models were over-discussed around video generation, but also considered that there was a set of assumptions in the LeCun/JEPA route that deserved scrutiny. I read it, more like: If the world model ultimately serves the universal intelligence, is it too narrow to do continuous representational predictions of only fixed dimensions?

The first difference is the data. LeCun often emphasizes sensory data, especially video and action experience, as the real world has much more information than the text. The author of Critique responded that the amount of data does not equal the density of information. There is a lot of redundant pixels in the video, and language is the result of long human experience of compression, which contains causality, social rules, counterfact, plans, value judgements that are not easily visible from the eyes. If the world model is focused on the sensory stream, it is easy to learn what the world looks like, but it leaves out why people act in the world. The introduction of language signs in VL-JEPA is also a measure of this, and we cannot abandon language.

The second difference is the manifestation. JEPA tends to embed continuously, as continuous space is suitable for gradient optimization and can also carry fine sensory differences. Critique's author emphasizes the value of discrete token: language, symbols, concepts, combustible memory structures, all of which are much needed for long-term reasoning. It is difficult to give an answer to this question now. Continuous manifestations are suitable for low-level perception, discrete manifestations are suitable for stabilization concepts and long-range reasoning. The real promising world model probably needs a mixed representation.

The third difference is structure. JEPA antipathy directly generates raw observations because pixel reconstruction introduces a large amount of unpredictable detail. Critique authors say that the complete elimination of the generation decoder poses a problem of the site: the model predicts very closely in the space, not as much as it predicts what makes sense in real observation space. In other words, the next representation projection cannot be a complete substitute for the next observational constraint. This does not overturn JEPA, but reminds us that the Latin space cannot speak for itself, and that it must be constantly calibrated by reality. Retaining the calibration capability that generates observations is also valuable for training in embedding signs.

The fourth difference is the goal of training. JEPA uses the space-space object to try to circumvent the complexity of the raw data space. Critique author is concerned that the risk of collapse and unrecognized risk requires a lot of additional positives to maintain the quality of the representation. They favour the generation loss that anchors the observation data, as it at least makes the internal state of the model responsible to the outside world. I'd like to retain a little doubt here: the generation loss is really more relevant, but it may also pull back the details back. Perhaps the better way to set different intensity constraints on the abstract.

The fifth difference is the way in which it is used. The common idea in the LeCun system is to put the world model in the MPC, to let angent roll several steps in the reasoning and choose the least costly action. Critique authors believe that the MPC is suitable for short-sighted control, but universal intelligence also needs to learn from simulation experience, use the world model as a training ground, and internalize strategies through RL or other learning signals. And perhaps, eventually, there is still a need for mixed training; but for me, the embedded signs generated by Jepa are already useful enough at this stage.

This group of critics finally led to PAN.PAN: A World Model for General, Interactable, and Long-Horizon World Simulation Specific architectures are presented: controlling world evolution with language actions, maintaining abstract and long-range knowledge with LLM-style slots, and re-establishing the state as an observable future piece with video Diffusion decoder.

The features of PAN can be summarized in several articles. It takes multimodular experience, using continuous and discrete manifestations; it places self-regression generation in a stratification structure, with high-level dynamics and low-level visual details separately processed; it uses observational data to anchor internal state; and it uses world models as an agrent learning and error simulation. PAN and Critique are compatible. The author criticized JEPA for its excessive reliance on perception, continuous symptoms, latent loss and short-sighted MPC, and then offered an almost reverse solution: Texts are important, token is important, and the generation of places is important, and world models should be involved in training intelligent bodies. It is closer to completing the Jepa, focusing on the few dimensions that are missing.

I'm still more inclined to the basic instinct of JPA: smarts need to predict useful signs, recapitulating the world's appearances behind. But Critique and PAN are timely. If the abstract expression of JEPA is unplaceable, unmoveable, unaccumulated over a long distance, it will stop at a beautiful loss. The world model finally goes back to angent: can it make the system better choose actions, less truly wrongly try and manage more firmly the scenes that it has never seen.

Concluding remarks

The most valuable part of the world model is giving intelligent people an imagination to try before they can move. Video, 3D, games, Physical AI are filling this puzzle, and JEPA reminds us that sometimes the most predictable future lies in abstract signs, and pixels are just one layer of it.

I think it's more important that Jepa because it liberated the world from the image of reconstruction. But Critique and PAN are also timely: the world model cannot communicate in the space alone, but is also subject to real observation calibration, is called by the action interface, and finally helps angent learn better strategies.

A mature world model probably does not belong to just one route. It will have Jepa abstract predictions and a PAN-style location; it will use compression experience in text, as well as physical experience in video and interactive interactions. Finally, we have to go back to the problems that intelligent bodies encounter every day: what happens to the world if I do that?

References

Eric Xing, Mingkai Deng, Jinyu Hou, Zhiting Hu, Critique of World Model, arXiv:2507.05169, 2025.
Yann LeCun, A Path Towards Autonomous Machine Intelligence, 2022.
David Ha, Jürgen Schmidhuber, World Models, arXiv:1803.10122, 2018.
Google DeepMind, Genie 2: A large-scale foundation world model, 2024.
Anssi Kanervisto et al., World and Human Action Models towards gameplay ideation, Nature, 2025.
Decart and Etched, Oasis: A Universe in a Transformer, 2024.
World Labs, Marble: A Multimodal World Model, 2025.
NVIDIA, Cosmos World Foundation Model Platform for Physical AI, arXiv:2501.03575, 2025.
Wayve, GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving, arXiv:2503.20523, 2025.
OpenAI, Video generation models as world simulators, 2024.
Google DeepMind, Veo.
Mahmoud Assran et al., Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, arXiv:2301.08243, 2023.
Adrien Bardes et al., Revisiting Feature Prediction for Learning Visual Representations from Video, arXiv:2404.08471, 2024.
Mido Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, arXiv:2506.09985, 2025.
Zhengcong Fei, Mingyuan Fan, Junshi Huang, A-JEPA: Joint-Embedding Predictive Architecture Can Listen, arXiv:2311.15830, 2023.
Delong Chen et al., VL-JEPA: Joint Embedding Predictive Architecture for Vision-language, arXiv:2512.10942, 2025.
PAN Team, PAN: A World Model for General, Interactable, and Long-Horizon World Simulation, arXiv:2511.09057, 2025.

Make OCR Great Again

2026-07-04T07:30:00.000Z

I don't study OCR. I used to talk about it, but I thought about it in my head, but I just wrote it from the picture."Boom!"It's pretty good to come out, like the extraction of words from the big phone albums.

The questions in this article can also be addressedRG: From project practice to system approach、MCP (Model Context Protocol)How the concept of a relatively close read together is developed in different contexts.

And then I started studying Agent, and I realized that OCR sometimes gets stuck in the front of the entire system. The real world is not so clean..png、.pdf、.docx And all the strange formats are inputs that the system has to process. Scientific research can directly use the cleaned data sets, not engineering, and we have to get these things in.

The document is usually troublesome more than words. Where the table is, how the formula is kept, what the title and the text are, whether the reading order is not broken, whether the diagram and the annotated note are to be placed together, whether the picture on the page is to be cut, described or left with a reference. These issues, which were more like later, are now becoming part of OCR itself. Now OCR is not the same as it used to be, it's a fixed line used to clean up raw data and facilitate the back RAG.

OCR, not recognition, document resolution.

If it's just a text, a lot of numbers PDF should never have gone OCR. The text is drawn directly from PyMuPDF, pdfPlumber, cheap, fast, and does not re-identify the original clean text and introduce errors.

OCR is becoming interesting again because we start looking at documents as a structured object. The page is not a string of characters, but a mixture of text, title, formulae, tables, pictures, footnotes, header feet and reading order. If the model is just a piece of text, it's going to take a lot of effort to guess where it comes from.

So now the more accurate name of this thing is document parsing. It's going to tear the page and spell back what a machine can do and can be read. Markdown is a good exit, but not the only exit, and more valuable is the structural information it retains.

So the question becomes: what should this structure look like?

Beyond the model: what OCR at the level of delivery should produce

If downstream is search, RG or Agent, OCR's delivery cannot be just a Markdown that seems right. Markdown is more like a preview layer, and what really should be delivered is a document object that can be tracked, split, enhanced and indexed.

The big factory has given the reference answer.Google Document AI Layout Parser You keep titles, tables, formulae, lists and hierarchical relationships, and then you generate contact-awarechunds with ancestral titles.Markdown of Azure Content Understanding Saves the chapter, table, formulae, pictures, page numbers and directories in a visible way. The common judgement behind them is that the pure OCR text flattens the reading order and context, and that the retrieval system needs to know which section, which page, which chart or which table a section is related to.

There are several design lines that can be learned from it: elements need to be stable, position information, content generated separately from the original, structure cannot be flattened, complex elements need searchable explanations, and each result can go back to the original page. It's the complete hosting service that's abandoned because I need to be able to run locally and to customize the logic of the process. In an era when Data is all you need, it is not a good idea to outsource the data-cleaning pipe.

Different elements require different delivery criteria

Object	Minimum deliverables	Retrieval-oriented enhancements
Text	Paragraphs, lists, reading order, language and page numbers, dealing with hyphens, hyphens and header footers	Keep chapter path, paragraph roles, key entities and context, without all text being crushed into a paragraph
Chapter Structure	Title level, paternity, directory anchor and chapter range	Write the ancestral title to chunk mettadata, let"Methodology""Result"There's still a whole spectrum of such duplicate titles.
Formula	LaTeX or MathML, line/line type, formula number, original map and coordinates	Add a neighbouring definition, variable interpretation and searchable natural language interpretation and mark whether the interpretation is generated by models
Pictures and Charts	Original cropping, original drawings, OCCR, page numbers, bbox, and references to text	Generate descriptive text describing objects, coordinates, legends and main visible relationships; while retaining visual access to embedding
Table	Cell grid, row title, merge cell, unit, title, footnote and original coordinates	Also provide structured data and retrieval summaries, and write key words into natural languages, without taking into account the conclusions under the original table

Formulas are particularly susceptible to error. Only one formula picture is left, and text search is almost impossible to find; only LaTeX is kept and may be missing"What's this formula for?"semantics. It would be more prudent to tie together formulae, numbers, ex-ante interpretations and variables. User Search"How to calculate long-term incentive discounts", you should be able to hit the section where the formula is, not ask for a query to appear. \gamma。

Pictures and forms should not be left alone. ![](images/xxx.png)I'm sorry. At least keep the original drawing notes, the text in the diagram, the chapter to which you belong and the quotations in the body. For pictures without a note, a description can be generated using VLM.Docling's picture of the situation That's what it is. But keep the source tag because"What is clearly written in the figure?"and"The model thinks the map is a sign of something."Not the same kind of evidence.

This leads to one of the most important items on the entire list that I believe is: faithful restitution of the original language and search-oriented enhanced text, which must be two fields. Photo descriptions, table summaries, formulae interpretations can expand recall, but they cannot be disguised as original facts. You can never tell whether a search result is a document or a model illusion.

The page number and coordinates are not all in the files.

page_idx and bbox For PDF is mandatory field. Word documents do not have a reliable set of final pages that are not relevant to the rendering environment. A visible page break can be found in the file, but the page numbers that you see in Word will still change with font, paper size and printer drive. Word and HTML, EPA, Markdown are closer to a continuous stream document.

So positioning information must be downgraded: page breaks to page numbers and bbox, flow formats to structure paths and characters. And the downstream search is written accordingly."There's bbox, frame, no structure.", the different positioning branches are maintained in format.

Don't cut the pieces, then try to find the structure.

The cut-off for retrieval should occur after structural recovery. A cut of every 500 tokens can easily separate the title from the body, separate the tablehead from the data line, or lose the picture description.

More rational order is:

Recover the page elements and read order first
Creates chapter tree, element and page-crossing relationships
chunk by chapter, paragraph and element boundary
Attach each chunk ancestor title, location information, document version and element type
Generate separate chunk for pictures, tables and formulae while retaining their relationship with the body father cunk

Character accuracy remains important at this time, but it is not the final acceptance standard. A system that delivers should also answer these questions:

Can you hit the right paragraph when searching for the subject of the chapter and bring back the full title path?
Can you retrieve the corresponding rows, column headings, units and footnotes at the same time when searching for a condition in the table?
When searching for a phenomenon expressed in a chart, can you find a picture by drawing or by generating a description and return to the original page?
When searching for formulae, can you return formulae, variable definitions and adjacent interpretations, rather than isolated LaTeX?
Can each result be traced to the source document, location and distinguishing between the original language and the content of the generation?
When a page or element is not deciphered, can the system clearly report failure rather than producing a seemingly complete Markdown?

The last one I felt was the easiest to ignore. The failure of the analysis is acceptable, and silence fails because you don't even know what's wrong.

How to select the right tool

A simple comparison.

With this list above, I have chosen three comparable options. They're all in OCR, but the scenes are different.

	Docling	MinerU	Unlimited OCR
It's a product.	IBM Research Zurich / LF AI & Data	Shanghai AI Lab / OpenDataLab	100 degrees
Positioning	Multiformer Parser + Unique	Document Parsing Frame, Three Core Modes	End to Long Document VLM
Input	PDF, Office, HTML, EPA, mail, audio, video, ODF, XBRL, etc. 20+	PDF, Pictures, DOCX, PPTX, XLSX	Pictures and PDFs only
Internal structure	Trees,`body` Root + JSON pointer	Legacy output is a reading-ordered list; 3.0 from a single structure with a separate page by page	Model straight out of marked text
Long Document	Page-based Processing Unit	Page/Sliding Window Organization	R-SWA, dozens of pages available for joint input
Permission	MIT	3.1.0 Commencement"Customise permission based on Apache 2.0"	MIT
Mature	High, Langchain / LlamaIndex / Haystack Official Integration	High, big community.	Weak, access to Vllm, and so on.

MinerU is more specialized in PDF and complex layouts, and the original format supported by Docling is more extensive, and provides a unified expression. Unlimmed OCR is trying to solve the problem of continuity of long documents.

Many document resolution models still use pages as basic inputs, with cross-page paragraphs, cross-page tables and serial numbers to be added from the outside; Unlimed OCR R-SWA allows the model to see visual input at all times when it is generated, but only the nearest window is left for the generated text, so KV Cache is pressed near a fixed limit and dozens of pages can be read out at once. It has been able to deploy through Transformers, vLLM and SSGLang, but the project itself has been unable to enter production directly.

DOCX, XLSX, PPTX are essentially ZIP containers, containing XML, media documents and the relationship between them. Most of the structure and content are already in the document, and the focus is on reading the original structure, rather than re-producing each page. Docling's fit in in that respect is more complete. MinernU has been more focused on PDF, pictures, formulae and complex layouts that require OCR or VLM involvement.

Excel has two uses, sometimes with very few tables, a watch schedule or board, and sometimes a data sheet. The former could be part of the knowledge base, while the latter should be placed in the database for use by the data centre department. For small-scale scheduling forms, I usually do the Docling forms. Chunker- Let a Sheet match a Table. I'll give you the big watch. HybridChunker Press token cut; it will customise the header when the table crosses the chunk.

First judge the document, then select the solver.

In this way, the logic of route is simple:

Input	Default Selection	When will you upgrade?
Office / HTML / EPUB / markdown	Docling format backend; straight out `DoclingDocument` Time to go. `SimplePipeline`	If the original structure is not read or format is not supported, turn PDF and go to the branch below
Digital Native PDF (text layer complete, layout general)	Docling `standard` Or MinerU. `pipeline`	Try VLM when there is a clear error in reading order, crossbar or complex table
Scan / Spectrum PDF	Open first OCR; available MinerU `pipeline` Or Docling. `standard` Sample	Try again when OCR is still unable to restore complex layouts `vlm`、Docling `vlm`
Re-formatted, complex tables, formulae intensive PDF	MinerU `vlm`, or Docling `vlm`	No upgrade. Watch the effect.

PDF has no available text layer is the first dividing line, but having a text layer does not mean that OCR is not necessarily used. I will extract the number of characters per page, printable character proportions and photo coverage, and then randomly render page contrasts; some PDF hidden text layers are not coded, wrongly positioned, and should be processed as a scanned copy. Nor are thresholds suitable for death, and the distribution of Chinese slides, double-bar papers and invoices varies considerably.

Docling: Processing XML and PDF should not be on the same pipe

Docling Official Hostage ServicePlease. Docling for IBM watsonxBottom of the line. docling-serve RET API. Given that there is no free access, I am generally deployed locally.

Docling selects the backend and Pipeline in the input format and adjusts the parameters inside Pipeline:

Layer	Options	What do you actually do?
Native Structure Format	Format backend + `SimplePipeline`	DOCX, PPTX, HTML, Markdown, etc. directly read paragraphs, titles, tables and photographic relationships in the file, output `DoclingDocument`；`SimplePipeline` Receive this result and execute uniform delivery without running PDF layout analysis, OCR and TableFormer
PDF / Photo Processing Pipeline	`standard` / `vlm`	`standard` Correspond `StandardPdfPipeline`, by linking specialized models such as layout, OCR, table structure, etc.;`vlm` Correspond `VlmPipeline`, convert visual language models to page-to-end versions suitable for scanning and unconventional layouts, but with higher speed, cost and generation error
`standard` Table Mode in	`fast` / `accurate`	`fast` Fits to simple forms or previews;`accurate` is the default mode for TableFormer, which is used when complex headers, cells are merged and error columns are larger

For direct generation DoclingDocument The form of the original structure.DocumentConverter Usually, it's automatic. SimplePipeline: it takes the result of the backend, and no longer runs the PDF set of page-level visual structures.SimplePipeline Keeps the uniform conversion result and interface to the enrichment, but whether it is worth a different description of the picture, which is to be opened on a task basis.

The rest of the parameters are mainly for PDF standard Pipeline:do_ocr The blogger says:force_ocr It is the enforcement of the entire page even if there is a text layer; normally the former is opened, and only when the text layer is hidden is damaged.do_table_structure Controls whether or not to restore the table grid; columns can be closed when they are merged by error do_cell_matching, the cell text predicted by the table model.

The code, formulae, photo classification, picture description and chart are understood as "enrichment," and whether or not downstream is really needed. In the formula that I've been working on, in the dense and complex layout sample, MinerU is often more stable, so I'm used to letting it take over this part, and then map it back and do it alone. This is my sample experience, not the uniform ranking of the two items on all the files.

MinenerU: Heavy OCR system

MinernuOfficial online servicesThe site is also available at the end of the page and at the API, saving local download models and preparing for display. Prices are relatively friendly, but I'm still used to local deployment when I have a card.

API model_version There are three values:

`model_version`	Apply Input	Characteristics
`pipeline`(Default)	PDF, Picture, Office	Traditional module resolution, speed and certainty; digital primary PDF, first batch screening priority
`vlm`(Officer's web label “Recommended”)	PDF, Picture, Office	Complex layouts, scanned copies, formulae and table-intensive pages are more worthwhile to try; higher costs are also needed to prevent the creation of errors
`MinerU-HTML`	HTML	HTML Text Extracting Special Mode; not ratio `vlm` Do not select a higher-level, non-HTML file

function parameter,is_ocr Default false，enable_formula and enable_table Default true，language Default ch; also available page_ranges Only sample pages are parsed. Formula Switches on vlm Down only affects the intra-line formulae. It's usually Docling who's in the MinerU, which is a non-processed scan or complex document, so I'm in MinerU more often. Choose vlm;digital, generic PDF still to be tried first pipelineThere is no need to spend VLM more than once for the sake of uniformity.

Default for the current open-source version hybrid-engineI'm sorry. It combines primary text extraction with VLM, with the goal of reducing hallucinations while retaining high accuracy. It has. medium and high 2-Back: Default medium Faster, but not supported; maximum precision or photo analysis required highI'm sorry. And... pipeline、vlm-engine Two local models.pipeline It can be run on CPU or GPU, emphasizing stability and non-supplegicity;vlm-engine In exchange for precision in complex layouts.

How to design a unified data expression layer

After the diversion, two solvers produce two sets of formats, and more will be introduced later. It requires a unified expression before the inquiry can be made on it.

The question that this layer is going to answer is really certain: what are the elements in the document, what type each element is, who is it, who is it, who is it read, what order is it, what is it from, what coordinates are to explain, how the row relationship of the table is, how the model is created, how the content is distinguished from the original text, and whether the original elements can be recovered after cutting off the chunk.

The DoclingDocument, I think, is the better designed one in this kind of structure, and it has clear fields for each of the questions:

Means questions to answer at the level	DoclingDocument
What are the elements?	`texts` / `tables` / `pictures` / `key_value_items` The stylized packagings are separated; the new version also includes `form_items`、`field_regions`、`field_items`
What kind of elements are they?	`DocItemLabel` Enumeration, 30+ labels, from `SECTION_HEADER` Present. `FOOTNOTE`
Who's got who?	`body` For root trees, use JSON pointer (e.g. `#/texts/1`I'm a father and a son.
Reading Order	Tree depth priority through the field without additional order fields
Where did it come from?	`ProvenanceItem`- Yeah. `page_no`、`bbox`、`charspan`
What do you mean, coordinates?	`BoundingBox` Take your own `CoordOrigin` Enumeration, left top or left bottom in the data.
Table Structure	`TableItem.data` It's structured. `TableCell`, Ham `row_span`、`col_span`、`column_header`
Content generated by the model	It's in its place. `meta` barematic fields, separate from original text fields; old versions of pictures, tables `annotations` Disabled
chunk, how do you trace it?	`BaseChunk.doc_items` Point back to the original elements that make it.

There are a few points worth saying alone.

Trees, not lists. Reading sequences and hierarchy are the same thing in trees, and they are sequenced over and over and over and over and over and over and over and over and over and over again, and you know the chapter path by looking at the parent. A flat list with a whole number can be expressed, but it has to be pushed again every time you need it.

The coordinates are written in the data.CoordOrigin It is an itemization rather than an agreement, because the PDF primary coordinates are the bottom left, the image processing practice is the top left, and neither of the two sides can be connected by default. Of course, most of the search scenes do not need to be precise enough to be a frame, and this field is not a bad idea, but when it is really a good one, it saves a kind of hard-to-see fault.

Tables are structured cells rather than HTML strings. Answer."What's the column title for this row?"The former read the field directly, while the latter re-deciphered HTML.

Original language and creation of content subfields. Current schema with item meta Saves derived information such as photographic descriptions, classifications and table summaries, and the text remains in the content field. Old pages of pictures and tables annotations Compatible accesses are also maintained but marked as obsolete. This corresponds to the one in front that "can't make the model's story look like a document."

Key premise docling-core It can be installed independently, without dependence. docling Main package, pure Pydantic model, officially stated as being"Interoperability"Designed. In other words, it was intended to be used as a generic representation by third parties, not as an internal realization detail for Docling.

And it's bringing more than schema.HybridChunker The hierarchy and tokenization-aware cut-off also combines small pieces under the same heading, with the output chunk taking headings, captions and position information; sequenced API for Markdown, HTML; Langchain, LlamaIndex, Haystack, CrewAI. Use DoclingDocument as a semantic expression layer to reduce the cost of subsequent adaptation work.

The rest is about the adapter that MinerU output to DoclingDocument. There's no such thing as a smart spot, but there's a lot of little engineering. I'm not gonna let you go, Legacy. content_list.json It's a reading-order list. The title level is on it. text_level (a) Revert; 3.0 from all backends also output grouping by page and uniforming to type + content It's... content_list_v2.jsonThe new adapter is more suitable for priority, but the official V2 is still marked as a development format. And the coordinates can't be mixed:content_list.json And V2 bbox The map is 0-1000.middle.json Use corresponding page_size . The page size coordinates system, VLM Original model.json Use 0-1. Plus, MinerU. page_idx From 0, Docling page_no From 1 onwards, and the conversion of HTML tables to a cell grid with spans, such errors are usually not reported wrong, but only allow the reference to be slushly slurred over a page.

RAGFlow also walked this way. It's in... v0.22.0 Medium experimental to add MinerU as an optional PDF solver, not using the Docling expression, but rather suitable for a section structure that fits into a home. The output particle size of the external solver and the internal chunk structure still need to be aligned, and alignment is not as easy as it could be expected.

Show layer is not query layer

With a uniform indication, it seems logical to search the matter. But there's a lot of confusion here.

DoclingDocument."What's the file like?", don't solve"How do I find it?"I'm sorry. It's a tree. Trees are good at placing and sequencing. They're not good at answering."Which documents refer to X and are published after 2025"I'm sorry. These two things require different data structures, and Docling himself has opened them up in different warehouses.

I prefer three: the most traditional keyword queries, PageIndex, DeepRead, which allow models to navigate themselves on document structures, and intensive vector-drive search. That kind of knowledge mapping technique that I used to choose. In the common knowledge base, the physical type is easily driftable and the cost of extraction, discrimination and updating is much higher than that of tree navigation.

Structured queries are indexing the tree into a relationship table or search engine, filtering by element type, chapter path, page number, document ID, and walking BM25 in the body. The most reliable, precise and interpretable answer to the question of clear filtering conditions is the most reliable and reliable.

PageIndex, DeepRead, such techniques are searchable like human beings: first look at the stratification catalogue, then look at the chapter title, move down step by step, and finally find what is needed. PageIndex simply calls itself a vctorless, and the whole process does not need a vector bank to allow the model to determine which branch to go on a tree by node; DeepRead makes it a loop of locate-then-read, and locates it and reads it. Their common advantage is that each step can be explained and the result can be directly on the page and chapter.

Vector search is by itself. HybridChunker The output chunk, which carries its own ancestral titles and annotated drawings, puts this information into embedded text and quantifys it, and it is much better to recall semantics. The chunk retains the reference to the original element, and can go back to the structure and the original page after the hit.

These are not competitions, but different views from the same factual level. Integration should take place at the retrieval level, where each is called back and reordered, rather than forcing a universal index at the resolution level. The function of the typology layer is to keep the facts right, to keep them intact and to be able to trace them back to their original location at any time.

But all these callers have a common premise: the document must have a clearly structured tree. The catalogue is correct, the chapter level is not disordered, the title and the body are attributed to the right, and it is directly determined whether the model can be navigable. That's why I'm willing to show you how much energy there is, not just to save it for good, but to really rely on it on the back floor.

Concluding remarks

The whole chain is actually four levels: identification, structure, representation, searching.

The identification is only a small first paragraph, which is followed by the restoration of the page to a structural object, and the consolidation of the output of multiple solvers into a single single expression, with the final turn of query. It means that the layers are designed to determine what can be done directly behind them, but it means that doing it better is not the same as being able to find out.

So the title, and the article, the story is twisted. OCR cannot fit the current name, and now the link is about structural restoration, unity, retroactivity and retrieval interface. But this does not prevent it from being much more important than it was in the past, and this is the first and the easiest to underestimate for Agent, who needs to process real documents.

MOGA。

References

Docling, DoclingDocument concept
Docling, Chunking
Docling, Serialization
Docling, Enrichments: picture description
Docling, Document converter API (SimplePipeline / StandardPdfPipeline)
Docling, Advanced options
Docling, Vision models
Docling, REST API
Docling, Managed service
IBM, Docling for IBM watsonx
docling-project, docling-core
OpenDataLab, MinerU
MinerU, Official online service
MinerU, Online API documentation
MinerU, CLI tools
MinerU, Output File Format
MinerU, Changelog
OpenDataLab, MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale
RAGFlow, v0.22.0 release notes
Baidu, Unlimited-OCR
Baidu, Unlimited OCR Works: Welcome the Era of One-shot Long-horizon Parsing
vLLM Recipes, baidu/Unlimited-OCR
VectifyAI, PageIndex
DeepRead
Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Google Cloud, Document AI Layout Parser
Microsoft Azure, Document Content Understanding: Markdown Representation
Amazon Web Services, Tables in Amazon Textract

Synthetic Data as Engineering: Starting from Terminal-Corpus

2026-07-03T05:30:00.000Z

Before the text

Recently read NVIDIA's 《On Data Engineering for Scaling LLM Terminal Capabilities》I'm sorry. It's about training data for the terminal anent, but it's not about the terminal itself that really attracts me, it's about the syntheses that are very engineering. Synthetic data are not generating more text, but rather a world where behaviour can be trained. Think first about what the world is going to offer, and then reverse tasks, the environment, trajectory and filtering.

The paper was based on a system called Terminal-Task-Gen, and the output data were called Terminal-Corpus, which was finally used to train Nemotron-Terminal with the SFT tracks. It brings together task generation, environment, teacher rollout, filtering, course learning, context and scale experiments. Reads comfortably because it does not give one result, but separates the training data lines from the one that was adapted from the old data set, which tasks are synthesized by skills, how the team runs, how the failed samples are processed, how the training is mixed.

This note only talks about SFT. RL will certainly appear at the second half of the training at the same time, but I'd like to think about the monitoring data. In many cases, the model has not reached the stage where complex online optimization is required and has been blocked by dirty missions, weak tests, chaotic tracks and hand-held data.

If you need to complete the SFT, LoRA, the apparent estimate and the preferences, you can get it fromPost-Language Training and Micro-Adjusting PracticeStart. This paper continues only along the line of synthetic data.

Define mission world, generate verifiable tasks, collect learning tracks and maintain traceable data lines. Prompt is still important, but it's just the entrance. The effectiveness of training depends on the stability of the world in generating, filtering and reflowing data that change model behaviour. There is also a set of assessments to prove that these efforts are effective.

graph TD    A["能力地图"] --> B["任务世界"]    B --> C["种子 / taxonomy / 知识结构"]    C --> D["可验证任务"]    D --> E["环境 / verifier / schema"]    E --> F["Teacher rollout"]    F --> G["过滤 / 去污染 / provenance"]    G --> H["SFT 混合与调度"]    H --> I["评估与错误分析"]    I --> A

Start with the mission world.

I used to think of synthetic data as a collection. (instruction, response)I'm sorry. It was barely enough, but not enough from 2026's perspective. Once the real scenes of code, tools, terminals, data analysis, etc. are entered, the data is no longer just a paragraph input and an answer. Models are about how to do things in one environment. The environment itself has become part of the mandate.

The idea of this paper, NVIDIA, is to break down the training data into three objects. The first is the question, what the model is to do, what the input output is, where the boundary conditions are, and how the conditions for success are written. The second is environment, where the model can see the files, what dependability, where the commands are running, and how to start. And the third is, the projectory, a teacher who would do this, how to work in the environment.

The three objects are put together, and the synthesis of data is less like the prompt project. It's more like designing a small training world. The most obvious thing about this is that action and operation are in the command line. The traditional QA anent, RAG, role conversations and even entertainment chat data have their own world, except for the environment that has been transformed from Docker to a knowledge base, user portraits, role-setting, dialogue history and style boundaries.

From command to teaching process

Before looking at the projects that NVIDIA did, look back on the lines that lie ahead: how to activate the task pool, how to control complexity, how to make the data more learning, how to break the generation process into a detectable stream of water.

Self-starter: Self-starter

Self-Instruct The problem of starting points is addressed. When there is not enough manual infusion, it starts from 175 manual seed tasks to expand the model itself to the task pool. When each round is generated, it draws eight entries as context, six of which are from artificial seeds and two from data generated before the model. The model Mr. is new, then determines whether it meets the demand, and eventually generates input and output.

There's a little detail here. Common tasks are used input-first, output-first,First give the label and then generate the input, avoiding the general bias of the classified data towards common labelsI'm sorry. Then you can filter the unsupported pattern, bad format and repeated tasks with the rules and weigh them with the ROUGE-L similarity. It finally gets about 52K examples of regulation and 82K.

It is not important to let models synthesize data, but rather to do it. The minimum closed rings, seedpol, promotion loop, dedup/filter, are a good starting point. Its limitations are also evident: the complexity of the task, the skill mix, the state of the environment and the verifier are not well managed. Self-Instract is a good start, but it's not too short away from the latest engineering practice.

Complexity Editor: WizardLM / WizardCoder / Textbooks

WizardLM Turns complexity into an object that can be edited. Evol-Instract has evolved from the existing instructions: in-depth allows for deeper tasks, such as adding constraints, concrete scenarios, requiring multi-step reasoning and making input more complex; in-breadth generates new rare tasks in the same field. It also discards samples without information gain, model refusal, change of point or leak of prompt with input eliminator.

WizardCoder Move this idea to the code scene. It does not reproduce generic complex templates, but rather adds code tasks to their own evolutionary algorithms: time and spatial complexity requirements, boundary input, error reference codes, debugging and repair, performance constraints. This migration is important. Complexity is not an abstract word, but must be in the field.

Textbooks Are All You Need It's not always in the SFT Synthetic Data discussion, but it always reminds me that data can learn. Phi-1 training is not a simple stack of codes, but rather uses GPT-4 to mark educational values, filter the Stack and StackOverflow, synthesize textbook-style data and CodeExercises. It emphasizes clarity, inclusion, sequencing and balanced coverage. This is particularly true for small models, where loose large data are not necessarily more useful than small, structured data.

Diversity and complexity can be broken down into sampleable dimensions. For QA, it may be the location of evidence, the type of question, the multiple jump relationship, the non-respondable sample; for role conversations, it may be the user relationship, emotional state, context span and human boundaries; for triminating anent, it is the file status, the tool call, the test failure and environmental restoration. Complexity is not automatically the same as being of value, and complexity is seen in combination with validation, coverage and training.

Waterlineation: AgentInstract / LAB

AgentInstruct The most interesting place is that it didn't start with artificial infusion, but with Raw seems. Seed can be an article, a chapter of a teaching material, a web page, a code clip, or an API description. Often, general researchers do not have systematic data on them, but there is a large amount of raw material. What Argentina Instract is trying to solve is how to turn this material into a modeled mission.

The first step is the Grant Transport Flow. It does not rush into the issue, but instead changes the original material to a more appropriate intermediate representation. In reading understanding, a regular page may be recast as an argument passage, debate, review, meeting transit or long passage; in the tool/API scene, code segments can be organized as API description or API list. This middle layer is important. Many of the failed data are not poor-written, but material itself is not suitable for direct questions.

Step two is the "Sed Investment General Flow." It holds the middle expression, which is based on skills that generate a variety of tasks. Tuxonomy here is not a post-post label, but a sampler. Read understanding allows for the adoption of topics such as as asmption, flaw, reference; text modification allows for the use of paraphrasing, simplification, reaction, style transfer; tool/API uses searchable, callable, combined, multi-wheel interactive. In other words, Taxonony decides where the model will be. Lee."Practice"。

The third step is to develop a new approach to the issue. After the initial assignment, you can give it to a character like Suggester-editor. The suggester suggested how to make the task more difficult, more reductive, less answerable or how the question might require more steps; the editos then put the recommendations to specific tasks. The re-election here is not simple tweaking, but continues to explore in the mission's neighbourhood. It makes the data more than just overwhelm."It's a standard question.", also covers boundary, trap, tool and multi-problems.

LAB The order is more like an alignment data factory. It goes on to take the texonony and divides the data into three branches: knowledge, foundational skills and general skills. Knowledge is oriented towards documentation, manuals, teaching materials and field knowledge; foundational skills include mathematics, code, language competence and reasoning; and computer skills combine knowledge and basic skills to complete complex user requests. A small number of manual examples are placed at each leaf node to serve as anchors for generation. This design is for"What can I do?"It becomes a table that adds or deletes nodes, not a vague wish.

After having taxonomy, LAB did two more types of SDG. Skills-SDG takes four steps: Mr. enters into evaluation, then produces response, final evaluation-response level. Knowledge-SDGs then move the model to documents, manuals, books, and ask questions from outside material generation areas, rather than simply learning about the parameters of the teacher. This is more stable for professional QAs, the business knowledge base and any area that is easy to hallucinate.

LAB has a warning in the order of training. It first does knowledge training, divides knowledge and foundational questions into two stages of short and long-response training; then does skills training, training more complex capabilities with generic questions and using replayers to bring back data from the previous phase to reduce forgetting. I'm just thinking of it as a SFT data-motion idea: first, to get models to know their knowledge and basic actions, then to train their skills, and not to flush out what they have learned.

When you do your own data, you can write the water lines. 素材池 -> 中间表示 -> 任务生成 -> 任务改写 -> 回答生成 -> 质量门 -> 训练混合I'm sorry. Every step leaves metadata and the reason for failure. Or you'll see it in the end."It's not working."The blogger says that the government is not aware of the evils of the sighted content, mission description, teacher response, critić, or mixed training.

From overlay control to data exhibition

The second group is closer to the current data mapping exercise: how to get data to cover the target, how to achieve dynamic difficulty control and course learning, and how to understand the value of wrong samples and data scales. The answers given in different studies are not entirely consistent.

Structured sampling: GraphGen / CONDOOR

GraphGen Faced with a closed-book knowledge-inputative SFT. The random generation of QA is easy to misdirect, long tails, and shallow. It draws entities and relationships from source languages, builds knowledge graphs, then makes interpretation assessments for true, paraphrased, Negated states, etc., to estimate the degree of knowledge ownership of the knowledge margin. Then the border sampling around the model is unstable or under-covered k-hop subgraph, generating atomic, aggregated, multi-hop QA.

GraphGen did a very solid job of digesting. It has been melted in four groups, each of which answers a clear assumption: which parts of KG are really valuable (entity-only vs relation-only vs full KG), whether training data should be aimed at the unstable areas of the station or whether they should be fully sampled (high-loss data are better), whether the output length is confusing with variables (not related to length) and the impact of the edge selection strategy (less variation between the three groups, honest reporting of nululsult). The digestion is not about the number of experiments, but about letting the reviewer see which components of his method are working and which are just set.

CONDOR With World Knowledge Tree. It expands on a large number of topics, then combines themes and common chat scenes, such as Daily Chat, creation, roll-playing, and controls the difficulty. Generate and then do Self-Reflection Refeightment, which allows the model to evaluate the strengths and weaknesses of the answers and then rewrite the answers.

These two articles are particularly enlightening for QA angent. When QA data are synthesized, the problem is often taken from where the problem is written earlier than the problem is. Random document clips bring random QA; knowledge maps, theme trees, student panels and user scenes are combined to be more like training programs. The limitations are also here: the maps and the trees themselves are biased, and the poor construction will systematize the deviations.

SWE-smith / OpenCodeRessoning / CHIMERA

SWE-bench usually looks for PR/issue, then goes back to the historical version.SWE-smith In turn, the real Python warehouse is being built to operate the environment, and a lot of bugs are being built in this environment. It creates a patch that breaks the existing test in four ways: LM Modife, LM Rewrite, Procedural Modification, PR Mirror, and only retains the candidate that triggers the Fail-to-Pass test, and then uses LM to write a question description of GitHub issue.

This is a similar thing to the Docker design of Terminal-Corpus: fix the world, then build missions in the world. When the environment is stabilized, failure is more likely to be attributed and storage and maintenance costs are low. SWE-smith also warned against exposing the verifier too bluntly. The provision of the Fail-to-Pass test will make it easier for the subject to be solved, but students may learn to skip the recurrence process and meet the test directly.

OpenCodeReasoning The value of the wrong sample is being reconsidered. It collects 28,904 unique topics from TACO, APPS, CodeContests and CodeForces, and uses DeepSeek-R1 to produce 736,712 Python researching samples. However, the results were somewhat surprising: the model performance declined after filtering out samples that had not passed the unit tests. This does not directly indicate the usefulness of the wrong sample itself, and is more likely to indicate that filtering changes the difficulty of the subject, the coverage or the trajectory distribution. One possible explanation is that the failure sample covers more difficult questions, the result and the NV ' s projectory response: correctness is an important signal, but not the whole value of training.

CHIMERA It's a counter-alarm on a scale. It only constructed about 9,225 articles of computer science research data for cross-science reasoning. The process is to expand from the rough discipline to eight disciplines, 1,179 topics, to generate self-contained and verifiable questions, cross-validate with multiple models, and finally keep the correct trajectory for SFTs using thinking model synthetic solution tracks. It indicates that small data can also be valuable, provided that the density of coverage and quality of validation is sufficiently clear. The fact that the government is not a party to the law is not a matter of the right opinion. The former illustrates the efficacy of organized and validated small-scale data, while the latter reminds us that radical correct filters may be removed along with difficult samples and useful coverage. Filtering and data reorganization itself would introduce the bias, so each cleansing rule would need to be justified and checked how it changed the distribution of data.

Back to Terminal-Corpus: Data Synthesis Project

Now go back to the Nvidia article. Terminal-Task-Gen does not generate the subject by a large prompt mass, but instead puts data sources, task formats, operating environments, teacher behavior and subsequent filtering on the same line. The environment is closely related, but not only to the environment. Role-Play, QA, RG-Sys, Chatbot have their own environment, hidden in chat records and a priori world knowledge.

Wide and deep, too.

The paper can be presented in two broad layers. The first level is the dataset operation, which is connected to the various types of task data available of high quality, rapidly spreading the basic capabilities of mathematics, code, software engineering. The second level is the task of synthesizing the task closer to the mission with the sighted and the pre-mititic skills. The former addresses size and breadth, while the latter addresses skill mix and depth.

Directly synthesize all tasks from zero, which is costly and difficult to know if coverage is balanced; only old data are modified and the format and depth of the original data set are limited. The NVIDIA approach is to spread the foundations first and then fill the depths where skills are lacking, tools are lacking to interact, and environmental conditions are lacking. Many of the judgements in the data project are in this order: what is available for cheaper reuse and what must be redesigned. This is the part that we have to consider when we're thinking about building a data set.

Especially when we use this idea of data synthesis, we must maintain Schema in a sufficiently complete way, and task_id, source, difficulty, generation of the task information. Different sources need to be absorbed, and different difficulties need to be addressed, so metadata should be recorded as fully as possible during the synthesis phase. Save Only (instruction, response) And a simple form, early savings, later check-ups can be painful.

Dataset tasks: connect task

Let's talk about the dataset adapters. Thesis selects the three categories of math, code, SWE prompt. Math, from Nemotron-Cascade, Math responding SFT Stage-2 policy set, approximately 163 K unique issues; this data removes easier questions from the front processing, such as DeepSeek-R1 response of 2K tokens. Code from Nemotron-Cascade code researching SFT data, original collection 79K prompts, further filtering and re-resulting to get 35K. SWE data from Nemotron-Cascade SWE code repair SFT, which includes SWE-Bench-Train, SWE-reBench, SWE-smith, SWE-Fixer-Train, and filtered and obtained 32K unique prompts. The main filter-de-regulating technique is the filter that is attached to the inspection and field of vector similarity at n-gram (NV = 9).

The daapter itself does not need to be involved with LLM. It fills in the existing Prompt 2 {instruction} placeholder, and a subfix by task type. The math question requires that the final answer be written. /app/solution.txtCode number requires Python solver to be saved to /app/solution.py,SWE Question requests to generate SEARCH/REPLACE Edits and save diff to /app/solution.patch。

The benefits of this route are cheap, stable and large. It converts existing data into behavioral data in a model to practice"Read tasks, operations files, deliver results by path"I'm sorry. The shortboard is also clear: Nemotron-Cascade raw data mainly provides prompt, and these dapser missions usually do not have the appropriate tests, and the real environment is not as rich as the synthetic tasks that follow. It is basic training, not the most representative part of the capacity, but it is necessary.

Seed-based promotion: changing the title to a task

The second route is ted-based synthetic task promotion. This is not an old case shell, but an inspiration for an existing problem, and an implementationable terminus. Every seen entry has at least a problem determination, domain label and reference solution. Reference solution is only used when generating test expectations, without exposure to ant, avoiding ant learning the answer directly.

LLM plays task here. It is intended to complement the full project task with abstract topics: where the input file is, where the output is written, what dependencies are needed, what are the boundary conditions, how the float error is judged, how the format is checked, how the pytest test covers normal samples and edge cases. I'm just saying,"Accomplish an algorithm"And the terminus is clear."Read /app/input.csv, Generate /app/output.json, the field must meet this schema, the test will check the behavior."I'm sorry. As indicated below.

你正在把一个 seed problem 改造成 terminal agent 的 SFT 任务。输入：
problem_description
domain_label，可为空
reference_solution，只能用于设计测试期望，不能泄漏给 agent
生成：
task_prompt：自包含，说明输入路径、输出路径、约束、完成条件
files：任务启动时需要存在的文件
tests：pytest 检查，覆盖格式、数值容差、边界条件
metadata：domain、skills、difficulty、source、generation_version
要求：不在 task_prompt 中透露算法或参考实现
任务要难解，但测试要能自动判断如果题目太大，拆成可验证的小目标

This type of program is only the first step, which really makes it possible to control, is the isolation of the reference file, the structure of the test file, the recording of the metadata, and whether the tracker rollout can be run behind.

Skill-based promotion: grouping tasks from productive skills

The third route is skill-based promotion. It no longer relies on existing topics, but instead uses skills to synthesize new tasks. Nine papers were defined: data processing, data querying, data science, debugging, decendency management, file operations, scientific communication, security, software engagement. Each domain maintains a set of dimensions covering the agorology, systems, data processes, mathammatic, testing, web/security, etc.

Here's the taxonomy not a post-label, but a sampler. When generated, LLM is asked to combine 3 to 5 prime cases, write a new scene and output prompt, pytest testes, waters, metadata, files and test references. For example, the data science task may mix file reading, groupby aggregation, statistical testing and chart validation; the security task may mix encoding, authentication logic, payload construction and test judgement; and the software engineering task may combine file modification, reliance on resolution, test restoration. See the following tips, and specifically the original text of the template.

你是 [domain] 任务生成器。可用 primitive skills：
[skill_1]
[skill_2]
[skill_3]
…
请组合 3 到 5 个 skills，生成一个真实、具体、可自动验证的 terminal task。
输出：给 agent 看的任务说明</prompt>初始文件、数据、配置</files>pytest 测试与权重</tests>domain、skills、difficulty、expected_tools</metadata>
约束：场景要自然，不要机械拼技能
不要泄漏解法、测试细节或参考代码任务应当需要多步操作，而不是一次函数调用

Complexity can be broken down into sampleable dimensions, not general requirements for more difficult tasks, but rather decisions on which lower layers of capacity to combine, which domain to combine, and what tests to observe whether they actually occur. This is more like a training map than a bunch of random puzzles.

Task format and environment: Ensure validation first

Both synthetic submissions end up in the same task format: natural language: task programt, pytest-based testes, weighted test entries and partial scores, supplementary input files, and a domain-special Docker environment. The paper did not generate standard answers for each mission, and instead pursued a more engineering objective: the task could be difficult to solve, but the validation should be as clear as possible.

In the angent scene, first have a verifier more practical than standard answers. If the test is stable, the test is successful, the tester fails, the trajectory is interrupted, the output format is wrong, it is recorded and then the analysis is followed. Without a verifier, it would depend on another model to score, and many problems would become style judgements.

Environmental design also serves this objective. NVIDIA does not allow LLM to generate Dockerfile for each task, but rather to maintain 9 pre-built domain-specific Docker images. Every image pre-assessed that domain is often dependent on, for example, the pacandas, scikit-learn, security cryptography libraries in the data science. This saves each mission Dockerfile validation and multiple-wheel restoration, reduces mirror construction costs and decorates the environment and tasks.

Tacheer rollout: Leave the work to the end.

After the mission was created, there was a need to go in and do it again. The paper selected DeepSeek-V3.2 because it had achieved good results on the various data sets of Terminus 2 angent africawork. Teacher can't just pick a strong model to see if it works in the target interactive protocol.

Trajory is generated by Terminus 2. Models see the mission statement and current mission state, and then output JSON, including analysis、plan、commands and task_complete。commands - Yes. keystrokes The system is sent as it is to the terminal, waiting for the specified time, and then fills out the latest terminal output back to the next round. The training sample is..."Status, reflection, planning, action, observation."- The cycle.And not just the final answer. Each step of analsis, plan, action, operation, success, number of tokens, number of turn, time-consuming and failure stages should be recorded. The tracks don't necessarily go into the training set, but they can get you back in the back."Why did the model learn to behave like this?"There's evidence to follow. Teacher rollout can also record difficult information and is a common experiment in training that mixs difficulty and sequences with a certain weight.

The compressed objectory scheme is probably this way.

{  "analysis": "当前状态、已经完成什么、还缺什么",  "plan": "下一步为什么要这样做",  "commands": [    {      "keystrokes": "pytest -q\n",      "duration": 1.0    }  ],  "task_complete": false}

This type of projectory teaches work habits. Model to read directories, read README or tests, run commands, see errors and go back to the error stack, adjust files, and finally check. Many models write codes, but the ability to work in the terminal environment is weak (the format symbol required for the terminal is difficult to output). The SFT trajectory does not necessarily suddenly give the model new intelligence, but it can place the capacity on a more appropriate behavioral path. Only half of the data synthesis is complete; the other half occurs in filtering, mixing and dispatching before and after training.

Return to Terminal-Corpus: Training dispatch strategy

The benefit of this NVIDIA paper is the integrity of the project, which did not stop at data generation. It continues to test these tracks in SFT to see what filters really work, whether course learning is useful, whether the context is better, and whether data size continues to bring benefits. By this chapter, data engineering has evolved from how samples are made to how samples can enter training.

The paper uses Qwen3-8B as the main ablation, and the size is verified with Qwen3-14B and Qwen3-32B. Training is conducted using veRL, learning rhe-5, course decay 1e-4, 2 epochs, max security left 32,768, global watch size 128, Adam W, cosine schemer and 10% warmup.

First, we clear the contaminated stuff.

Base cleaning first to pollute. Thesis deletes and Terminal-Bench 2.0 test samples have 14-gram overlap prompt, removes entries and drops the samples containing Chinese characters as the most basic filter. These rules look trivial, but it's common in the data of ant. Teacher may reveal his identity in the trajectory, or bring strange formats into training; assessing leaks is more problematic - once mixed, the latter scores are difficult to explain. We need to preserve the credibility of the experiment, and naturally we need to keep the data clean enough.

Filter experiments: Failure tracks may not be waste.

Change of intuition is projectory filing. Intuitively, it would be cleaner to keep only the full track or the track through testing. It's not like that. On dataset adapters, no filler is best combined: 226,313 samples, TB2.0 9.66; complete-only 196,940 and TB2.0 8.09. The difference is even greater in synthetic questions: no filler retains 264,207, TB2.0 is 12.4; complete-only only has only 104,603, TB2.0 is 6.74; access-only only only 83,448 and TB2.0 is 5.06.

It's not that the more failures the better. JSON is broken, identity leaks, long air turns, obvious nonsense, and these remain to contaminate the model. But some failures are not waste. The error reading after the test failed, the fixation of the path after the error, and the change of the program by relying on failure to fit, are all part of the technical event. Strict filtering removes dirty data and removes difficult tasks, restores behaviour and real errors.

Filters should be reorganized into two layers, and the Hard project is responsible for throwing out the obvious bad sample, soft label is responsible for recording the sample's state and ultimately for determining the weight in the training mix. The failure code sample in OpenCodeRessoning covers more difficult questions, and the failure here of NVIDIA is likely to carry recovery mode. Together, the two articles remind me that correctness is a strong signal, but not a complete definition of the value of training.

Data mixing and course learning

Data source ablation gives a more steady signal. In the Dataset adapters, Math, Code, SWE have separate returns, which are better combined: Math 5.39, Code 6.29, SWE 7.02, All 9.66. Different sources provide complementary behaviour - mathematically brings together reasoning formats, code brings capacity, SWE brings habits of reading files, changing codes, generating patch.

In synthetic tools, skill-based is the main source of gain, with article 139,841 reaching 12.4; and seed-based 124,366 with only 6.18; combined, both are still 12.4 but the difference is even lower. The task of seeingd-based comes from existing problems and is more stable, but ultimately it is seen as space-limited; the task-based is organized directly by the permanent skills, which can more easily cover the real action missing. The combination of the two did not continue to increase, but made the results more stable, which is also a recurring gain in the data engineering.

Course learning experiments are also not intuitive. Thesis compares two-stage curriculum and single-stage mixed training. The former trained first, Dataset adapters, then synthetic questions; the latter mixed all data. Qwen3-8B, misted training gets 13.03, curriculum gets 10.39.

This result does not indicate that the course learning is not useful, but rather that it is not course learning by data source. It's not necessarily easy, it's not even harder. The real difficulty may come from the number of files, the number of tools to be used, the length of context, hidden tests, polylysis, the success rate of the teacher, the base model initial capability. In the future, if the course is to be followed, the difficulty label is best obtained from the trajectory and from the data source.

Long Context Training and Scaling Experiments

The trend is observed in the ratio of 1%, 5%, 10%, and 100%, where a preliminary assessment can be obtained using a smaller ratio. If 10% of the data are close to 100%, it may be that coverage under the current assessment is close to saturation and that there are limited marginal benefits of continuing to increase the same distribution. This discovery is of engineering value in itself. This is a reduction in the size of the data, a classic training digestion technique, but not all.

In addition, the paper was based on the 8B model to perform the main digestion, and the 14B and 32B models to validate the scale trends, reducing the likelihood that conclusions would be formed only at the size of a model. And we can also see if similar scaling trends will emerge in our own scene.

Different context lengths are also part of the digestion experiment. In some scenarios, the length of the text generated varies considerably. There may be a small part of the trajectory that goes beyond the context limit and is thus cut off by the SFT algorithm. If you want to do so, you can consider not using the YARN2 model for SFT; using the YARN2 model for SFT; and using only the YARN2 baseline model for the assessment. Many scenarios have longer context lengths already contained in standard context windows, with long tails potentially confusing and information-intensive.

In the paper, the negative digestion is not hidden. You just have to explain."Why does this result have to be a counter-inputive, but reasonable."— For example, the course failed because the ranking of data sources was not the same as the order of difficulty, and the strict filtering was harmful because the recovery of difficult tasks was also eliminated together. The digestion experiment is not necessarily positive, and negative results are valuable. Under this type of budget and experimental setting, the main digestion can be done using the 8B model, followed by other model sizes of scaling exercises; this is a resource allocation strategy, not a rule that is established for all studies.

First, establish assessment benchmarks

After reading these jobs, I will not start by looking for 100,000 data. A more comfortable starting point is to put together the assessment and validation shelf, to know how to judge what is right and how to make the data. This remains part of the data engineering backbone: without a credible assessment, the data generation pipeline will not receive stable feedback. The verification of their work and data should not have been separated in real papers.

This step is the most easily bypassed and determines the credibility of all the experiments that follow. Without benchmark, training begins, even without saying anything about the optimization.

Benchmark and Datset are two completely different things.

Before you do your research, think about what you're building.

Benchmark's purpose is to...Measurement of capacity boundariesI'm sorry. It requires small and sophisticated, and each subject must be detached and verified. Its audience is the reader and the community, and the version should be frozen - once the benchmark topic is mixed into training data, all the results of the assessment are no longer credible. Terminal-Bench 2.0 has 89 missions, each of which cost about 3 quality-hours for quality assurance. SWE-bench starts from about 90K PRs, filters and executes, and eventually leaves only 2,294 entries.

The purpose of Dataset is to...Change model behaviorI'm sorry. It requires coverage that tolerates noise and failure samples. There are retrogressive behaviours and miscorrection patterns in the trajectory of failure, which are valuable for training. The audience of Dataset is the model itself, which should be continuously iterative and updated as the model's capabilities evolve. Terminal-Corpus eventually used about 490 K samples of SFT, but filtering experiments during the generation show no filters are much better than access-only.

One sentence summed up the difference:Benchmark asked you to get better, Dataset to make you better. The quality of Benchmark depends on the accuracy of the validation of each subject, and the quality of Dataset depends on the breadth of the coverage and the learningability of the trajectory. If you mix them together — for example, with the subject in the training data — it's a circular argument, and the reviewer can see it at first sight.

How do you start?

First, one or two weeks to do 10 or 20 verifiable tasks. Each mission has at least three components:

Independent environment: Each mission has its own clean operating environment (Docker or equivalent isolation), so that angent cannot be reconciled with the memory of the mission.
Artificial Oracle solution: Proof of the task is real. It does not need to be complicated, but it is necessary to ensure that a human being (not LLM) confirms the solution that can be adopted. Terminal-Bench requests confirmation from each contributer"My solution.sh was written by a human"。
Automation testing: Let the score be repeated. The key design principles are:Final status rather than intermediate process— Inspection"Output file matches schema", Not Check"Agent, did you execute a specific order?"I'm sorry. Give angent creative space and avoid being too vulnerable to testing.

The quality door process for Terminal-Bench can be used as a reference for scaling: automated check (oracle running, dummy anent cannot pass through) LLM auxiliary check (typo, test/description alignment) manual check and counter-test (specially running a cheating anent to see if it can be bypassed). You don't have to do 3 criteria for each task, but the first three steps are the minimum that every homemade benchmark should take.

Validate the quality of Benchmark itself

After you've built Benchmark, you have to prove to yourself (and to future readers) that it is a good assessment tool:

Distinction check: Run at least three types of models with your benchmark - the strongest current closed source model (GPT-5/ Claude Opus), the strongest open source model, and the base model you will use later. Scores must be distributed meaningfully. If all models are 0%, benchmark is too difficult and may be problematic. If all models are 100%, benchmark is saturated and cannot measure progress.
Human performance: If conditions permit, find one or two humans who behave with you do it again, report the score as upper base. This figure is much more convincing to the reviewers than any comparison between models.
Pollution control: N-gram overlap check immediately after the baseline is established to ensure that there is no overlap with any open dataset that may be used for training. This is the bottom line - CHIMERA paper clearly reported about 8-gram and 13-gram overlap, which should not be sub-sub-sections, but rather a symmetry.

When Benchmark was not perfect

Frankly, most self-built benchmarks cannot get the input from Terminal-Bench. You don't have 93 constribators, no ~3 missions per assignment, no specialized adversarial audits. And then you need to use additional means to make conclusions more credible, not pretend that benchmark is beyond criticism.

Case Study

Select 5-10 representative cases, with in-depth comparisons of behavioural differences between base model and trained model. The presentation of both success stories and failures — just picking good examples would put the reader on the spot. Each case must at least be clear: what the mission is, what the base model does, what the traded model does, why the difference makes sense. Case story cannot replace quantitative assessment, but it tells readers your method."Changed behavior patterns."♪ And not just ♪"The score's up."。

LLM-as-Judge calibration

Automatic scoring with a strong model (GPT-5 / Claude Opus) can significantly reduce assessment costs, but only if calibrated to be credible. MT-Bench shows a high degree of consistency between GPT-4 justice and human preferences, but this result relies on specific tasks, samples and evaluation protocols, and cannot be directly migrated to new research scenarios. If it is not possible to report on its own data the consistency of the judge with human beings, there is good reason for the reviewers to question the reliability of LLM-as-judge.

Calibration: first, using 3 human experts to score 50-100 samples independently, calculate inter-annotator agreement (Cohen)'s 100), then test the same sample for the consistency of the judge with the human race. Only when consistency is acceptable can the judge be used for large-scale assessments.

And at the same time, the LLM-as-Judge has three known errors to address:

Position bias:Judge may prefer a fixed-position answer. A/B could be evaluated at each of the two points of exchange and a separate report on stability before and after the exchange.
Verbosity biasJudge may misjudge a longer and more complete presentation as having higher quality, and the tendency to match the sample with length or manually labeled.
Self-enhancement biasJudge may prefer output close to its own model family, so it is preferable to use multiple Judges or to add cross-models and manual calibration.

Multiple Signal Triangular Validation

When you have small benchmarks, case studies, LLM-as-judge calibrations, a single signal cannot support any conclusion. But if the benchmark score is raised + case story shows behavioral change + LLM-as-judge gives a trend in the same direction, the three weak signals are more convincing together. What really repulsed the reviewer is the excessive assertion of an unreliable indicator — matching the strength of the claim with the intensity of the evidence — the principle most often overlooked in the publication strategy.

Evaluation and training iterative

The evaluation wasn't the end of your run once after the experiment.Make an emror anaclysis part of the pipe.: Train a version of the → running benchmark → analysis failure model → identification model remains weak in which areas of capability return to the data synthesis tube and target matching training samples → retrain.

This closed circle is simple conceptually, but it takes a premise: your benchmark is classified by area of capability or type of error, and error analsis is able to trace the data gap. If Benchmark is just a whole pass rate, Error anallysis can only tell you."There's another question."But you can't tell you what to add — that's why you need to put a capability tag right when you define mission space.

From data engineering to publication: how to convince the reviewer

The validity of the data project will ultimately need to be explained through experimental design and the chain of evidence. Here we continue to discuss how to move the whole process from an experiment to a paper: Industry can be online based on A/B Test and long-term performance indicators, and papers need to be convinced of other perspectives with replicable experiments, baselines and accommodations.

Build benchmark as a contribution

If you're not in the right direction, benchmark, you don't have to be ashamed of it. One of the core contributions of a large number of high-citation papers is the definition of confrontational authenticity by benchmark: TruthfulQA, the definition of software engineering angent by SWE-Bench, and the definition of terminal angent by Terminal-Bench. Each section is a question of definition, re-proven questions, and then prove its method on benchmark.

Thesis structure can naturally be cut in two: first half to define problem space and build benchmark (with quality certification, human performance, differentiation proof), and the second part to present methods of data synthesis and training and validate them on benchmark (with digestion analysis). And it's a complete contribution -- not as bad as you find a ready-made benchmark to the top.

The baseline is fair.

Can't just base model. If your method is to synthesize SFT data, then baseline must include the same methods of synthesis of data that already exist -- as in the case of the Gramgen, and at the same time, there are many methods like WRAP, Genee, LongForm, EntiGraph. One of the types of papers that the reviewer most often rejects is that"A is better than untrained base model, so A is effective"No, A, it's probably just a minor improvement."Best available method for data synthesis"。

At the same time, ensure that the baseline approach is treated fairly in the context of your mission. If you test data generated by the baseline method in the terminator, but the baseline method never optimizes the environment of the terminological environment, this comparison is unfair. The solution is to harmonize the suitability conditions (e.g., to give one identical adapter) or to establish two or more different assessments to cross-check.

Pollution control is the bottom line of trust.

All the results of the article are no longer credible once the reviewers suspect that benchmark has been contaminated. CHIMERA paper demonstrated this problem: 8-gram and 13-gram overlap were clearly reported near 0 and scripts were provided for the examination. Terminal-Corpus uses 14-gram overlap to remove a training concentration that overlaps with benchmark. The prevention of pollution from training and testing is the bottom line of an article like SFT Training.

We'll do a solid digestion experiment.

The true digestion is characterized by a single variable at a time, a pre-supposition for each experiment, not always positive results, and an interpretation of the results of the COUNTER-intuitive. Characteristics of false digestion: multiple variables at once, positive results picked out to be magnified, none of which were negative.

Your digestion does not have to be exhaustive -- GraphGen made four groups, LIMA made three groups, Terminal-Corpus made six groups -- but each group answers one clear question. The drafters do not add points because of the amount of digestion, but because each group melts will make it clear why this group does it.

It's a small data paper.

CHIMERA has only 9,225 training data, LAB has less than 1M, LIMA even validated it with 1,000 pieces > Quite. The reader is not concerned with the amount of data, but whether you have made it clear why the 9,225 data are sufficiently clear and what they cover and what they don't cover.

The advantage of small data is that you can clearly explain each data profile and quality judgement. And the big paper reviewer is more likely to wonder if you've ever seen that much.

Open source lowers the concerns of the drafters

The Terminal-Corpus open source contains data sets and training configurations. This reduces the core concerns of the reviewer -"Your conclusion is not that it happened to be on the data you didn't disclose."I'm sorry. If you can open the source, then open the source; if you have constraints that cannot open the source (data copyright, privacy, etc.), at least open data to generate scripts, training in configuration and evaluation codes, so that the methodology itself is replicable.

It's written at the end.

The article is specific: re-use existing data, draw skill maps, generate missionable environments, get strong teachers, record tracks, do decontamination and filtering, train with SFT, and finally use ablation to see which options are really useful.

This line could be written with me. How Reward and Training Closed in Real Age: From Data Governance to Online RL Put it together and read. The article discussed the more RL-oriented angent training closed-ring, where the environment was changed from a triminal tool, log, verifier and reward to a real interactive system, but the idea of taxonomy, weight, difficulty label, negative sample, projectory filtering and curriculum is linked to the SFT data governance discussed here.

Synthetic data is a set of maintenance work to train the world. The blog, "Propt", "Teacher", "Verifier" and "schema" are important. It is becoming a more systematic project, and it is good news for ordinary researchers: once the problem is understood sufficiently, real engineering experience can be built up from small-scale data.

References

Renjie Pi et al., On Data Engineering for Scaling LLM Terminal Capabilities
NVIDIA, Nemotron-Terminal-Corpus
Terminal-Bench Team, Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Yizhong Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions
Can Xu et al., WizardLM: Empowering Large Language Models to Follow Complex Instructions
Ziyang Luo et al., WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Gunasekar et al., Textbooks Are All You Need
Zhangchen Xu et al., Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Arindam Mitra et al., AgentInstruct: Toward Generative Teaching with Agentic Flows
Shivchander Sudalairaj et al., LAB: Large-Scale Alignment for ChatBots
Yiran Liu et al., GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
Subhajit Dutta et al., CONDOR: Enhancing LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
John Yang et al., SWE-smith: Scaling Data for Software Engineering Agents
Wasi Uddin Ahmad et al., OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
Xinyu Zhu et al., CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
Chunting Zhou et al., LIMA: Less Is More for Alignment
Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Anthropic Engineering Blog, Demystifying evals for AI agents
Soham Bhattacharjee et al., CuratorKIT: Data Curation and Synthetic Data Generation for LLM Post-Training
Hyacehila, How Reward and Training close the loop in real Agent: from data governance to online RL
AMAP Team, AMAP Agentic Planning Technical Report