Making Agents Practical: Structured Output and Constrained Decoding for LLMs

This English version was machine-translated from the Chinese original. Technical terms may require verification.

The Large Language Model (Large Language Models, LLMs) must interact with traditional software systems when acting as a control hub for intelligent bodies (Agent). Such interactions usually require model outputs in strict format, such as JSON Schema, XML or AST in a given field language. The problem is that the model is token self-regression sampling, and the software system requires that the entire object be legal once.

The requirement that the model stabilizes the complex structure with a hint only leads to two problems: the formatting errors in the long sequence accumulate, and the format constraints take away the context and attention that should have been used for mission reasoning. When the model is to be semantically deduced, and remembers brackets, fields, enumeration values and closed rules, the failure rate increases with output length.

If the overall location of the model generation phase needs to be completed first, reference may be madeLLM Life Cycle Overview; if you are concerned with how the task, the example and the output contract are written, from" Intense engineering and context learning "Start. This paper only discusses how format constraints sink from Prompt to API and reason engines.

Agent needs a stable structured output, as it is relied on for tools, state updates and backend interfaces. It's expensive to try again after a hint and a failure, and it's easy to get Agent loop stuck on the same error. Now, more useful is to bind some of the formats down to the API layer or the reasoning engine layer.

Prompt Engineering (2023 +)

In early applications, developers rarely intervene in the reasoning engine and only use the Prompt Engineering to guide models to generate structured outputs. This phase is based on inspirational rules and context learning (In-Context Learning).

Context-based Distribution Induced

The early-direction formats include, inter alia, zero sample commands (Zero-shot Industries), examples of fewer samples (Few-shot Studies) and template-based output guides. Developer usually writes very detailed formatting instructions in System Prompt, for example: "You must output strictly in the following JSON format, do not output any additional natural language interpretation, and must include key values 'data'、'confidence' and 'reasoning'I'm not sure. Then, by providing several perfect JSON questions and answers (Few-shot), an attempt was made to build a strong pattern matching inertia in context windows.

In terms of mathematical and statistical mechanisms, the essence of this approach is to use Transformer ' s self-directional mechanism to deflect the probability distribution of conditions. The standard probability model generated by self-return is:

$$P(Y \mid X) = \prod_{t=1}^{N} P(y_t \mid y_{<t}, X, \theta)$$

of which $X$ The context of Prompt and Few-shot is the one that contains the text.$y_t$ Is generated token. By the $X$ Token (e. g. {), 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, {, 高频, 高频, 高频, 高频, {, 高频, 高频, 高频, 高频, 高频, 高频, {, 高频, {, {, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, 高频, {, 高频, 高频, {, {, 高频, 高频, 高频, 高频, 高频, {, {, {, {, {, {, {, {, {, {, {, {, {, {, {, {, {, {, ",) the self-regulating mechanism will give these formats Token a higher weight in calculating the points of Query and Key, thus allowing models to predict $y_t$ , the probability of exporting the desired syntax symbol is in local proximity to 1.

Limitations in statistical and attention mechanisms

Prompt Engineering is still useful in simple missions, and when models become stronger, it is not completely ineffective in a number of complex missions. However, when a long context, a nested JSON Schema or a long code is generated, only the hints expose two problems:

First, model output is the probabilistic sampling process. Assuming that each format is fully guided by Prompt, the probability of being the right token is high, for example. $P(y_t \text{ is valid syntax}) = 0.999$I'm sorry. But a complex JSON object could contain hundreds or even thousands of tokens. As long as the sequence is long enough, local errors accumulate into total failure.

Second, the format constraints will be context-driven. When models are to be mathematically extrapolated, multistep tools are to be called or code generated, the lengthy examples of JSON Schema and Few-shot are inserted into the context, which crowds out the task information and places the burden on the model both on “understanding questions” and “remember formats”.

Meanwhile, JSON Schema’s constraints are poor on Token Efficity, leading to a large number of Tokens being wasted on JSON formats, so there has been a wave of new formats in the academic community, but basically, JSON and YAML have been used as classic formats to facilitate communication between different developers, and it may be only new projects within the enterprise that have the opportunity to promote the new format as a first class citizen of the Agent framework. This has not taken into account the impact of the lack of pre-training to drive the new format.

JSON Mode and Targeted fine-tuning (2023-2024)

Outside the tipping programme, research and industry began to structure outputs from modelling training, API system layers and reprocessing layers of intervention. The Tool Calling study was also concentrated during this period.

For example, the core idea of Toolformer is to allow language models to learn to use tools through self-monitoring, and its training language is embedded in a large number of structured calls that are strictly structured in [API NAME(args)] format. The Gorilla 7B model goes further, fine-tuning the large API collection and introducing the Retriever-aware training method in the fine-tuning process, forcing the model to align the output generated with the specific API parameter document.

The problem of fine-tuning programmes is also evident. Models may remember Schema, which is common in training concentration, but stability may not be reliable when encountered with new definitions, deep-seated or frequently changed business interfaces. The continuous SFT for a large number of APIs also entails cost of computing, data maintenance and version management.

Meanwhile, the head manufacturer, represented by OpenAI, launched the original JSON Mode at the API level. Its approximate bottom logic is based on a highly optimized system-level fine-tune, which is accompanied by simple inspirational output filtering: It ensures only that the string that is the final output of the model can be successfully deconstructed by json.loads() that the brackets are closed, comma separators and string transliteration at the basic grammar level are legal. However, its bottom is only a broad syntax tree of JSON, the generic data interchange format, and does not understand the specific business logic of the developers and cannot handle particular Schema.

Python Level Limited Decoding and Library (2023-2024)

And with JSON Mode, there's also restricted decoded technology. This phase is represented by the Python-level intervention library (e.g. early Outlooks, LMQL, Guidance), which brings Automota Theory to the LLM sampling process.

Achieving logic

The restricted decoder method is to calculate the valid token assembly in the current context by using an external limited-state machine (Finite-Stat Machine, FSM) and remove the illegal token by masking (Mask) before each step is generated. Models are still sampled, but sampling space is limited by grammar.

In the case of Outlooks, the underlying algorithms first convert the developers ' defined JSON Schema into a large regular expression (Regex), then compile it into FSM, and map it to an absolute legal collection, thus obtaining a mask vector$m(S_{t-1}, Regex) \in {0, 1}^{\mid V \mid}$I'm sorry. The logarithmic vector of the first logarithmic logarithmic logarithmic logarithmic value of the next Token is calculated for the pre-scatter to the model$\alpha \in \mathbb{R}^{\mid V \mid}$ 。

Before Softmax, perform Logit Masking's mathematical expression:$$\tilde{\alpha} = m \odot \alpha$$ In strict implementation, the mask logic is to set the illegal Token Logit to a negative size: $\tilde{\alpha} i =\begin{cases}\alpha i & \text{if } m_i = 1 \ -\infty & \text{if } m_i = 0 \end{cases}$$最后，通过 Softmax 函数进行采样：$$\tilde{s}_t \sim \text{Categorical}(\text{Softmax}(\tilde{\alpha}))$$由于 $e^-info} = $0. As long as the Grammar and mask are correct, illegal token at the grammar level will not be sampled.

Performance bottlenecks

The Outlooks FSM mask solves the grammatical constraints, but early pure Python achieves easily slow reasoning. CPU-GPU sync, GIL costs, incompatibility with high performance batch reasoning, and FSM ' s difficulty in handling embedded regression structures make it expensive in production reasoning.Structured output cannot be determined by the correct rate of format, but also by its effect on throughput and delay.

High performance bottom restricted decoded engine (2024 - 2026)

To address the performance bottlenecks at the Python level, the successor programme began to sink restricted decode logic into an implementation environment such as C++/Rust/CUDA, and replaced simple FSM with a downward thrust from the Pushdown Automaton, PDA and pre-compiled grammar tree.

From FSM to below from PDA

To decipher the semantics of embedded structures such as JSON, the modern bottom engine converts Schema to a context-independent grammar (Context-Free Grammar, CFG). The core data structure for implementing the CFG is the downward motivation (PDA).

PDA equals the addition of a "Stack" on a traditional limited-state machine. When left brackets are encountered, PDA presses a state into the stack; when right brackets} is encountered, the firing is performed. The introduction of this stack structure allows PDA to track the embedded depth of JSON at unlimited levels.

The status of the PDA is determined by the "FSM Status + Stack Contents". Because of unknown insulation, state space can theoretically grow indefinitely and cannot be expected to be static Mask table as a simple FSM. The follow-up engine is mainly addressing the same question: how to complete the PDA token screening at low cost while keeping the swallowing.

Xgrammar: Pre-compiled and Cache Life Medium

Developed by CMU/MLC team Xgrammar is the default structured engine for the current frame integration of vLM and so onI'm sorry. It draws on the engineering fact that while the PDA status can grow indefinitely, most tokens do not rely on the current insular depth when decoded.

Xgrammar has strictly divided the vocabulary into two categories at the C++ level:

Context independent Token (Context-indepent tokens): The legitimacy of these Tokens is determined only by the current top level of the PDA FSM state, and no canteens are required (e.g. to generate string content) "value" The normal English letter is independent and legal at all times.

Context depends on Token (Context-dependent tokens): Token (e. g. closed brackets) must be checked to determine the validity of the stack, only if there is a corresponding open object in the stack.

Xgrammar generates a static mask for Token at the pre-compilation stage, which is independent of context. During the decoded run-off (Runtime), its C++ engine maintains a self-adaptation to the Token mask cache. For most of the steps generated, Xgrammar directly $O(1)$ Complex independent token cache, skipping a large number of PDA checkpoint calculations, compressing the CPU cost per Token to just $30-80 \mu s$ 。

SSGLang (Commpressed FSM): Forward leap decoder based on compressor status

SSGLang, which has started from the extended structure of the limited-state machine, proposes Compressed-FSM and Forward Jumping Code.

There are a large number of inevitable, defined format strings in the frolling of JSON Schema. For example, when generating a certain step, the next character must be {)"status": "I'm sorry. In the traditional restricted decoding, LLM wastes a dozen years of the first transmission cycle, predicting each of these characters without information entropy.

Technical dismantling: SSGLang compression algorithms analyze FSM scale maps at offline stages, looking for a strange transfer margin (Singular Transmission Edge): that is, a path that has a state and has only one legal follow-up state. SGLang compresses these consecutive single paths into a coarse particle size polychard border (Commpressed Edge).

During decoded runtime, once the engine reaches the starting point of the compression, the system will skip the front-handing of the large model, perform tokenize the whole string on the CPU level, and use the bottom interface to write directly to the GPU KV Cache depository tree.

Integration with the bottom memory management mechanism

The performance of the bottom-down restricted decoding system is enhanced and relies on modern visible page management mechanisms.

The traditional VLM uses PagedAttention to divide KV Cache into memory blocks of fixed size (e. g. 16 Tokens), which greatly mitigates the visible fragmentation. However, PagedAttention is not optimal for prefix matching and reuse when processing the process of structuring and requesting a large number of identical Systems Prompt or Schema templates.

SSGLang introduced RadixAttention (the base tree focus mechanism), a structure designed for KV Cache reuse. In RadixAttention, all requests for KV Cache are organized into a giant dictionary tree (Radix Tree / Prefix Tree). When multiple Agents request and follow the same complex JSON Schema, their pre-commands and the Token sequence defined by Schema are fully consistent. RadixAttention $O(\text{prefix_length})$ The complexity instantaneously matches the shared prefix of these requests and achieves the non-copy sharing of fine-grained particles in the GPU display.

Summary

Reviewing changes in structured output technologies from 2023 to 2026, binding positions are constantly moving backwards. The early placement of format pressure on prompt and model sampling is easy to fail and waste context. Python-level restricted decoder moves the format to the decodering process, but introduces the costs of the sync and interpreter. The bottom engine combines the grammatical resolution, jump and cut to the C++/Rust executive level, and even KV Cache management, to make models less effort to process brackets and field names.

For those who are developing complex Agent frameworks (e.g. multi-intelligence collaboration, context finance Calling, code generation), two things can be highlighted: Give priority to format binding to API or reason engine: Do not pile too many brackets, fields and examples of format in Agent System Prompt. The context window should be left more for task dismantling, role definition and search results. Where you can transmit the type of Schema, you do not have to use natural language as a reminder.If the manufacturer or local engine used does not support restricted decodement for the time being, then back to the hint and retry mechanism.

Decoded Engines According to Schema DynamicsIf operations require large-scale simultaneous operations and Schema is relatively fixed, such as standardized voice-list extraction or fixed log JSON structures, priority may be given to Xgrammar co-operation vLM. Pre-compilation and caches can be more effective. If the scene is more dependent on jump decoded, prefixed or specific running time capabilities, then comparison of the reasoning engines, such as SSGLang.

JSON Output does not have to rely entirely on the hint and retry. Some of the output formats can now be written into Schema, Grammar and the reasoning engine configuration, allowing the system to be directly involved in the decodement.

Some examples of codes

vLLM defaults to a restricted decoded backend for Xgrammar, Outlines and LM-Format-Enforcer. Passing into the requesting body through OpenAI-Compatible Server.

from openai import OpenAI
from pydantic import BaseModel

class Sentiment(BaseModel):
    sentiment: str
    confidence: float
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-3B-Instruct",
    messages=[{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}],
    # 启用受限解码，后端自动生成基于 XGrammar 或 Outlines 的 FSM
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "sentiment_response",
            "schema": Sentiment.model_json_schema()
        }
    }
)
print(completion.choices.message.content)

Embedded JSON Formatting Output Example For complex embedded structures (e.g. progressive extrapolation of the thought chain), Pydantic is also the basis for building Schema. The bottom of the vLM PDA (lower push from the motivation) engine can interpret the retrogression and embedded relationship.

from openai import OpenAI
from pydantic import BaseModel

定义嵌套的底层模型
class Step(BaseModel):
    explanation: str
    output: str
定义顶层模型
class MathResponse(BaseModel):
    steps: list[Step]
    final_answer: str
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
completion = client.chat.completions.create(
    model="your-model-name",
    messages=,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "math_reasoning",
            "strict": True,
            "schema": MathResponse.model_json_schema()
        }
    }
)

SSGLang also provides OpenAI compatible API services, support {"structured_outputs": {"json":...}} Syntax: But its most powerful feature is the use of its own SSGLang front-end language (SGLang Frontend Language), which is able to integrate deep with the bottom-down compressed FSM and the Forward Jumping Decode, which produces zero-cost parallels.

Simple example of original regular rules to bind to

import sglang as sgl

@sgl.function
def extract_info(s, text):
    s += f"Extract the name from this text: {text}\n"
    s += "Name: "
    # 使用正则表达式强制约束输出格式，底层会自动跳过确定性前缀
    s += sgl.gen("name", regex=r"[A-Z][a-z]+ [A-Z][a-z]+")
state = extract_info.run(text="The user is John Doe.")
print(state["name"])

Embedded JSON constraints (based on Junp-Forward optimization) When generating embedded JSON, GSLang can enable structured output by entering Regex or directly into JSON Schema. Hard-coding strings (e. g. embedded parenthesis and static fields) that exist in Schema are detected at the bottom Synchronising folder "wand": { "wood": ) When the engine is written directly to KV Cache by jumping forward calculation.

import sglang as sgl

可以使用符合 JSON 结构的嵌套 Regex
nested_json_regex = r"""{
  "name": "[\w\s]+",
  "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",
  "wand": {
    "wood": "[\w\s]+",
    "core": "[\w\s]+",
    "length": \d+(.\d+)?
  }
}"""
@sgl.function
def generate_character(s):
    s += "Generate a Harry Potter character profile in JSON:\n"
    s += sgl.gen("character", regex=nested_json_regex)
SGLang 会使用压缩 FSM 加速这段深层嵌套的生成
state = generate_character.run()
print(state["character"])

If API services are purchased directly, reference is made to official documents, and Gemini, OpenAI, provides the API, largely based on the Schema of Pydantic to bind the output format. But there are differences in the parameter names of each family.