The AI Replacing Front-End Engineers Can't Build the Simplest Panel in a Game

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Order

The questions in this article can also be addressedWhen we talk about AI Native Game, what are we talking about?、How the game industry is introduced AI AgentHow the concept of a relatively close read together is developed in different contexts.

This is a phase-out of my Internet-friendly internship, and I'm exploring how AI should get involved in the game development process.

AI is already a daily way to generate a web page. The same model gives it a complete visual copy, which gives it a panel in the game: a background, buttons, a reward panel. I've been doing this for two months, and I can't say it's working, and it's a long way from the web.

Why?

The game UI is more complicated? It's more like Web in the ceiling. The Corhen Game is the middle of the UI-writing game with HTML/CSS/JS, which is used by Minecraft, Civilization VII, Alan Wake 2; the format level is also not at all, and the new UXML and USS are written by Unity as labels and style sheets; the cocos and XGUI interface files are simpler JSON or variants. Unreal's UMG is really the most complex, but it also exposes a lot of standard interfaces that we can programmed to spell out.

The game UI needs an art design? If the starting point is planning, it is indeed a problem. But in the real division of labour, the visual drafts are a step that the GUI designers have already done. The rest is to translate it into UI engineering files in the engine, without beauty or creativity, with care and a lot of repetition. And this is exactly where AI should be stronger than anyone.

I can't do it. Neither the AI tools of the commercial engine nor the internal programs that the teams are doing, any one of them can say that they have solved the problem and then go out and get it used by everyone.

Before the text is expanded, two of the texts that will be used throughout the text will be here:

• The ease of the Internet may be the best company to do UI workflow AI research, and the least suitable company to actually do it. One company has a combination of Unity, Unreal and a number of self-study engines, and almost all the games in the market UI you can find the team. This means that the problem of commonality will be exposed more quickly. But the more technology is routed, the less end of fit work, the harder it is to get to the point.
• Before you do an AI Agent, the most important question is the target. Is it a Copilot for the people now, or is it Agent to take over a post or a process? These two objectives sound like a single step away, and may actually be completely different technical routes and completely different costs.

What have I done lately?

UI Agent tried to solve the problem

The game UI production chain is not short. Plan for demand, UX gives interactive programs, GUI output visual design, UIP builds projects in engines, VX adds effects and animations, and finally program access to game logic. What UI Agent wants to do is not add a new assistant to it, but to shorten the production chain across the job.

The ideal form is today's Coding Act: Gives a textual demand and gets a product that can run. But every link in the UI production chain is not suitable for AI. The aesthetic and design intent still require judgement, and AI is better suited to take over the later work. So we put the boundary at UIP: enter the finalized draft, output the UI Project and the Compass Code that can enter the project.

Before developing such systems, it is necessary to clearly enter and export them. The input here is a Figma or PSD design, with a small amount of planning information; the output is a complete UI project file and the corresponding program code.

Put input and output together, and the problem is clear. The design describes "what it looks like," and the UI Project document also says "How it works." Which layers form a button, press what status they switch to, where the data from the incentive list comes from, who stretches after the resolution changes, who stays, and which information does not exist in the static design.

This is also why it was difficult to establish an automation programme in the past. The output of the certainty script can only come from input, information not found in the design, and the script cannot be added to it. The requirement for designers to add agreed fields to the name of the layer would alleviate some of the problems, but at the same time, the new work would be pushed to the designer.

The generation model brings another source of information: a priori accumulation in pre-training. Models see a row of equal distances and identical structures, and can be assumed to be a list; and a small fork at the top right corner can be presumed to be a closed button. It is still guessing, but this speculation is for the first time making the matter theoretical: without increasing the workload of the designer, it can reduce the manpower in the latter part.

Stage 1: Do not let the model write the final file

The idea was straightforward: since there was no difference between the UI project and the Web UI project, let AI write it like a web page. We're writing in Skill, telling Agent how to analyze PSD, export cut-ups, initialize projects, generate prefab and perform validations. After reading the manual, Agent calls in a dozen command line tools. This version is intuitive and does generate documents. The problem is that most of the documents generated cannot be opened. A whole page of UI works is usually dozens of nodes, thousands of lines, JSON. Each node has its own list of transforms, anchors, offsets, dimensions and components. In cases where the model has hardly seen such a format, it must have miswritten certain fields and then spent a great deal of time on changing the format. In the initial test, approximately 10 minutes were required for one generation, almost half of which was in the process of re-engineering documents.

Then let's just stop having the model write the final file. We added a smaller semantic IR, so that the model would only describe "It's a button" "The labels are here" "These elements belong to the same group" and coordinates converted, fuid, component lists and type maps to the definitive compiler. The format error thus disappears, and IR becomes a generated record. People can check and modify it directly without facing thousands of line target documents.

In white, it is a more advanced language (DSL) that allows models to write advanced languages, with the rest to be handed over to the compiler. Advanced languages are better taught than lower languages, as are models for people.

Stage 2: Turn prototype into engineering pipe

IR and compiler solve "files are right," but how does the real script fit into the system and how does the result be verified?

Stage2 did not replace the technical route ahead, but replaced the existing prototype with a pipe that would handle real inputs, generate real work and accept real validation.

The first paragraph is definitive pre-treatment. System solvers PSD/PSB and smart objects, extract the layer trees, position dimensions, text, hybrid mode and transparency, generate synthetic and cut-off resources, and prediscover the replicating clusters and large sections of the background with geometry.

The second paragraph is the model. It makes only two semantic judgements: first, to determine whether a set of layers should be preserved, unopened or merged, each of which controls corresponds; then to organize these elements into father-son structures and extrapolate anchor points. The results are written into the list of elements and the structure tree, one of them. .md The document's a little bit of a mix. .yaml The file is composed of IR. A person can check it directly, and the model does not need to write thousands of lines of target files.

The third paragraph is a definitive compiler. IR after validation, the compiler is responsible for coordinates conversion, anchor calculation, type mapping and assembly of components, which eventually produces complete .uiprefabI'm sorry. Of course, we actually kept two compilations in the real project, a purely script-based tool and a project assembly based on the Function Calling, and then we talked.

Finally, it's validation. Static checks cover issues such as missing resources, nodal naming and structural errors, which are then handed over to the real editor to load the document, screenshot and read the log. JSON does not legally represent that the project is really working, and the visual reduction of the editor is also an important feedback signal.

At this point, the system has been able to do more than generate a document. It has a clear input contract, an intermediate layer that can be read by a person and model, a back end where results can be repeated, and a mechanism for self-certification. It runs the draft in its entirety, achieves the engineering closure of the existing route and exposes new problems.

Stage 3: Turn the process itself into code.

Stage3 has run through the pipe, but it's slow and unstable. Logs are full of errors, repairs and retests; changes in Skill's hints may cause the whole process to collapse in unexpected places.

Where's the source?

The process is actually a manual. The order of the call of the ten steps, what to do after failure, when to try again, when to give up, are all written in a manual for models. The manual can only tell the model what to do and cannot be forced to implement it sequentially. The consequences are twofold: irrecoverable, with the same draft running twice, with different paths and different outcomes between them; failure without attribution. The transmission timeout, model output rejected by the calibration, cut-chart resources missing, editor unable to open, many errors without clear processing paths, and models can only be repeated once in many cases.

So Stage3 does a simple thing: move the process from the natural language into the code and build a workflow system.

We've made a map of the entire generation, 54 nodes, of which only 10 are actually called. Each node has a clear input, output and reversible action, and the process can only advance along the previously stated side. The model can still be judged, but it cannot change the order of implementation at will or invent a treatment after failure. Each action contains clear semantics, such as applicant worker access, failure of external service calls, error of function request, error of assembly.

Re-test and re-repair need to be treated separately.

The transmission layer fails, such as timeout, disconnection, incomplete response or JSON cannot resolve, and the system is retrying at the current node. If the model output returns in its entirety, but is not checked, the system will take the inspection report back to the duty node according to the wrong type, which is the back-up. The checker is no longer solely responsible for declaring failure, but also for telling the process what is wrong and where it should be repaired. (At the time being, the checker is a script that returns to the real and false: it can stop the error, but does not tell the follow-up process what to do.)

Run status will keep the disk log running, and checkpoint will remain at each node. Logs save a full run track, and checkpoint supports recovery and recurrence from an error node without running from the header at each time. These changes make it easier for UI Agent to operate and to be iterative.

But stability and utility are two things. This process can reliably end a draft from scratch at the cost of having it done from scratch every time; in real projects, the interfaces that are built from the beginning are only a few.

Stage 4: UI in Real Project, not every time you build it.

The UIs in mature projects rarely start from scratch. The longer a project runs, the more changes are made using the interface already in place. "Replace the original interface with this design and change it to a new one" is more common than that. Stage4 therefore no longer asked how to generate an interface, but how to start processing existing projects.

This was done by introducing a declaration document, which we call the blueprint. It states the following categories:

• Reference: Project documents as skeletons
• Source of output: Which layer of path is in the draft design for each redo
• Target binding: which node of the output is loaded on the reference and which assembly method is used
• Reservations and exclusions: which nodes must be retained as they are and which draft paths do not participate

Runs when the system reproduces the reference, recreates only the contents within the blueprint statement and then reloads them back to the given location. The part that needs to be modified is usually less than half the entire page and the nodes, layouts and logical interfaces in the existing project can continue to be used.

The assembly is made on a white list basis. Each of the five assemblers allows changes to only a small number of clearly listed fields. For example, the list assembly can only replace the XML reference, and the hang-up assembly can only modify the XControl path and cannot touch the position, rotation and scaling of the mount itself. Even if the model is miscalculated, its potential for destruction is limited to the blueprint statement. This is the same idea as the previous one, which bound the final document with the compiler.

The blueprint is much more stable. The four recent blueprints have crossed many authentic designs, all running through the entire process, taking time less than the one that was generated from the beginning (the blueprint generation took about one minute, and the animations took a little longer, less than the new 3-10 minutes). This is not a model that's getting stronger. We're giving it more knowledge, just making it do less.

This transaction is not cost-effective, depending on how many times the blueprint can be reused. Creating blueprints also entails costs (with automatic creation but not good enough) and one project and one set of norms, none of which can be taken away.

Let's talk about our understanding of the problem from the differences between different programmes

How to reuse control: Tasks should not be transferred

An important difference between the game UI and WebUI is how the control is reused. Web projects usually develop interfaces and logic simultaneously, and models can generate both directly. The video assets and program logic of the UI game are performed by different positions, so we will seal the high frequency structures of buttons, lists, bullet windows and so forth into prefab to reduce the cost of subsequent re-laying and access.

The system was originally designed for people. It takes a few days to get acquainted with the template warehouse of a project, but the model has difficulty obtaining the same knowledge. Open WebUI is very popular, game UI engineering is very small, and self-study engine data is less likely to enter model training. The more well-encaped and easily used the past, the more knowledge the model needs to add to the project.

We want to use the zero-shot panduration of the generation language model to solve UIP problems, which is not trained in the data at all, and which is certainly much more difficult than doing WebUI.

So how do we solve this problem?

1. Do large-scale multimodular pre-training to allow models to learn directly about engines and controls
2. Create a master-speech knowledge base to make project knowledge available to models at running
3. Search for reusable controls from inventory assets by multi-modular search
4. In Figma or PSD, indicate the control type earlier

Large-scale pre-training is the most direct, but also the highest cost. Only the developers of commercial engines such as Unity, Unreal may have collected sufficient data, and small technology houses and self-study engines are hardly in a position to follow this path. Even commercial engines have not yet solved the problem of UI generation with pre-training.

The knowledge base is a more realistic approach in the short term. The model does not know what controls are in the project and provides it with instructions, conditions of use and interfaces. We have done it, and we can solve some of the problems. The problem is that the number of prefabs in the project may be very large, and some of the knowledge must be combined with pictures to understand, and all of it would soon get out of control. Visual retrieval can be added to this: the model does not need to determine the type of control accurately, but simply to find assets that look and structure close to each other from the existing interface before reading the corresponding project information.

The last thing I'd like to see was a front-poster. It is usually able to produce the best results, as the designer has already completed the semantics missing in the system in advance. But it didn't eliminate jobs, it just transferred a part of UIP's work to the GUI. LLM has been in the history of similar layer naming, structuring and control labelling programs many times before their appearance, often ending up in the Demo phase. Evaluation of UI automation cannot be based solely on the generation of results, but also on whether the whole line has been reduced.

I've been looking around for the other team's options. The project team's procedures are largely based on a knowledge base: the small amount of exposure in the training language supports the zero-shot and injects some of the norms within the project (i.e. SPEC). AI Lab and XGUI, in turn, rely on the interventions of labeling and pre-stratification down LLM in return for faster and more steady generation.

The more people close to the project are willing to feed their knowledge to the model, the more people far from the project are inclined to block uncertainty before the model, which is an interesting discovery.

File format determines the technical route and how Agent interacts with it

As mentioned earlier, in addition to the definitive compiler, we have also retained a set of realizations based on the Function Calling. Agent no longer generates complete target files, but instead calls a set of tools to build and modify UIs over time. Why do you keep two routes in the same project? This question starts with the UI Project file itself.

The marketable UI project formats can be broadly divided into three categories:

1. Sequenced text represented by Unity UGUI..prefab、.unity、.csd and .uiprefab All of them belong to this route. The project is modified by the editor and re-sequenced into long text similar to that of JSON or YamL.
2. The statement format is represented by WebUI and Unity UI Toolkit. Structures, styles and renderings are relatively separate, and documents originally considered manual, version management and branch consolidation.
3. In Unreal .uasset is the binary asset represented. The model cannot read and write directly, but only operates through the editor and the interface to which it is exposed.

The article title says AI can write well for WebUI, but it's not good for UI, and it's also about the file format itself. UGUI prefab is also text, but models are difficult to modify directly. Because these documents are only the result of the serialization of the editor, they are never prepared for handwritten writing. A field or reference is wrong, and the entire document may not be loaded. And here, the title party in the title needs to take it back a little: the text format is not the same as the model.

The three formats naturally lead to three automated routes. UGUI, which is a sort of serialized text that is readable but not suitable for direct editing, is more suitable for adding an IR or DSL to produce the final file by a definitive compiler; WebUI can directly use the code capabilities and training language available in the model; face-to-face .uasset Such binary assets can only be implemented gradually by the editor interface if Agent develops a plan for their modification.

But it's only half the problem. The progress of Coding Age in the last six months has been made not only by the stronger underlying models, but also by the cheap and intensive feedback that code ownership naturally takes place. The web page contains a syntax check, running error, DOM status and browser screenshots, and the model can be quickly updated to understand what happened. The UI compiler usually only tells you whether the document is legal, but it is difficult to judge whether the control type is correct, the structure is reasonable and the visual reduction is met. These feedbacks need to be designed separately.

The construction of the UI project by nodes with tools may lead to more rounds of decision-making and token consumption (the programmmatic tool Calling will mitigate part of it), but it is easier to design a good set of tools than to compile them. A good set of tools is not just a tool, but a world where models can be known and reached. For AI, learning tools on zero-shot are easier to learn than learning DSL. Based on tools to connect to UI projects, the landing threshold is lower than the compiler route.

Because of the company's huge use of the project .csd So when you have a serialized text UI project, we can find that you have basically chosen the DSL idea, so that LLM can generate JSON or other forms of IR and then produce the final result through the compiler. G85 has an interesting attempt: to get Agent to put the UI structure out of Figma first, as the only credible source, and to export still away from the compiler, but it does work better. I guess the reason is that Figma's interface is more mature than the MCP of the self-study editor, and the model knows what's going on faster than it does with every change. That is also the reason I prefer the tool route, although only one case is not conclusive.

From running back to Benchmark: How does the system know this is better?

Tools exposed UI projects to Agent. But how much can it see, and then can it be judged that it did wrong?

This depends on the feedback system. The visual similarity can only answer the final image. The UI project also requires checking node structure, resource references and control types, and entering the editor is also dependent on loading logs and interactions. Feedback needs to indicate where the error occurred and which nodes should be repaired. The feedback in the previous section refers to this.

Run-time feedback is used to correct the current generation, and Benchmark is using the same set of checks to compare different versions with a representative set of cases. Changes in the hints, tools or workflows should allow the system to re-run, if there is progress, if returns are introduced, and if errors are moved from one layer to another. Operating and assessment environments are best shared. Feedback was originally the same as that of Benchmark, the same engineering input.

Only run-time feedback and Benchmark are not entirely sufficient. Anthropic. C Compiler ExperimentUsing test sets, CI and GCC oracle to drive Agent iterative, Linux can eventually be compiled, but the code is not of sufficient quality. Feedback can move the system towards a functional target, but does not automatically define maintenanceability and human taste.

As models continue to grow stronger, the injection of generic knowledge and fixed processes will decrease, while project specifications, component habits and human judgement will not automatically enter the model. For businesses that need long-term maintenance, multi-person collaboration and steady access, the whole of Vibe remains unrealistic. For the AI Agent system developed for games UI, both forward to hand over project restraints and human tastes to the model, back to run and Benchmark to check whether it is actually in compliance; what is more easily available in the absence of any side is Demo.

In this section we emphasize that multi-source feedback should be part of the feedback signal, except for the ultimate visual effects. The mature project develops its own habits and norms, which need to be used as a validation feedback and as a source of knowledge. There are also options to spell UIs with only final visual effects and cut maps. The image is right, but the draft, the interactive draft, the project code is all thrown out, and can be seen in the editor.

Multistate components, fonts, action and programs

Fonts are a small number of information that is suitable for completion from the side of the draft. Designers would have chosen fonts that would have bound them to the project font library or to a unified Text Steel, which would have been more reliable than Agent based on pixel guessing. Font information needs to be brought out from Figma or PSD, which is a few things that do not add much to the work of designers, who would have chosen fonts.

Multi-state components and action effects are more difficult to address. The static draft does not have a time axis and does not tell the system how to switch between suspension, pressing and disable, and the information does not exist in itself. For simple multi-state components, this can be achieved from the point of view of the re-use of the control, but the complex multi-state remains manual.

VX is a more complex issue, and key frame animations of properties require simple interactive drafts that are initially generated, and few UI Shaders can be reused. But for complex scenes and particles, Live2D, and sequence frames, AI is virtually impossible to do directly. Some jobs are not duplicated work, but art, and we can only stop here.

The development of UI-related codes is a stand-alone issue, but AI writing codes are more powerful than completing UI components. Like traditional program development, the UI program code needs to solve two problems. The way to inject project norms and tastes, which used to flow only between people, SDD is necessary here, and Figma and UI projects are to be added to the context. The second is how to get the code written by AI to be validated more quickly, which requires better automation of the QA.

UI developed a long Pipeline, which contains a lot of aesthetic and internal norms, and internal processes are much more complex to automate than a web page in Vibe Coding.

Worklow or Age: UI Age is what it should be.

The difference between WorkFlow and Age is how much decision-making is left to models and how much is left as scripts. When the output is stable, the steps to be implemented and the way to check are clear, the Workwork can write the project knowledge and the handling of failures directly into the code; it is only when the target is clear, but the path cannot be listed in advance that Agent needs to choose the tool, observe the results and continue to explore. UI produces both jobs, so both sides have to do it.

There is no need to do a set of roles according to the jobs of Planner, GUI, UIP, and Program. The human division of labour is based on skills and energy constraints, and moving the organizational structure to the model will only reintroduce context exchange and communication costs. The expansion of Agent should be only a task, not a role created to imitate human functions.

A more realistic form is to fix pre-processing, compilation, assembly and validation with the Workflow, leaving semantic judgement, asset retrieval and failure repair to Agent; manual responsibility is to supplement design intent and to identify results that cannot be judged by rules and feedback. It could be used in a UIP-based way, with a complete process gradually being taken over within the system to the extent that Benchmark has proven reliable. The part that relies on project experience and human taste also needs to preserve human collaboration.

In the discussions that preceded, we were thinking about the positioning of our products in the context of the existing UI workflow. The real organizational structure will be useful in the development of Agent, and the emergence of AI Agent will be counterproductive to the organizational structure itself. The game industry itself does not have a clear division of labour like the traditional mobile Internet, and in some small studios UX and GUI are normal and the process often combines the tasks of the fusion. UI Agent is much faster in automation than UIP, and when the support tools are done well enough, the entire UI work chain may need only two jobs to plan and develop, the former to handle ideas and achieve visual effects (in combination with the current AIGAgent), and the latter to think about UI structures and program logic, which may be the organizational structure of the future.

Take Home Messages

• The entire semantic divide between the draft and the UI project Control type, interactive status, data binding, self-adaptation rules, are not in the input. This information is not supplemented by a definitive script, which requires the designer to indicate in advance that the activity is being pushed upstream. The generation model gives this missing part the opportunity to be filled for the first time. It remains guessing, but it is the first solution that does not increase the workload of others.

• Don't let the model touch something that it's not good at. Do not write thousands of lines, JSON, rewriting an IR; do not use a manual string to use a chart from a statement; do not always generate from scratch, and change to a part that moves only the blueprints on the reference. Each time it's the same thing: narrow the action space of the model and give the identified part to the certainty code. Provides as much information as possible for Agent as is available instead of relying entirely on visual effects.

• The format determines the route and the feedback determines the ceiling. Serialized text is suitable for IR-adder compilers, with declaration formats that can be generated directly, and binary assets can only be used as tools. But it can only be produced in half. A large part of the progress made in the six months is due to the fact that the code has a cheap and intense feedback, and UI does not. Building better feedback signals is a task that needs to be considered in the future, and this route is more universal and reliable than compilation.

• What we should do today is Copilot, not take over. Workflow fixes, compiles, assembles and validates, Agent is responsible for semantic judgement, asset retrieval and failure repair, and human design intent, and human collaboration is the direction that should be focused at this time. The total autonomy of the work stream of the stock project UI is still far away.

PPT:Downloads design2ui-talk-sanitized.pptx