JoyAI-VL-Interaction: From Chat Back to Continuous Interaction

I've seen it lately. JoyAI-VL-InteractionAt first glance, it was just fun: it was fun to watch the stream, to judge when to talk, and to delegate complex issues that could not be dealt with at once to a backstage model or angent. Looks like a very complete real-time assistant. And then, I thought it was not the "can do so much" thing that was the most interesting thing to do, but the question that I'm used to being less suspicious was to put it back in front: Is it true that Chat is the place where the generation model is supposed to be?

The questions in this article can also be addressedFrom LLM to VLM, how language models achieve visual understanding、What should the world model look like and how should it be used?How the concept of a relatively close read together is developed in different contexts.

Chat is not a natural form of a model.

We're too familiar with Chat now. The user says, "Model," "User," "Model". It's like running longer: when you get an order, you finish the job and then you report back. For a long time, the language model seems to be the Chatbot. Round of inputs, round of outputs, end of the EOS token answer, as if this was the shape of the model itself.

But it's more like the rules that produce together with training data, not the model structure itself that forces us to do it. Self-regressive models do very simple things: give the front token, predict the next token. EOS token is just a special symbol in the sequence to tell the decodator "This part can stop." It is also natural if the scene does not need to be here, but rather to keep receiving audio, video, text and action.

The wheel, Chat, is an interface designed to cut the world apart in pieces. It does work, but it is not the only rational form. In real interaction, people listen, see, interrupt, supplement, silence, and while watching, decide whether or not to say the next sentence.

So I don't think that I'm a new species that's just emerging in space. It's more like releasing the generation model from the Chat product and re-placing it back into the stream of continuous events: input is no longer a single paragraph of user news, output is no longer a single paragraph of assistant answers, but a constant sense, judgment and action.

Interaction Model, where the hell is it?

Interaction Models of Thinking Machines Lab That is the same issue. They think that many of the AI systems today are stuck not in models that do nothing, but in a narrow interface. Users need to organize their intentions into complete input; models are often created in their own output, and new voice, images, interruptions and feedback are not available. We can certainly put a new state in the end of a tool call, or we can try to do something about the break in the model output, but these are always like patches on the existing Chat mechanism.

TML says that interation Models should be pre-existingly interactive and not hand over all critical capabilities to external scaffolding. The model is designed to handle both audio, video and text, and to receive information, responses and actions on a continuous basis in real-time collaboration. The multi-stream, micro-turn design they refer to is to rediscover the world by cutting the big rounds into smaller pieces of time, so that the model does not wait for the end of a full sentence or a full paragraph of the answer. It is still multiple, but the amount of time grain has dropped to hundreds of milliseconds; for humans, it is close enough to continuity.

What really changed here was not the numbers of delays, but the structure of collaboration. Users can interrupt, models can listen and think, images change models to change plans, and backstage angent can interact with the front desk model. Interaction Model does not necessarily have a dramatic structural change, but more changes are made to training objectives, data organization, input-output protocols and system patterns.

Back to the earlier generation model.

I would have preferred to interpret this direction as a return to a generation model, not a rebellion to a generation model. Early language models learn about continuity in the sequence. Then the model became a very useful dialogue assistant. The side effect is that we're also too tight on modeling and chatting.

Once you get into a multimodular real-time scene, the boundaries of Chat are revealed. Cameras are not fixed prompt, voice is not finished text, and user action does not necessarily appear with clear command. The key is whether the model can learn a new set of action token or behavioral labels: to listen, to remain silent, to send a reminder, to call a tool, to assign a back-office model.

And so, of course, Interaction Model is still a generator. It simply does not necessarily produce natural languages, but may also be timing, movement, control signals, or job descriptions for back-office systems. When you extend the output to these things, it remains a VLA problem, except that the actions do not necessarily occur on the arm of a machine or in a real-time interactive system.

What did JoyAI-VL-Interaction do?

JoyAI-VL-Interaction is a very intuitive example. And it was after I saw it that I started thinking about Interaction Model and moved the idea of writing this article; it changed the way I looked at Agent. It's an interactive model of 8B size, visual priority. The model is judged between three actions per second: silence, direct response, or assignment. Silence here is not a failure, but an act of training.

This is not like the normal video interpretation model. The traditional VLM is more concerned with what is in the video, "Please summarize this video." JoyAI-VL-Interaction is more like answering another question over and over again: "Is it worth interrupting now?" If it is worth it, should it be an immediate reminder or a long-range mission model that leaves complex issues to the backstage?

Its behaviour is derived from a cross-referenced sample of over 4 million times and is further optimized through enhanced learning. This data pattern is critical because the question of interaction is natural and time-consuming: a reminder is right, but five seconds late, and the interaction may have failed. At the system level, JoyAI-VL-Interaction is not just a model weight. It opens models, training formulations, time-matching data and deployable systems; service side includes reasoning, WebUI, ASR, TTS and backstage anent, and is compatible with vLM ecology.

The official also made a manual pairing of 58 real-event-driven visual interactions, depending on the quality of the response and the timing of the response. This assessment is quite correct, since the common failure of interactive models is not an error but a failure when it should occur. And, in general, this is a rather interesting new attempt.

GPT-Live-1: Voice side Interaction Mode

If JoyAI-VL-Interaction is putting Interaction Model in visual language stream, GPT-Live-1 is more like a voice-side version of the same problem. It is not a reverse of the logic of this paper, but a different piece: real-time interaction takes place not only in cameras and videos, but also in the most everyday human conversations. The visual side is more concerned with “what I see, whether I should step in now” and the voice side is more concerned with “what I hear, if I want to answer now”.

Traditional voice assistants are often cascade systems: they turn their voice into text first, then they give it to the language model to generate answers, and then they read them with TTS. This chain certainly works, but it naturally makes dialogue several separate stages. OpenAI describes GPT-Live as full-duplex speech-to-speech model, which can hear and speak at the same time, and be closer to the state of listening and adjusting and talking about the reaction of the other. The focus is not just on " support for voice input output " , but on interactive protocols starting from turn-based chat to a continuous flow of voice.

Interaction Model, the core behavior is not necessarily answerable. It may give a backchannel when the user stops, may remain silent while the other party is not finished, may interrupt lightly when a misunderstanding is imminent, or may delegate more complex tasks to the back-office system. Models generate more than sentences, including rotation, silence, confirmation, interruption and continued listening to these conversations. They are similar to the "quiet, direct, accredited" in JoyAI-VL-Interaction, but the modulation is a two-way voice from visual language.

This is why GPT-Live-1 is a suitable reference for Interaction Model. JoyAI-VL-Interaction judges “should we see and act now”, and GPT-Live-1 judges “should we say, hear, wait or delegate now”. The common denominator is not the multiplicity of the patterns per se, but the model begins to incorporate the timing of the interaction into its own behaviour and is no longer entirely left to the script of the external product.

I'd rather now consider Interaction Model as a set of emerging issues than a single architecture name. VL input, voice input, robot input, desktop input may all look different, but they are all trying to put the model back into a continuous event stream from the discrete Chat box. JoyAI-VL-Interaction gives me an example of a visual language side, and GPT-Live-1 reminds me that voice itself is also enough to constitute an important real-time interactive world.

Why is this direction worth looking at?

I don't want to say that JoyAI-VL-Interaction has solved the real-time assistant. 8B Models must still be limited in terms of knowledge, complex reasoning, long-tail requests and individualization. The TML Interaction Model is also in the research preview phase, with long sessions on context, deployment costs, secure boundaries and backstage collaborations, with many more issues to resolve.

But I still think this direction is worth looking at. It didn't invent a new structure that was completely different from the generation model, it just pulled us out of Chat's inertia. Visual and speech-based interactions are all illustrative: models do not necessarily wait for user-to-user speech, generation does not necessarily produce only natural languages, and EOS does not have to be an interactive boundary. We used to talk about real-time in the VLA, autopilot and robots, but people's interaction with Agent does not necessarily require such real-time. The existing Chat alignment and Agent design is not necessarily the last answer.

If Chat is the most successful interface for the basic models in the past few years, more models may then be directly located in a changing environment: cameras, live streaming, robots, desktops, and wherever humans are needed. They are not confronted with “what do I see” questions and answers, but “do you want to do something now or not”.

This is also an interesting place for JoyAI-VL-Interaction, GPT-Live-1 and other Interaction Models. JoyAI-VL-Interaction is a visual example, GPT-Live-1 is a voice-side example; they are not just trying to connect the model with an input-output channel, but rather trying to put interactive timing into model behavior. This seems to be a new concept, and finally, it goes back to a very simple question: If the generator model could have been followed by a continuous sequence, why do we have to keep it in the question-and-answer box of Chat?