MineCLIP, Visual Signals, and Reward Design

Minecraft AGent's ultimate goal is often clear: milla cow, craft an iron pickaxe, dig a hole. Trouble in the middle. It is difficult for the environment to give valuable feedback before the mission is completed. RL knows the end, but doesn't know if that step is closer to the end.

The questions in this article can also be addressedHow Reward and Training close the loop in real Agent: from data governance to online RL、Actic RL: Why is training closed rings more important than training algorithms?How the concept of a relatively close read together is developed in different contexts.

When long-range missions are detached into skills, the problem is smaller, but it will not disappear.find a cow、harvest milk_bucket、place crafting_table The task is shorter than the full one and feedback is still needed. Handwritten for each skill, will soon become a bunch of rules of vulnerability. MineCLIP is here: train visual-linguistic models with video and subtitles from players to judge whether this recent image is like this skill description.

MineDojo The MineCLIP is proposed to cut in from this gap. It uses YouTube video clips and time to train video-language compatible mode. It's like the CLIP, just a short video.

Give Video Window $V_t$ and task text $G$Video encoder $\phi_V$ Output $v_t=\phi_V(V_t)$, text encoder $\phi_G$ Output $g=\phi_G(G)$I'm sorry. Reward head to subsynthetic cosine similarity, multiplied by a learning temperature:

$$ s(V_t,G)=\exp(\alpha)\left\langle \frac{\phi_V(V_t)}{|\phi_V(V_t)|}, \frac{\phi_G(G)}{|\phi_G(G)|} \right\rangle . $$

During the training, every video clip in the battling is subtitled. The positive sample is the same video-text pair, while the negative sample is from other text or video in the bat. InfoNCE:

$$ \mathcal{L}{v\rightarrow g} =-\frac{1}{B}\sum{i=1}^{B} \log \frac{\exp(s(V_i,G_i))} {\sum_{j=1}^{B}\exp(s(V_i,G_j))}. $$

After training,$s(V,G)$ Not just service search, but also as a soft judgement: is this observation consistent with the target description?

When entering RL, the smart body takes the nearest 16 frames at each step $V_t$I'm sorry. If the candidate text is $\mathcal{G}={G,G_1^-,\ldots,G_{N_T-1}^-}$MineCLIP first converts the similarity of the target text to softmax probability:

$$ P_{G,t}= \frac{\exp(s(V_t,G))} {\sum_{G'\in\mathcal{G}}\exp(s(V_t,G'))}. $$

The appendix to the paper discussed two ways to convert the target amount. The first one is direct reward:

$$ r_t=\max\left(P_{G,t}-\frac{1}{N_T},0\right). $$

$1/N_T$ It's a random guess baseline. Visual matching below the baseline is not rewarded to send the models themselves to the optimizers. The second one is delta reward:

$$ r_t=P_{G,t}-P_{G,t-1}. $$

This is more like a progress reward. It does not reward standing on the same thing, but it rewards the visual change closer to the goal. Direct is more effective for moving animal missions; static targets may simply allow angent to stare at the target, but forget to continue interacting.

Change is simple: traditional rare rewards are given only when the mission is completed. $1$ or $100$; MINECLIP gives a similarity of visual language at each time step. It eases the problem of exploration and saves a lot of handwritten work. But it's still proxy. Visually, it's like a Sheep, not the wool actually goes into the backpack.

Plan4MC It's a good combination of traditional RL and LLM Agent. It does not use MineCLIP as a reward, but it starts by tearing the Minecraft skills into three categories: Finding-kills, Manipulation-kills dig, kill, place, collect, Crafting-skills synthesis. The upper level produces skill graph graph, then the graphics are used to search for skill sequences; only the bottom skills are still RL-based.

When training Manipulation-skills, Plan4MC uses MineCLIP to train inspiric reward. Approach and MineDojo are close: take past 16 frames, score with current skill sample and 31 negative samples, and get the target proft max probability $p$, and then to:

$$ r_{\mathrm{CLIP}}=\max\left(p-\frac{1}{32},0\right). $$

Plan4MC also writes the border very well. MineCLIP rewarded is useful for some visually recapable skills, but does not cover all behaviours. Add distance and attack reward, log / cobblestone, iron ore / diamund, and depth reward. VLM provides some signals, but not a complete rewardback.

Visual rewards are also easy to drill into. Optimizers look for the state that makes the model feel like, not necessarily for the state that actually enables the environment to be completed. Angent may have aimed its perspective at an entity, creating a high-level image of similarity without correctly interacting. Visual signals are dense, but no stocks, formulations, durable tools and long-term causal chains are visible.

MineCLIP is a partial answer: it solves the problem of signal density in skills training, does not solve the whole of the program, state validation and long term credit accreditation. Plan4MC just showed it. Pre-trained visual-linguistic models can turn a target into a reward, but available anent has to do more, reward.

CLIP4MC Similar studies, with similar overall logic, could be used as a reference. It's not going to start here.

References: