Pantograph’s 4B Minecraft model learns goals from video

Pantograph has unveiled Pan, a 4B-parameter Minecraft model trained to follow goals by learning from internet-scale video rather than relying primarily on action-labeled datasets. The company says the approach improves generalization for long-horizon tasks and could eventually transfer to broader robotics work.

The setup is straightforward in concept: treat large video corpora as reinforcement learning trajectories that contain observations but not actions or rewards. Pantograph uses goal-conditioning during pretraining, taking later frames in a video as targets for earlier parts of the sequence — a form of hindsight relabeling. After that action-agnostic phase, the models are post-trained on a much smaller dataset with action labels to make them act in the environment.

For this project, Pantograph pretrained a family of models on about 500,000 hours of Minecraft gameplay video, then post-trained them on roughly 2,000 hours of contractor trajectories containing both video and actions. It evaluated the models across 104 environments, each defined by an initial Minecraft world and a goal world, with a single rendered goal image used as the prompt.

Pan-4B results against STEVE-1 and a VLA baseline

Pantograph compared Pan-4B with STEVE-1 and a VLA baseline initialized from Gemma 4, which the article describes as one of the strongest open-source language models with video input support. The company says Pan’s goal-conditioned pretraining led to stronger performance, especially on varied and out-of-distribution tasks.

A few of the reported results:

Recommended reading

Hassabis says STEM makes you 10x better at AI

• Walk forward to village: Pan-4B 85.7%, STEVE-1 16.5%, VLA 18.7%
• Small tower: Pan-4B 70.8%, STEVE-1 6.7%, VLA 14.9%
• Wool target: Pan-4B 99.4%, STEVE-1 1.0%, VLA 3.2%
• Add wood floor: Pan-4B 38.3%, STEVE-1 5.0%, VLA 4.1%
• Creeper hallway: Pan-4B 45.3%, STEVE-1 44.9%, VLA 21.0%
• Gauntlet: Pan-4B 51.2%, STEVE-1 21.4%, VLA 6.4%

The company also reports strong scaling effects inside its own model family. On building tasks, Pan-4B scored 34.7% with a 95% CI [32.0%, 37.4%], versus 26.9% for Pan-2B and 13.9% for Pan-200M. On a set of 30 mechanism-focused environments, Pan-4B reached 24.4% with a 95% CI [22.3%, 26.5%], compared with 15.1% for Pan-2B.

What Pan can and cannot do

Pantograph says the model can fight mobs, explore for specific objects, handle platforming, and build structures from goal images. It highlights a behavior it calls the “goal dance”, where the model reaches a target position and then oscillates to match the prompted image more precisely.

The limits are just as clear. Pan still struggles to recover from mistakes, sometimes searches for structures instead of building them, and has patchy understanding of Minecraft mechanisms. The company says the models learned some systems — including buckets, flint and steel, crops, and fishing — but not others such as levers, ender pearls, and animal taming. In combat, Pantograph says Pan-2B and Pan-4B can effectively fight creepers and zombies, but often fail against skeletons and do not use shields well.

Pantograph also points to examples of reward hacking in offline-trained models. In one case, it says the model reproduced a horse-riding viewpoint without actually riding the horse; in another, it reached the target scene after falling through a gap instead of building the intended bridge.

For now, Pan acts only inside Minecraft. Pantograph says it plans to scale the models over the coming months and train on a broader mix of video games, computer use, general real-world data, and robotics.