Interactive explainer · vision-language-action

How a robot foundation model turns a sentence into motion

The recipe behind π0, GR00T N1.5 and the rest of the flow-matching VLA family — pulled apart into the four pieces that decide whether the thing actually runs on a robot. Every number below is a slider; every figure is live.

written for people who already have a robot to put this on

01

A slow model on a fast robot

A manipulation controller wants a new joint command every 20–30 ms. A vision-language model needs considerably longer than that to look at an image. Everything distinctive about the VLA recipe follows from refusing to accept that as a dealbreaker.

Three concessions make it fit. The policy emits a chunk of H future actions instead of one, so a single slow forward pass covers many control ticks. The network is split into a large vision-language trunk that runs once per decision and a small action expert that runs K times on top of its cached output. And the chunk is produced by flow matching, which generates continuous actions in a handful of steps rather than autoregressively decoding them one token at a time.

Those three moves interact inside one hard deadline. Drag the sliders until it breaks.

FIG 01The real-time constraintone control decision, in milliseconds
COMPUTE — backbone ×1, expert ×Kbackbone62 msBUDGET — n = 8 control ticks before the chunk is exhausted267 ms✓ 205 ms headroom — the next chunk lands in time
policy rate3.7Hz
headroom205ms
Runs once. The dominant fixed cost.
Runs K times. Why the expert is kept small.
Every design choice downstream is really a move inside this budget. Freezing and caching the backbone takes its cost off the K-multiplied path; rectified flow shrinks K; chunking buys a wider budget by committing to n steps of open-loop execution. Defaults here are plausible for a mid-size VLA on a single accelerator — set them to your own measurements.
02

Two towers, different clocks

The asymmetry is the whole design. The backbone carries the semantic work — what a mug is, what “on the tray” means — and is expensive, so it runs once and its tokens are cached. The action expert carries the motor work and is cheap, so it can afford to run K times against those cached tokens.

Cross-attention is the seam between them: action tokens attend into the frozen vision-language features, never the other way around. That direction matters. It means gradient pressure from action regression never reshapes the representation that grounds language — the failure mode where a policy quietly stops reading its instruction and just replays the most common motion for that scene.

FIG 02Architecture walkthroughclick any block

⟲ dashed span repeats K times — backbone output is cached

Pass
Action expert
runs K× per inference · ≈ 0.1–0.5 B params
innoised chunk A^τ + τ embeddingcross-attention into VL tokens + state token
outvelocity v_θ(A^τ, τ, c) ∈ (B, H, D_act)
  • Self-attention runs across the H action timesteps, so the chunk is generated as one coherent trajectory rather than H independent predictions.
  • τ enters through AdaLN modulation — the same weights behave differently at the noisy and clean ends of the path.
  • Deliberately small. It is the only part that runs K times per control decision, so its width sets your achievable control rate.
The split that defines the family: a large, slow, mostly-frozen vision-language trunk that runs once, and a small action expert that runs K times on top of its cached output. Shapes and parameter counts are representative of published configurations rather than any single checkpoint.
03

Generating the chunk

Take a demonstration chunk A¹ ∈ ℝ^(H×D), draw noise A⁰ ~ 𝒩(0, I), and connect them with a straight line. Flow matching trains a network to predict the velocity along that line, given how far along it you are.

A^τ = (1 − τ)·A⁰ + τ·A¹ — the probability path
u = A¹ − A⁰ — constant along it
ℒ(θ) = 𝔼 ‖ v_θ(A^τ, τ, c) − u ‖² — c = VL tokens + state

The target is constant, so there is no noise schedule to design and no per-step supervision to construct — one τ, one forward pass, one MSE. The sampling loop exists only at inference:

A ← A + (1/K)·v_θ(A, τ, c), τ : 0 → 1

Two comparisons are worth holding onto. Against discrete action tokens (RT-2, OpenVLA), this keeps actions continuous — no binning error, no vocabulary to tune — and produces all H×D numbers in K passes instead of decoding them one at a time. Against DDPM-style diffusion policies, the straight-line path is the point: the field is nearly constant, so Euler integration converges in single-digit steps where a curved diffusion path needs tens.

The curvature slider below is that argument, made adjustable. At zero curvature a single step is exact. Turn it up and watch how many steps you suddenly need.

FIG 03Flow-matching playgroundH = 16 steps × D = 6 dims
+2.3−2.3tt + 15
A^τ — current chunkA¹ — demonstration
τ0.00
step0/4
live err0.820
final err0.067
Euler steps from noise to action. Each one is a full pass through the action expert.
How far v_θ departs from a straight path. At 0, one Euler step is exact.
final error vs K4
Training never runs this loop — it regresses v_θ against the constant target A¹ − A⁰ in a single pass. The loop is an inference-time cost, and the curvature slider is what buys it: drive curvature to zero and one step is exact, which is the limit that rectified flows are engineered toward and the reason production policies get away with K in the single digits.
04

Committing to the future

A chunk is a promise about the next H timesteps made from a single observation. Executing all H is maximally efficient and completely blind. Executing one and replanning is maximally reactive and unaffordable. Everything real sits between: execute n steps, throw away the rest, replan.

The cost of that promise is staleness — by the time the last executed step of a chunk reaches the motors, the observation behind it is n + latency ticks old. Temporal ensembling softens the seam by averaging overlapping chunks instead of hard-switching, which trades a little reactivity for a lot of smoothness at the boundaries.

Drag the target around. The red tether shows where the target was when the currently-executing chunk was computed.

FIG 04The policy loopdrag the target
control 30 Hz · policy 3.8 Hzcontrol ticks @ 30 Hznow
tracking err0px
staleness0ms
replan267ms
worst react367ms
How many future steps the expert emits in one shot.
Replan every n steps. 50% of each chunk is thrown away.
Wall-clock cost of the backbone plus K expert passes.
Two clocks, not one. The controller wants an action every 33 ms; the policy can only answer every 267 ms. Chunking covers the gap by predicting ahead — and pays for it in staleness, which you can see as the red tether between where the target is now and where it was when the executing chunk was computed. Raise n and the arm goes smooth and deaf; lower it and the arm reacts but the policy runs hot.
05

Where it goes wrong

The interesting failures are rarely in the loss curve. In rough order of how much time they cost:

  1. 01
    Normalization statistics drift

    State and action stats are per-embodiment and computed over the training slice. Deploy with a different set — a rebuilt dataset, a changed joint order — and the policy emits confident, smoothly-denoised garbage. Nothing in the metrics catches it.

  2. 02
    The backbone stops listening

    Fine-tune the trunk unfrozen on a narrow task set and language grounding erodes. The policy still succeeds on the training tasks, which is exactly why it takes so long to notice. Test it by giving a wrong instruction for the scene and checking that behaviour actually changes.

  3. 03
    n tuned on the wrong axis

    Chunk length gets tuned for smooth-looking rollouts, which pushes n up, which makes the policy deaf to anything that moves. If your scene is static this never surfaces in evaluation and immediately surfaces in deployment.

  4. 04
    Chunk-boundary discontinuities

    Hard-switching between chunks puts a step change into the joint targets every n ticks. It shows up as an audible tick in the drivetrain long before it shows up in success rate. Ensembling or blending the overlap fixes it.

  5. 05
    Absolute vs delta action spaces

    Mixing embodiments that disagree about whether an action is a target pose or an increment produces a policy that works on one robot and drifts on another. Decide once, per dataset, and check it at ingest.

—

The papers behind this

This page is a synthesis, not a reproduction of any one system. Shapes, parameter counts and latencies are representative defaults chosen to make the tradeoffs legible — replace them with your own measurements before drawing conclusions.

  • Flow Matching for Generative ModelingLipman et al. · 2022
  • Flow Straight and Fast: Rectified FlowLiu et al. · 2022
  • Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT)Zhao et al. · 2023
  • Diffusion Policy: Visuomotor Policy Learning via Action DiffusionChi et al. · 2023
  • RT-2: Vision-Language-Action ModelsBrohan et al. · 2023
  • OpenVLA: An Open-Source Vision-Language-Action ModelKim et al. · 2024
  • π0: A Vision-Language-Action Flow Model for General Robot ControlBlack et al. · 2024
  • GR00T N1 / N1.5: An Open Foundation Model for Generalist Humanoid RobotsNVIDIA · 2025