VLA Foundry: One Codebase From Language Model to Robot Policy
Toyota Research Institute released an open framework that trains language, vision-language and robot action models in one codebase. Its Qwen3-VL-based policy beat an older model by 23 points in simulation.
TL;DR
- Toyota Research Institute's VLA Foundry trains language, vision-language and robot action models in one open codebase, from pretraining to action fine-tuning [^1].
- Its policy built on Qwen3-VL beat the institute's earlier closed multi-task model by 23 points on average in simulation; real-robot results are still missing [^1].
Background
A vision-language-action model, or VLA, takes camera images and a task description and outputs robot movements. It is usually built in stages: a language model is trained, extended with an image encoder into a vision-language model, and then adapted to predict actions [^1]. The authors argue that most open-source VLA projects cover only that last stage and stitch together incompatible pretraining pipelines, even though data choices made early on affect how well the robot performs later [^1].
What happened
Researchers at Toyota Research Institute released VLA Foundry, an open-source framework with one data-loading and training stack for all three stages [^1]. Models, data pipelines and encoders are created by name from YAML configuration files, so swapping a vision encoder or a language backbone is a single command-line change, and pretrained backbones can be loaded from Hugging Face [^1]. Training builds on FSDP2 and AWS tooling, and the team benchmarked throughput on up to 128 GPUs across 16 nodes [^1]. The release also packages the open LBM simulation benchmark, with 49 table-top tasks for two-armed robots, as a Docker image, plus a dashboard for comparing policies statistically [^1].
To show the pipeline works end to end, the team trained Foundry-VLA-1.7B from scratch: a 1.2-billion-parameter language model trained on 1 trillion tokens of the open DCLM dataset, an added 86-million-parameter vision transformer, and a 325-million-parameter action head trained with flow matching to produce a sequence of robot actions [^1]. A second model, Foundry-Qwen3VLA-2.1B-MT, puts the same action head on top of the pretrained Qwen3-VL 2B [^1]. Both learned from simulated and real teleoperated demonstrations from the institute's earlier LBM work, covering 42 tasks in simulation and 361 in the real world; unlike that work, they did not use the open OXE dataset [^1].
On 16 simulation tasks seen during training, the Qwen-based model beat the earlier closed-source LBM multi-task policy by 23 percentage points on average, a statistically significant gap [^1]. A from-scratch variant trained only on simulation data was statistically on par with LBM, while the full from-scratch model was the weakest of the four compared [^1]. For the Qwen model, multi-task training beat single-task training and per-task fine-tuning helped further; for the from-scratch model, the multi-task and fine-tuned versions were statistically worse than single-task [^1]. Both showed some zero-shot success on 3 held-out tasks [^1].
Why it matters
The clearest signal is about the backbone. The stronger pretrained vision-language model made the better robot policy, which the authors read as consistent with the hypothesis that stronger backbones improve policy outcomes [^1]. That is the kind of question the framework is built to test, because pretraining and action training share one codebase, and the intermediate Foundry-LLM-1.2B and Foundry-VLM-1.3B checkpoints are released so others can change any stage [^1].
The limits are spelled out in the paper. All reported results come from simulation on a narrow slice of robot types, with no real-hardware numbers yet, and every experiment uses the flow-matching action head [^1]. The authors also do not yet say which data recipes work best across stages, and they do not address safety or failure detection [^1]. One result shows the gap between the two worlds: the model trained only on real-robot data reached almost 0% success in the simulator, which the authors expected because the simulation lies outside its training distribution [^1].
Practical example
Say Tomás runs a small robotics lab with a two-armed table-top station and wants to know whether a stronger vision-language backbone gives a better policy. With separate tools, the backbone and the action training would live in different codebases, and any difference in results could come from the plumbing [^1].
In VLA Foundry, Tomás trains one policy on the released Foundry-VLM-1.3B checkpoint with the paper's flow-matching action head, then swaps in Qwen3-VL 2B from Hugging Face with a configuration change [^1]. Both runs share the same training loop, data loader and preprocessing, so the backbone is the only thing that differs [^1].
Next, Tomás runs both policies in the Docker-packaged LBM simulator and follows the dashboard. Its statistical tool, STEP, lets Tomás stop an evaluation early or collect more rollouts than planned, which would amount to "p-hacking" with a standard test such as Barnard's [^1]. If the lab's runs look like the paper's, the Qwen-based policy wins by a wide margin [^1].
The catch comes at the real station. The paper reports no real-hardware numbers, and its real-data-only model barely worked in the simulator [^1]. So the simulation tells Tomás which backbone to try first on the bench, not how the arm will do there; that still takes real trials.
Related gear
We recommend this book because it walks through pretraining a language model from scratch, the first stage of the pipeline VLA Foundry extends to vision and robot actions.