Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Language models learn to control robots from web data

General-purpose language models are becoming robot controllers. Instead of training robotics-specific models on robot data, Waddle Labs and RoboCurve use models pretrained on web data and code, which already carry spatial and procedural knowledge. Code lets the model emit compact actions, and a harness of tools lets it generalise with a VLM in the loop. Latency and data remain bottlenecks, but the founders expect general-purpose robots within two years.

Takeaways

• General-purpose models can control robots. The RT-2 paper fine-tuned a language model pretrained on web text and images to output effector poses instead of English. Code as Policies let a coding agent call Python functions like pick up an object, achieving one-shot control without extra robot data.
• Web and computer-use data teach spatial reasoning. Han Ming says dragging a cursor to orbit a CAD object in Blender teaches reasoning about space and directions. Feeding computer-use and coding data into one model transfers better than robot-only data.
• Code is a compact way to express actions. Francois says a compact piece of code has lower Kolmogorov complexity than the equivalent action output. Voyager created tools on the fly and compressed experience into reusable tools, showing how agents can build skill libraries.
• In-context learning is cheap but saturates. Francois says it needs no SGD or FLOPs but caps after about 20 to 40 examples and a context window of roughly 50,000 of a 100,000-token training length. A company with infinite data like Tesla would not use it.
• Harnesses combine code with VLMs. Vincent says repetitive parts like unscrewing can be automated with code, and variation is built into the coder's policy graph. A VLM inside the loop detects objects or failures so the deterministic graph generalises.
• Strong language models make strong robots. The Platonic Representation Hypothesis holds that models converge on a shared mapping of the world, so Jay says one really strong model, regardless of architecture, would outperform weaker specific models. Han Ming cites computer-use data as evidence.

What comes next

• Jay predicts consensus among frontier labs that general-purpose robots, able to follow any natural-language instruction and do what a competent teenager could do by hand, will arrive within two years.
• He says Fable-class LLM latency is improving about 2x per month, so real-time control could come by the end of the year.
• Han Ming says Waddle Labs plans to get there in about two years but must consolidate Astra's first pass into faster skills.
• Francois compares this to sleep and the DAgger data aggregation framework, where new state-action-reward goes back into context and the system compresses skills.