Large language models had the internet. Robots never had an equivalent. Every hour of robot experience has traditionally come from a person teleoperating a real machine, which is slow, expensive and tied to one robot body in one lab. That is why the most important robotics story of this autumn is not a new humanoid but a change in where the training data comes from. At the Conference on Robot Learning (CoRL 2026) in Austin, NVIDIA presented GR00T 2, a robot foundation model pretrained on 402,730 hours of first-person video of people at work. Panasonic and its university partners presented T-Rex, which also starts from human video. And a new open dataset, HubEgo-100K, released more than 10,000 hours of skilled work filmed from the worker's point of view. Egocentric video for robot learning has moved from a research idea to the largest single ingredient in some of the most capable robot models.
What egocentric video for robot learning means
Egocentric video is footage recorded from the point of view of the person doing the task, usually with a head-mounted camera or phone and sometimes with wrist cameras or motion sensors. For robotics it is attractive because it shows what a robot needs to learn: hands approaching objects, tools being used, and the sequence of steps in a real task, filmed from roughly where a robot's own cameras would sit.
The catch is that human video contains no robot actions. A person's hand is not a gripper, and a video does not record joint angles or motor commands. Researchers therefore use human video in a few different ways: to learn visual representations of manipulation, to reconstruct 3D hand motion and retarget it to robot hands, and to co-train policies on human and robot data together. The HubEgo-100K paper summarises the state of the evidence: egocentric human video improves visual representations for manipulation and, when co-trained with a small amount of robot data, improves policy generalisation to new scenes, objects and tasks.
What the new evidence shows
GR00T 2: human video as the bulk of a robot foundation model
NVIDIA describes GR00T 2 as a "world-action model" that predicts future video and robot actions together, so that it learns how physical interactions unfold along with the actions that produce them (I covered the broader idea of models that predict the world in World models in 2026). Its pretraining mixture is the clearest picture yet of how a frontier robot model is fed:
- 402,730 hours of egocentric video of people at work, from NVIDIA's Human-1M dataset, of which 100,000 hours are openly available on Hugging Face as Human-1M-100K.
- 10,184 hours of demonstrations with a handheld UMI gripper across 2,897 tasks.
- 23,500 hours of robot teleoperation across parallel-gripper arms, dexterous hands and the Unitree G1 humanoid.
In other words, human video is more than ten times larger than all of the robot data combined. NVIDIA also measured whether it actually helps. It pretrained four models on none, 1%, 10% and 100% of the human video and evaluated them on 72 downstream tasks. Task progress on the real robot rose at every step, from 50% to 70%.
The transfer results are the striking part. In chemistry-lab tasks demonstrated only by people wearing gloves and head and wrist cameras, with no teleoperation of the task at all, a robot with two five-fingered hands lined a lab funnel with filter paper and wetted it in 65% of trials. On 50 tasks that the target robot had never practised, seen only in human video or other robots' data, 40 succeeded at least once zero-shot, and five demonstrations per task, under three minutes of robot data, raised success to 95%. NVIDIA also released supporting tools: a Video to Data workflow that turns human video into simulated robot demonstrations, and an open EgoHand model that reconstructs 3D hand motion, which NVIDIA says passed 40,000 Hugging Face downloads in the two weeks after launch. On the industrial side, NVIDIA says Foxconn is applying GR00T 2 to factory automation for server manufacturing and Siemens is evaluating it across industrial use cases.
T-Rex: human video first, touch second
Panasonic Holdings and Panasonic R&D Company of America, with UC Berkeley, NVIDIA, Stanford, Sapienza University of Rome and ItalAI, announced T-Rex on 5 November 2026. It is a tactile-reactive vision-language-action model for bimanual dexterous work, and it follows the same recipe of starting broad and narrowing down. T-Rex first learns visual and motion knowledge from 22,889 hours of human egocentric video, then learns contact-rich robot behaviour from about 100 hours (more than 7,700 episodes) of robot data with fingertip tactile signals, and is finally fine-tuned with roughly 100 demonstrations per task. Across 12 real tasks such as turning pages, transferring eggs and screwing in a light bulb, it reached an average success rate of 65%, against 35% for the best previous baseline; without tactile input it dropped to 42%. The lesson is that human video supplies breadth, but it does not replace the sensing that video cannot see, a point I explored in Tactile sensing for robots in 2026.
HubEgo-100K: skilled work, filmed in the wild
Most earlier egocentric datasets show daily life, kitchens or scripted tabletop tasks. Hub Labs' HubEgo-100K, described on arXiv in November 2026, targets skilled work instead. Its first release, v1.0, contains 10,124 hours of 1080p video with synchronised 100 Hz inertial data from a single head-mounted phone. It was recorded by 12,227 contributors in 15,548 environments, with 83% of hours in workplace settings, and ships as 72,754 episodes with machine-generated action and language annotations in LeRobot and MCAP formats under a CC BY 4.0 licence. The authors plan further releases towards 100,000 hours. Their design argument is worth noting: published data-scaling studies find that generalisation grows with the diversity of training environments and objects rather than with more demonstrations per environment, so the dataset is built for diversity first.
EgoVerse and World In Your Hands: what makes human data useful
Two earlier open datasets add the caveats. EgoVerse, a consortium led by Stanford, Georgia Tech, ETH Zürich and UC San Diego with industry partners including Meta Reality Labs Research and Scale AI, released 1,362 hours of human demonstrations across 1,965 tasks and 2,087 demonstrators, and replicated its experiments across several labs and three robot embodiments. Its headline finding is that policy performance generally improves with more human data, but effective scaling depends on alignment between the human data and the robot's learning objective. In one case, a robot whose demonstrations used a different strategy from the humans (propping a bag open with one gripper instead of opening it with two hands) saw co-training performance degrade. Scene diversity was the most reliable driver of generalisation.
World In Your Hands (WIYH), from TARS Robotics with academic partners, offers more than 1,000 hours of in-the-wild manipulation with millimetre-level 3D motion. Its results show that verified annotations give substantially stronger pre-training signals than unverified egocentric data, even when the unverified data has more samples. Quantity alone is not the answer.
Where human video for robot training breaks down
- The embodiment gap is real. Human hands, human strategies and human camera motion do not map one-to-one onto a robot. EgoVerse's strategy-mismatch case shows that the wrong human data can hurt.
- Video misses force and contact. T-Rex needed tactile robot data on top of 22,889 hours of human video for delicate tasks.
- Labels matter more than hours. WIYH found verified annotations beat larger unverified sets, and HubEgo-100K's annotations are machine-generated, which is scalable but should be validated for your task.
- Privacy and consent are part of the dataset. HubEgo-100K reports that contributors agreed to use for AI and robotics training, and that segments showing children, other people's faces or private text such as documents and licence plates were cut before release. WIYH's licence explicitly forbids re-identification, face recognition and surveillance uses.
- Lab numbers are not production numbers. Most results here come from the model builders' own evaluations, on their own task sets.
Robot training data: lessons for technology leaders
For leaders who sponsor robotics or physical AI programmes, the practical shift is that the data strategy now matters as much as the robot. Three implications stand out:
- Human video can cut the cost of the long tail. If skills seen only in human video can transfer, the expensive part of a deployment becomes the last few demonstrations on your own robot, not thousands of hours of teleoperation. GR00T 2's few-demonstration results point in that direction.
- Your own workers' expertise is a data asset, and a governance question. Workplace egocentric recording is exactly what HubEgo-100K is built from. Any internal programme needs consent, privacy filtering and clear licensing from day one.
- Openness is shifting the build-or-buy question. With open weights, open datasets under permissive licences and open tools for hand reconstruction, teams can fine-tune rather than build from scratch, as I argued about vision-language-action models.
A practical playbook for egocentric robot data
- Start from an open foundation model and open data. Evaluate a pretrained model on your tasks before you plan any data collection.
- Record a small pilot of your own task from the worker's view. Head and wrist cameras, consistent protocols, and the same strategy you want the robot to use.
- Check alignment before scaling. Compare how people and your robot do the task; if the strategies differ, fix the demonstration protocol first.
- Budget for verified labels. Spot-check machine annotations and treat label quality as a metric.
- Keep some robot data in the loop. Plan for a handful of robot demonstrations per task and, for contact-rich work, tactile or force signals.
- Write the privacy rules first. Consent, face and document filtering, retention and licence terms should be settled before the first hour is recorded.
- Measure the scaling curve yourself. Train with increasing shares of human data, as NVIDIA did, and keep the data only if your own task progress rises.
Closing thought
For years the honest answer to "when will robots generalise?" was "when we have enough data, and we don't know how to get it". This autumn suggests a partial answer: much of that data can come from people simply doing their jobs while a camera watches from their point of view. It will not remove the need for robot data, touch or careful evaluation, and it raises real questions about consent and privacy. But it changes the economics of teaching a robot a new skill, and that is the part technology leaders should start planning for now.
References
- GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation, NVIDIA GEAR Lab, project page, November 2026
- Practice Makes Precise: At CoRL, NVIDIA and Partners Reveal How Robots Rehearse Delicate Work in Simulation, Sasa Docca, NVIDIA Blog, 9 November 2026
- Panasonic HD Co-Develops a Robot Manipulation AI That Performs Dexterous Bimanual Tasks by Reacting Instantly to Fingertip Touch, Panasonic Holdings press release, 5 November 2026 (paper: T-Rex: Tactile-Reactive Dexterous Manipulation)
- HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning, Javad Amirian, Armin Kiani, Loïc Schaeffer, Tim Sprecher, Mohamed Taha and Karthik Yadati, Hub Labs, arXiv:2611.21938, November 2026
- EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World, Ryan Punamiya, Simar Kareer, Zeyi Liu et al., Stanford University, Georgia Tech, ETH Zürich, UC San Diego and partners, arXiv:2604.07607, 2026
- World In Your Hands: A Large-scale Ego-centric Dataset for Learning Robotic Manipulation In the Wild, Yupeng Zheng et al., TARS Robotics and partners, arXiv:2512.24310
