NVIDIA Cosmos3: When AI Learns the Real World
Hey Sarah, welcome back to "Learn English with Podcasts"! Today I want to talk about something really cool. Have you ever wished your robot could understand the real world?
What do you mean? Like, not just follow commands, but actually know what's happening around it?
Exactly. That's what NVIDIA just built. It's called Cosmos 3, and it's a foundation model for what they call Physical AI.
Physical AI? That sounds like science fiction. What does it actually do?
It helps machines understand the physical world. Think about a home robot cleaning a table. It needs to see the dishes, understand physics, and plan its actions. Cosmos 3 does all of that in one model.
Wait, one model? Isn't that usually done with separate systems?
That's the breakthrough. Before, you needed a vision model to see, a world model to predict, and an action model to move. Cosmos 3 combines everything into a single architecture.
That's impressive. How does it work under the hood?
It uses something called Mixture-of-Transformers, or MoT. There are two towers inside. The Reasoner tower thinks about what it sees, and the Generator tower creates videos or actions.
So one part thinks, and the other part does?
Pretty much. The Reasoner is like the brain that interprets images, video, and text. The Generator then produces physically accurate simulations or robot movements.
And this is open source?
Yes, fully open. NVIDIA released the model weights, training code, and even datasets. Anyone can download and use it.
That's huge. What can people actually build with it?
A lot of things. Autonomous cars can use it to predict traffic. Warehouse robots can learn to avoid obstacles. It even generates realistic videos for training other AI systems.
So it's like a training ground for robots? Learning in a virtual world before going real?
Exactly. NVIDIA calls it a world foundation model. It creates simulated environments where robots can practice safely without breaking anything in the real world.
That makes sense. How big is this model?
There are two versions. Nano has 16 billion parameters and runs on a regular workstation GPU. Super has 64 billion and needs a data center.
64 billion? That's enormous.
It is. But here's the thing. Both versions are ranked number one on several benchmarks. They beat much bigger closed models.
Open source beating closed models? That's the real story here.
It really is. NVIDIA also created a new benchmark called HUE to test how well these models understand physics. Cosmos 3 leads there too.
What about real-world applications? Is anyone actually using it?
Yes. Robotics companies are post-training it for specific tasks. Autonomous driving teams use it to generate rare edge cases that are hard to capture in real life.
Like what kind of edge cases?
Imagine a self-driving car encountering a ball rolling into the street. You can't wait for that to happen naturally. Cosmos 3 can simulate thousands of those scenarios for training.
That's actually really smart. Safer than testing on real roads.
Much safer. And much faster. You can generate years of driving experience in hours.
You know what I find funny though?
What's that?
The model is called Cosmos. Like the universe. But it's really just learning about... tables and chairs and robots.
Ha! That's a good point. But I guess understanding a table and a cup is actually understanding the universe, in a way.
OK, that's deep. I'll give you that one.
And the best part? It's all free and open. So anyone, anywhere, can start building physical AI today.
Here's to open source and robots that finally understand the real world.
Cheers to that, Sarah.