Nvidia unveils the Cosmos 3 simulation platform designed to serve as the foundational operating system for the era of embodied AI.

In 2026, the large language model industry is undergoing an unprecedented perceptual shift. The scaling laws that have dominated the purely digital realm are facing a harsh reckoning as they attempt to cross into the physical world, constrained by the laws of physics, spatial limitations, and long-tail data. The core contradiction within the AI sector is becoming increasingly stark: the exponential surge in cloud computing power stands in sharp contrast to the stumbling pace of robot deployment at the edge.

This divide reached its peak on the final day of RSS 2026 in Sydney. During the “Robot World Models” workshop, the traditional robotics community’s reverence for physical causality collided head-on with the AI industry’s faith in scale effects. At the epicenter of this paradigm clash, NVIDIA’s Physical AI expert Max Li presented a deep dive into their newly released flagship product: Cosmos 3.

Cosmos 3 is no longer just a more realistic video generator. It is the world’s first world foundation model built on a single Transformer architecture that fully unifies text, vision, audio, and action modalities. If the past two years of embodied AI resembled a kit-building era—where developers struggled to piece together vision models, language models, and control algorithms like Lego blocks—Cosmos 3 marks the dawn of an integrated-platform era.

During his presentation, Max Li signaled a clear strategic pivot: the next phase of embodied AI will belong to the edge, not data centers. By migrating AGI brains to edge devices like Jetson for real-time inference, NVIDIA aims to eliminate the burden of robots carrying server-grade hardware behind them. This reflects NVIDIA’s ultimate ambition as a computing powerhouse: to become not just a hardware monopolist, but the Android operating system of the embodied AI era. Through standardized protocols for “brains and limbs,” NVIDIA seeks to unify the fragmented landscape of robotic forms, enabling them to evolve on a common logical foundation.

Below is the edited transcript of Max Li’s speech, adapted from the original English presentation while preserving all original meaning and technical details.

Max Li: Thank you to the moderator for the introduction. It is an honor to share our latest research here today: Cosmos 3, a multimodal world foundation model designed for physical AI.

One of the greatest challenges in physical AI right now is data. Compared to computer vision agents or large language models, acquiring physical-world data is heavily restricted by linear constraints of time and space. We cannot simply scale physical data the way we scale compute. This is why we believe teleoperation or synthetic data alone cannot solve the problem; we need more fundamental technology. That is why we introduced the concept of a World Foundation Model, aiming to serve as the true cornerstone for many physical AI applications.

We also spent considerable time defining what constitutes a qualified world foundation model, particularly from an agent’s perspective. What capabilities must a model possess to interact with the world in a foundational way? We identified three critical abilities:

First, understanding the world. As previous speaker Laura noted, if you want to retrieve food from a microwave, the model must first comprehend the current environmental state before it can act. Second, simulating outcomes. Agents must be able to predict the results of specific actions. In the physical world, trial-and-error carries high costs; breaking a glass cup is irreversible. In simulation, however, you can break things repeatedly at zero cost. Therefore, the ability to simulate future trajectories is crucial. Third, taking actions. This is the most familiar aspect for robotics researchers: materializing intent into the physical world.

Building on Laura’s framework, we integrated all modalities into a single model: text, images, video, audio, and actions, enabling generation across all of them. Let us look at what Cosmos can actually do.

For understanding the real world, given a video observation of a robot performing a task, the model operates similarly to a Vision-Language Model (VLM), reasoning over what happened between specific timestamps or predicting observational outcomes. We consider world understanding the bedrock of any model; without it, action is impossible.

Generation capability is equally vital. We jointly model audio, video, and action. In this example, audio-video generation serves as a validation mechanism, confirming that the model correctly understands future outcomes under specific control conditions. Incorporating audio conditioning is particularly interesting because physical interactions often provide cues through sound, such as collisions or friction. This is why we included audio in the foundation model training.

Regarding world simulation, here is an example of Cosmos simulating complex human-environment interactions. The bottom-left shows head-mounted camera trajectory, and the bottom-right shows hand pose. The model simulates the entire process of heating an object and placing it on a table.

We firmly believe that ego-centric data is the most effective pathway for foundation models to acquire deep knowledge of physical interaction. We are collaborating with numerous teams and open-source communities to inject this knowledge into the model.

We have also implemented inverse dynamics. This goes beyond predicting future visual observations from given actions; using multi-view observations, it can reverse-engineer the underlying actions driving those visual changes. If you possess massive volumes of unlabeled video data, this serves as an excellent data mining tool. In this demonstration, I show how predicted actions can drive a robot’s URDF model to move forward.

Once the model sufficiently captures all possible physical outcomes—dynamics, video evolution, textual understanding—it can be directly converted into a policy model to execute actions in the real world. While the current demos may not be the most visually stunning, the core logic is straightforward: if the model deeply models states and their transition mechanisms, it naturally possesses policy capabilities, receiving specific instructions and executing them physically.

Upon release, we launched several ready-to-use versions. The server-side Super series offers massive scale but is less friendly for real-time robotic tasks. Consequently, we dedicated significant effort to developing Cosmos Edge. We found it achieves real-time inference on edge devices like Jetson without requiring server-grade GPUs, which should prove highly practical for developers in this field.

Since launch, a key advancement involves collaboration with the Pi team to demonstrate policy evaluation. We are not claiming this fully replaces real-world evaluation yet, but we have successfully offloaded a portion of assessment work offline. Policies can interact with Cosmos, leveraging its inverse dynamics for novel evaluations. The top of the screen shows real-world policy performance; during online policy evaluation, the model decided against placing the ball in the trash bin, resulting in failure—a scenario our current technology can accurately simulate and assess.

To summarize the preceding points: the model encompasses VLM capabilities, particle tracking, motion planning (“Move it”), multimodal input/output (text, image, video, audio), global policies, forward dynamics models, and inverse dynamics models. We configure different input-output combinations to unify these functions within a single model.

Architecturally, this is a Mixture-of-Experts (MoE) design. For attention masks, we employ causal self-attention for world understanding (reasoning) and bidirectional attention for generation.

Given this is a robotics conference, I must address how we unify action vectors across different embodiments. We support various form factors: autonomous vehicles (AVs), camera motion (first-person movement), single-arm robots, dual-arm robots, and humanoids. We use heuristic rules to couple different action vectors, allowing them to share semantic space as much as possible.

In Inverse Dynamics Mode, given video, the model predicts the underlying actions that caused it.

A persistent challenge when processing different modalities is handling positional embeddings. Here is our solution for unification:

Video: We apply incremental indexing across all dimensions P, H, W (defined as a 3D space), programming along horizontal and weight axes.

Audio & Action: Lacking height and width, we rely solely on the temporal axis.

This ensures all deep generation remains compatible with the underlying loops of large language models, preserving their general foundational knowledge.

A key difficulty lies in synchronization: language tokens lack a time concept, yet video, audio, and action require temporal alignment. Our approach uses 24 FPS as the baseline. When inputting videos at different frame rates, we recalculate their indices based on 24 FPS, aligning everything to a true timeline rather than introducing drift.

Regarding data, we filtered approximately 22 million trainable samples for pre-training, covering OCR, visual grounding, and other domains. For supervised fine-tuning (SFT), we extracted subsets specifically tailored for physical AI.

Training proceeded in three phases:

1. Pre-training: Focused on image generation, video generation, and image-to-video. This data equates to roughly 10 million hours of video for intuitive scale reference.

2. Mid-training: We introduced additional modalities, notably action, and cross-domain transfer data popular in robotics. For instance, we generate asset simulation videos. Though they appear artificial in simulation, we render them to photorealistic quality without altering physical properties, incorporating them into the dataset.

3. Post-training: At paper release, we trained only the Nano and Super versions. We applied post-training for text-to-image, text-to-video, and image-to-video tasks, alongside strategy-focused outputs relevant to this audience, demonstrating how Cosmos converts into a policy model and operates.

Model Specifications:

Cosmos Edge: A 4B-parameter MoE model with two expert towers (each 2B).

Cosmos Nano: A 16B-parameter MoE model (8B for the reasoner, 8B for the generator).

Cosmos Super: Our largest version, a 64B-parameter MoE model (32B each), serving diverse application scenarios.

We observe a clear scaling law. Setting aside policy requirements for real-time inference, larger models demonstrate extremely pronounced advantages in image and video generation.

Finally, a critical point: Cosmos is fully open-source. We strive for maximum openness. I encourage everyone to explore our project. I personally commit to manually responding to every issue on GitHub. No AI agents were used; I reply myself. I dedicate daily time to debugging the model and codebase. If you encounter questions or bugs, please report them, and I will do my utmost to provide support within a 24-hour window.

We released the models, code, and a comprehensive paper packed with granular details. This was a remarkable collaborative effort involving over 100 contributors, culminating in Cosmos. We hope it lays a solid foundation for the many applications the community aims to build.

Moderator: Thank you, Max, for the insightful presentation. I have a question: You compressed all modalities into one massive model. Is there evidence that this significantly narrows the Action Grounding Gap? Are the actions generated by the policy—particularly the Edge version—merely plausible-looking, or are they physically viable?

Max Li: Excellent question. The paper contains detailed breakdowns. For example, during Cosmos’ pre-training, we entirely excluded PID control information or similar data. It was trained purely as a state transition machine. Control signals were only introduced during post-training. When we conducted joint post-training for specific robots, integrating control information made it truly practical. This reflects a philosophical stance: pre-training should maximize generality and generalization, while post-training tailors it for specific tasks.

Audience Member: How do you measure Cosmos’ precision regarding physical laws? Specifically, how do you quantify the gap between predicted physical trajectories in video generation and actual physical laws?

Max Li: We utilize specialized benchmarks for physical evaluation, such as VBench, which are explicitly designed to assess the physical plausibility of generated videos.

Regarding whether incorporating action improves physical precision: while this iteration lacks robust global statistics proving action integration boosts physical prediction across all domains (since not all internet videos carry robot action labels), we consistently observe that in robotics and embodied AI contexts, adding action significantly enhances the model’s predictive accuracy for future visual trajectories. In these scenarios, the model proves highly valuable.

Audience Member: Thank you for the talk. I noticed multiple size variants (Edge, Nano, Super) and mentioned training utilized millions of hours of video data. Did you train different model sizes using different amounts of video data, or was the dataset consistent across all versions?

Max Li: A highly perceptive question. Typically, when studying scaling laws, you aim to maintain a compute-optimal ratio between token count and parameters. However, for the Edge model, we had to make deliberate trade-offs. For instance, we completely removed the audio modality from the Edge version. We found audio unnecessary for most edge tasks, and removing it freed up resources. When scaling to 16B (Nano) and 64B (Super), resource constraints are less severe.

This highlights a critical pain point and an open research question: technical trade-offs ultimately depend on your available data. My experience may not be directly replicable by other teams due to differing datasets. Overall, yes, there are data and modality trade-offs across model scales. I remain convinced that larger models yield better results, and I hope we eventually train a definitive physical AI foundation model.

RSS 2026 is approaching. We are currently assembling a high-caliber group of AI researchers focused on real-time paper tracking and rigorous technical discussion, with zero fluff.

Join via QR code or add WeChat: Luyoyo_2026. Note: Paper Group + Your AI Focus Area.

Copyright © 2011-2026 Leiphone / Shenzhen Yingpeng Information Technology Co., Ltd. All rights reserved. ICP License: 粤ICP备11095991号. Office: 0755-26581864.