Second Order AI

NVIDIA’s Cosmos3-H-Dreams simulates real surgery with generative AI

NVIDIA released a research simulator that generates realistic video of gallbladder surgery in real time, responding as a user moves the surgical robot’s instruments.

1 min read

Hugging Face

At a glance

parameters in the base model
16B
control inputs, from arm poses to energy controls
28
NVIDIA RTX PRO 6000 GPU to run it live
1
What’s new
Trained on recordings of real gallbladder removals on CMR Surgical’s Versius robot, it is, to NVIDIA’s knowledge, the first real-time generative surgical simulator built from clinical footage that responds to live controls.
Who it affects
Surgical robotics researchers, and developers building robots that could one day learn surgical tasks in simulation.

Instead of modelling anatomy and tissue physics by hand, the model learns how a surgical scene looks and changes from video paired with the robot’s movements. NVIDIA trained a larger teacher model, then distilled it into faster students that generate video in short blocks as new commands arrive, released in four-step and two-step versions that trade image quality for speed. Code, checkpoints and a recipe for training on your own data are on GitHub and Hugging Face.

Why it matters

Realistic simulators could let surgical robots practise tasks safely before they touch a patient, and help researchers test ideas faster. NVIDIA stresses that this is a research tool, not a system for patient care: looking realistic is not the same as being physically accurate, and evaluation with surgeons is still to come.

Original source

NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

Hugging Face

huggingface.co

Read the original

Summary by Second Order AI. Full reporting is at the original source.

Related stories

More in →