AI Choreography

Zhejiang University and Unitree release 5 million robot-executable motion clips, plus one model that reads text, speech, music and video

November 19, 20261 views阅读中文原文
Zhejiang University and Unitree release 5 million robot-executable motion clips, plus one model that reads text, speech, music and video

On November 17, a GitHub repository called Movens-Zero logged two commits in a row, both titled "Update model release." In the early hours of the same day, the Movens-Zero-XL weight file on Hugging Face was refreshed. On November 19 the repo pushed another update to its inference scripts. As it stands today, the repository hosts four sizes of pretrained checkpoints, full training code, an 8,192-pair evaluation benchmark, and an entry point to a dataset called Movens-5M.

The team behind it is the embodied-intelligence joint lab of Zhejiang University and Unitree Robotics, with contributors from Bit Oasis, HiThink Research, Microsoft Research, East China Normal University, Peking University and the National University of Singapore. The paper, "Movens-Zero: Scaling Humanoid Motion Foundation Models with Large-Scale Human Videos," is an 86-page technical report that went up on arXiv on October 30.

So the timeline needs to be stated plainly: the paper is not new; what is new this week is that outsiders can actually use the thing — code, four checkpoint sizes, dataset and benchmark, all open. Facts below follow the project page, the GitHub repository, Hugging Face and the arXiv abstract; press coverage is compiled from multiple media reports (Machine Heart, RoboHot and AIGC.BAR all covered the release on November 18).

The short version

The point is not that the robot dances more gracefully. The point is that it turns ordinary internet video into motion data a robot can actually execute, and then trains a whole-body motion model on that data that understands four different kinds of input.

Four numbers:

  • 5 million humanoid motion sequences that passed simulated execution checks (upstream of those sit 15 million human motion sequences)
  • Over 10,000 hours of behavior data, more than 108 million motion frames in total
  • 297 fine-grained motion types across 16 scene groups
  • 4 input modalities: text, speech, music, RGB video

On the dancing question, the demo is direct: type "Dance some hip-hop" and it produces a dance with no reference clip. Feed it a song and it generates footwork and upper-body movement that follow the rhythm and energy. Feed it video of a person dancing and it transfers that person's movement onto a Unitree G1.

The hard part is not the model

There have traditionally been two ways to teach a humanoid to move, and each has a fatal flaw.

Motion capture. Precise, clean, immediately usable — and impossible to scale. How many minutes can one performer produce in a day? What does a mocap stage cost per day? Reaching "a hundred thousand hours" that way is not realistic.

Estimating motion from internet video. Video is infinite, but 3D motion recovered from pixels jitters, sinks through the floor, drifts off balance, and pushes joints past their limits. It looks fine to a human. A robot that copies it falls over.

Movens inserts an execution check in the middle. The pipeline does not stop at "we recovered the human's motion." It retargets that motion onto the G1 skeleton and joint limits, then has a simulated G1 actually run it. Clips the robot cannot track stably are dropped; only the ones that survive stay. That is what "executable" in the title is doing.

This filter is expensive, and it is exactly why the result can be called pretraining data rather than yet another grab-bag of mocap.

Five stages from raw video to training fuel

The project page breaks the data engine into five steps, each aimed at a specific failure mode:

  1. Clip generation — split long videos at shot boundaries and keep provenance. Skip this and a single clip with three scene changes leaves annotations and motion completely misaligned.
  2. Quality filtering — check image quality, whether the person is clearly visible, and whether there is enough motion information. Blurry footage, distant tiny figures and static shots are all removed.
  3. Semantic annotation — add scene and action descriptions, and crucially describe body parts over time. This is the biggest difference from ordinary video captioning. Not "a person wipes a door," but "0–4 s: leans forward, left hand with a cloth wipes the lower-middle of the door, right arm bent behind the body; 4–6 s: straightens up; 6–7.25 s: both arms rise toward the upper part of the door."
  4. Motion recovery — estimate 3D human pose from the frames and record how the body moves over time.
  5. Physical refinement — fix jitter and instability, retarget to the G1's body proportions, then verify in simulation that the robot can execute it stably. The project page states the configuration for this gate: six physical checks and four tracking policies — the same motion has to survive several different controllers before it counts as passing.

The output comes in two layers: 15 million human motion sequences, and 5 million robot motion sequences that cleared verification.

Architecture: an understanding module plus an action expert

Nothing here is a brand-new primitive, but the combination is worth spelling out.

In front sits Qwen2.5-Omni-3B as the understanding module, encoding text, speech, music and video into one semantic space. Behind it sits a generative module called the action expert, which produces motion via flow matching.

The key design choice is task-mixed training: rather than one model per input type, seven generation tasks are trained together so that a single action expert learns text-to-motion, speech-to-motion, music-to-motion and video-to-motion at once. The benefit is cross-modal transfer — beat awareness learned from music helps with timing under spoken commands; spatial structure learned from video helps posture plausibility under text prompts.

The action expert comes in four sizes spanning an order of magnitude:

  • S: 64.2M (3 layers, width 768)
  • M: 187.9M (4 layers, width 1,152)
  • L: 500.1M (6 layers, width 1,536)
  • XL: 910.5M (7 layers, width 1,920)

The understanding module stays fixed across all four; only the action expert scales. That is what makes their scaling study clean — there is exactly one variable.

Real-time loop: generate a short chunk, carry the tail forward

The most awkward failure in human-robot interaction is not ugly motion. It is that the robot is still executing an instruction you gave three seconds ago.

Movens-Zero uses a chunked scheme the paper calls Multimodal Real-Time Chunking. Each step generates a short motion segment; the robot begins executing the head of it, while the unexecuted tail is retained as the seed for the next segment and regenerated against the latest instruction and the robot's current state.

Two consequences follow directly:

  • Instructions can change mid-stream. In the demo the operator swaps commands on the spot and the robot picks up from there instead of finishing the whole prior sequence.
  • Video demonstrations can be given live. A camera captures a person moving, and the model keeps generating subsequent motion from the newest frames plus robot state.

The mechanism closes the loop across understanding, generation and execution rather than playing back open-loop. The hardware demonstrations run on a Unitree G1: a GPU inference server does the computation, an onboard whole-body controller executes, and a client streams robot state back.

Quantifying whether more data actually helps

The fair criticism of any foundation-model claim is: what exactly did your scaling scale?

They ran two groups of experiments, and the numbers are concrete.

Group one: fix the XL action expert, raise data from 1% to 100%. After deduplication, total motion duration grows from 122 hours to 12,231 hours.

  • Text-motion matching (R@3): 12.72% to 62.57%
  • Video-conditioned motion error (MPJPE): 132.82 mm to 40.90 mm, a 69.2% drop (13.3 cm down to 4.1 cm)
  • Speech-command matching: 13.98% to 74.30%
  • Music beat alignment (BeatAlign): 0.23 to 0.61
  • Motion tracking score: 0.21 to 0.72

And between 30% and 100% of the data the curve has not flattened yet: R@3 gains another 20.77 percentage points and MPJPE falls a further 53.5%. At the 12,000-hour mark they have not hit a data ceiling.

Group two: full data, only the action expert changes size.

  • Validation flow-matching loss: 0.487 to 0.159, down 67.4%
  • Motion tracking score: 0.18 to 0.72

Group three is the sharper one: does a bigger model get more out of new data? Going from 10% to 100% of the data, the S expert's R@3 rises from 6.34% to 10.30%, a 62.5% relative gain; the XL expert goes from 24.92% to 62.57%, a 151.1% relative gain. The other five metrics show the same pattern: the larger the capacity, the larger the relative return from added data.

That last result matters more to model builders than the first two — it says data and capacity have to grow together, and scaling only one wastes the other.

What this means for AI choreography

First, place it against adjacent work.

  • Music to 3D dance motion (EDGE, Bailando, FineDance and similar): output is skeletal animation for human viewing, with no requirement that a robot can execute it.
  • Audio stream to robot improvisation (the UCL work where a G1 dances along as it listens): the emphasis is low-latency online reaction, not motion library quality.
  • Monocular video to robot imitation (MoViC): the emphasis is ditching mocap gear and transferring a person's motion to a robot in real time.

Movens-Zero takes a fourth corner: one unified foundation model plus a large corpus of motion that a robot has been verified to execute. Its music pathway does not generate a pretty dance animation; it generates G1 joint trajectories that have been checked in simulation.

That difference is hard-edged in practice. With the former you still retarget, fix interpenetration and tune balance yourself. With the latter the output is motion a G1 can consume, and the repository ships both Sim2Sim (MuJoCo) and Sim2Real (physical G1) deployment scripts.

For choreographers and stage work it opens a specific loop: generate a first pass from text or music, watch the robot perform it in simulation, and only then move to hardware. That loop used to be either expensive (mocap, retargeting, tuning) or unreliable (pure video generation that a robot cannot perform).

How to use it — it actually runs

The repository gives an executable path, not a "coming soon":

  1. Environment: conda create -n movens-zero python=3.10, then pip install -e .. Music, speech and video input need FFmpeg and FFprobe.
  2. Weights: python scripts/download_models.py pulls XL (910.5M) by default. Use --model-size S/M/L to save memory. Note the Qwen2.5-Omni-3B backbone is not distributed in the repo and must be obtained separately.
  3. Inference: one command per mode.

- Text: movens-zero --mode text --description "A person walks forward calmly."

- Music: movens-zero --mode music --input-path song.wav --prompt "Generate a dance motion that follows the rhythm and energy of this music."

- Speech: movens-zero --mode speech --input-path speech.wav

- Video: movens-zero --mode video --input-path clip.mp4

  1. Defaults: 300 frames at 50 FPS (six seconds), 10 flow-sampling steps, CFG 1.5 over the 0.2–0.8 noise interval. Raise --num-output-frames for longer output; in video mode the main subject is auto-zoomed when small (--no-auto-zoom disables it).
  2. Outputs: each run directory contains joint_pos.csv, joint_vel.csv, body_pos.csv, body_quat.csv, motion.npy and metadata.json — joint and rigid-body position, velocity and quaternion, ready for a controller or for analysis.
  3. Onto a robot: sim2sim/ runs MuJoCo with the whole-body controller and a real inference-latency benchmark; sim2real/ is the physical G1 deployment with a GPU inference server and onboard controller, streaming robot state from the client.
  4. Train your own: training/train.py supports single and multi-GPU and resume; the single-GPU example is batch 4 with accumulation 4 over 30,000 steps.
  5. Evaluate: Movens-Intent has text, speech, music and video tracks totaling 8,192 expert-verified condition-motion pairs, with scoring scripts per track.

Good fit: who it's for

  • Teams building humanoid motion foundation models: this is one of the few stacks with data, weights, training code, an evaluation set and deployment scripts all present. Start here instead of climbing from zero.
  • Music-driven performance and choreography work: the music pathway emits executable trajectories you can preview in simulation, at a fraction of mocap cost.
  • Teleoperation and imitation-learning research: video mode is literally "film a person, have the robot copy it," from ordinary RGB footage with no mocap suit.
  • Scaling-law research: four action-expert sizes over a public data fraction makes a reproducible scaling testbed.
  • Teaching: outputs are CSV and npy, so students can inspect joint trajectories without first building a render pipeline.

Not a good fit: who should stay away

  • Commercial products: the repo says plainly that "project-specific code and checkpoint license terms have not yet been specified," and the Qwen2.5-Omni-3B backbone falls under the Qwen Research License, which permits non-commercial research and evaluation only. Shipping this into a commercial product today will not clear legal review.
  • Non-Unitree platforms: outputs use the G1's 36-dimensional motion representation. Porting to another robot means doing your own retargeting; no other body is adapted in the repo.
  • Precise hand manipulation: this is a whole-body model, not a dexterous-hand model, and there is no fine manipulation benchmark.
  • Sourcing latency numbers: the README gives no end-to-end latency or hardware success rate, only a sim2sim latency benchmark entry. You will have to measure it yourself.
  • Tight compute budgets: XL is a 910.5M action expert on top of a 3B Qwen backbone, the README assumes cuda:0, and there is no CPU or quantized path.
  • People who just want a video: the output is motion data, not rendered frames. You supply the renderer.

Pitfalls

  • Do not call it a paper release. arXiv v1 is October 30 — an 86-page technical report. What shipped this week is the open-source release. Combining the two turns an old paper into fake news.
  • Do not call it fully open source. Code and weights are there; the license terms are blank. Those four words mislead in this state.
  • Do not count the Qwen backbone as open. Qwen weights are not redistributed with the repo and are non-commercial.
  • Do not read 1.08B as a parameter count. It is a frame count (over 108 million frames). Parameters run 64.2M to 910.5M for the action expert, plus the 3B backbone.
  • Do not mix up 297 and 16. 297 is motion types; 16 is scene groups.
  • Do not treat 50 FPS as a control rate. That is the motion sequence output rate; control frequency depends on the onboard controller and is not specified.
  • Do not compare it to video generators on visual quality. It outputs joint and rigid-body trajectories, not pixels.
  • Weigh the phrase "execution verified" carefully. Five million means verified in simulation, not guaranteed on hardware — there is a sim-to-real gap, and the hardware demonstrations are qualitative with no large-scale success statistics.
  • Check secondhand numbers against the source. Aggregators routinely swap 15 million and 5 million, or turn hours into clips. Trust the project page and the arXiv abstract.

Cost and time

  • Reading: 15 minutes on the project page covers the structure and the four checkpoint sizes; five minutes for the arXiv abstract; one to two hours to work through the method and experiments in the 86-page report.
  • Getting inference running: environment plus downloads (XL weights and the 3B backbone, tens of gigabytes) plus a first command is roughly half a day if nothing fights you. Most of it goes to model downloads and FFmpeg dependencies.
  • Training your own: 30,000 steps on one GPU is the README example, but a usable model on your own data is a multi-GPU, multi-week job. Data preparation — retargeting and physical refinement — costs more than training.
  • Hardware: a G1, a GPU inference server and a working onboard controller. Scripts exist; a hardware bill of materials and troubleshooting guide do not, so debugging time depends on your robot's state.
  • Money: data and code are free; the cost is compute, storage and hardware.

Next steps: where to start

First, watch the five-minute-sixteen-second hardware film on the project page. It demonstrates all four input modes — video-to-robot, live speech commands, music-driven motion, text prompts. That tells you whether this is the thing you need, before you download tens of gigabytes.

Second, if you only want to test whether music can drive a robot to dance, pull the S expert (64.2M) and run --mode music on a six-second track, then look at the trajectory in MuJoCo. S is cheap and sufficient to judge direction.

Third, if you are building a product, settle licensing before you invest. The repo currently says the terms are unspecified — that is "pending," not "open." Track the commercial licensing path on the Qwen side as well.

Fourth, use Movens-Intent's 8,192 pairs as your benchmark. If you want comparative experiments, this is a ready four-track evaluation set that saves weeks over building your own.

Fifth, plan around the gaps: G1-specific output, no hands, no hardware success rates. If your scenario is a non-Unitree platform with fine manipulation needs, treat this as a reference architecture, not a finished product.

Sources

  • Project page (author affiliations, dataset statistics, four action-expert sizes, 5:16 hardware film): https://movens-zero.github.io
  • GitHub repository (installation, four-modality inference, training, Sim2Sim, Sim2Real, license note): https://github.com/wendell0218/Movens-Zero
  • Paper "Movens-Zero: Scaling Humanoid Motion Foundation Models with Large-Scale Human Videos" (arXiv:2610.41877, 86-page technical report, v1 October 30, 2026): https://arxiv.org/abs/2610.41877
  • Hugging Face dataset Movens-5M (5 million executable humanoid motion sequences, multimodal pairing): https://huggingface.co/datasets/wendell0218/Movens-5M
  • Hugging Face benchmark Movens-Intent (four tracks, 8,192 condition-motion pairs): https://huggingface.co/datasets/wendell0218/Movens-Intent
  • Hugging Face weights Movens-Zero-XL (910.5M action expert, updated November 17, 2026): https://huggingface.co/wendell0218/Movens-Zero-XL
  • Machine Heart report (November 18, 2026; scaling experiments and pipeline detail): https://ainrk.com/article/article-31370
  • RoboHot entry in Chinese and English (November 18, 2026): https://robohot.news/items/gc67c3us4kpma3wditahyvqvu
  • AIGC.BAR analysis (November 18, 2026): https://www.aigc.bar/en/AI%E8%B5%84%E8%AE%AF%E6%96%87%E7%AB%A0/2026/11/18/embodied-ai-gpt-moment-movens-5m

Note: core facts — data scale, the four parameter counts, scaling experiment numbers, inference defaults and license status — follow the project page, GitHub repository, Hugging Face and the arXiv abstract. Press-side details are compiled from multiple media reports.

Sources

Comments

0 published
Log inorsign up to leave a comment.

Loading comments...