Source confidence: medium
See also: Deeper Dive — Mocap as Reward — teaching cheap robots to move like you →

Mocap as Reward — teaching cheap robots to move like you

Insight class: Emergent Capability · Difficulty: Experimental · Quality: 3/5 Connected entries: 7 · Tools: Blender, JupyterLab, Microduck, Unity, Unreal Engine, XInputReader, ppf-contact-solver

In short: Running this system replaces reward-function trial and error with 'show it what you mean' on a cheap open robot.

Key fact Detail
What it covers 3D, open source, animation, art
Tools Blender, JupyterLab, Microduck, ppf-contact-solver, Unity
Posts covered 7 posts (3 S-tier, 2 A-tier) · top: Blender rigid-body spaceship — motors… · PPF Contact Solver — ZOZO's open-source GPU… · Blender — • 500+ body, hand & face landmarks…
Creators @marc.kaz, @6ab3, @ianhubertz
Sources dated May 2024 – Sep 2026
Difficulty Experimental
Prompt system Run-it-yourself prompts
Last verified 2026-10-22

Thesis

The Microduck RL demo shows the classic failure: a loose reward function makes the robot 'stand' in a way nobody intended. The catalog also holds the fix, scattered across collections. FreeMoCap turns a webcam into markerless motion capture, HY-Motion produces motion from text on robot skeletons, and Blender plus a GPU contact solver can rehearse physical contact. Use captured or generated motion as an imitation reward on a cheap open robot. The reward-design trial and error becomes 'show it what you mean'.

Why it matters

Hobby robotics is gated by reward engineering more than by hardware. A webcam reference clip gives an individual the motion-prior pipeline that only labs used to have, for small expressive robots, animatronics and installations.

The evidence — connected entries

Microduck robot — reinforcement learning with PPO and reward shaping

Creator: @summerr.exe · Tier: A Tools: Microduck

The problem: PPO on Microduck reward-hacks a loose objective

Training Pollen Robotics' open-source Microduck to stand: define a reward function (e.g. points for target body height), simulate 4,096 parallel robots, and let PPO update the policy toward higher-reward actions. The result 'stands' in an unintended way, a clean demo of reward hacking and why reward design is iterative.

View original post

Blender — • 500+ body, hand & face landmarks per frame

Creator: @marc.kaz · Tier: S Tools: Blender, Unity, Unreal Engine

Reference source: webcam markerless mocap, 500+ landmarks

FreeMoCap turns ordinary webcams into professional markerless 3D motion capture. — @marc.kaz; uses Blender, Unity, Unreal Engine; demonstrates motion capture.

View original post

HY-Motion 1.0 — Tencent's text-to-3D motion generation

Creator: @6ab3 · Tier: A

Reference source: text-to-motion on human and robot skeletons

by Tencent Hunyuan — featured by @marc.kaz | Billion-parameter text-to-3D motion generation. Type "a person walks forward then starts jumping" and get skeleton-based character animation. 200+ action types, 700hrs of pro mocap training. Works on human + robot skeletons. Free on HuggingFace. | DiT + Flow Matching 200+ action types Free

View original post

Blender rigid-body spaceship — motors, constraints and drivers for flight

Creator: @ianhubertz · Tier: S Tools: Blender, XInputReader

Rehearsal: Blender rigid bodies, motors and drivers to prototype motion physically

Ian Hubert shows physics-driven vehicle animation in Blender: active/passive rigid bodies, hinge and spring constraints, motor constraints pushing against the ground as thrusters, gravity off, and custom properties wired via drivers with scripted expressions for throttle and lights. Mentions Erindale's XInputReader addon for gamepad control.

View original post

PPF Contact Solver — ZOZO's open-source GPU cloth and soft-body physics

Creator: @marc.kaz · Tier: S Tools: Blender, JupyterLab, ppf-contact-solver

Rehearsal: GPU contact solver for soft/cloth contact cases

ZOZO open-sourced ppf-contact-solver, its internal virtual try-on physics engine: penetration-free fabric, rope and soft-body simulation with strain limits, up to 180M contacts, full GPU acceleration, a Blender plugin and a JupyterLab interface.

View original post

Self-stabilizing robots are reaching a fascinating stage whe — The content is presented purely for learning and infor...

Creator: @s_sapphire_robotics · Tier: B

Counterpoint: passive mechanical stability needs little AI — design the body first

Self-stabilizing robots are reaching a fascinating stage where they can walk and balance with little to no AI, relyin... — @s_sapphire_robotics.

View original post

Animatronic eye sculpture — hexagonal plate with moving eyes and macramé mandala

Creator: @miselyum.art · Tier: B

Target use: animatronic sculpture that should move with intent, not just stand

A stage-design art piece: a grey hexagonal plate of moving animatronic eyes framed by a macramé mandala, conceived as a bridge between people and a moon deity. Built for an event production; silent showcase.

View original post

Action sketch — how to implement this

Reproduce the Microduck standing run in its sim with 4,096 envs; confirm the reward hack. Record a 10-second webcam clip of a 'bob and look around' motion with FreeMoCap (or generate one with HY-Motion), retarget to Microduck's joints, and add a pose-tracking imitation term to the reward. Compare policies: reward-only vs reward + imitation, scored by a blind viewer on 'looks intentional' and by joint-tracking error; deploy the winner to hardware if you have it. Falsifier: if retargeting human motion to a duck-scale robot body produces infeasible joint targets that the policy can't track better than the plain reward, the motion prior is the wrong shape for small robots.

Prerequisites

Key tools and resources

Creators to follow

Prompt system — run this yourself

Everything below is pasteable into Claude Code, Claude, or any agent. Use the one-shot when you want the whole result in one run; use the step prompts when you want to steer. Every step carries its own audit, and the gap audit at the end makes the agent find and fill what it missed.

Mission. Running this system replaces reward-function trial and error with 'show it what you mean' on a cheap open robot. It reproduces the Microduck reward hack, captures or generates a reference motion (FreeMoCap from a webcam or HY-Motion from text), retargets it to Microduck's joints, adds an imitation term to the reward, and compares reward-only against reward-plus-imitation. It is for a hobbyist or artist with a GPU who wants expressive small robots or animatronics. Done means two trained policies, a joint-tracking error table, a blind 'looks intentional' rating, and a falsifier verdict on whether human motion is the wrong shape for a duck-scale body.

1 · Deeper, not bigger

Before running anything, make the task deeper rather than larger. Ask: what would a top professional do here that I am not asking for; what is the rawest material; which methods and tools; what, other than the model or me, proves it right. Then ask for exactly that.

Rewritten prompt — ask for exactly that:

I want a small open robot, Microduck, to move with intent instead of reward-hacking a loose objective. Act as a legged-robotics researcher. Read the Microduck repo's configs and robot description, DeepMimic and AMP, and Disney Research's bipedal character paper first. Then: (1) reproduce the standing reward hack with 4,096 envs and record what the policy actually does; (2) capture a 10-second 'bob and look around' clip with FreeMoCap from my webcam, or generate it with HY-Motion, and retarget it to Microduck's joints with inverse kinematics respecting joint limits, reporting any frames that violate limits; (3) preview the retargeted motion in Blender or the sim with physics on to check feasibility before training; (4) add a phase-conditioned imitation term to the reward and train reward-only versus reward-plus-imitation, 3 seeds each, with domain randomization; (5) score joint-tracking error, foot slip, survival and a blind viewer rating. If retargeted targets are infeasible and the imitation policy tracks no better than the plain reward, report that the motion prior is the wrong shape for small robots. Deploy to hardware only if sim passes.

2 · One-shot prompt

ROLE AND STANCE
You are a legged-robotics researcher running a controlled comparison on a small open robot. You assume the reward will be exploited and that human motion may not fit a duck-shaped body. You measure feasibility before training.

CONTEXT
- summerr.exe trains Pollen Robotics' open-source Microduck to stand: a reward for target body height, 4,096 parallel simulated robots, PPO updating the policy. The result 'stands' in an unintended way: classic reward hacking.
- marc.kaz: FreeMoCap turns ordinary webcams into markerless 3D motion capture with 500+ body, hand and face landmarks per frame, open source, with Blender, Unity and Unreal export.
- HY-Motion 1.0 from Tencent Hunyuan: billion-parameter text-to-motion, 200+ action types, trained on 700 hours of mocap, works on human and robot skeletons, free on Hugging Face.
- ianhubertz: Blender rigid bodies, hinge and spring constraints and motor constraints driven by custom properties, useful for previewing motion physically.
- ppf-contact-solver from ZOZO: GPU contact simulation for soft and cloth contact, relevant if the robot wears a costume or skin.
- s_sapphire_robotics: a running robot that self-stabilizes mechanically with no sensors; design the body first.
- miselyum.art: an animatronic eye sculpture as the target use: motion that reads as intent.

HARD CONSTRAINTS
- Use the repo's own sim and trainer; do not swap frameworks without a reason written down.
- Retargeted reference must respect joint limits and velocity limits from the robot description; clip and report violations.
- Same seeds, env count, training steps and domain randomization for both arms.
- 3 seeds per arm minimum.
- Hardware deployment only after sim survival above 95% over 100 episodes with randomization.

DELIVERABLES, IN ORDER
1. baseline/: reproduced reward-hack run with video and a one-paragraph description of the exploit.
2. reference/: the FreeMoCap or HY-Motion clip, the raw landmark or skeleton file, and the retargeted joint trajectory at the sim's control rate.
3. feasibility.md: joint-limit violations, max joint velocity versus motor limit, and a physics playback in sim or Blender showing whether the motion is dynamically plausible.
4. reward/: the imitation term (phase-conditioned exp of negative squared joint error, plus root orientation and foot contact terms) and its weight, added to the existing reward.
5. runs/: reward-only and reward-plus-imitation, 3 seeds each, with configs and logs.
6. eval.md: joint-tracking RMSE, foot slip, survival, energy proxy, and a blind 'looks intentional' rating by 10+ viewers.
7. hardware.md if hardware exists: 60-second run result.

GAP ANALYSIS (BUILT IN)
Keep gaps.md: which sim the repo uses and its version; whether the robot description's limits match the real servos; whether FreeMoCap from one webcam gives stable depth; HY-Motion's licence and skeleton mapping; retargeting choices that changed the motion's meaning (head motion carried by neck versus body); unmeasured sim-to-real factors. Fill what you can now.

SELF-QUESTIONS
- What exactly did the baseline exploit, and does my imitation term close that loophole or open a new one?
- Which human joints have no duck counterpart, and where did their motion go?
- Is the retargeted motion feasible at the servos' speed limits?
- Does the imitation weight dominate so much that the robot falls while tracking?
- Would a blind viewer pick the imitation policy as intentional, or just as different?

STOP CONDITIONS
Stop and report if more than 20% of reference frames violate joint limits after scaling, if the baseline cannot be reproduced, or if training diverges in both arms. Otherwise stop when all deliverables exist.

RETURN FORMAT
Table of deliverable, path, status; the eval table per arm with mean and spread across seeds; the blind rating result; the falsifier verdict; top five gaps.

3 · Step-by-step prompts

Step 1 — Ground truth: repo, robot model, baseline hack

Goal: Reproduce the reward hack from the repo's own configs.

Clone the Microduck repository (and Open Duck Mini if Microduck references it). Identify: the simulator and version (MuJoCo Playground, MJX, Isaac Lab or other), the PPO implementation, the standing task's reward terms with weights, observation and action spaces, control frequency, and the robot description file with joint names, limits, velocity limits and actuator gains. Write repo_notes.md.
Run the standing task with 4,096 parallel envs and the repo's default steps. Record a video of the final policy and describe in one paragraph exactly what it does to satisfy the height reward. Log reward curves.
Also read DeepMimic, AMP and Disney Research's bipedal character paper; summarize each one's reward form and how they handle infeasible reference motion in papers.md.
Finally note GPU, VRAM and wall-clock time for the baseline run, so later runs can be budgeted, and record the exact commit hash of the repo you trained from.

Expect: repo_notes.md, papers.md, baseline/ with video, logs and an exploit description.

Audit before moving on: If the baseline does not reproduce a hack, document what it does and re-run with a second seed; the comparison needs a known failure to improve on. Confirm joint limits in the description against the servo datasheet if hardware exists.

Step 2 — Capture or generate the reference motion

Goal: Get a 10-second 'bob and look around' motion in a usable skeleton format.

Option A, FreeMoCap: record 10 seconds of yourself doing a small 'bob and look around' motion with one or ideally two calibrated webcams. Export 3D landmarks. Smooth with a Butterworth low-pass at 6 Hz. Report landmark jitter (frame-to-frame std of a static point).
Option B, HY-Motion 1.0: from Hugging Face, check licence, then generate from the text 'a small creature bobs up and down and looks around curiously, standing in place' at several seeds. Export the skeleton sequence.
Pick the better clip by eye and by smoothness. Save raw output, the processed sequence, frame rate and skeleton definition in reference/.
Write down what the motion should communicate, so the blind rating later has a target. Keep the clip short and in place: no walking, no turning more than 45 degrees, feet planted, because a standing duck-scale body cannot follow locomotion it was never trained for.

Expect: reference/ with raw and processed motion, jitter or smoothness metrics, and intent.md.

Audit before moving on: Play the processed sequence on a stick figure; any foot sliding, popping joints or depth flips mean re-record or re-generate. Confirm frame rate and units (metres, radians).

Step 3 — Retarget and check feasibility

Goal: Map the motion to Microduck joints without breaking limits.

Write retarget.py: define correspondences from the reference skeleton to Microduck (pelvis to base, hips, knees and ankles to leg joints, neck and head to the duck's neck and head joints). Scale root height and stride by the ratio of leg lengths. Solve inverse kinematics per frame for foot and head targets with joint limits as bounds (scipy least-squares or a robotics IK library). Resample to the sim's control rate.
Write feasibility.md: percentage of frames hitting a joint limit, max joint velocity versus the actuator velocity limit, and a list of human motions that have no duck equivalent and where you mapped them.
Play the retargeted trajectory in the sim with physics on and a PD controller tracking it, and optionally in Blender with rigid bodies and motor constraints. Record whether the robot stays up.

Expect: retarget.py, reference/retargeted.npy at control rate, feasibility.md, a physics playback video.

Audit before moving on: If more than 20% of frames hit limits or velocities exceed motor limits, slow the clip or reduce amplitude and re-run before training. If PD playback falls immediately, the reference is dynamically infeasible; note it and consider AMP instead of exact tracking.

Step 4 — Imitation reward and matched training runs

Goal: Train reward-only and reward-plus-imitation under identical conditions.

Add an imitation term to the existing task: r_imit = w_q * exp(-k_q * sum((q_ref(phase) - q)^2)) + w_root * exp(-k_r * root orientation error^2) + w_contact * foot contact match. Add phase (sin and cos of clip time) to observations. Keep the original height and alive terms.
Start with imitation weight equal to the sum of existing terms; sweep two other weights (0.5x and 2x) on one seed first, then fix it.
Enable domain randomization on friction, link masses, motor strength and action latency, same for both arms.
Train reward-only and reward-plus-imitation, 3 seeds each, 4,096 envs, the same number of steps. Save configs, checkpoints and logs in runs/.
Generate both configs from one template with a single use_imitation flag. Log each reward term separately so you can see which term the policy is earning from, and save a video every 20% of training to catch a new exploit early.

Expect: reward/ code diff, runs/ with 6 completed trainings and configs, the weight sweep result.

Audit before moving on: Diff the two arms' configs; any difference other than the imitation term and phase observation invalidates the comparison. Watch for a new hack: high imitation reward while falling or sliding the feet; check videos, not just curves.

Step 5 — Evaluate with metrics and blind viewers

Goal: Score both arms on numbers and on perceived intent.

For each policy run 100 randomized episodes. Measure: joint-tracking RMSE against the reference (computed for both arms, even reward-only), foot slip (horizontal foot velocity during contact), survival rate, mean torque squared as an energy proxy, and head motion amplitude.
Render 10-second clips of each policy from the same camera. Shuffle and label them A to F with a script that keeps the key hidden. Ask at least 10 viewers who did not train the policies to rate 'looks intentional' 1-5 and say what the robot seems to be doing. Compare answers with intent.md.
Report means and spread across seeds. Include one clip of the raw retargeted reference played kinematically, so viewers' ratings of the policies can be compared with the ceiling set by the reference itself. Use a Mann-Whitney U test on ratings between the two arms.

Expect: eval.md with the metrics table, the blind rating table and viewer descriptions.

Audit before moving on: Confirm the key was hidden until ratings were in. If seed spread overlaps heavily between arms, state that the difference is not established.

Step 6 — Validate, deploy if earned, write the gap report

Goal: Answer the falsifier and record what remains unknown.

Apply the falsifier: if retargeting produced infeasible targets and the imitation policy does not track better than the plain reward (lower joint RMSE and higher blind rating, outside seed spread), the motion prior is the wrong shape for small robots. Write the verdict.
If the imitation arm wins and survival exceeds 95% over 100 randomized episodes, deploy to the physical Microduck if available: export the policy, run 60 seconds on a soft surface with a spotter, film it, and log falls.
If it loses, try AMP style imitation as a single follow-up and record whether it changes the verdict.
Write gaps.md: each claim from the source posts (4,096 envs, 500+ landmarks, 200+ HY-Motion actions, robot-skeleton support) checked or not; sim-to-real factors unmeasured; retargeting choices that changed meaning; anything not reproduced. Fix what you can now.

Expect: verdict.md, hardware.md if applicable, gaps.md.

Audit before moving on: Confirm the verdict uses the pre-stated criteria, not a new metric chosen after seeing results. Confirm hardware ran only after the sim gate passed.

4 · Gap analysis and audit

Run this after the build. It makes the agent enumerate its own claims, mark what is verified, rank the gaps and fix the top ones.

You are auditing a mocap-as-reward experiment you did not run. Read repo_notes.md, baseline/, reference/, feasibility.md, reward/, runs/, eval.md, verdict.md and hardware.md.
1. List every claim: baseline exploit, reference quality, retargeting feasibility, tracking error, survival, blind ratings, hardware result.
2. Mark each verified, unverified or false with evidence: a video, a log, a metric file, a config diff.
3. Re-check: both arms share every setting except the imitation term and phase; 3 seeds per arm exist; joint limits were enforced; the blind test key was hidden; hardware ran only after the 95% sim gate; HY-Motion's licence permits the use.
4. List missing pieces: servo latency measurement, real joint limits, foot contact sensing, a second reference motion.
5. Rank gaps by whether they could change the falsifier verdict.
6. Fix the top three now.
7. Report what you could not fix and why.

Questions the agent must ask itself before it says done:

5 · What proves it right

6 · Failure modes

7 · Agent direction (unattended runs)

Begin by reading the Microduck repo and reproducing the baseline; every later comparison depends on that. Cache the robot description, reference clips and retargeted arrays under reference/, and checkpoints under runs/. Run the weight sweep and the 6 training runs as background jobs; spawn a sub-agent to build the blind-rating harness while training runs, since it touches only eval/. Stop and ask when the sim stack in the repo needs hardware or licences you lack, when more than 20% of frames violate limits, or before any hardware run. Never run the physical robot without a spotter. If FreeMoCap capture quality is poor after two attempts, switch to HY-Motion rather than spending the session on camera calibration. Return README.md linking repo_notes.md, baseline/, reference/, feasibility.md, reward/, runs/, eval.md, verdict.md and gaps.md, each with a one-line status and the falsifier verdict.

Prompt system updated 2026-10-22 · all prompt systems · JSON

Browse more deep dives at DejaViewed — 846 entries across the full catalog.

Generated 2026-10-22 · curated by @6ab3 for DejaViewed