A-TIER demo
Microduck robot — reinforcement learning with PPO and reward shaping
100k ft — What is this?
Microduck robot — reinforcement learning with PPO and reward shaping
50k ft — Why care?
Training Pollen Robotics' open-source Microduck to stand: define a reward function (e.g. points for target body height), simulate 4,096 parallel robots, and let PPO update the policy toward higher-reward actions. The result 'stands' in an unintended way, a clean demo of reward hacking and why reward design is iterative.
10k ft — Key takeaways
• Agents exploit loose rewards; expect to iterate on reward terms.
• Microduck is an approachable open-source RL playground.
• RL = reward function + many parallel simulated attempts + policy updates.
5k ft — Tools & techniques
Microduck
PPO, parallel simulation, reinforcement learning, reward shaping
1k ft — Deep-dive / try it
MICRODUCK and what I learnt about reinforcement learning !
@pollenrobotics @seeedstudio @huggingfacecommunity @nvidiarobotics
#robotics #reinforcementlearning #cyberdeck #engineeringexplained #microduck
Transcript excerpt
Let's teach my dog how to stand. This is Microdog from Poland Robotics. It's basically an open source playground for reinforcement learning. Imagine training a dog. You don't tell her how to move every joint to sit. Instead, you give her a treat. Whenever she does something closer to sitting, our L works on the same idea. We define a weird function that gives it points for things like reaching the right body height. Then I fill the simulation with 4096 dogs trying to figure out how to stand inside this jesson zone. The PPO algorithm sees every attempt and updates the policy so actions with higher rewards become more likely. And finally, we got a standing dog. Except it has a very different definition of standing. I got a bunch of suggestions. So now it's time to run some experiments and fix the reward function. Follow to see how it goes.