The trials and tribulations of training a physical robot with reinforcement learning

This is a follow up to my article “Training a real robot to play Puckworld with reinforcement learning”. In that one, to make it a little punchier, I showed the overview and end results of the project, but left out the insane number of little hurdles and decisions I had to figure out.

So this article will be about those details instead, partly for me to justify the pain, but maybe more charitably to show that for any project with a neat (hopefully?) presentation, there’s probably a harrowing saga of hair-pulling roadblocks behind it. Here, it’s roughly in the order I encountered things. There are lots I’m leaving out too. read more

The Mostly Printed CNC (MPCNC), mostly

I’m not sure when I got the urge to make a CNC… or maybe it was always there. I did a summer job in a machine shop when I was 19, where I was given a minimum of training by the 88 year old machinist: “keep the pink things [wiggling his fingers] away from the sharp things [pointing to the milling machine’s cutting edge], boy.”

(I’m not joking. He did show me more things later, but only because I asked. He would mostly just motion for me to come over so he could tell me filthy jokes from the silent film era and cackle to himself.) read more

Training a real robot to play Puckworld with reinforcement learning

After I trained an agent to play “puckworld” using Q-learning, I thought “hey, maybe I should make a real robot that learns this. It can’t be that hard, right?”

Hooooooooo boy. I did not appreciate how much harder problems in the physical world can be. Examples of amateurs doing Reinforcement Learning (RL) projects are all over the place on the internet, and robotics are certainly touted as one of the main applications for RL, but in my experience, I’ve only found a few examples of someone actually using RL to train a robot. Here’s a (very abridged!) overview of my adventure getting a robot to learn to play a game called puckworld. read more

The Red Lama (Red Llama clone)

After making the worst fuzz pedal ever (that’s for another post)¬†and Orange Ya Glad (which was fine, but didn’t add quite as much fuzz as I wanted and adds a weird buzz even when you’re not playing on some speakers), I just wanted a normal fuzz pedal. After doing a bit of reading, I found that the Red Llama overdrive pedal (by Way Huge) is a classic, and after watching a few YouTube demos, it seemed good (to be honest, people are crazy about the “different” sounds of various fuzz/distortion/overdrive that various antique/obscure transistors or configurations will give you, but they all sound¬†pretty similar to me, and I suspect people think they’re hearing differences more often than there actually are).

Anyway, I wanted to tribute the original Red Llama circuit I was cloning, so I went for… read more