The trials and tribulations of training a physical robot with reinforcement learning

This is a follow up to my article “Training a real robot to play Puckworld with reinforcement learning”. In that one, to make it a little punchier, I showed the overview and end results of the project, but left out the insane number of little hurdles and decisions I had to figure out.

So this article will be about those details instead, partly for me to justify the pain, but maybe more charitably to show that for any project with a neat (hopefully?) presentation, there’s probably a harrowing saga of hair-pulling roadblocks behind it. Here, it’s roughly in the order I encountered things. There are lots I’m leaving out too. read more

In case hexapods weren’t creepy enough: the centipede robot!

Similar to…most of? my ideas, I don’t remember why I thought of this. I think after I made the reinforcement learning robot, I was on a robot kick, and came up with this. Hexapods are of course a robot classic, but I don’t think I had ever seen a centipede robot.

Why a centipede? Well… I can make up a few “practical” reasons: because of its length, it could potentially bridge gaps, or bend “upwards” to have height, or possibly even climb. But the real reason is because they haven’t been done that much and I thought it would be cool, funny, and creepy. read more

Training a real robot to play Puckworld with reinforcement learning

After I trained an agent to play “puckworld” using Q-learning, I thought “hey, maybe I should make a real robot that learns this. It can’t be that hard, right?”

Hooooooooo boy. I did not appreciate how much harder problems in the physical world can be. Examples of amateurs doing Reinforcement Learning (RL) projects are all over the place on the internet, and robotics are certainly touted as one of the main applications for RL, but in my experience, I’ve only found a few examples of someone actually using RL to train a robot. Here’s a (very abridged!) overview of my adventure getting a robot to learn to play a game called puckworld. read more

First project with the new 3D printer: a TOF sensor mount

I’m pretty late to hop on the 3D printing bandwagon, but I heard its siren song and couldn’t stay away much longer!

After the briefest search online and asking a few friends, I decided to go with the Monoprice Delta Mini. My main reasons were that I didn’t want to have to tinker and build much (at least to start), I wanted to be able to get decent quality, I didn’t want to spend a ton, and I don’t especially need a large print size. The MPDM matched all of this (supposedly works out of box, can do 0.05 mm layer height, 160 bucks, 4″ height by 3″ diameter print volume), so I went for it. read more

The Red Lama (Red Llama clone)

After making the worst fuzz pedal ever (that’s for another post) and Orange Ya Glad (which was fine, but didn’t add quite as much fuzz as I wanted and adds a weird buzz even when you’re not playing on some speakers), I just wanted a normal fuzz pedal. After doing a bit of reading, I found that the Red Llama overdrive pedal (by Way Huge) is a classic, and after watching a few YouTube demos, it seemed good (to be honest, people are crazy about the “different” sounds of various fuzz/distortion/overdrive that various antique/obscure transistors or configurations will give you, but they all sound pretty similar to me, and I suspect people think they’re hearing differences more often than there actually are).

Anyway, I wanted to tribute the original Red Llama circuit I was cloning, so I went for… read more

Motion detection with the Raspberry Pi, part 1

Okay Declan, let’s try making this post a short and sweet update, not a rambling Homerian epic about simple stuff.

I got a Raspberry Pi (RPi) and an RPi camera because I wanted to learn about them and mess around with them. If I could do image recognition with them, that’d be a good platform to do ML, NN, and if I got enough data, maybe even DS type stuff. Luckily, there’s a ton of resources and code out there already. I drew upon heavily from www.pyimagesearch.com, which is a REALLY useful site, explained very great for beginners. Two articles that I basically copied code from and then butchered were this and this. read more

Squall Moan: Small Clone clone

squallmoan

Ahhh, where it all started.

I was jamming with a friend in his basement and he had a bunch of pedals, which I was noodling around with. None really stuck out to me until this little guy. If you want a sample of what it sounds like, there are plenty of test drives on YouTube. You may recognize its sound from Nirvana songs (only 90s kids will myeh myeh myeeehhh). read more

A spooOOOOoooky project!

bloodhead1

This is a fun one.

It’s also a testament to how nifty and easy it is to quickly whip up a project with Arduinos, provided you have enough of a “critical mass”, as I’ve called it before, of other stuff that you might end up needing. read more

Back to the blog and the NYC Maker Faire

Hey there, nonexistent reader! You may have (not)iced that I’ve been gone for a while. That’s because I just defended my PhD, and I was pretty balls to the wall busy for the last few months getting ready for that. I made a few posts here and there, but towards the very end I realized I really had no time for a blog that no one reads anyway. Maybe I’ll make a post about the whole before-and-after experience at some point.

Anyway, my friends and I just got back from a weekend trip to the NYC Maker Faire. Max has a friend who got us a few free tickets, which was sweet; they’re not inexpensive, something like $35 for a single day pass I think? So she definitely saved us some buxx. I had never been before. It was pretty perfect weather and in Queens, so all said and done not too crazy to get there. read more

Low power Arduinos, part 1

pro_mini_led

As part of an ongoing project, I wanted to see how low I could get the power consumption of Arduinos to go. The reason is as follows. When getting back into Arduinos a few months ago, I wanted to try a telemetry project of some sort, collecting data remotely and sending it back. Ideally, the idea would be to collect data from different places and analyze the aggregate in some cool way, but that’s a story for another post.

The point I was going for, though, is that I wanted to put these Arduinos in places that wouldn’t have constant access to power, so that already means using a battery. Using a battery to power an Arduino isn’t a big deal (plenty of people do it for portable projects), but once you’re looking at long term powering without recharging, it’s a different story. read more