MQTT to rf24 Translator (Home Automation Part 1)

-or- Learning to listen to your Arduinos…

I’m an Arduino enthusiast. By that I mean I freakin’ love Arduinos. There is so much you can do with them. You want to convert an old PS/2 keyboard over to USB? The ProMicro has you covered. You want a $6 temperature sensor? The nano paired with a cheap temp probe can do that. You want to turn something on and off? Swap that temp probe out for an inexpensive relay and you’re golden.

Easy enough, but how do you get those temperature readings off of that sensor? Should you leave your laptop plugged into the controller and just walk by occasionally and check it? That doesn’t seem very useful. What if you’re on vacation and need to turn off that lamp you accidentally left on? What you need is network communication.

Most Arduinos (there are exceptions, we’ll look at those in a minute) can’t talk on the same WiFi network that your laptop uses for endless Facebook feeds. In fact, by default they can really only speak over the serial lines on pins 0 and 1. We can still use that though. The NRF24L01+ is a very inexpensive radio that can enable an Arduino to talk to other Arduinos over a mesh network. A sort of IoAT (Internet of Arduino Things? I’m still working on the name part.) All in all this mesh network can handle more than 3,000 nodes on a single channel. I’d like to see WiFi do that!

It does this by using a tree topology where you have one “trunk” with up to 5 leaves, each of those five leaves can have 5 leaves and so on until you get 5 total layers deep. This allows namespace for exactly 3,125 devices. The creator of the library explains it well in the documentation.

All this is well and good, but my laptop can’t speak rf24. Neither can yours. We need to translate that into something a little more universal. To get a good overview, I’ll explain my use case. There are many other use cases but this one is mine.

  • Motion/Temperature sensor in kitchen monitoring freezer temps and room activity.
  • HomeAssistant reacting to this information in meaningful ways: adjusting lights and sending alerts.

The problem is that HomeAssistant runs on a standard server, on a standard network. It also doesn’t speak rf24. The Arduino has no LAN connection and only speaks rf24. We need a translator. We need more than that. We need a translator and a way to send these translated messages back and forth.

What we need is a pub/sub. No, that isn’t some beer of the month club. Publish/Subscribe, very creative, I know. A pub/sub simply acts as a community bulletin board. Some things post to it, other things can read from it, most things do both. It’s really very social; a sort of Twitter for our IoAT (that name is growing on me). I use Mosquitto (MQTT Client/Broker) for this purpose because it’s free and has binaries available for pretty much any platform you can get your hands on. This makes it stupid simple to install. Like sudo apt-get install mosquitto kinda simple. HomeAssistant integrates with MQTT out of the box so that side is good, that puts us back to the whole rf24 mess.

So to really simplify it, the Arduinos live on the rf24 network while the MQTT broker lives on the LAN. Additionally, the two protocols are not compatible. Clearly, the only solution is some sort of super-Arduino that can talk on the LAN and rf24, and speaks both protocols. Enter the NodeMCU. The NodeMCU is a special kind of Arduino, no doubt. In addition to all the normal Arduino goodness, it has a WiFi card strapped to its back. Or perhaps it is a WiFi card with a simplified Arduino on board. Chicken? Egg? I’m not sure anymore. It does get us where we need to go though. We can teach it to speak both protocols and give it hardware for both networks. So our new network map looks something like this:

Temp/Motion Sensor –rf24–> NodeMCU –WiFi–> MQTT Broker –LAN–> HomeAssistant

It works in reverse as well, HomeAssistant can send a command to turn on/off a relay through the MQTT broker to the NodeMCU which hands it off to the responsible Arduino controlling the relay. Life is good. Now let’s build some sensors…

Parts Listing

These are listings for the actual parts that I have ordered and used. There are plenty of places to buy these from and most of them are going to be similar enough to work the same way. If you’re looking for an easy place to start, I can recommend the following parts.

Up Next: Building the sensor nodes…

Let’s Split Planck – Hard Mode

Handwiring a Split Planck

-or- the adventures of being cheap.

finishedProduct

Why?

I don’t know. I mean, it’s not like I need another keyboard. But people like you and me don’t rationalize things like that. Yes, I have plenty of keyboards. I even have more than one Planck. But, I don’t have a split Planck! As I was doing the yearly cleaning of the desk ritual, I noticed that I had a bunch of parts around. Like probably enough to build a whole keyboard. Well, you can guess where it went from there. I ended up spending about $3 on case screws to complete the project.

What you’ll need

  • A (micro) controller
    • I used an Arduino Pro Micro but a Teensy or any other ATmega32u4 would work. The generic Pro Micros are dirt cheap on Amazon/eBay and have plenty of pins for a 40% build.
    • Important: it must be a ‘u’ series. The Arduino Nano won’t work as it’s an ATmega328p. The ‘u’ means that it can act as a USB device so your computer actually thinks it’s a keyboard.
  • Keyswitches
    • You’ll need 48 of these.
    • Cherry MX or Matias, your choice. Since we’re hand wiring we don’t have to worry about board/switch compatibility.
    • You can often find them cheap/used on forum sites in the classifieds.
  • Keycaps
    • You’ll need 48 of them in 1u size. Grab the profiles you like and make sure they are compatible with the switches you chose.
    • I used some leftovers from old sets. Sure, it’s a little shady looking but it was free.
    • You can often find basic/OEM sets cheap on forum sites in the classifieds.
  • Diodes
    • You’ll need 48 of these as well.
    • Any cheap diode will work, I used 1N4148 but the 1N4001 would work as well.
  • Wire
    • Leftover wire or upcycled stuff is fine. Fairly thin on the gauge.
  • Plate and case
    • I 3D printed a combination plate and case. Other options include having a fabrication shop cut one from metal or acrylic.
  • A button
    • You know how there’s always that one piece that seems weird or more trouble to obtain than it’s worth? Yeah, we need one of those too. If you’re a maker on the regular you probably have some momentary contact buttons/switches laying around. If not, you can order them but they tend to come in packs of 100 for about 10 bucks. You can get creative and source them out of broken things you find around the house. I used to use the power and reset switches out of old computer cases.

Stage 1: The case

I ended up enlisting my 3D printer for this. As I mentioned, my goal was to build a keyboard for as close to free as possible. I went on to Thingiverse and searched for ‘keyboard’. This is what I found:

emptyCase

You print two tops and two bottoms to make a complete set. One piece of advice I’d add to this specific model: when you print the second set, mirror them horizontally in your slicer. The way I printed them, the holes on the top aren’t in the optimal position.

There was a bit of filing and scraping to get the switches to fit just right. I’d imagine that my printer could be tweaked up a bit. I know I’m smashing the first layer, but when it seems to work, I hate to mess with it too much. After a little work, we have something starting to resemble a keyboard:

switchesInstalled

Stage 2: The wiring

To put it simply, you’re building a grid. On an ortholinear keyboard, the grid is easier to see and therefore somewhat easier to wire. We need to hook the switches together horizontally to form rows and vertically to form columns. In order to prevent what’s known as ‘ghosting’, we use diodes to build the rows. The reasoning is interesting but beyond the scope of what we’re working on. If you’re interested at all in electrical engineering and how it applies to human interface devices, I’d encourage you to google ‘keyboard matrix ghosting diodes’ and read up a bit.

The rows

The rows are built with diodes, as mentioned earlier. As a neat trick you can actually use the long legs of the diodes to connect the switches and save a bit of space and wire. The secret is to bend them all into shape first so they all match, that’ll give you nice clean lines when you solder them up. I use an extra switch as a sort of form to bend them around like so:

bendingDiode

A diode has two sides, and polarity matters. One side is designated by that small black band. We want that side pointing away from the leg of the switch it’s soldered to. Place the non-banded leg of the diode between the two legs of the switch and bend it around until it’s nice and lined up down the side. Then, towards the bottom of the switch, bend the banded leg around until it points to the right. This leg will eventually attach to the next diode in the series. I’d recommend going ahead and bending all 48 out now. It’s tedious, but it’ll ensure that they all look pretty much the same and that the black band is in the right position.

Once you get them all bent into shape, it’s time to break out the soldering iron. Remember, if it smells like chicken, you’re holding it wrong. The side without the band goes to one leg of the switch, and clip off the extra. The other leg bends awkwardly in your way as you’re trying to solder the next one. No, there isn’t any real secret to this part. Just go from left to right, take your time, and bend things out of the way when needed. Sometimes rotating the project around will make it easier to reach the harder to reach bits. Just don’t burn yourself. Or your desk. Or things sitting on your desk. Like your other keyboard. Hypothetically, it could happen. I’m just looking out for you.

diodesDone

The columns

The columns are just connected with straight wire, but of course nothing is that simple. I seemed to have missed taking any photos of preparing the wires, but you can see the end result in the finished photos. What you’re looking at doing is stripping the wire, only in the middle of the run as well as both ends. The process for doing this isn’t as magical as you might think. You take a good sharp knife or razor blade and roll it over the wire on the top and bottom of the part you want to remove. Then just cut a slit across the part you want to strip off and it should peel off. Looks like magic but it gets pretty easy by the second or third wire. Once properly stripped, solder your columns together like so:

columnsDone

Connect the sides

There are a few ways of accomplishing this. The fancier builds make use of IO Expanders or I2C in order to reduce the number of wires between the halves to two. This requires two controllers and some additional parts. We’re on a budget build here. I took an old HDMI cable, which has 19 conductors, to just wire it up as if it were a single piece. Use four of the wires to connect the rows of the left side to the rows of the right side. Use six wires to connect the columns of the non-controller side to the controller on the other side. Use two of the wires to hook up a reset button in the leftover hole in the non-controller side.

nonControllerSide

If you’ve made it this far, you’ll want to hook up the controller side next. Take the wires that are soldered to the rows on the other side and tack them down to the same rows on the controller side. Hook the two leads from the reset button to ‘RST’ and ‘GND’ on the controller. That leaves you with the columns from the other side. First you have to decide where to put them on the controller. I used physical placement that ensured the shortest wires. Here’s my notes, for what they’re worth:

wiringNotes

Hook the columns from the other side into the pins you decide on. Then hook the columns from this side up. Finally, hook the rows into their designated pins. Remember, the way we wired this up, the rows stretch across the divide and work as if they weren’t separated. You’ll end up with 12 columns and 4 rows. As you can see below, this is where my neat wiring went a little, well, a little less neat.

controllerWired

You’ll want to test it before closing it up but the only real construction left is to give the cable entrance/exits and controller a good coating of hot glue and secure them in place. You’ll also need to figure out how to close up your case. For the model I used, I just glued some knurled nuts into the holes in the corners and then screwed it all together. Your approach will vary depending on what your case looks like.

Stage 3: The firmware

I’ve built a bunch of keyboards and converters and I’ve always used variants of the same firmware. A few years back a person wizard named Hasu created the TMK firmware. A little while later a gentleman by the name of Jack Humbert created The Planck keyboard and forked TMK into QMK. Both are more-or-less the same. QMK has some added features but supports less keyboards. Since both support a handwired build, we’ll just go with QMK.

QMK actually almost supports what we’re trying to do. They have a 5×12 handwired project in the firmware that could be modified to work but it actually gets even easier. The Planck is a 4×12 ortholinear keyboard. It’s the other keyboard in the photos above. You can’t help but notice the similarities. Thankfully, QMK can’t tell the difference either so that’s a win for us. We just need to tweak the planck firmware ever-so-slightly.

First file: qmk_firmware/keyboards/planck/config.h

There are a few things in this file that you could/should change:

Optional:
Manufacturer/Product/Description
I mean, you made this thing. Put something awesome there. It only shows in device listings such as ‘About this Mac’, Device Manager, or System Details so it isn’t exactly vital, just an extra point of customization.

Required:
MATRIX-ROW-PINS and MATRIX-COL-PINS
This is where you tell the controller how you wired up the rows and columns and on which pins to find things. Now when you chose which pins to use on the controller, you chose them as ‘Hardware Pins’ what the firmware wants to know is the software or ‘AVR’ names of those pins. With a little digging, I was able to find this:

ProMicroPinoutAvrNames

So now we can convert our hardware pin names to AVR pin names and then just to add another degree of what-were-they-thinking, we’ll strip the beginning ‘P’ off of each. That led me to a section that looks like this:

#define MATRIX_ROW_PINS { B6, B2, B5, B4 }
#define MATRIX_COL_PINS { D1, D0, D4, C6, D7, E6, F4, F5, F6, F7, B1, B3 }

If you used the same pin assignments from my notes above, you can safely use the above lines for your firmware.

Second File: qmk_firmware/keyboards/planck/rules.mk

If you’re using a Teensy, you’ll want to change -DBOOTLOADER_SIZE to ‘512’. To use a ProMicro, you don’t have to change anything here.

Third File: qmk_firmware/keyboards/planck/keymaps/…

So here is where it gets cool. You get to define what every key on your new board does. And it goes way beyond anything you can do on a regular keyboard. You want a key that automatically spits out your email address? No problem. You have a need for a Euro symbol on an ANSI layout? We can do that. This is a deep rabbit hole to fall into and the perfect layout is very personal to you. I’m a long time Planck user so I know MY perfect layout, it’s up to you to find yours.

To get you going, let’s start with the default layout. It’s reasonable and usable. You’ll quickly figure out what parts of it do and don’t work for you and you can modify it to fit your needs. Start by making a copy of the ‘default’ folder and changing the name from ‘default copy’ to your name. The only thing to notice here is that the ‘KC_SPACE’ is repeated on each of the layouts. This is because some Plancks (MIT layout) have a 2u spacebar and some (Grid layout) have 2 1u keys in its place. We just made the grid layout so we have the option of setting each of the space bars to different things. Just replace one or the other and enjoy the extra functionality.

Stage 4: Firmware, meet Controller

This is where articles like this get less useful. The process to build and upload your firmware is completely different depending on if you use Windows, Mac, or Linux. I’m a Mac user and I know how I did it, but that might not help you. Fortunately, QMK offers some good advice if you know where to look.

To get your ‘build environment’ setup, look at ‘qmk_firmware/docs/BUILD_GUIDE.md’. It will lead you through the setup of WinAVR, or DFU, depending on your OS. It also includes directions on how to build and upload. I’ll go ahead and detail what I did on macOS Sierra as it is a little different than what they outline.

  • $ cd qmk_firmware/keyboards/planck
  • $ make clean
  • $ make clint
  • $ ls /dev/cu.*
  • Hold reset button on pro micro for 5 sec
  • $ ls /dev/cu.*
  • make note of new number, insert into |x|x|x| section below
  • $ cd ../../.build
  • $ avrdude -p atmega32u4 -P /dev/cu.usbmodem|X|X|X| -c avr109 -U flash:w:./planck_rev4_clint.hex
  • Hit enter within 8 seconds (on a different keyboard, in case that wasn’t obvious.
  • Unplug and plugin the new keyboard to power cycle it.

This is what worked for me. Your mileage may vary. If you’re on Windows or Linux, it certainly won’t work for you. I’ve used this same procedure on OS X Mavericks/Yosemite and macOS Sierra. The instructions provided by QMK outline a different path that should work for you if my method doesn’t.

Stage 5: Enjoy

Now that you have a fancy new keyboard, give it a spin. Do a few runs on TypeRacer. Learn to type if you haven’t already. Start a blog and type your first article. The world is yours.

more images @ imgur