Automating Raspberry Pi setup (and ESP32, and Linode) with Ansible

Back in 2017 when I first moved off the old and busted NinjaBlocks platform to a Raspberry Pi for my temperature sensor setup, I said:

[…] if the hardware itself died I’d be stuck; yes, it was all built on “open hardware” but I didn’t know enough about it all to be able to recreate it.

I definitely have no problems with the hardware now (and with my move to ESP32s and MQTT for the temperature sensors themselves it’s even simpler), but I recently realised that the software configuration on the Raspberry Pis was still a problem: the configuration and setup of everything had had a steady pace of organic updates and tweaks and if one of the SD cards died or had a problem and I had to reformat it, I’d have a hell of a time recreating it afresh. On the “main” Raspberry Pi 4B+ alone I would need to:

  • Install all the various software packages (vim, git, tmux, Docker, etc.)
  • Add my custom shell configuration and bashmarks
  • Install Chrony and configure it to allow access from the ESP32s
  • Configure the drivers for the JustBoom DAC HAT and install and configure shairport-sync to allow for AirPlay to the big speakers in the lounge room
  • Configure and run the Mosquitto Docker container to allow all my temperature, humidity, air quality, and power data to flow to all the various places it needs to go
  • Configure and run the pi-home-dashboard Docker container so the Raspberry Pi Zero Ws can display our little temperature dashboards
  • Configure and run the powerwall-to-pvoutput-uploader Docker container so our power usage data can be both sent to PV Output and also be read by InfluxDB and the Raspberry Pi Zero W dashboard

In addition to all of that, the Raspberry Pi Zero Ws that display our little dashboards required a whole bunch of constant tweaks and changes to get them working properly (they run a full Linux desktop and display the dashboard in Chromium, and Chromium has an extremely irritating habit of giving a “Chromium didn’t shut down properly” message instead of loading the page you tell it to if there were basically any issues at all; fixing that required a whole lot of trial and error until I narrowed down the specific incantations to get it to stop).

Setting all of that back up from scratch would have been an absolute nightmare so I figured I should probably automate it in some fashion. And in addition, it means I would be able to keep future changes under version control as well so no matter what I’ve done I’d be able to restore it if necessary. At work we use a piece of software called Ansible to bring up the required software and configuration on the Amazon EC2 instances we run, so I thought that’d be a good place to start.

Even though it’s developed by RedHat and isn’t just some random open-source piece of software (although it is open-source), I still found the documentation to be not great in terms of explaining everything step-by-step and building on that knowledge in subsequent steps. But after a bunch of reading and trial and error, plus several weeks of getting it all working, I have my entire Raspberry Pi setup for all the Pis we have at home fully automated! I can wipe the SD card, reflash it with a fresh copy of Raspbian, then run Ansible and it gets everything installed and configured and working exactly how I need it.

I uploaded the whole shebang to GitHub to hopefully help others out as well. It’s obviously completely customised for our setup, but it at least gives a reasonable idea of how everything works.

It starts by creating an inventory, basically a list of the hostnames you want to run playbooks against (a “playbook” is a file that describes the list of steps to run in order to get to the desired state you need). Alongside that, you can group the hosts together so you can target a playbook to run on a specific set of hosts. For example, the server group only consists of the main Raspberry Pi 4B+ described above, but the dashboards consists of all three of the Pi Zero Ws that are running the dashboards.

One of the really handy things with Ansible is that you can set variables that will be reused in various places, and you can configure them for all hosts, or per group, or per individual host. I’m using a combination of those, and inside the server group it will actually look up items from within 1Password so I can commit the code to source control and not be storing secrets in it. You can also set variables per individual host as well, which I use to specify the dashboard URL that each of the Pi Zero Ws should load.

The playbooks themselves live in the playbooks directory, and they specify a set of hosts to run against, and a series of roles to run. The roles are reusable sets of tasks, so for example I run the initialisation role against all of the Raspberry Pis no matter what they’re ultimately going to be doing, for doing the initial things like setting the hostname and updating all the software packages, configuring Git, etc.

After the initialisation is done, the server playbook will run all the various steps to get Docker installed, NVM and Node.js installed, then get everything else configured and installed that needs to be configured and installed. Compare to the dashboards playbook that will also run the same initialisation steps, but then runs the dashboard role which installs the drivers for the HyperPixel display and various other things that need doing, and will configure the autostart file so on boot Chromium comes up with the correct URL depending on which Raspberry Pi it’s running on! The dashboard_url variable in that template file is set in the host_vars directory per specific hostname, so I can customise it for each one.

After my complete success here, I decided I wanted to do the same for when I needed to reflash my ESP32s, because previous it relied on me remembering to update a configuration file with the name of the ESP32 and I had definitely messed that up on a couple of occasions. That was relatively straightforwards as well, and I added it to my esp32-sensor-reader-mqtt repository (and included a playbook to just erase a board because I never did it quite often enough to remember what the specific steps were).

And then finally after that, I decided I should also automate my Linode setup. I’d posted back in 2019 about using Linode’s StackScripts to set everything up, but the problem with that is that you run it at the very beginning and your Linode is set up appropriately at that point, but any changes you make after that aren’t saved anywhere, so you’re essentially back to square one again. The final sentence in that blog post was this:

As long as I’m disciplined about remembering to update my StackScript when I make software changes, whenever the next big move to a new VM is should be a hell of a lot simpler.

But in news that will surprise nobody, I was not at all disciplined. 😅 The other problem is that you can’t test the StackScript as you’re going (they only run when you first spin up the Linode afresh), you have to update it and hope those steps work in future. With Ansible, the idea is that everything is idempotent, so you can run everything as many times as you want and it won’t change if something has been configured already, so it enables you to easily test out parts of the playbook updates without needing to wipe the whole damn thing. It’s taken a bit over a month of working on it after dinner and on weekends, but now the whole Linode setup is fully automated as well.

However, the other wrinkle is that where the Raspberry Pis don’t store any data on them and can just be wiped without problem, my Linode hosts this blog, my website and all its images, all the various images I’ve posted to Ars Technica over the years, and a bunch of other things too. I ended up splitting the Ansible process into two, there’s the configuration and then another separate playbook that copies everything over from the old Linode onto the new one and then registers a new SSL certificate with Let’s Encrypt and updates Cloudflare to point the DNS of my website and blog and the general server hostname to the new Linode.

That bit was a bit nerve-wracking, I tested the process a bunch of times and pulled the trigger for real last weekend, then had a minor panic attack when I realised that the database dump of my website hadn’t been reimported since I originally tested the process back on the 9th (I suspect it didn’t import because there was already data in the database, but unfortunately it didn’t actually error out so I didn’t know), so I didn’t have any of the posts I’d made nor any of the temperature data since then. 😬 Fortunately I realised this the morning after I’d done the migration and had wisely left the old Linode up and running, so I renamed the database on the new Linode so I could harvest the temperature data that been sent since the migration, dumped the old database from the old Linode and imported it into the correct location on the new Linode, and then exported and reimported just the missing temperature data, and we’re back in business.

This time I should be able to revisit this in three or four years when I next do a big upgrade and it should actually be quite painless (famous last words, I know, but I’m much more confident this time).

Upping my monitoring game with MQTT

Previously on Monitor All The Things:

(The display that used to show the air quality has been changed to show a clock instead, and the air quality monitoring is done via another ESP32 now. I’m also sensing a definite theme with my blog post titles here).

I hadn’t blogged about it, but I also have all of this (indoor and outdoor temperature and humidity, power usage and generation plus battery charge, and outdoor air quality) going into InfluxDB for visualising in Grafana. The dashboard I made looks like this:

Pretty spiffy, eh?

It’d very much evolved rather organically as I went though, so lots of different things on different hosts sending HTTP calls all over the place, including my own slightly dodgy system for getting the ESP32s that are connected to the temperature sensor to save their readings onto the local filesystem if my website couldn’t be contacted (for example if we had an internet outage), as well as two separate things hitting the Powerwall’s local API every five seconds to pull the power data (one for the little HyperPixel display at the front of the house, and one for the visualisation stuff above).

I figured there had to be a cleaner and more elegant way of doing this. At work I deal with Amazon’s Simple Queue Service (SQS) quite a lot and use it in one of the services I built, so I wondered if there was a way to accomplish something similar myself, so I could just have everything drop messages onto a queue and have the things that need to read them pick those messages up from the queue.

Turns out there is, and it’s called MQTT!

It’s an absurdly simple and lightweight protocol, you have a central server called a “broker”, a publisher that sends messages to a given topic on the broker, and as many subscribers as you want that also connect to the broker and each listen on a topic or topics, and the broker ensures those messages get from the publisher to each subscriber. There’s also quality of service settings where you can have it guarantee that the message is received by the subscriber at least once, and it’ll queue up the messages for the subscribers if they drop offline and the messages will all be sent once the subscriber comes back.

Interestingly, you can also have a broker on one machine connect to a broker on another machine, and have it send messages on a particular topic to the remote broker, which seemed like it’d be a good way to get weather updates to my website.

There’s a guy who wrote an MQTT client library in MicroPython for the ESP32, mqtt_as, so that would take care of the ESP32 side of things, I’d use a popular open-source MQTT broker called Mosquitto, and there’s a Javascript MQTT client called MQTT.js that would be used for my website and all the other TypeScript parts of the setup.

I did a bunch of brainstorming in draw.io and came up with this elaborate diagram:

(Mechanise is the hostname of my Linode, which my website runs on, and PVOutput is a website for sending your solar power generation data to, a bunch of people at work also do the same and we’re all in the same “team” so we can see how much combined we’ve all generated together).

After that, it just involved a whole bunch of coding (as well as ordering two spare ESP32s so I could test that my code worked without having to pull apart my existing setup), which I’ve uploaded to GitHub:

Despite having written up a careful plan and done what I thought was getting all my ducks in a row to make a quick switchover this morning, there were a number of things I ran into that caused it to take a few hours to get going (things like forgetting to configure PostgreSQL on the Raspberry Pi 4B to allow things running in Docker to access it, needing to add an extra published port on the Linode so my website could connect to Mosquitto, and most annoyingly of all, a recent VSCode update breaking Pymakr and having to revert to an old version of both pieces of software). I got everything up and running in the end, and now if I add any new monitoring things, it’ll be quite simple to publish the data to Mosquitto and slurp it up into InfluxDB!

More fun with temperature sensors: ESP32 microcontrollers and MicroPython

More fun with temperature sensors: ESP32 microcontrollers and MicroPython

I’ve blagged previously about our temperature/humidity sensor setup and how they’re attached to my Raspberry Pis, and they’ve been absolutely rock-solid in the three-and-a-half years since then. A few months ago, a colleague at work had mentioned doing some stuff with an ESP32 microcontroller and just recently I decided to actually look up what that was and what one can do with it, because it sounded like it might be a fun new project to play with!

From Wikipedia: ESP32 is a series of low-cost, low-power system on a chip microcontrollers with integrated Wi-Fi and dual-mode Bluetooth.

So it’s essentially a tiny single-purpose computer that you write code for and then flash that code onto the board, rather than like with the Raspberry Pi where it has an entire Linux OS running on it. It runs at a blazing fast 240MHz and has 320KB of RAM. The biggest draw for me was that it has built-in wifi so I could do networked stuff easily. There’s a ton of different boards and options and it was all a bit overwhelming, but I ended getting two of Adafruit’s HUZZAH32s which come with the headers for attaching the temperature sensors we have already soldered on. Additionally, they have 520KB of RAM and 4MB of storage.

Next up, I needed to find out how to actually program the thing. Ordinarily you’d write in C like with an Arduino and I wasn’t too keen on that, but it turns out there’s a distribution of Python called MicroPython that’s written explicitly for embedded microcontrollers like the ESP32. I’ve never really done much with Python before, because the utter tyre fire that is the dependency/environment management always put me off (this xkcd comic is extremely relevant). However, with MicroPython on the ESP32 I wouldn’t be having to deal with any of that, I’d just write the Python and upload it to the board! Additionally, it turns out MicroPython has built-in support for the DHT22 temperature/humidity sensor that I’ve already been using with the Raspberry Pis. Score!

There was a lot of searching over many different websites trying to find how to get all this going, so I’m including it all here in the hopes that maybe it’ll help somebody else in future.

Installing MicroPython

At least on macOS, first you need to install the USB to UART driver or your ESP32 won’t even be recognised. Grab it from Silicon Labs’ website and get it installed.

Once that’s done, follow the Getting Started page on the MicroPython website to flash the ESP32 with MicroPython, substituting /dev/ttyUSB0 in the commands for /dev/tty.SLAB_USBtoUART.

Using MicroPython

With MicroPython, there’s two files that are always executed when the board starts up, boot.py which is run once at boot time and is generally where you’d put your connect-to-the-wifi-network code, and main.py which is run after boot.py and will generally be the entry point to your code. To get these files onto the board, you can use a command-line tool called ampy, but it’s a bit clunky and also not supported anymore.

However, there is a better way!

Setting up the development environment

There are two additional tools that make writing your Python code in Visual Studio Code and uploading to the ESP32 an absolute breeze.

The first one is micropy-cli, which is a command-line tool to generate the skeleton of a VSCode project and set it up for full autocompletion and Intellisense of your MicroPython code. Make sure you add the ESP32 stubs first before creating a new micropy project.

The second is a VSCode extension called Pymakr. It gives you a terminal to connect directly to the board and run commands and read output, and also gives you a one-click button to upload your fresh code, and it’s smart enough not to re-upload files that haven’t changed.

There were a couple of issues I ran into when trying to get Pymakr to recognise the ESP32 though. To fix them, bring up the VSCode command palette with Cmd-Shift-P and find “Pymakr > Global Settings”. Update the address field from the default IP address to /dev/tty.SLAB_USBtoUART, and edit the autoconnect_comport_manufacturers array to add Silicon Labs.

Replacing the Raspberry Pis with ESP32s

After I had all of that set up and working, it was time to start coding! As I mentioned earlier I’ve not really done any Python before, so it was quite the learning experience. It was a good few weeks of coding and learning and iterating, but in the end I fully-replicated my Pi Sensor Reader setup with the ESP32s, and with some additional bits besides.

One of the things my existing Pi Sensor Reader setup did was to have a local webserver running so I could periodically hit the Pi and display the data elsewhere. Under Node.js this is extremely easily accomplished with Express, but using MicroPython the options were more limited. There are a number of little web frameworks that people have written for it, but they all seemed quite overkill.

I decided to just use raw sockets to write my own, though one thing I didn’t appreciate until this point was how Node.js’s everything-is-asynchronous-and-non-blocking makes doing this kind of thing very easy, you don’t have to worry about a long-running function causing everything else to grind to a halt while it waits for that function to finish. Python has a thing called asyncio but I was struggling to get my head around how to use it for the webserver part of things until I stumbled across this extremely helpful repository where someone had shown an example of how to do exactly that! (I even ended up making a pull request to fix an issue I discovered with it, which I’m pretty stoked with).

One of the things I most wanted to do was to have some sort of log file accessible in case of errors. With the Raspberry Pi I can just SSH in and check the Docker logs, but once the ESP32s were plugged into power and running, you can’t easily do a similar thing. I ended up writing the webserver with several endpoints to read the log, clear it, reset the board, and view and clear the queue of failed updates.

The whole thing has been uploaded to GitHub with a proper README of how it works, and they’ve been running connected to the actual indoor and outdoor temperature sensors and posting data to my website for just under a week now, and it’s been absolutely flawless!

More Raspberry Pi-powered monitoring: air quality!

Here in New South Wales, last year’s bushfires over late spring and into summer were astoundingly bad, and there were days where Sydney had the poorest air quality on the entire planet. Everyone was watching the PM2.5 values, and there were days where Kristina couldn’t go outside because of her asthma. I figured it’d be neat to set up a Raspberry Pi-powered air quality sensor and had ordered the sensor back in February but didn’t get around to putting it into service until now.

This is the bit that lives inside so we can easily see the latest reading:

A small 4" LCD display showing the air quality values for PM1.0, PM2.5, and PM10.

It uses the same sort of setup as my Pimoroni display, and I updated my pi-home-dashboard to add a second page to display the values from the air quality reader.

The sensor itself is a Plantower PMS5003 sensor and is attached to the same Raspberry Pi that the outdoor temperature sensor is on. Adafruit’s instructions on getting it set up were pretty straightforward, and they also give some sample code for how to read it, but it’s in Python which I intensely dislike (I don’t really even have any strong feelings about the language itself one way or the other, but I’ve never had a good experience with the damn package management around it, so I do my damnedest to avoid it). I was able to write the same logic in TypeScript instead — though had to consult the clever people on Ars Technica because parsing the output from the sensor involves things like bit-shifting which is quite low-level and something I’m utterly unfamiliar with — and chucked the whole thing up on GitHub. It takes ten readings and averages them, and has an HTTP endpoint for pulling the latest values.

I’ve set the front-end up so the colour of the numbers will change to orange and red depending on how bad the air quality is, but hopefully it’s a long while before we actually see that in action!

More space: the Pimoroni HyperPixel4 display on a Raspberry Pi Zero W

Back at the start of 2018 I blogged about my Raspberry Pi temperature display setup and it’s been pretty excellent and utterly reliable since then, but because of its small size — the display is only 2 inches — it wasn’t particularly visible from across the room. That, combined with the discovery that the Envoy power consumption monitoring system we had installed with the solar panels has a locally-accessible API that you can use to get real-time production and consumption data (which lives at http://<ip-of-the-envoy-box>/production.json?details=1), made me start looking into larger displays so I could include both temperature/humidity data and our power consumption.

My first port of call was the 2.7-inch version of the original 2-inch display. I ordered it on the 6th of April then… nothing showed up. I’d assumed the PaPiRus was MIA and had instead ordered a 4-inch, 800×480-pixel display in the form of Pimoroni’s HyperPixel4 display, the non-touch version. The Raspberry Pi registers it as a regular display so you run a full desktop environment windowing system on it rather than the way the PaPiRus works.

Of course, about a week after ordering the HyperPixel 4, the PaPiRus finally arrived! The 2.7-inch version of the PaPiRus is 264 pixels wide by 176 pixels high, so not exactly high-resolution. There’s actually quite a lot of freedom to tweak the position of the elements on screen pixel-by-pixel, but I quickly discovered that that’s extremely tedious when doing it directly on the Raspberry Pi itself because it takes several seconds for it to contact the required endpoints to pull in the data and then refresh the whole display. As well as writing text, the display can also display (1-bit) bitmap images, so I decided to change tack and instead of using the PaPiRus’s text API I wrote a probably-slightly-overengineered Node.js application that would run on the Raspberry Pi 4B, fetch the data from the outdoor and indoor sensors as well as the Envoy, use the Javascript Canvas API to lay everything out, and then convert it to a bitmap image that the Python script on the Pi Zero W would fetch every minute and then update the display with.

The biggest advantage of this system is that I could run it locally on my regular computer to quickly tweak the positioning without having to wait for the PaPiRus display to refresh each time, and I set it up so I could invert the colours to be white on black instead so I could clearly see the boundaries of the canvas. I put the code up on GitHub if anyone is interested in poking through it, and the end result looks like this:

Having over-engineered my Node.js solution, the HyperPixel4 display arrived maybe a couple of weeks later! It’s extremely slick-looking, but unfortunately the little plastic nubs that are meant to keep the screen in place in the house aren’t actually big enough to hold it in, and I managed to have the display itself pop out and crack some of the wires that feed the display and it caused all sorts of display weirdness. I emailed the place that makes the HyperPixel display about it and they were super nice and helpful and sent me out a replacement display with no questions asked! While I was waiting for the new one to arrive, the old broken one was partially working enough that I could at least get everything up and running how I wanted it, anyway.

Because using the HyperPixel is the same as if you’d hooked up an HDMI display and were using the Pi as a regular computer, I started from the full-blown Raspbian desktop image, not the Lite one. It was relatively straightforward to get everything going (mostly just installing and configuring the driver from Pimoroni’s GitHub repository), but there were some additional things I needed to do to get everything working as I wanted. I settled on a Node.js backend and React frontend setup (the separate backend was necessary because CORS; I couldn’t hit the Envoy URL directly from the browser on the Pi, so I have to have the Node.js backend pull in the data and then feed it to the React app), both of which are running in a Docker image on the Raspberry Pi 4B.

  • By default the HyperPixel4 runs at full brightness, so I followed this to turn it way down, and also to set up a cron job to entirely turn the display off at midnight and turn it back on at 8am.
  • To get the Pi to open Chromium full-screen on boot, I followed these instructions.
  • To disable the annoying “Restore pages” dialog in Chromium, this on the Raspberry Pi Stack Exchange was helpful.
  • Raspbian comes by default with a VNC server installed, just not enabled. To enable it and allow access directly from macOS’s “Connect to Server” dialog in the Finder:
    • Run sudo raspi-config, go to Interface Options > VNC and enable it.
    • Run vncpasswd -service to set a VNC password (note if it’s longer than eight characters, only the first eight are used when connecting).
    • Create the file /etc/vnc/config.d/common.custom with the contents: Authentication=VncAuth
    • Then Restart the VNC service with sudo systemctl restart vncserver-x11-serviced
  • And lastly, to disable the Pi from turning the screen off after activity, I followed these steps.

My ~/.config/lxsession/LXDE-pi/autostart ultimately ended up looking like this:

@lxpanel --profile LXDE-pi
@pcmanfm --desktop --profile LXDE-pi
point-rpi
@chromium-browser --start-fullscreen --start-maximized --app=http://fourbee:3003
@xset s off
@xset -dpms 
@xset s noblank
@sudo /home/pi/Source/rpi-hardware-pwm/pwm 19 1000000 135000

And the whole setup looks like this:

A photo of a small LCD display showing outdoor and indoor temperature and current power consumption and production. The text is white on black.

It’s quite the improvement in visibility and I can easily read it from all the way in the kitchen! It updates itself automatically every 30 seconds, and there’s no e-ink full-display-refresh screen-blanking when it does.

HomePod, Docker on Raspberry Pi, and writing Homebridge plugins

Apple announced the HomePod “smart speaker” in 2017, and started shipping them in early 2018. I had zero interest the smart speaker side of things — I’d never have Google or Amazon’s voice assistants listening to everything I say, and despite trusting Apple a lot more with privacy compared to those two companies, the same goes for Siri — but the praise for the sound quality definitely piqued my interest, especially having set up shairplay-sync on the Raspberry Pi as an AirPlay target and enjoying the ease of streaming music to a good set of speakers. For AU$499 though, I wasn’t going to bother as the setup for the stereo system in our home office did a reasonable enough job. It consisted of an amplifier that was sitting next to my desk, going into an audio switchbox that sat next to my computer and could be switched between the headphone cable attached to my computer, and one that snaked across the floor to the other side to Kristina’s desk so she could plug into it, with the speakers were sitting on the bookshelves on opposite sides of the room (you can see how it looked in this post, the speakers are the black boxes visible on the bottom shelves closest to our desks).

Fast-forward to last week, and someone mentioned that JB Hi-Fi were having a big sale on the HomePod and it was only AU$299! The space behind my desk was already a rat’s nest of cables, and with the standing desk I’ve ordered from IKEA I was wanting to reduce the number of cables in use, and being able to get rid of a bunch of them and replace it with a HomePod, I decided to get in on it (it’s possible to turn the “Listen for ‘Hey Siri'” functionality off entirely).

It arrived on Tuesday, and to say I’m impressed with the sound quality is a bit of an understatement, especially given how diminutive it is. It has no trouble filling the whole room with sound, the highs are crystal clear, and if the song is bassy enough you can feel it through the floor! It shows up just as another AirPlay target so it’s super-easy to play music to it from my phone or computer. I took a photo of our new setup and you can see the HomePod sitting on the half-height bookshelf right at the bottom-left of the frame (the severe distortion is because I took the photo on our 5D4 with the 8-15mm Fisheye I borrowed from a friend, which requires turning lens corrections on to avoid having bizarrely-curved vertical lines, which in turn distorts the edges of the image quite a bit).

The setup and configuration of the HomePod is done via Apple’s Home app, which uses a framework called HomeKit to do all sorts of home automation stuff, and the HomePod is one of the devices that can work as the primary “hub” for HomeKit. I have no interest in home automation as such, but a selling point of the HomeKit is that’s a lot more secure than random other automation platforms, and one of the things it supports is temperature sensors. Someone wrote a Node.js application called Homebridge that lets you run third-party plugins and even write your own ones to appear and interact with in HomeKit, so I decided I’d see if I could hook up the temperature sensors that are attached to the Raspberry Pi(s)!

I’d ordered a 4GB Raspberry Pi 4B last month because I wanted to have a bit more grunt than the existing Pi 3B — which only has 1GB RAM — and to start using Docker with it, and it arrived on the 1st of this month. With that up and running inside in place of my original Raspberry Pi 3B, I moved the Pi 3B and the outside temperature sensor much further outside and attached it to our back room that’s in the backyard, because the previous position of the sensor underneath the pergola and next to the bricks of the house meant that in summer the outdoor temperatures would register hotter than the actual air temperature, and because the bricks absorb heat throughout the day, the temperatures remain higher for longer too.

Installing and configuring Homebridge

Next step was to set up Homebridge, which I did by way of the oznu/docker-homebridge image, which in turn meant getting Docker — and learning about Docker Compose and how handy it is, and thus installing it too! — installed first:

  1. Install Docker — curl -sSL https://get.docker.com | sh
  2. Install Docker Compose — sudo apt-get install docker-compose
  3. Grab the latest docker-homebridge image for Raspberry Pi — sudo docker pull oznu/homebridge:raspberry-pi
  4. Create a location for your Homebridge configuration to be stored — mkdir -p ~/homebridge/config

Lastly, write yourself a docker-compose.yml file inside ~/homebridge

version: '2'
services:
  homebridge:
    image: oznu/homebridge:raspberry-pi
    restart: always
    network_mode: host
    volumes:
      - ./config:/homebridge
    environment:
      - PGID=1000
      - PUID=1000
      - HOMEBRIDGE_CONFIG_UI=1
      - HOMEBRIDGE_CONFIG_UI_PORT=8080

Then bring the Homebridge container up by running sudo docker-compose up --detach from ~/homebridge. The UI is accessible at http://<address-of-your-pi>:8080 and logs can be viewed with sudo docker-compose logs -f.

The last step in getting Homebridge recognised from within the Home app is iOS is to open the Home app, tap the plus icon in the top-right and choose “Add accessory”, then scan the QR code that the Homebridge UI displays.

Writing your own Homebridge plugins

Having Homebridge recognised within the Home app isn’t very useful without plugins, and there was a lot of trial and error involved here because I was writing my own custom plugin rather than just installing one that’s been published to NPM, and I didn’t find any single “This is a tutorial on how to write your own plugin” pages.

Everything is configured inside ~/homebridge/config, which I’ll refer to as $CONFIG from now on.

Firstly, register your custom plugin so Homebridge knows about it by editing $CONFIG/package.json and editing the dependencies section to add your plugin. It has to be named homebridge-<something> to be picked up at all, I called mine homebridge-wolfhaus-temperature and so my $CONFIG/package.json looks like this:

{
  "private": true,
  "description": "This file keeps track of which plugins should be installed.",
  "dependencies": {
    "homebridge-dummy": "^0.4.0",
    "homebridge-wolfhaus-temperature": "*"
  }
}

The actual code for the plugin needs to go into $CONFIG/node_modules/homebridge-<your-plugin-name>/, which itself is a Node.js package, which also needs its own package.json file located at $CONFIG/node_modules/homebridge-<your-plugin-name>/package.json. You can generate a skeleton one with npm init — assuming you have Node.js installed, if not, grab nvm and install it — but the key parts needed for a plugin to be recognised by Homebridge is adding the keywords and engine sections into your package.json:

{
  "name": "homebridge-wolfhaus-temperature",
  "version": "0.0.1",
  "main": "index.js",
  "keywords": [
    "homebridge-plugin"
  ],
  "engines": {
    "homebridge": ">=0.4.53"
  }
}

index.js is your actual plugin code that will be run when Homebridge calls it.

Once I got this out of the way, the last bit was a LOT of trial and error to actually get the plugin working with Homebridge and the Home app on my iPhone. The main sources of reference were these:

After several hours work, I had not the nicest code but working code (Update 2020-04-12 — moved to ES6 classes and it’s much cleaner), and I’ve uploaded it to GitHub.

The final bit of the puzzle is telling Homebridge about the accessories, which are the things that actually show inside the Home app on iOS. For this, you need to edit $CONFIG/config.json and edit the accessories section to include your new accessories, which will use the plugin that was just written:

{
    "bridge": {
        "name": "Traverse",
        [...]
    },
    "accessories": [
        {
            "accessory": "WolfhausTemperature",
            "name": "Outdoor Temperature",
            "url": "http://pi:3000/rest/outdoor"
        },
        {
            "accessory": "WolfhausTemperature",
            "name": "Indoor Temperature",
            "url": "http://fourbee:3000/rest/indoor"
        }
    ],
    "platforms": []
}

The url is the REST endpoint that my pi-sensor-reader runs for the indoor and outdoor sensors, and the name needs to be unique per accessory.

Homebridge needs restarting after all these changes, but once you’re done, you’ll have two new accessories showing in Home!

They initially appear in the “Default Room”, you can add an “Indoor” and “Outdoor” room to put them into by tapping on the Rooms icon in the bottom bar, then tapping the hamburger menu at the top-left, choosing Room Settings > Add Room, then long-pressing on the temperature accessory itself and tapping the settings cog at the bottom-right and selecting a different room for it to go into.

What’s next?

As part of doing all this, I moved all of my public Git repositories over to GitHub where they’re more likely to be actually seen by anybody and will hopefully help someone! I also updated my pi-sensor-reader to use docker-compose, and fully-updated the README to document all the various options.

Next on the Homebridge front is going to be tidying up the plugin code — including moving to async/await — and adding the humidity data to it!

More Raspberry Pi adventures: the Pi Zero W and PaPiRus ePaper display

I decided I wanted to have some sort of physical display in the house for the temperature sensors so we wouldn’t need to be taking out our phones to check the temperature on my website if we were already inside at home. After a bunch of searching around, I discovered the PaPiRus ePaper display. ePaper means it’s not going to have any bright glaring light at night, and it also uses very little power.

The Raspberry Pi is hidden away under a side table, and already has six wires attached to the header for the temperature sensors, so I decided to just get a separate Raspberry Pi Zero W — which is absurdly small — and the PaPiRus display.

Setting it up

I flashed the SD card with the Raspbian Stretch Lite image, then enabled SSH and automatic connection to our (2.4GHz; the Zero W doesn’t support 5GHz) wifi network by doing the following:

1) Plug the flashed SD card back into the computer.

2) Go into the newly-mounted “boot” volume and create an empty file called “ssh” to turn on SSH at boot

3) Also in the “boot” volume, create a file called “wpa_supplicant.conf” and paste the following into it:

country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="WIFI_SSID"
psk="WIFI_PASSWORD"
key_mgmt=WPA-PSK
}

4) Unmount the card, pop it into the Pi, add power, and wait 60-90 seconds and it’ll connect to your network and be ready for SSH access! The default username on the Pi is “pi” and the password is “raspberry”.

(These instructions are all thanks to this blog post but I figured I’d put them here as well for posterity).

The PaPiRus display connection was dead easy, I just followed Pi Supply’s guide after soldering a header into the Pi Zero W. If you want to avoid soldering, they also offer the Zero W with a header pre-attached.

Getting the Python library for updating the display was mostly straightforward, I just followed the instructions in the GitHub repository to manually install the Python 3 version.

I wrote a simple Python script to grab the current temperature and humidity from my website’s REST endpoints, and everything works! This script uses the “arrow” and “requests” libraries, which can be installed with “sudo apt-get install python3-arrow python3-requests”.

Next step is to have the Pi 3 that has the sensors run a simple HTTP server that the Zero W can connect to it, so even if we have no internet connection for whatever reason, the temperatures will still be available at home. I’ve updated my Pi Sensor Reader to add HTTP endpoints.

Temperature sensors: now powered by Raspberry Pi

The Weather section on my website is now powered by my Raspberry Pi, instead of my Ninja Block! \o/

Almost exactly three years ago, I started having my Ninja Block send its temperature data to my website (prior to that, I was manually pulling the data from the Ninja Blocks API and didn’t have any historical record of it). Ninja Blocks the company went bust in 2015, and there was some stuff in the Ninja Blocks software that relied on their cloud platform to work and I ended up with no weather data for a couple of days because the Ninja Block couldn’t talk to the cloud platform. I ended up hacking at it and the result was this very simple Node.js application as a replacement for their software. It always felt a bit crap, though, because if the hardware itself died I’d be stuck; yes, it was all built on “open hardware” but I didn’t know enough about it all to be able to recreate it. I’d ordered a Raspberry Pi 3 in June last year, intending on replacing the Ninja Block and it’s sometimes-unreliable wireless temperature sensors with something newer and simpler and hard-wired, but I found there was a frustrating lack of solid information regarding something that on the surface seemed quite simple.

I’ve finally gotten everything up and running, the Ninja Block has been shut down, and I’ve previously said I’d write up exactly what I did. So here we are!

Components needed

  • Raspberry Pi 3 Model B+
  • AM2302 wired temperature-humidity sensor (or two of them in my case)
  • Ethernet cable of the appropriate length to go from the Pi to the sensor
  • 6x “Dupont” female to either male or female wires (eBay was the best bet for these, just search for “dupont female”, and it only needs to be female on one end as the other end is going to be chopped off)
  • 1.5mm heatshrink tubing
  • Soldering iron and solder
  • Wire stripper (this one from Jaycar worked brilliantly, it automatically adjusts itself to diameter of the insulation)

Process

  1. Cut the connectors off one end of the dupont cables, leaving the female connector still there, and strip a couple of centimetres of insulation off.
  2. Strip the outermost insulation off both ends of the ethernet cable, leaving a couple of centimetres of the internal twisted pairs showing.
  3. Untwist three of the pairs and strip the insulation off them, then twist them back together again into their pairs.
  4. Chop off enough heatshrink tubing to cover the combined length of the exposed ethernet plus dupont wire, plus another couple of centimetres, and feed each individual dupont wire through the tubing (there should be three separate bits of tubing, one for each wire).
  5. Solder each dupont wire together with one of the twisted pairs of ethernet cable, then move the heatshrink tubing up over the soldered section and use a hairdryer or kitchen blowtorch to activate the tubing and have it shrink over the soldered portion to create a nice seal.
  6. Repeat this feed-heatshrink-tubing/solder-wire/activate-heatshrink process again but with the cables that come out of the temperature sensor (ideally you should be using the same red/yellow/black-coloured dupont cables to match the ones that come out of the sensor itself, to make it easier to remember which is which).
  7. Install Raspbian onto an SD card and boot and configure the Pi.
  8. Using this diagram as a reference, plug the red (power) cable from the sensor into Pin 2 (the 5V power), the yellow one into Pin 7 (GPIO 4, the data pin), and the black one into Pin 6 (the ground pin).

AdaFruit has a Python library for reading data from the sensor, I’m using the node-dht-sensor library for Node.js myself. You can see the full code I’m using here (it’s a bit convoluted because I haven’t updated the API endpoint on my website yet and it’s still expecting the same data format as the Ninja Block was sending).

I’d found a bunch of stuff about needing a “pull-up” resistor when connecting temperature sensors, but the AM2302 page on adafruit.com says “There is a 5.1K resistor inside the sensor connecting VCC and DATA so you do not need any additional pullup resistors”, and indeed, everything is working a treat!

Better Raspberry Pi audio: the JustBoom DAC HAT

I decided that the sound output from the Pi’s built-in headphone jack wasn’t sufficient after all and so went searching for better options (a DAC—digital-to-analog converter).

The Raspberry Pi foundation created a specification called “HAT” (Hardware Attached on Top) a few years ago which specifies a standard way for devices to automatically identify and configure a device and drivers that’s attached to the Pi via its GPIO (General Purpose Input/Output) pins. There’s a number of DACs now that conform to this standard, and the one I settled on is the JustBoom DAC HAT. It’s a UK company but you can buy them locally from Logicware (with $5 overnight shipping no less).

The setup is incredibly simple: connect the plastic mounting plugs and attach the DAC to the Pi, then start it up and edit /boot/config.txt to comment out the default audio setting:

#dtparam=audio=on

Then add three new lines in:

dtparam=audio=off
dtoverlay=i2s-mmap
dtoverlay=justboom-dac

Then reboot. (If you’re running Raspbian Stretch or newer, the i2s-mmap line should not be added).

To say that I’m impressed would be an understatement! I didn’t realise just how crappy the audio from the Pi’s built-in headphone jack was until I’d hooked up the new DAC and blasted some music out. I’m not an audiophile and it’s hard to articulate, but I’d compare it most closely to listening to really low-quality MP3s on cheap earbuds versus high-quality MP3s on a proper set of headphones.

If you’re going to be hooking your Pi into a good stereo system, I can’t recommend JustBoom’s DAC HAT enough!

Raspberry Pi project: AirPlay receiver

I bought a Raspberry Pi almost exactly a year ago, intending on eventually replacing my Ninja Block and its sometimes-unreliable wireless sensors with hardwired ones (apart from the batteries needing occasional changing, there’s something that interferes with the signal on occasion and I just stop receiving updates from the sensor outside for several hours at a time, and then suddenly it starts working again). To do that, I need to physically run a cable from outside under the pergola to inside where the Raspberry Pi will live and I don’t really want to go drilling holes through the house willy-nilly. I want to eventually get the electrician in to do some recabling so I’m going to get him to do that as well, but until then the Pi was just sitting there collecting dust. I figured I should find something useful to do it with, but having a Linode meant that any sort of generic “Have a Linux box handy to run some sort of server on” itch was already well-scratched.

I did a bit of Googling, and discovered Shairport Sync! It lets you use the Raspberry Pi as an AirPlay receiver to stream music to from iTunes or iOS devices, a la an Apple TV or AirPort Express. We already have an Apple TV but it’s plugged into the HDMI port on the Xbox One which means that to simply stream audio to the stereo we have to have the Xbox One, TV, and Apple TV all turned on (the Apple TV is plugged into the Xbox’s HDMI input so we can say “Xbox, on” and the Xbox turns itself on as well as the TV and amplifier, then “Xbox, watch TV” and it goes to the Apple TV; it works very nicely but is a bit of overkill when all you want to do is listen to music in the lounge room).

Installing Shairport Sync was quite straightforward, I pretty much just followed the instructions in the readme there then connected a 3.5mm to RCA cable from the headphone jack on the Raspberry Pi to the RCA input on the stereo. It’s mentioned in the readme, but this issue contains details on how to use a newer audio driver for the Pi that significantly improves the audio output quality.

The only stumbling block I ran into was the audio output being extremely quiet. Configuring audio in Linux is still an awful mess, but after a whole lot of googling I discovered the “aslamixer” tool (thanks to this blog post), which gives a “graphical” interface for setting the sound volume, and it turned out the output volume was only at 40%! I cranked it up to 100% and while it’s still a bit quieter than what the Apple TV outputs, it doesn’t need a large bump on the volume dial to fix—there’s apparently no amplifier or anything on the Raspberry Pi, it’s straight line-level output. The quality isn’t quite as good as going via the Apple TV, but it gets the job done! I might eventually get a USB DAC or amplifier but this works fine for the time being.

On macOS it’s possible to set the system audio output to an AirPlay device, so you can be watching a video but outputting the audio to AirPlay, and the system keeps the video and audio properly in sync. It works extremely well, but the problem we found with having the Apple TV hooked up to the Xbox One’s HDMI input is that there’s a small amount of lag from the connection. When the audio and video are both coming from the Apple TV there’s no problem, but watching video on a laptop while outputting the sound to the Apple TV meant that the audio was just slightly out of sync from the video. Having the Raspberry Pi as the AirPlay receiver solves that problem too!

UPDATE: Two further additions to this post. Firstly, and most importantly, make sure you have a 5-volt, 2.5-amp power supply for the Raspberry Pi. I’ve been running it off a spare iPhone charger which is 5V but only 1A, and the Pi will randomly reboot under load because it can’t draw enough power from the power supply.

Secondly, the volume changes done with the “alsamixer” tool are not saved between reboots. Once you’ve set the volume to your preferred level, you need to run “sudo alsactl store” to persist it (this was actually mentioned in the blog post I linked to above, but I managed to miss it).