Upping my monitoring game with MQTT

Previously on Monitor All The Things:

• 26th July, 2021 — More fun with temperature sensors: ESP32 microcontrollers and MicroPython
• 25th November, 2020 — More Raspberry Pi-powered monitoring: air quality!
• 7th June, 2020 — More space: the Pimoroni HyperPixel4 display on a Raspberry Pi Zero W
• 17th February, 2018 — More Raspberry Pi adventures: the Pi Zero W and PaPiRus ePaper display
• 30th December, 2017 — Temperature sensors: now powered by Raspberry Pi

(The display that used to show the air quality has been changed to show a clock instead, and the air quality monitoring is done via another ESP32 now. I’m also sensing a definite theme with my blog post titles here).

I hadn’t blogged about it, but I also have all of this (indoor and outdoor temperature and humidity, power usage and generation plus battery charge, and outdoor air quality) going into InfluxDB for visualising in Grafana. The dashboard I made looks like this:

Pretty spiffy, eh?

It’d very much evolved rather organically as I went though, so lots of different things on different hosts sending HTTP calls all over the place, including my own slightly dodgy system for getting the ESP32s that are connected to the temperature sensor to save their readings onto the local filesystem if my website couldn’t be contacted (for example if we had an internet outage), as well as two separate things hitting the Powerwall’s local API every five seconds to pull the power data (one for the little HyperPixel display at the front of the house, and one for the visualisation stuff above).

I figured there had to be a cleaner and more elegant way of doing this. At work I deal with Amazon’s Simple Queue Service (SQS) quite a lot and use it in one of the services I built, so I wondered if there was a way to accomplish something similar myself, so I could just have everything drop messages onto a queue and have the things that need to read them pick those messages up from the queue.

Turns out there is, and it’s called MQTT!

It’s an absurdly simple and lightweight protocol, you have a central server called a “broker”, a publisher that sends messages to a given topic on the broker, and as many subscribers as you want that also connect to the broker and each listen on a topic or topics, and the broker ensures those messages get from the publisher to each subscriber. There’s also quality of service settings where you can have it guarantee that the message is received by the subscriber at least once, and it’ll queue up the messages for the subscribers if they drop offline and the messages will all be sent once the subscriber comes back.

Interestingly, you can also have a broker on one machine connect to a broker on another machine, and have it send messages on a particular topic to the remote broker, which seemed like it’d be a good way to get weather updates to my website.

There’s a guy who wrote an MQTT client library in MicroPython for the ESP32, mqtt_as, so that would take care of the ESP32 side of things, I’d use a popular open-source MQTT broker called Mosquitto, and there’s a Javascript MQTT client called MQTT.js that would be used for my website and all the other TypeScript parts of the setup.

I did a bunch of brainstorming in draw.io and came up with this elaborate diagram:

(Mechanise is the hostname of my Linode, which my website runs on, and PVOutput is a website for sending your solar power generation data to, a bunch of people at work also do the same and we’re all in the same “team” so we can see how much combined we’ve all generated together).

After that, it just involved a whole bunch of coding (as well as ordering two spare ESP32s so I could test that my code worked without having to pull apart my existing setup), which I’ve uploaded to GitHub:

• esp32-sensor-reader-mqtt — Temperature and humidity readings
• esp32-air-quality-reader-mqtt — Air quality readings (Update April 2024: This has been retired and the code in it rolled into the esp32-sensor-reader-mqtt repository above)
• pi-home-dashboard — The central service that runs the display from More space: the Pimoroni HyperPixel4 display on a Raspberry Pi Zero W and Powering our house with a Tesla Powerwall 2 battery
• powerwall-to-pvoutput-uploader — The combination PVOutput uploader, which already existed but now optionally will publish its data to an MQTT topic
• influxdb-firehose — The combination Docker container stack that brings up InfluxDB, Grafana, and my TypeScript app to pull in all the data from the four sources above, put it into line protocol format, and shoot it into InfluxDB
• featherwing-temperature-displayer — The CircuitPython code that runs on an Adafruit ESP32-S2 with ThinkInk display, for giving us a display of the temperature and humidity while we’re in the home office; this only refreshes every three minutes so it just hits the API endpoint of the pi-home-dashboard above to read its data

Despite having written up a careful plan and done what I thought was getting all my ducks in a row to make a quick switchover this morning, there were a number of things I ran into that caused it to take a few hours to get going (things like forgetting to configure PostgreSQL on the Raspberry Pi 4B to allow things running in Docker to access it, needing to add an extra published port on the Linode so my website could connect to Mosquitto, and most annoyingly of all, a recent VSCode update breaking Pymakr and having to revert to an old version of both pieces of software). I got everything up and running in the end, and now if I add any new monitoring things, it’ll be quite simple to publish the data to Mosquitto and slurp it up into InfluxDB!