More Raspberry Pi adventures: the Pi Zero W and PaPiRus ePaper display

I decided I wanted to have some sort of physical display in the house for the temperature sensors so we wouldn’t need to be taking out our phones to check the temperature on my website if we were already inside at home. After a bunch of searching around, I discovered the PaPiRus ePaper display. ePaper means it’s not going to have any bright glaring light at night, and it also uses very little power.

The Raspberry Pi is hidden away under a side table, and already has six wires attached to the header for the temperature sensors, so I decided to just get a separate Raspberry Pi Zero W — which is absurdly small — and the PaPiRus display.

Setting it up

I flashed the SD card with the Raspbian Stretch Lite image, then enabled SSH and automatic connection to our (2.4GHz; the Zero W doesn’t support 5GHz) wifi network by doing the following:

1) Plug the flashed SD card back into the computer.

2) Go into the newly-mounted “boot” volume and create an empty file called “ssh” to turn on SSH at boot

3) Also in the “boot” volume, create a file called “wpa_supplicant.conf” and paste the following into it:

country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="WIFI_SSID"
psk="WIFI_PASSWORD"
key_mgmt=WPA-PSK
}

4) Unmount the card, pop it into the Pi, add power, and wait 60-90 seconds and it’ll connect to your network and be ready for SSH access! The default username on the Pi is “pi” and the password is “raspberry”.

(These instructions are all thanks to this blog post but I figured I’d put them here as well for posterity).

The PaPiRus display connection was dead easy, I just followed Pi Supply’s guide after soldering a header into the Pi Zero W. If you want to avoid soldering, they also offer the Zero W with a header pre-attached.

Getting the Python library for updating the display was mostly straightforward, I just followed the instructions in the GitHub repository to manually install the Python 3 version.

I wrote a simple Python script to grab the current temperature and humidity from my website’s REST endpoints, and everything works! This script uses the “arrow” and “requests” libraries, which can be installed with “sudo apt-get install python3-arrow python3-requests”.

Next step is to have the Pi 3 that has the sensors run a simple HTTP server that the Zero W can connect to it, so even if we have no internet connection for whatever reason, the temperatures will still be available at home. I’ve updated my Pi Sensor Reader to add HTTP endpoints.

Finally, some actual miniature painting

So despite having gotten the back room set up for miniature painting over three and a half years ago, I hadn’t actually done any of it since then. 😛 I also realised I hadn’t actually taken a photo of the setup.

I bought Games Workshop’s latest game Shadespire early last month, it does have miniatures to paint but only eight in the core set, and it’s a board game where the games last about half an hour or so versus the multi-hour affairs that are traditional Warhammer/Warhammer 40,000 games. I figured that with the holidays around and time to kill, and not having the prospect of endless amounts of miniatures to paint, I’d give it a go. I’m pleased to say that I clearly still have the painting skills!

I’ve finished five of them so far, so only three to go, and took some proper photos of them with the full external flash/umbrella setup.

Blooded Saek

Angharad Brightshield

Targor

Karsus the Chained

Obryn the Bold

(I’ll admit that I cheated slightly and didn’t actually paint any of these in the back room, however… during the week and a bit that I was doing them, the weather was really hot and the dinky little air conditioning unit in the back room wasn’t remotely up to keeping things cool, so I ended up bringing all the paints and bits inside and did them at the dining table).

The game Shadespire itself is really neat as well. I’ve only played a handful of games, but rather than just “Kill the other team” you also have specific objectives to accomplish as well. Have a read of Ars Technica’s review of it, they’re a lot more thorough and eloquent than I could be. 😛

Temperature sensors: now powered by Raspberry Pi

The Weather section on my website is now powered by my Raspberry Pi, instead of my Ninja Block! \o/

Almost exactly three years ago, I started having my Ninja Block send its temperature data to my website (prior to that, I was manually pulling the data from the Ninja Blocks API and didn’t have any historical record of it). Ninja Blocks the company went bust in 2015, and there was some stuff in the Ninja Blocks software that relied on their cloud platform to work and I ended up with no weather data for a couple of days because the Ninja Block couldn’t talk to the cloud platform. I ended up hacking at it and the result was this very simple Node.js application as a replacement for their software. It always felt a bit crap, though, because if the hardware itself died I’d be stuck; yes, it was all built on “open hardware” but I didn’t know enough about it all to be able to recreate it. I’d ordered a Raspberry Pi 3 in June last year, intending on replacing the Ninja Block and it’s sometimes-unreliable wireless temperature sensors with something newer and simpler and hard-wired, but I found there was a frustrating lack of solid information regarding something that on the surface seemed quite simple.

I’ve finally gotten everything up and running, the Ninja Block has been shut down, and I’ve previously said I’d write up exactly what I did. So here we are!

Components needed

  • Raspberry Pi 3 Model B+
  • AM2302 wired temperature-humidity sensor (or two of them in my case)
  • Ethernet cable of the appropriate length to go from the Pi to the sensor
  • 6x “Dupont” female to either male or female wires (eBay was the best bet for these, just search for “dupont female”, and it only needs to be female on one end as the other end is going to be chopped off)
  • 1.5mm heatshrink tubing
  • Soldering iron and solder
  • Wire stripper (this one from Jaycar worked brilliantly, it automatically adjusts itself to diameter of the insulation)

Process

  1. Cut the connectors off one end of the dupont cables, leaving the female connector still there, and strip a couple of centimetres of insulation off.
  2. Strip the outermost insulation off both ends of the ethernet cable, leaving a couple of centimetres of the internal twisted pairs showing.
  3. Untwist three of the pairs and strip the insulation off them, then twist them back together again into their pairs.
  4. Chop off enough heatshrink tubing to cover the combined length of the exposed ethernet plus dupont wire, plus another couple of centimetres, and feed each individual dupont wire through the tubing (there should be three separate bits of tubing, one for each wire).
  5. Solder each dupont wire together with one of the twisted pairs of ethernet cable, then move the heatshrink tubing up over the soldered section and use a hairdryer or kitchen blowtorch to activate the tubing and have it shrink over the soldered portion to create a nice seal.
  6. Repeat this feed-heatshrink-tubing/solder-wire/activate-heatshrink process again but with the cables that come out of the temperature sensor (ideally you should be using the same red/yellow/black-coloured dupont cables to match the ones that come out of the sensor itself, to make it easier to remember which is which).
  7. Install Raspbian onto an SD card and boot and configure the Pi.
  8. Using this diagram as a reference, plug the red (power) cable from the sensor into Pin 2 (the 5V power), the yellow one into Pin 7 (GPIO 4, the data pin), and the black one into Pin 6 (the ground pin).

AdaFruit has a Python library for reading data from the sensor, I’m using the node-dht-sensor library for Node.js myself. You can see the full code I’m using here (it’s a bit convoluted because I haven’t updated the API endpoint on my website yet and it’s still expecting the same data format as the Ninja Block was sending).

I’d found a bunch of stuff about needing a “pull-up” resistor when connecting temperature sensors, but the AM2302 page on adafruit.com says “There is a 5.1K resistor inside the sensor connecting VCC and DATA so you do not need any additional pullup resistors”, and indeed, everything is working a treat!

Internet history

On Twitter recently, Mark had downloaded the whole archive of his Twitter account’s history and had been poking through it and randomly retweeting amusing old tweets. I downloaded my own Twitter history and quickly realised that a lot of the old things I’d linked to weren’t accessible because I’d been using my own custom URL shortener (this was before the days of Twitter doing their own URL shortening) and it wasn’t running anymore. Fortunately I’d had the foresight to take a full copy of all of my data and databases from Dreamhost before I shut down my account, and one of those databases was the one that had been backing my URL shortener. A quick import to PostgreSQL Workers KV and a hacky Node.js Cloudflare Workers application later, it’s all up and running! I’m under no illusions that it’s almost ever going to be accessed by anyone except me, but it’s nice to have another part of my internet history working. I’ve been hosting my own website and images and whatnot (things like pictures I’ve posted on my blog née LiveJournal, or in threads on Ars Technica) in one form or another since about 2002, and the vast majority of those links and images still work!

Speaking of my website, about four years ago now I went and tried to collect all my old websites into a single archive so I could look back and see the progression. The majority of them I actually still had the original source code to, though my very first one or two have been totally lost. The earliest I still have is from March of 1998 when I was not quite fifteen years old! I started out with just HTML, then discovered CSS and Javascript rollover images, and then around 2001 I started using PHP. I had to go in and hack up some of the PHP-based sites in order to get them to work, and oh dear god 18-year-old me was a FUCKING AWFUL coder. One of the sites consisted of a bit over three thousand lines in a single file, with all sorts of duplication and terribleness, and every single one of the sites that was hooked into MySQL had SQL injection vulnerabilities. I’m very proud of just how much my code has improved over the years.

I went back this weekend and managed to recover another handful of sites, and also included exports of the Photoshop files where the original site source wasn’t available. I’ve packed them all up into a Docker container (I’ll write another post about my experiences with Docker at some point soon) and chucked them up on archive.virtualwolf.org for the entire Internet to marvel at how terrible they all were! There’s a little bit more background there, but it’s a lot of fun just looking back at what I did.

Better Raspberry Pi audio: the JustBoom DAC HAT

I decided that the sound output from the Pi’s built-in headphone jack wasn’t sufficient after all and so went searching for better options (a DAC—digital-to-analog converter).

The Raspberry Pi foundation created a specification called “HAT” (Hardware Attached on Top) a few years ago which specifies a standard way for devices to automatically identify and configure a device and drivers that’s attached to the Pi via its GPIO (General Purpose Input/Output) pins. There’s a number of DACs now that conform to this standard, and the one I settled on is the JustBoom DAC HAT. It’s a UK company but you can buy them locally from Logicware (with $5 overnight shipping no less).

The setup is incredibly simple: connect the plastic mounting plugs and attach the DAC to the Pi, then start it up and edit /boot/config.txt to comment out the default audio setting:

#dtparam=audio=on

Then add three new lines in:

dtparam=audio=off
dtoverlay=i2s-mmap
dtoverlay=justboom-dac

Then reboot. (If you’re running Raspbian Stretch or newer, the i2s-mmap line should not be added).

To say that I’m impressed would be an understatement! I didn’t realise just how crappy the audio from the Pi’s built-in headphone jack was until I’d hooked up the new DAC and blasted some music out. I’m not an audiophile and it’s hard to articulate, but I’d compare it most closely to listening to really low-quality MP3s on cheap earbuds versus high-quality MP3s on a proper set of headphones.

If you’re going to be hooking your Pi into a good stereo system, I can’t recommend JustBoom’s DAC HAT enough!

Raspberry Pi project: AirPlay receiver

I bought a Raspberry Pi almost exactly a year ago, intending on eventually replacing my Ninja Block and its sometimes-unreliable wireless sensors with hardwired ones (apart from the batteries needing occasional changing, there’s something that interferes with the signal on occasion and I just stop receiving updates from the sensor outside for several hours at a time, and then suddenly it starts working again). To do that, I need to physically run a cable from outside under the pergola to inside where the Raspberry Pi will live and I don’t really want to go drilling holes through the house willy-nilly. I want to eventually get the electrician in to do some recabling so I’m going to get him to do that as well, but until then the Pi was just sitting there collecting dust. I figured I should find something useful to do it with, but having a Linode meant that any sort of generic “Have a Linux box handy to run some sort of server on” itch was already well-scratched.

I did a bit of Googling, and discovered Shairport Sync! It lets you use the Raspberry Pi as an AirPlay receiver to stream music to from iTunes or iOS devices, a la an Apple TV or AirPort Express. We already have an Apple TV but it’s plugged into the HDMI port on the Xbox One which means that to simply stream audio to the stereo we have to have the Xbox One, TV, and Apple TV all turned on (the Apple TV is plugged into the Xbox’s HDMI input so we can say “Xbox, on” and the Xbox turns itself on as well as the TV and amplifier, then “Xbox, watch TV” and it goes to the Apple TV; it works very nicely but is a bit of overkill when all you want to do is listen to music in the lounge room).

Installing Shairport Sync was quite straightforward, I pretty much just followed the instructions in the readme there then connected a 3.5mm to RCA cable from the headphone jack on the Raspberry Pi to the RCA input on the stereo. It’s mentioned in the readme, but this issue contains details on how to use a newer audio driver for the Pi that significantly improves the audio output quality.

The only stumbling block I ran into was the audio output being extremely quiet. Configuring audio in Linux is still an awful mess, but after a whole lot of googling I discovered the “aslamixer” tool (thanks to this blog post), which gives a “graphical” interface for setting the sound volume, and it turned out the output volume was only at 40%! I cranked it up to 100% and while it’s still a bit quieter than what the Apple TV outputs, it doesn’t need a large bump on the volume dial to fix—there’s apparently no amplifier or anything on the Raspberry Pi, it’s straight line-level output. The quality isn’t quite as good as going via the Apple TV, but it gets the job done! I might eventually get a USB DAC or amplifier but this works fine for the time being.

On macOS it’s possible to set the system audio output to an AirPlay device, so you can be watching a video but outputting the audio to AirPlay, and the system keeps the video and audio properly in sync. It works extremely well, but the problem we found with having the Apple TV hooked up to the Xbox One’s HDMI input is that there’s a small amount of lag from the connection. When the audio and video are both coming from the Apple TV there’s no problem, but watching video on a laptop while outputting the sound to the Apple TV meant that the audio was just slightly out of sync from the video. Having the Raspberry Pi as the AirPlay receiver solves that problem too!

UPDATE: Two further additions to this post. Firstly, and most importantly, make sure you have a 5-volt, 2.5-amp power supply for the Raspberry Pi. I’ve been running it off a spare iPhone charger which is 5V but only 1A, and the Pi will randomly reboot under load because it can’t draw enough power from the power supply.

Secondly, the volume changes done with the “alsamixer” tool are not saved between reboots. Once you’ve set the volume to your preferred level, you need to run “sudo alsactl store” to persist it (this was actually mentioned in the blog post I linked to above, but I managed to miss it).

Farewell Dreamhost

After 12 years of service, I’m shutting my Dreamhost account down (for those unaware, Dreamhost is a website and email hosting service).

My very first—extremely shitty—websites were hosted on whichever ISP we happened to be using at the time—Spin.net.au, Ozemail, Optus—with an extremely professional-looking URL along the lines of domain.com.au/~username. I registered virtualwolf.org at some point around 2001-2002 and had it hosted for free on a friend’s server for a few years, but in 2005 he shut it down so I had to go find some proper hosting, and that hosting was Dreamhost.

The biggest thing I found useful as I was dabbling in programming was that Dreamhost offered PHP and MySQL, so I was able to create dynamic sites rather than just static HTML. Of course, looking back at the code now is horrifying, especially the amount of SQL injection vulnerabilities I had peppered my sites with.

Around the start of 2011, I started using source control—Subversion initially—and finally had a proper historical record of my code. I used PHP for the first year or so of it, then ended up outgrowing that and switched to a Perl web framework called Mojolicious. The only option to run a long-lived process on Dreamhost is to use Fast-CGI, which I never managed to get working with Mojolicious, but fortunately Mojolicious could also run as a regular CGI script so I was still able to use it with Dreamhost, albeit not at great speed.

At the same time I started using Subversion, I also signed up with Linode who offer an entire Linux virtual machine with which you can do almost anything you’d like as you have full root access. I originally used it mostly to run JIRA so I could keep track of what I wanted to do with my website and have the nifty Subversion/JIRA integration working to see my commits against each JIRA issue. I slowly started using the Linode for more and more things (and switched to Git instead of Subversion as well), until in 2014 I moved my entire website hosting over to the Linode.

At that point the only thing I was using Dreamhost for was hosting Kristina’s website and WordPress blog, and the email for our respective domains. Dreamhost’s email hosting wasn’t always the most reliable and towards the end of 2015 they had more than their usual share of problems, so we started looking for alternatives. Kristina ended up moving to Gmail and I went with FastMail (who I am extremely happy with and would very highly recommend!), I moved her blog and my previously-LiveJournal-but-now-Wordpress-blog over to the Linode, and that was that!

Moving my website hosting to the Linode also allowed me to move over to Node.js and I’ve been going full steam ahead ever since. Since that posted I’ve moved over from callbacks to Promises (so much nicer), I wrote myself a HipChat add-on to keep an eye on the temperature that my Ninja Block is reporting, and I moved my dodgy Twitter image upload Perl script functionality into my site and added a nice front-end to it. Even looking back at my code from 6 months ago to now shows a marked increase in quality and readability.

So in summary, thanks for everything Dreamhost, but I outgrew you. 🙂

Missing history

So it turns out that the LiveJournal to WordPress Importer didn’t actually import everything. I’d been going through and updating links to old entries to point to their relevant entry here in WordPress, and there were several pages that didn’t actually make it across (it also imported every single comment back to around 2010 twice, so I had to go through and delete all of those duplicates; prior to 2010 it was fine, for some weird reason). That wouldn’t have been too bad, but in between my having imported my LiveJournal originally and me discovering this, Kristina’s old LiveJournal account was deleted, which also meant that every single comment of hers on my LiveJournal was now gone, and any fresh import I did directly from the LiveJournal API wouldn’t have them at all. 🙁 Looking back through my old entries was kind of sad, with just “Deleted comment” everywhere in place of Kristina’s actual comments.

I wanted to have the complete history of my LiveJournal here in WordPress, but I also didn’t want to have all of Kristina’s comments missing. I figured SOMETHING had to be able to be done!

I used ljdump to hit LiveJournal’s API and download each entry there into a raw XML file, and that had grabbed all the journal entries, so clearly something had fucked up in the WordPress import part.

The situation was this:

  • I had most but not all of my old LiveJournal entries imported into WordPress
  • Those entries that made it across did have Kristina’s old comments on them
  • I had all of my own entries downloaded to raw XML
  • ljdump also grabbed all the comments for each entry as well (sans Kristina’s, obviously)

I manually went through and compared the entries in WordPress to those on my LiveJournal month-by-month, and found that there were 68 missing ones in total. I hacked at the LiveJournal to WordPress Importer plugin until I was able to get it to read the raw XML files that’d come directly from ljdump, then spun up a new temporary WordPress install and was able to import just those missing entries. Next, I erased that temporary instance, imported the full backup from this blog, then ran the importer again to bring in just those 68 missing entries from XML, and it worked a treat.

Unfortunately there were a handful of those entries that also had had Kristina’s comments on them previously, so they were still missing. Thankfully, me being the digital hoarder that I am, I still had all of the email notifications that LiveJournal had sent me for each and every comment on my journal, and the LiveJournal API actually shows even deleted comments in their properly threaded state, just with no body or detail beyond the username who posted it. So I was able to copy the content and timestamp for each comment of Kristina’s that’d been on those missing entries that weren’t imported, and update the raw comment XML with that detail!

This is still a work in progress and my next step is to hack at the importer further to read the comments directly from XML (currently it’s reading the journal entries from the XML files, but the comments are still pulled from LiveJournal’s API directly). It’ll definitely be do-able, it may just take a little while because everything related to WordPress is in PHP and I’ve not done any PHPing for quite a number of years now!

This may seem a bit odd, but given I have 13 years of history in LiveJournal, and it’s where Kristina and I initially started chatting a lot more before she visited and we got together, I didn’t want to have these weird entries where Kristina was just essentially erased from my blog history.

I might even put my modifications to the plugin up on Bitbucket if it seems to be working well, given the current LiveJournal to WordPress Importer is a bit shit.

New computer!

I’d been looking to upgrade to a 27″ iMac from my mid-2010 MacBook Pro, and five years from a machine is not bad at all! I was hoping that the non-retina iMacs would get one last upgrade but it was not to be. I checked the Apple Store after the cheaper model of retina iMac was released last week, and they had in fact removed all of the custom options from it other than RAM and storage, so the only video card available was the base-level 1GB one. Bleh.

However! There was fortuitously an almost-maxed-out 27″ Late 2013 iMac available on the refurb store: 4GB Nvidia 780M GTX video card (top of the line), and 3TB Fusion Drive. Only 8GB RAM but it’s user-replaceable in this model anyway, so I have another 16GB on the way. The thing with the refurb machines is that what you see is what you get, there’s no upgrades or anything available, but that was okay. The only change I’d have made is a 1TB SSD (the Fusion Drive is a large hard disk combined with a relatively small SSD—128GB) but ah well. I saved probably more than a grand all up!

kungfupolarbear has inherited my old machine as it’s a decent step up from her MacBook Air, and we’re going to use the MacBook Air as a travel computer, and if I want to do some coding from the lounge or some such.

\o/

kungfupolarbear is in the US visiting her family, she left yesterday and arrives back in the country two weeks later. It’s weird not having her here, I feel like I’m just biding my time until she’s back.

The whole stereotype of “The wife is away, I’m going to do all of these things that she doesn’t let me do normally!” just doesn’t apply to us at all. We both will happily play video games or putter around on the computer or whatever. I’m having gypocalypse, wobin, and some other friends over on Saturday to play this Warhammer 40,000-themed D&D-type thing, and that wouldn’t matter if kungfupolarbear was here, she’d happily hang out despite having zero interest in actually playing it and would absolutely not begrudge us spending time playing it. Everything is just so totally effortless, it still amazes me. <3

I had yesterday and today off (yesterday because we woke up at 3:30am in order to get to the airport on time D: ), and am working from home the rest of the time she’s gone since Beanie needs looking after and obviously can’t be by himself for 11 hours a day.

In other news we bought an Xbox One, a colleague of mine was selling his as he’s moving to Paris. We’ve got Forza 5 (a racing game) to tide us over until the Diablo III expansion comes out, and man the graphics are nice. This is only a launch title too, so they’ll be even more impressive once developers get a handle on squeezing the most out of the system. One very slick thing is that you can have the Xbox turn the TV and amplifier on when it turns on, and it’ll turn them off when the Xbox turns off too.