Another year of Node.js (now also featuring React)

I posted last year about my progress with Node.js, and the last sentence included “I’m very interested to revisit this in another year and see what’s changed”.

So here we are!

There’s been a fair bit less work on it this year compared to last:

$ git diff --stat 6b7c737 47c364b
[...]
77 files changed, 2862 insertions(+), 3315 deletions(-)

The biggest change was migrating to Node 8’s shiny new async/await, which means that the code reads exactly as if it was synchronous (see the difference in my sendUpdate() code compared to the version above it). It’s really very nice. I also significantly simplified my code for receiving temperature updates thanks to finally moving over to the Raspberry Pi over the Christmas break. Otherwise it’s just been minor bits and pieces, and moving from Bamboo to Bitbucket Pipelines for the testing and deployment pipeline.

I also did a brief bit of dabbling with React, which is a frontend framework for building single-page applications. I’d tried to fiddle with it a couple of years ago but there was something fundamental I wasn’t grasping, and ended up giving up. This time it took, though, and the result is virtualwolf.cloud! All it’s doing is pulling in data from my regular website, but it was still a good start.

There was a good chunk of time from about the middle of the year through to Christmas where I didn’t do any personal coding at all, because I was doing it at work instead! For my new job, the primary point of contact for users seeking help is via a room on Stride, and we needed a way to be able to categorise those contacts to see what users were contacting us about and why. A co-worker wrote an application in Ruby a few years ago to scrape the history of a HipChat room and apply tags to it in order to accomplish this, but it didn’t scale very well (it was essentially single-tenented and required a separate deployment of the application to be able to have it installed in another room; understandable when you realise he wrote it entirely for himself and was the only one doing this for a good couple of years). I decided to rewrite it entirely from scratch to support Stride and multiple rooms, with the backend written in Node.js and the frontend in React. It really is a fully-fledged application, and it’s been installed into nearly 30 different rooms at work now, so different teams can keep track of their contact rate!

The backend periodically hits Stride’s API for each room it’s installed in, and saves the messages in that room into the database. There’s some logic around whether a message is marked as a contact or not (as in, it was someone asking for help), and there’s also a whitelist that the team who owns the room can add their team members to in order to never have their own messages marked as contacts. Once a message is marked as a contact, they can then add one or more user-defined tags to it, and there’s also a monthly report so you can see the number of contacts for each tag and the change from the previous month.

The backend is really just a bunch of REST endpoints that are called by the frontend, but that feels like I’m short-changing myself. 😛 I wrote up a diagram of the hierarchy of the frontend components a month or so ago, so you can see from this how complex it is:

And I’m in the middle of adding the ability to have a “group” of rooms, and have tags defined at the group level instead of the room level.

I find it funny how if I’m doing a bunch of coding at work, I have basically zero interest in doing it at home, but if I haven’t had a chance to do any there I’m happy to come home and code. I don’t think I have the brain capacity to do both at once though. 😛

A year of Node.js

Today marks one year exactly since switching my website from Perl to Javascript/Node.js! I posted back in March about having made the switch, but at that point my “production” website was still running on Perl. I switched over full-time to Node.js shortly after that post.

From the very first commit to the latest one:

$ git diff --stat 030430d 6b7c737
[...]
177 files changed, 11313 insertions(+), 2110 deletions(-)

Looking back on it, I’ve learnt a hell of a lot in that one single year! I have—

  • Written a HipChat add-on that hooks into my Ninja Block data (note the temperature in the right-hand column as well as the slash-commands; the button in the right-hand column can be clicked on to view the indoor and outdoor temperatures and the extremes for the day)
  • Refactored almost all of the code into a significantly more functional style, which has the bonus of making it a hell of a lot easier to read
  • Moved from callbacks to Promises, which also massively simplified things (see the progression of part of my Flickr– and HipChat-related code)
  • Completely overhauled my database schema to accomodate the day I eventually replace my Ninja Block with my Raspberry Pi (the Ninja Block is still running though, so I needed to have a “translation layer” to take the data in the format that the Ninja Block sends and converts it to what can be inserted in the new database structure)
  • Added secure, signed, HTTP-only cookies when changing site settings
  • Included functionality to replace my old Twitter image hosting script, and also added a nice front-end to it to browse through old images

Along with all that, I’ve been reading a lot of software engineering books, which have helped a great deal with the refactoring I mentioned above (there was a lot of “Oh god, this code is actually quite awful” after going through with a fresh eye having read some of these books)—Clean Code by Robert C. Martin, Code Complete by Steve McConnell, The Art of Readable Code by Dustin Boswell and Trevor Foucher.

I have a nice backlog in JIRA of new things I want to do in future, so I’m very interested to revisit this in another year and see what’s changed!

Stubbing services in other services with Sails.js

With all my Javascript learnings going on, I’ve also been learning about testing it. Most of my website consists of pulling in data from other places—Flickr, Tumblr, Last.fm, and my Ninja Block—and doing something with it, and when testing I don’t want to be making actual HTTP calls to each service (for one thing, Last.fm has a rate limit and it’s very easy to run into that when running a bunch of tests in quick succession which then causes your tests to all fail).

When someone looks at a page containing (say) my photos, the flow looks like this:

Request for page → PhotosController → PhotosService → jsonService → pull data from Flickr’s API

PhotosController is just a very thin wrapper that then talks to the PhotoService which is what calls jsonService to actually fetch the data from Flickr and then subsequently formats it all and sends it back to the controller, to go back to the browser. PhotosService is what needs the most tests due to it doing the most, but as mentioned above I don’t want it to actually make HTTP requests via jsonService. I read a bunch of stuff about mocks and stubs and a Javascript module called Sinon, such but didn’t find one single place that clearly explained how to get all this going when using Sails.js. I figured I’d write up what I did here, both for my future reference and for anyone else who runs into the same problem! This uses Mocha for running the tests and Chai for assertions, plus Sinon for stubbing.

Continue reading “Stubbing services in other services with Sails.js”

Learning new things: Javascript and Node.js

We’ve used Node.js (specifically with a framework called Sails.js) at work for a number of projects but I never really felt I properly understood one of Node’s fundamental concepts, that of the callback. It’s absolutely pervasive throughout Node and I was able to muddle on through at work without totally grasping it, but it wasn’t ideal.

Back at the end of January I decided to try rewriting my website using Node.js (it’s currently written in Perl using the Mojolicious framework) as a learning experience. It’s now almost two months later and my site is actually completely rewritten with Node/Sails (sans tests, which are currently being written; I know about test-driven development but I wasn’t about to start bashing my head against failing to understand how to get the tests to do what I wanted on top of learning a whole new language :P) with all the same functionality of my Perl one, and although I’m still far from an expert I actually feel like I have a proper handle on what’s going on.

The problem I found when trying to find examples was that they were all very contrived; I felt like they were missing fundamental underlying parts that apparently everybody else was able to understand but I couldn’t. For me, the “ah ha” moment was this post on Stack Overflow about using callbacks in your own functions. It didn’t assume anything or use an example of some module that apparently everyone is already familiar with (the most common one was fs.read() to read data from the filesystem). Once I had that straight, it was full steam ahead. It’s also significantly easier to deal with Javascript objects compared to Perl’s array/hash references.

My actual live website at virtualwolf.org is still on the old Perl version, but I don’t want to put the Node one up until I’ve actually got it properly covered with tests. Speaking of tests, I’m using a thing called Istanbul for code coverage, the reports it generates look like this, and it’s really satisfying having the numbers and bars go up as your coverage increases. It’s basically gamification of tests, really!

All in all, I’m pretty pleased!

Introducing the LiveJournal XML Importer

Continuing on from my previous post about my LiveJournal to WordPress experience, and how the importer managed to miss a bunch of entries, it turns out I didn’t have every notification email still around. The ones prior to February of 2004 I’d apparently deleted so sadly there’s no recovering Kristina’s really early comments from the missing posts, but from what I could see there weren’t too many of those anyway, thankfully.

However, I’m happy to say that I’ve been able to hack the importer to import all the entries and comments from an ljdump archive! I’ve put the code up on Bitbucket, I’m sure there’s bugs and edge-cases and things that don’t work properly, but it worked perfectly for me. I’ve changed it from the original importer to still import comments from journals that have been deleted so the threading remains intact and you don’t end up with weird comments seemingly replying to nothing. They’re easily identified by the fact that the date on the comment is set to the time you performed the import, so they show up at the top of the Comments section in WordPress’ admin.