Introducing the LiveJournal XML Importer

Continuing on from my previous post about my LiveJournal to WordPress experience, and how the importer managed to miss a bunch of entries, it turns out I didn’t have every notification email still around. The ones prior to February of 2004 I’d apparently deleted so sadly there’s no recovering Kristina’s really early comments from the missing posts, but from what I could see there weren’t too many of those anyway, thankfully.

However, I’m happy to say that I’ve been able to hack the importer to import all the entries and comments from an ljdump archive! I’ve put the code up on GitHub, I’m sure there’s bugs and edge-cases and things that don’t work properly, but it worked perfectly for me. I’ve changed it from the original importer to still import comments from journals that have been deleted so the threading remains intact and you don’t end up with weird comments seemingly replying to nothing. They’re easily identified by the fact that the date on the comment is set to the time you performed the import, so they show up at the top of the Comments section in WordPress’ admin.

Lily’s visit

Lily normally lives with her mother up near Nimbin (which is an almost-nine-hour drive north from our house), but they came down for Christmas this year. We saw Lily last in October of 2014, and she didn’t really like Beanie at that point because he kept trying to lick her, and Beanie himself wasn’t quite sure what to make of her as she was running all around and generally being fairly hyper. This time was a night-and-day difference, Lily was really good with him and knew how to pet him properly, and was significantly calmer than last time. When I would read to Lily before bed, Beanie would come in and curl up on the floor while we’re doing so. The only slight issue was that Beanie kept trying to hump Lily! We’re not entirely sure why (we get the occasional hump attempt, but nothing like this), the only conclusion we came to was that it was because she’s so much smaller than us.

My sister and her husband and two girls (Scarlett, 5, and Arya, 1-and-a-bit) live three hours away in the opposite direction from Nimbin, but they were up for Christmas this year too, which we had at my parents’ place.

New bears

"THAT WAY!"

Reading with Nanna Ruth

We had Lily for a week all up, from Christmas Day to New Year’s Day. Kristina pointed out that it’s so much easier to have a proper conversation with Lily now, she actually listens to what you’re saying rather than just talking at everyone all the time and not really caring if anyone else is listening. There were a few pretty hilarious exchanges, too. My favourites were these two:

Lily: Why are we doing this?
Me: Better safe than sorry.
Lily: I don’t understand.
Me: It means it’s better to do something and perhaps not have needed to, than to not do it and then have regrets later.
Lily (in a very matter-of-fact tone): You are speaking absolute nonsense. I don’t understand a word you’ve just said.

It was a serious struggle not to burst out laughing. The other one was when we were reading before bed, Lily still has some of her older learn-to-read books around, and one of them had a thing on the front:

Lily (reading the cover of the book): “Synthetic phonics.”
Me: Synthetic! That’s a big word!
Lily (again very matter-of-fact): That’s not a big word. “Intergalactic” is a big word.

Lily’s reading level is pretty impressive. Mum bought her The Animal Book, which is a big book about all sorts of different animals and plants, and Lily was just zooming through it, even reading the proper scientific names of things with barely a pause. She’d also had the rite of passage that I’m pretty sure all children go through (me included), which is where a relative buys you a big book of REALLY CORNY jokes. We were inundated with jokes the whole time, but I think the most impressive thing was how well she was remembering all the different jokes.

I’d forgotten that I had Minecraft on the Xbox One, so we had a pretty good chunk of time playing that as well, either Lily by herself or the both of us with splitscreen. She’d only played on her grandma’s iPad previously but she picked up how to control everything with a twin-stick controller really quickly! She was fairly obsessed with it and would tell me even when she wasn’t playing about what she wanted to build next or how she wanted to modify her various houses she’d created.

Scarlett, my niece, was staying with my parents for five days while everyone else went back down to Nowra, so we were able to go over and catch up a couple more times. We went to Featherdale Wildlife Park on Sunday, which mum and dad hadn’t ever been to before (naturally, photos were taken), and on Tuesday we went down to Collaroy Beach to generally chill out (where I put suncream on but managed to burn both kneecaps and the shin on one leg. >_<).

We also realised that we didn’t have any recent photos of all of us, so I set up all my crazy lighting equipment yesterday and took some really quite nice photos of us.

Family photo!

All in all, it was just a really nice time. She’s such a good kid, her manners (both regular and table) are great, and she’s extremely bright. It sucks that we don’t see her more often, but c’est la vie.

Missing history

So it turns out that the LiveJournal to WordPress Importer didn’t actually import everything. I’d been going through and updating links to old entries to point to their relevant entry here in WordPress, and there were several pages that didn’t actually make it across (it also imported every single comment back to around 2010 twice, so I had to go through and delete all of those duplicates; prior to 2010 it was fine, for some weird reason). That wouldn’t have been too bad, but in between my having imported my LiveJournal originally and me discovering this, Kristina’s old LiveJournal account was deleted, which also meant that every single comment of hers on my LiveJournal was now gone, and any fresh import I did directly from the LiveJournal API wouldn’t have them at all. 🙁 Looking back through my old entries was kind of sad, with just “Deleted comment” everywhere in place of Kristina’s actual comments.

I wanted to have the complete history of my LiveJournal here in WordPress, but I also didn’t want to have all of Kristina’s comments missing. I figured SOMETHING had to be able to be done!

I used ljdump to hit LiveJournal’s API and download each entry there into a raw XML file, and that had grabbed all the journal entries, so clearly something had fucked up in the WordPress import part.

The situation was this:

  • I had most but not all of my old LiveJournal entries imported into WordPress
  • Those entries that made it across did have Kristina’s old comments on them
  • I had all of my own entries downloaded to raw XML
  • ljdump also grabbed all the comments for each entry as well (sans Kristina’s, obviously)

I manually went through and compared the entries in WordPress to those on my LiveJournal month-by-month, and found that there were 68 missing ones in total. I hacked at the LiveJournal to WordPress Importer plugin until I was able to get it to read the raw XML files that’d come directly from ljdump, then spun up a new temporary WordPress install and was able to import just those missing entries. Next, I erased that temporary instance, imported the full backup from this blog, then ran the importer again to bring in just those 68 missing entries from XML, and it worked a treat.

Unfortunately there were a handful of those entries that also had had Kristina’s comments on them previously, so they were still missing. Thankfully, me being the digital hoarder that I am, I still had all of the email notifications that LiveJournal had sent me for each and every comment on my journal, and the LiveJournal API actually shows even deleted comments in their properly threaded state, just with no body or detail beyond the username who posted it. So I was able to copy the content and timestamp for each comment of Kristina’s that’d been on those missing entries that weren’t imported, and update the raw comment XML with that detail!

This is still a work in progress and my next step is to hack at the importer further to read the comments directly from XML (currently it’s reading the journal entries from the XML files, but the comments are still pulled from LiveJournal’s API directly). It’ll definitely be do-able, it may just take a little while because everything related to WordPress is in PHP and I’ve not done any PHPing for quite a number of years now!

This may seem a bit odd, but given I have 13 years of history in LiveJournal, and it’s where Kristina and I initially started chatting a lot more before she visited and we got together, I didn’t want to have these weird entries where Kristina was just essentially erased from my blog history.

I might even put my modifications to the plugin up on Bitbucket if it seems to be working well, given the current LiveJournal to WordPress Importer is a bit shit.

Tasmania, and LiveJournal to WordPress

So I’ve finally ditched LiveJournal and migrated it all over to my own WordPress site. I wanted to keep blogging things, but where people might actually read them. 😛

We decided to scale down our ambitions with the five-year anniversary trip, as Kristina is currently contracting and so would be losing out on quite a lot of money, and so went to Tasmania for five days instead. We’d heard great things but had never actually been before.

We stayed in the Hobart Art House, which was fantastically close to everything. We rented a car from Budget, and specified “Suzuki Swift or similar”. A Swift would have been fine, but instead we got a 1.2-litre Nissan Micra. It’s the most gutless car I’ve ever driven, and to get up the hill to where we were staying I had to put my foot all the way down to the firewall, and even then it still struggled.

At the bar

Day 1 was wandering around the Salamanca Markets, burgers for lunch at Jack Greene (their beef and blue cheese burger is amazing), then some more wandering until dinner at Monty’s on Montpelier. Their website says “fine dining” and my god, if anything that’s underselling it. The food there is absolutely incredible.

Cliffs

Day 2 started with a three-hour boat ride around the south-eastern coast with Pennicott Wilderness Journeys. According to the tour guide it was one of the calmest days they’d seen this year, but unfortunately that didn’t help because about half-way through the trip I got horribly sea-sick, though thankfully (barely) not to the point of throwing up, but it meant I was mostly sitting there feeling absolutely miserable and not even being able to look at any of the scenery. Kristina was suffering as well, so in hindsight the whole boat trip was a terrible idea. It redeemed itself with lunch at Port Arthur Lavender (again, incredible food), then dinner at Rockwall. One of their specials was a coffee-rubbed wagyu, and it was definitely the best steak I’ve ever had. It was pretty much falling apart in my mouth, it was so tender.

The other side of the lookout

Day 3, we went to Bruny Island. The “attractions” there were rather over-hyped, but the scenery is absolutely stunning. It’s like a tropical island in places, the clarity of the water is unbelievable. The afternoon was spent walking around the streets around where we were staying and checking out all the old houses, then dinner was at Mures. Mine was quite tasty, but Kristina’s risotto was crunchy, and even after she sent it back and got a new one, it wasn’t brilliant. She got a cold seafood tasting platter afterwards though, which was REALLY good.

Looking down to Hobart

Day 4, the last full day we had, we wandered around the botanical gardens, had lunch at Blue Eye (in keeping with everything else, the food was excellent), then went for a drive up to Mount Wellington. On the drive up, there was someone in a van who was doing 40km/h in a 70km/h zone and kept drifting over entirely into the other lane. It was terrifying! Thankfully there was a solid section of broken line on the road where I could overtake them, and the little Micra managed it surprisingly well. Mount Wellington itself was absolutely incredible, I’ve never been up a mountain where there’s alpine foliage before, and it felt like an entirely different world. Dinner was at Don Camillo, which was quite tasty but not on the level of some of the other restaurants.

On the last day we walked around Salamanca Place again, and had lunch back at Rockwall before heading to the airport. Our own car is only a 1.8-litre 2000 Corolla, but it was pretty funny feeling like it had a whole lot of grunt driving back from Sydney airport after having spent five days driving the Micra around. 😛

All in all, Tasmania was excellent and we’ll definitely be back!