Orpheus News

[banglabhai420]

Recent downtime

Most of you will have noticed the unannounced downtime we had had over the past few days. Nothing serious happened, but a chain of unfortunate and slightly annoying events delayed our comeback.

It all started with an innocent systemctl restart networking in a maintenance script. As it turns out, this command works well on virtual machines, which I tested it on before, but will in many cases not on real hardware. It shut down the network connection on our most important servers, requiring a manual reboot of the servers. Nothing bad enough to justify a few days of downtime you will say. And you are right. The real problems only began there. For one, it shut down Spine's IRC connection and therefore our main communication channel, which caused a fair bit of inconveniences and delay. After all, from Spine's point of view, all our major servers were suddenly offline for no good reason.

Once that issue was sorted out and the servers had been rebooted we decided to utilise the already ongoing downtime for some more maintenance on the servers, applying updates (debian dist upgrade). You guessed it, it didn't go just as smoothly as we hoped it to, basically causing a new downtime after the main issue had been resolved. While I was busy fixing the now actually broken server, Spine made use of the time by running some database updates. Those, again, took a lot longer than expected and caused even more problems (curse you, MySQL collations!). The new bugs many of you have noticed on the site since we are back are caused by those, and if it wasn't for character encodings, MySQL collations will be the eventual downfall of humanity. I am happy Spine has taken considerable efforts already to, among other things, at some point, migrate away from MySQL to PostgreSQL and I rather wish for that to happen sooner than later. But it's a huge undertaking that still requires a lot of invested time and we can only hope. After all, there are far more matters in Gazelle that need addressing.

The downtime wasn't entirely wasted, as we also were able to deploy some overhaul to our internal services, aside from the already mentioned system updates.

Anyway, that's the long and short of the recent downtime, and we're happy to be back online! Enjoy your music!

[Silent Watcher]

Avoid unnecessary posts such as 'Thank you', 'Welcome', etc. Such posts will be deleted and user will be warned if it happens again. If caught spamming, the following actions are applicable -

• First time - Warning
• Second time - 5000 Points will be deducted
• Third time - Ban for 7 days
• Fourth time - Permanent Ban

If the post helped you, reward the user by reacting to the post like this -