FA does another thing, screws it up

[Conan]

Early this morning, FA went down for planned database maintenance. In FA tradition, this was talked about in as general terms as possible.

Administrator notice:
Fur Affinity will be going down for database maintenance on Aug 13, 2013 at 8:00 AM UTC / 4:00 AM EDT.
Downtime is estimated to be less than an hour.

An hour and a half after the downtime began, it came back up. But as you can tell by the graph below, it didn't stay up for very long.


Just hours later the site began throwing 503 and 502 errors, before giving out entirely (for what appears to be an hour before someone does something about it). The site was brought back online with the explanation:

An issue was identified as related to the earlier database maintenance.
 Changes are being reverted. Unfortunately, this means that FA will be in for another downtime in the near future to actually see the changes applied.
 There will be an announcement about that downtime in advance.

Why all the problems? Why the downtime? We turn to Twitter:

We're working on resolving the issue. The downtime is due to an issue with our  database migration to the new servers.

What new servers? The new ones people donated. In February. It took them nearly six months to finally get around to setting up the hardware they desperately needed, and they still botch something in the process of migrating to them.

What exactly happened? Our sources from IRC say:

19:23 < Pi> <yak[away]> bah. binlogs filled up the current db server's harddrive faster than a snapshot could be moved to the new one and have it sync up as a slave.

[Conan]

Looks like they're giving the migration another go, and once again: six months of prep time and they have only narrowed down the "read only mode" to 24-48 hours.

Administrator notice: On Thursday, Sep. 12 08:00 AM UTC FA will be upgrading our database servers. Server migration is expected to last between 24-48 hours. The site will be put in Read Only mode during this time. We apologize for any inconvenience this may cause. Please plan accordingly.

This after Plan A failed spectacularly

[winserv03fan]

FA will be upgrading our database servers.

Woah, pretty impressive having FA upgrade their servers for them...

I'm wondering how this would take anywhere near 2 days to complete though. All FA typical bullshit aside, can't it be as easy as copying the database from one drive to another? I don't think that should take anywhere near 2 days, unless they're using Wi-Fi, or like, Bluetooth to do it.

[nrr]

I'm wondering how this would take anywhere near 2 days to complete though. All FA typical bullshit aside, can't it be as easy as copying the database from one drive to another? I don't think that should take anywhere near 2 days, unless they're using Wi-Fi, or like, Bluetooth to do it.

Typically, in well-run ops shops, these kinds of migrations happen in stages that span a couple of weeks because of how ITSM change control works, amongst other things. FA eschews all of that in favor of just fucking doing it, which is nice, but you need to have competent people in place _and_ failsafes (like database backups and a lot of extra disk) handy in order to keep from causing a catastrophe.

[Conan]

FA will be upgrading our database servers.

Woah, pretty impressive having FA upgrade their servers for them...

I'm wondering how this would take anywhere near 2 days to complete though. All FA typical bullshit aside, can't it be as easy as copying the database from one drive to another? I don't think that should take anywhere near 2 days, unless they're using Wi-Fi, or like, Bluetooth to do it.

The database is known to be several hundred gigabytes in size, to the degree that it was once said the notification table was nearly 400GB alone. That's probably millions of rows. All because they refuse to force delete ancient notifications "just in case" someone needs one someday.

It takes 24-48 hours to complete because FA seems to want to do this sort of thing in the most inefficient ways possible, because Yak is a developer first and foremost who roleplays a sysadmin from time to time.

[winserv03fan]

The database is known to be several hundred gigabytes in size, to the degree that it was once said the notification table was nearly 400GB alone. That's probably millions of rows. All because they refuse to force delete ancient notifications "just in case" someone needs one someday.

I find this fucking hilarious. Were they saving them with the whole string and HTML formatting for each row? Haha.

[Fate]

The database is known to be several hundred gigabytes in size, to the degree that it was once said the notification table was nearly 400GB alone. That's probably millions of rows. All because they refuse to force delete ancient notifications "just in case" someone needs one someday.

good god. just periodically erase notifications every six weeks, fuck.


Then again, this is FA, bastion of inefficiency and using gaming rigs as servers.

[JigsawJones]

THAT'S NINTENDO POWER!!!

[Ketsuban]

FA is out of readonly mode, what does the load graph look like?

[Pi]

Bad.

The response times shot through the roof, and the SQL times got worse.

[Conan]

Left side of the graph is from prior to the outage. Far right is after. Yeah, it got worse, and it looks like PHP is partially to blame.

[winserv03fan]

Via Twitter:

We made some changes this morning. How is performance holding up today for everyone? Better, faster?

FA's been completely unresponsive for near an hour now...

[Fate]

so this is an "upgrade" in the same way I "upgraded" my old PC with bullets.

[Ketsuban]

Some two or three hours later, there's finally a recognition that the site isn't working.

We are aware of the current site outage and looking into the cause. We will keep you updated as more information comes to light.

We're not going to actually tell you anything, though - what do you take us for, professionals? _{FURAFFINITY AND THE FENDER AND REDNEF CHARACTERS ARE THE PROPERTY OF FUR AFFINITY LLC}

[Conan]

After six hours offline, it just came back up. As usual, no one seems to want to explain what happened.

Based on past events, I'd say the database crashed and since the redundancy of their new redundant servers isn't operational yet, the site toppled over.

Oh, the irony of that all.

[ProvincialTwit]

BUT YOU GUYS ITS A FREE SITE AGLAHJGAHUWGHAULWGHARELUGAH FUCK MY FACE

ahem.

Christ if that place were being run like any sort of real business these people would've been replaced ages ago.  If this is how Piche thinks you're supposed to run a web service it's no wonder he needed favors from mommy and daddy to get a job.

[Ketsuban]

The database move has fucked up profiles for anyone who made use of Unicode characters e: as well as everywhere else - anytime someone said "character © person" the copyright symbol got mangled.

e: and as soon as it was there it's gone again.

[Conan]

SQL times and response times are still slower than they were prior to the upgrade.

BUT YOU GUYS ITS A FREE SITE AGLAHJGAHUWGHAULWGHARELUGAH FUCK MY FACE

It's like you predicted a post one of their forum mods made earlier today.

I love how everyone assumes FA is run like a full time business with employees instead of a hobby site that exists off ads and donations and relies on volunteers.
 
 Taking a day to fix something isn't some ridiculous amount of time, especially when there's only a small team knowledgeable about what's going on behind the scenes.

Geez guys it's completely unacceptable to think that the largest furry website with all sorts of really big numbers they like to throw around might restart the database in under six hours!!!!!!! what do you think this is?!?!?!?!?!

[Pi]

BUT YOU GUYS ITS A FREE SITE AGLAHJGAHUWGHAULWGHARELUGAH FUCK MY FACE

It's like you predicted a post one of their forum mods made earlier today.

I love how everyone assumes FA is run like a full time business with employees instead of a hobby site that exists off ads and donations and relies on volunteers.
 
 Taking a day to fix something isn't some ridiculous amount of time, especially when there's only a small team knowledgeable about what's going on behind the scenes.

And if there were a larger team that were more knowledgeable about what is going on behind the scenes, they might fix shit a little more quickly. But hey.

[vigilante777]

And the site is tanking again with loading times of a minute. I don't really have any good analysis tools, so here's a random site with some info's: http://loadimpact.com/load-test/www.furaffinity.net-7bf296068380db52e8b81b1c03e10b6c