Author Topic: Performance issues prompt "unplanned maintenance". Maybe. (Read 6276 times)

Fate · « **Reply #40 on:** February 12, 2012, 09:19:42 am »

Six hours of actual downtime to install, configure, and test a new database engine entirely.

There's two or three ways I can think of to have NO downtime and do this shit. But that would again require intelligence or foresight.

zed · « **Reply #41 on:** February 12, 2012, 10:07:34 am »

Just out of curiosity, what would be an acceptable option?

Jim Demintia · « **Reply #42 on:** February 12, 2012, 11:33:54 am »

Quote from: zed on February 12, 2012, 10:07:34 am

Just out of curiosity, what would be an acceptable option?

I'm gonna guess, not having a lot of DB admin experience, that if they had some kind of acceptable backup/replication setup they could switch the site onto the backup, maybe in read-only mode, for a while while the DB is upgraded. Or maybe there's a zero-downtime way to migrate the DB, who knows (with MySQL I doubt it).

It boils down to the fact that they run that site like it's a tenth of the size it really is, and it shows.

Fate · « **Reply #43 on:** February 12, 2012, 12:08:38 pm »

Quote from: zed on February 12, 2012, 10:07:34 am

Just out of curiosity, what would be an acceptable option?

Having seen and been utterly baffled by their network diagram despite a handful of years as a network engineer, starting with some actual redundancy in the servers would be nice. Have the site set up as a redundant pair of clusters - We'll call them Cluster 0 and Cluster 1 for reference.

Run your sync cycle and send Cluster 0's database servers into read-only mode while you upgrade and test Cluster 1. When complete, set the upgraded Cluster 1's databases to read-only mode and throw the switch that changes which cluster is serving data. Then lock and upgrade Cluster 0, test it, and when done flip back to Cluster 0. Congraulations - If all has gone well you're still serving image requests without any downtime.

Jim Demintia · « **Reply #44 on:** February 12, 2012, 12:13:57 pm »

IIRC, part of the problem is that while FA has a "read only" mode, it's not actually, completely "read only". Some things still write to the DB. So my guess is that the code will not correctly cope with being unable to write to the database.

They were also on a hilarious ancient version of MySQL for many years, which I'm sure lacks usable replication features.

Fate · « **Reply #45 on:** February 12, 2012, 01:22:53 pm »

Quote from: Jim Demintia on February 12, 2012, 12:13:57 pm

FA is problems.

Well, yes. I was starting from the assumption that FurAffninty had been built and maintained by intelligent, competent people with knowledge and skills pertaining to everything from network design and hardware selection to software engineering, which is why it's wishful thinking.

Ketsuban · « **Reply #46 on:** February 12, 2012, 01:25:11 pm »

Quote from: Fate on February 12, 2012, 01:22:53 pm

Quote from: Jim Demintia on February 12, 2012, 12:13:57 pm
FA is problems.

FA is more than just problems - it's a collection of security flaws held together with congealed ejaculate.

Jim Demintia · « **Reply #47 on:** February 12, 2012, 03:19:19 pm »

Creeping up on 12 hours here. Anyone hearing anything on IRC or on FAF?

Fate · « **Reply #48 on:** February 12, 2012, 03:44:19 pm »

Nothing at all, but my brain has started adding the ending music for Deadly Space Action every time I see or hear the word "Furaffinity."

Conan · « **Reply #49 on:** February 12, 2012, 04:23:30 pm »

Quote from: Jim Demintia on February 12, 2012, 03:19:19 pm

Creeping up on 12 hours here. Anyone hearing anything on IRC or on FAF?

They're stuck on "Importing the database".

Jim Demintia · « **Reply #50 on:** February 12, 2012, 04:56:51 pm »

At 12 hours in? I guess I shouldn't be surprised, but haha wow. I would imagine those four hundred million rows or whatever will take a while to import.

loki · « **Reply #51 on:** February 12, 2012, 06:24:33 pm »

Quote from: Jim Demintia on February 12, 2012, 04:56:51 pm

At 12 hours in? I guess I shouldn't be surprised, but haha wow. I would imagine those four hundred million rows or whatever will take a while to import.

A rational DBA or software engineer would take time to figure out why they have 400 millions row in anything and take steps to fix that....

But hey this is FA we're talking about! More hardware! MORE!

Pi · « **Reply #52 on:** February 12, 2012, 06:53:21 pm »

I've got a shiny new Susan B. saying that the import is not gonna work right and they'll have to roll back.

Conan · « **Reply #53 on:** February 12, 2012, 06:57:52 pm »

Quote from: Jim Demintia on February 12, 2012, 04:56:51 pm

At 12 hours in? I guess I shouldn't be surprised, but haha wow. I would imagine those four hundred million rows or whatever will take a while to import.

Quote from: Yak

Spoke too soon.
While most of the database - if you count by the number of tables - was imported, about half a dozen tables left to import comprise a larger part, if you count by data volume. We're looking at several more hours here.

Ketsuban · « **Reply #54 on:** February 13, 2012, 02:40:00 am »

Quote from: Yak

Still here, still importing.
messagecenter_submissions.ibd, 38452M, will you import already.

Every time I see one of their UI ~~cockups~~ mockups using some genericfur's account showing thousands upon thousands of submissions, journals, watches and notes I want to vomit.

Conan · « **Reply #55 on:** February 13, 2012, 03:11:48 am »

Quote from: Ketsuban on February 13, 2012, 02:40:00 am

Quote from: Yak
Still here, still importing.
messagecenter_submissions.ibd, 38452M, will you import already.

Every time I see one of their UI ~~cockups~~ mockups using some genericfur's account showing thousands upon thousands of submissions, journals, watches and notes I want to vomit.

Personally, I've stopped deleting mine just to make that table larger.

Also coming up on 24 hours here.

Fate · « **Reply #56 on:** February 13, 2012, 04:29:45 am »

Twenty-seven hours into the six-hour upgrade. I figure, what the hell, right? It's only 450% of the time estimate. Totally reasonable.

The Inverse Scotty. Nobody's ever gotten outright fired for that in the past, right?

Fate · « **Reply #57 on:** February 13, 2012, 06:02:46 am »

Doublepostin' because of the time gap.

"Database has been imported and everything looks fine.
Doing a few last checks before bringing FA back online."

So that's... Twenty-nine hours for a six hour upgrade. I don't know whether to applaud, masturbate with glee, or wait until the tests take a further four hours.

Ben · « **Reply #58 on:** February 13, 2012, 07:03:29 am »

26 hours, actually. The downtime started around 8:20am EST, they promised it would begin at 4:40am PST.

Ketsuban · « **Reply #59 on:** February 13, 2012, 01:33:17 pm »

And now people are reporting 500 Internal Server Error responses when they try to access the message centre.

Gee, I wonder if this has anything to do with that forty-gig table that yak specifically called out as taking too long to import?