Author Topic: Performance issues prompt "unplanned maintenance". Maybe. (Read 6277 times)

loki · « **Reply #20 on:** January 28, 2012, 02:28:56 pm »

Quote from: GreenReaper on January 27, 2012, 10:23:58 pm

Uh, that's four hundred million rows . . .

Yea, I didn't even look at the number, was just going off what Conan said. If you have 400 million rows in a single table, you're doing something very wrong. Their DB schema is probably a huge mess. Hell, I'm surprised MySQL can even support that.

ProvincialTwit · « **Reply #21 on:** January 28, 2012, 04:46:57 pm »

Like most other poorly-constructed open-source projects, if you throw enough resources at it...

*(resources required scale exponentially)

Conan · « **Reply #22 on:** February 06, 2012, 05:09:08 pm »

After a week and a half of "Unplanned maintenance this coming week" it's finally been scheduled for "Feb 11-12" (which is about as broad as you can get when scheduling something) to "make upgrades to our the SQL server and perform other scheduled maintenance".

Conan · « **Reply #23 on:** February 07, 2012, 10:11:56 pm »

And an update:

Quote

Reminder: We've an outage scheduled this weekend (Feb 11-12) to do upgrades, general maintenance and improve site performance. We're upgrading our SQL database and applying system patches. We estimate it to take 8-12 hours, but it may take longer. Plan accordingly!

It may be a coincidence, but exactly a year ago the same things were going on. Maintenance was scheduled for the first weekend in February to update the OS and MySQL. It was scheduled, but then delayed a week, and then delayed another 24 hours. When it was all done, the "3-6 hour" maintenance window had stretched to 12 hours.

Jim Demintia · « **Reply #24 on:** February 08, 2012, 04:04:43 pm »

So their DB is on a Sun box? Did I see someone say that in another thread? Is it a SPARC box? Because if so, I find this, from Wiki, to be interesting.

Conan · « **Reply #25 on:** February 08, 2012, 05:01:06 pm »

Quote from: Jim Demintia on February 08, 2012, 04:04:43 pm

So their DB is on a Sun box? Did I see someone say that in another thread? Is it a SPARC box? Because if so, I find this, from Wiki, to be interesting.

They have a Sun x4150.

Also, Yak now claims they're "replacing" the database software, though that could just be a mistake since he is not too good with English.

Jim Demintia · « **Reply #26 on:** February 08, 2012, 05:19:27 pm »

So it's an Intel box. Oh well.

Yak is still going on with the scrapers thing, I see. I really wonder what that's about. Unlike Piche, I don't really get the idea that pathological, straight-faced lying is his thing. You are left to wonder if these are not bots scraping the site because either the bots, or the site, are badly configured (or both).

Has someone really gone into all the trouble of writing a scraper for FurAffinity but not writing it well enough to make it hard to tell that it is, in fact, a scraper? I mean, is someone just running 'wget -R' on furaffinity.net? I'd really like to know.

(and are there so many of these people, behind communal NAT boxes, that a simple IP block won't suffice? I suspect incompetence here, that the activity being described are bots, not scrapers, and Yak really doesn't understand the difference. That would be odd...though, since I would think, given the kinds of things we might imagine Yak does in his spare time, he would be well aware of such things.)

ProvincialTwit · « **Reply #27 on:** February 08, 2012, 06:21:39 pm »

You suspect incompetence?

In relation to FA?

BRA FUCKING VO JIM

Pi · « **Reply #28 on:** February 08, 2012, 07:27:20 pm »

Near as I can tell: they're upgrading MySQL because they have no idea what's actually going on and they hope it'll fix it.

Those of us who live on round Earth know that this is a great way to irrevocably screw your database up if handled inappropriately.

Fate · « **Reply #29 on:** February 08, 2012, 07:50:36 pm »

So, best case scenarios are that they botch the mysql upgrade and either trash their database that way, or someone forgets to set old_passwords=1 and FA grinds to a screeching halt until Piche or someone trashes the database in fury.

ColonThree · « **Reply #30 on:** February 09, 2012, 08:00:49 am »

It wouldn't surprise me if it fucked up in an even more bizarre manner, like the existing FA code only working because it inadvertently makes use of a bug/'feature' in mySQL which Percona has fixed.

Freehaven · « **Reply #31 on:** February 09, 2012, 08:35:24 am »

Quote from: ColonThree on February 09, 2012, 08:00:49 am

It wouldn't surprise me if it fucked up in an even more bizarre manner, like the existing FA code only working because it inadvertently makes use of a bug/'feature' in mySQL which Percona has fixed.

Given all the other problems with FA's code, this wouldn't surprise me in any way.

At least a prolonged shutdown might get users migrating to other sites.

Jim Demintia · « **Reply #32 on:** February 09, 2012, 08:39:17 am »

The "best case" is that they have some kind of incomplete backup, which results in some level of data loss (if memory serves, that happened once before). Even in that case, there is the issue that, the backup may be useless in the new MySQL version they are upgrading to.

And whatever happens, I don't think anyone believes this is going to fix anything, and may very well make things worse. It'll be entertaining, at any rate.

Pi · « **Reply #33 on:** February 09, 2012, 09:01:51 am »

after a goddamn year of asking, yak has finally provided some rationale about scrapers:

Quote

<yak[away]> Pi: scrapers load gallery pages with large numbers, very fast and sometimes in parallel; with pagination in mind this generates sql queries with `LIMIT $big_number_here, 60`. half a decade old mysql query optimizer bug interprets that as "read $big_number_here rows from disk into the buffer pool, discard them, then read 60 more and return".
<yak[away]> this both thrashes the disks and wipes a large portion of the in-memory table data cache to make room for rows that are immediately discarded.

See, again, to me this doesn't scream "upgrade the database", it screams "fix my fucking query". But that'd require, y'know, competence.

Fate · « **Reply #34 on:** February 09, 2012, 10:12:31 am »

tbh, the entirety of FurAffinity screams "we need competent software engineering, hardware choices, and network design" but it's just as likely to get Westboro Baptist to host the servers.

loki · « **Reply #35 on:** February 09, 2012, 07:26:59 pm »

Quote from: Jim Demintia on February 09, 2012, 08:39:17 am

The "best case" is that they have some kind of incomplete backup, which results in some level of data loss (if memory serves, that happened once before). Even in that case, there is the issue that, the backup may be useless in the new MySQL version they are upgrading to.

And whatever happens, I don't think anyone believes this is going to fix anything, and may very well make things worse. It'll be entertaining, at any rate.

I wonder if they even ever bothered to test if their restore process works. It's going to be one of those "What a Terrible Failure" stories where they lose all their data despite having backups - it's great that you have backups but it's pretty important to test that your restore process actually works too.

Conan · « **Reply #36 on:** February 09, 2012, 11:13:53 pm »

Yak doesn't even know if this will fix the problems, but they're sure going to try. It also sounds as if they haven't, y'know, tested anything yet.

Quote

The upgrade to Percona is something I've been planning on for a while now; it is not directly related to fixing the current situation, although being a performance oriented fork of MySQL with fewer limitations and more tunable parameters - it will provide a passive benefit for hopefully long enough to have the slowdowns problem fixed directly.
The reason for the switch to Percona is that it allows for easy online binary incremental backups as well as sane replication, load balancing and failover scenarios when used in conjunction with MMM. We are looking into scaling out with a secondary master.

Jim Demintia · « **Reply #37 on:** February 10, 2012, 06:23:28 am »

So they're going to upgrade their Pentium 4 of a database to an overclocked Pentium 4. How very them.

Conan · « **Reply #38 on:** February 12, 2012, 04:24:48 am »

Quote

Administrator notice:
Outage Notice: FA to go down approximately 4:40am Pacific Time, approximately, for at least 6 hours. Please check the Site Status forum for further updates.

And here we go! Place your bets! I say this goes late into the evening, and when it comes back something is broken.

Jim Demintia · « **Reply #39 on:** February 12, 2012, 08:04:16 am »

Quote from: Conan on February 12, 2012, 04:24:48 am

Quote
Administrator notice:
Outage Notice: FA to go down approximately 4:40am Pacific Time, approximately, for at least 6 hours. Please check the Site Status forum for further updates.

Why can't he write anything without hilarious mistakes?