[THE CREATURE STIRS] or "A rare FA code update"

[Conan]

FA is looking for testers (mainly popular artists) to test out a "patch" that makes FA not choke on trying to shove 10,000 INSERT queries through the database when someone posts something.

http://www.furaffinity.net/journal/5269529

Good news, everyone!

We have news that's NOT a banner update notification!

We're looking for testers to help put a patch through the ringer, and we need volunteers! Specifically, volunteers who meet a specialized criteria:

* Volunteers MUST have 8,000+ watchers
* Volunteers MUST have original, fresh content to post (submissions and/or journals)
* Volunteers MUST have an account in good standing

Interested? Send a note to THIS ACCOUNT! We're accepting 20 total volunteers to help out!

We're testing a patch that significantly modifies how FA sends out notifications. Currently, artists with large amounts of watchers may face timeouts when submitting journals or uploading new submissions, and in some cases notifications of the submission may get lost when this happens. This new update in testing instantly submits the journal or submission, then breaks up notifications into smaller batches. Watchers will be notified gradually until the list is complete.

The hideous Spider People of Taurus VII need not apply.

EDIT: Lowered watch count slightly to make it available for a wider audience.

This "patch" comes just a few months shy of the site being online for 8 years. 8 Years. It took them eight years to figure out that this was a problem and figure out to fix it.

[magus]

The best part, of course, is they could've just done a query to see who has 8k watchers or more, and then note all those people - it can't be a very long list.

But they have to look like they're doing SOMETHING.

[pmart]

The best part, of course, is they could've just done a query to see who has 8k watchers or more, and then note all those people - it can't be a very long list.

But they have to look like they're doing SOMETHING.

For the curious and lazy, the number is roughly 338.

[Conan]

The best part, of course, is they could've just done a query to see who has 8k watchers or more, and then note all those people - it can't be a very long list.

But they have to look like they're doing SOMETHING.

According to popufur.com there are fewer than 200 people that met the original 10k watchers requirement, and 340 with 8,000 watchers or more.

Contacting just those people is too hard and they probably didn't want to be accused of "playing favorites" even though they'd have a reasonably valid excuse in this situation.

[Kindrift]

I'm also getting the feeling that FA cannot possibly not have an announcement in the header.

[Conan]

Yesterday, these changes were put into a public beta.

Today, the site began whitescreening and MySQL crashed. Now it's slow as shit and will need to go down "to do maintenance to alleviate the issue." sometime tomorrow. This is also showing that their new redundant database servers are still not set up or are set up incorrectly.

Never change, FA.

[zed]

Yesterday, these changes were put into a public beta.

Today, the site began whitescreening and MySQL crashed. Now it's slow as shit and will need to go down "to do maintenance to alleviate the issue." sometime tomorrow. This is also showing that their new redundant database servers are still not set up or are set up incorrectly.

Never change, FA.

I can't believe what I am reading. Are you implying that Dragoneer chose to deploy untested changes on a production site? Because that would be moronic. Obviously the admins have a staging server deployed somewhere that they can use for this kind of load testing. Yes, that's it. I am quite sure that one of the coders has written a script that could be used for the kind of stress testing which they are definitely not unleashing upon their production environment. And surely--surely--the admins used a battery of unit tests and even some integration tests before deciding to deploy to staging. And the positive outcome from their continuous integration tests supported this decision.

Or not.

[Conan]

FurAffinity is forced to switch into readonly mode due to the heavy database traffic taking place over the past couple of weeks building up a large backlog of unprocessed data. This amount of data is negatively affecting performance and has to be given time to be processed. No ETA on the removal of readonly mode at this moment.

http://forums.furaffinity.net/threads/795080-2013-12-09-Database-maintenance?p=4151485#post4151485

Discovered a huge backlog of unpurged InnoDB transactions spanning the timeframe of more than a day.
That explains the slowdowns and extra space being taken up by the ibdata file.

Forcing the purge as fast as the RAID10 array of 15k drives allow for.

The coding talents of Yak shine through once again.

[graeme]

They're just positively awful.  They have no idea how to do any level of test.  "Unprocessed data built up in the database over the past weeks"

What the fuck does that even MEAN?  How do you have that much power hardware wise and still fuck it up this badly?  I would ask if there are integration tests or load tests or such.. but.. let's be serious, we know the answer.  This is awful, and people need to be beating down Dragoneer's door on this.

But who would take the job if Yak got fired?  Really?

[winserv03fan]

What the fuck does that even MEAN?

It sounds like they have some script that breaks apart the notification load into parts. If I had to guess, whatever they are using is skipping over some data causing their database cache to keep growing. This probably means that at some point they will have to rewrite their new notification system before the site will be functional again.

Pretty much just typical FA. They update a few lines of code and the whole site goes to shit.

people need to be beating down Dragoneer's door on this.

Uh, maybe not.

[mahadri]

It sounds like they have some script that breaks apart the notification load into parts. If I had to guess, whatever they are using is skipping over some data causing their database cache to keep growing.

That the ibdata file is growing indicates database updates are being written to the InnoDB transaction log but the disks don't have enough IOPS to commit the transactions to the database. The new notification queueing shouldn't have increased the load more than the previous notification method. Since the database response time has been getting progressively higher after the upgrade, I suspect it's now reached a point where the disks can no longer keep up.

Curious that the new database server has rotating disks instead of SSDs...

[nrr]

Curious that the new database server has rotating disks instead of SSDs...

SSDs aren't the cure-all for applications like this, and unless you really do your homework, you could end up with some seriously shitty disk that will end up tanking on you when you least suspect it. The track record with FA's hardware purchases leaves a lot to be desired, and I wouldn't trust them to be able to spec out decent SSDs, even despite Mr. Piche's fetish with gadget shit.

FWIW, they have a serious boner for using FreeBSD on barely-supported hardware. I'd look there first before saying, "Oh, yeah, just plop in SSDs."

[JigsawJones]

How long would FA have to be frozen to see an uptick in other furry board activity?

[nrr]

How long would FA have to be frozen to see an uptick in other furry board activity?

Something on the order of about a year? FA was down for months in the past, and that made maybe a negligible dent in things. Maybe.

[graeme]

Well, massive security breaches haven't done it, the knowledge that rogue admins read all the notes hasn't done it, the downtimes haven't done it.

It'd probably have to go down for a year or longer to get some migration, but I bet people would come right back to it. 

If there were a one click "migrate away" button/website/scraper/inviter that could pull your art down, post them at a new board, check and see what friends are there (somehow?) and readd those contacts MAYBE you could overcome the network effect.  Maybe.  But that's a lot of work and would likely get the FA crew pretty mad at you, and then their fanboys would  just DDOS the "transfer" site or something.

I think personally a bigger crash is coming.  You don't have small events in isolation in broken systems.   These things are symptoms, not individual root causes.

[Conan]

How long would FA have to be frozen to see an uptick in other furry board activity?

Other sites see a significant increase in traffic almost immediately. It lasts about as long as FA is down. This has been the case with Weasyl the last few times FA has gone down and I'd imagine it's even more prevelant on sites like e621, SoFurry, Inkbunny, etc.

[JigsawJones]

How long would FA have to be frozen to see an uptick in other furry board activity?

Something on the order of about a year? FA was down for months in the past, and that made maybe a negligible dent in things. Maybe.

You seem to have forgotten the Furry Art Place incident. The rush to that site was SO pronounced, it prompted an astonishing PR campaign on Dragoneer's part, with near daily updates on hardware purchase and setup. FAP really lit a fire under those fat haunches...

[graeme]

How long would FA have to be frozen to see an uptick in other furry board activity?

Something on the order of about a year? FA was down for months in the past, and that made maybe a negligible dent in things. Maybe.

You seem to have forgotten the Furry Art Place incident. The rush to that site was SO pronounced, it prompted an astonishing PR campaign on Dragoneer's part, with near daily updates on hardware purchase and setup. FAP really lit a fire under those fat haunches...

I'm not entirely convinced that the purchases or setup happened.   The moment FAP stopped being an issue, all the commentary stopped.

[GreenReaper]

How long would FA have to be frozen to see an uptick in other furry board activity?

Other sites see a significant increase in traffic almost immediately. It lasts about as long as FA is down. This has been the case with Weasyl the last few times FA has gone down and I'd imagine it's even more prevelant on sites like e621, SoFurry, Inkbunny, etc.

To put a few solid numbers to that, Inkbunny's daily incremental file backup is usually 1.0GB (plus another 1GB for the compressed DB). Yesterday morning it was 1.5GB, and today, 1.7GB. Site traffic is also higher, but only by about 10-20%. If the downtime continues to the weekend, I'd expect these numbers to increase. It tends to last for at least a little while after downtime as people decide it's a good idea to have a backup gallery.

As for FA:

The processing rate of the backlogged data made it clear that migrating to a new server would be a faster thing to do.
Migration status is at 53%. Until the migration is over FA will have to remain in readonly mode.
Using the opportunity to upgrade from MySQL 5.5.x to 5.6.x which brings a number of performance improvements across the board, some of which are directly related to our current problem.

[Pi]

GR, you talk too much.