Database Server Failure

[Conan]

Perhaps the most predicted failure to date, FA is offline this weekend because of a "bug in the RAID controller" for the database server and one or more failed hard drives.

They've overnighted parts, apparently not realizing because it's the weekend they won't be shipped until Monday and won't arrive until Tuesday. Or they simply don't want to have to admit the site will be down until Tuesday.

[winserv03fan]

Another problem caused by out of date software. Or firmware. Whatever. FA is shit, you get the point.

[nrr]

I can actually buy the RAID controller firmware bug bit. We had a couple of SAN members very recently tank at CNN, which we found was attributed to a firmware issue that just bit us with no warning. Furthermore, that kind of shit happened all the time with our SAN members on the JIRA Studio and Atlassian Enterprise Hosted infrastructure when I was managing that.

What I find absolutely inexcusable, however, is the fact that they didn't have some sort of a slave database replica that they could've immediately promoted by way of Heartbeat or CARP or similar. Instead, they're left waffling about with downtime. Really?

This goes to show you that you cannot trust the hardware running your ship to stay up, so design your software around this fact.

[magus]

RAIDs can be randomly fragile. And buggy. And all those things.

But go look at the 'SFW' thread. Where yak (assuming it's him - also assuming the unset iptables rule was blocking Moldova or some such) admits he ought to properly back up the db, but hasn't taken the time to figure out how to run xtrabackup. Which isn't exactly fucking rocket science. And the inexcusable lack of a slave to take over, but we know from the great noncooperative hardware inventory that there's nothing the FA admins love better than having spare hardware... and then doing jack with it.

[Conan]

And the inexcusable lack of a slave to take over, but we know from the great noncooperative hardware inventory that there's nothing the FA admins love better than having spare hardware... and then doing jack with it.

I seriously wonder if they think they need to have identical hardware to build a cluster. They tweeted that they've "been trying to save up to procure a new DB server, but we lack funding." but I'm sure the plan is to buy another overpowered 64GB of RAM monstrosity instead of buying two reasonable servers and building a cluster out of them.

[magus]

Well, you do need something in the ballpark, so that peak load doesn't bog one of them down and not the other, but considering their overkill approach, that's unlikely.

On that note, what's the betting pool like on the completely retarded hardware they'll try to blackmail the users into buying this time? I'm going to go with a big fuckoff PCIe SSD. As shiny as possible, fusion-io if they can swing it. Because it has the latency. And the IOPS. And 'It's not a RAID, so it won't fail this way again!'

[Lynx]

...how long until we have another 'SAVE TEH FA!' fund raiser?

I'm certain some dipshits out there will toss money at Princess Piche to get their porn back 24/7.

[a pigeon]

Might be hard to do a fund raiser, unless they can use kickstarter or indiegogo or whatever:

@WufflesWorgen You do know that we get next to no donations anymore, right?

@zidarsk8 Lack of online method to collect. We can only do check/money order. People don't rush out to do those.

https://twitter.com/furaffinity/status/244652093011533825

@beach_fox I wish, but we can't quite afford another DB server right now.

https://twitter.com/Dragoneer/status/244621619618799616

Sciggles re-tweeted some tweets by Silverautomatic to the effect of it isn't FAs fault and they've overnighting parts look at how saintly they are. 

The approved response from the userbase, as ever, is flattery and excuse making.

[nrr]

The approved response from the userbase, as ever, is flattery and excuse making.

Frankly, I think it's adorable, in a way only a weasel could love.

[ProvincialTwit]

Frankly, I think it's adorable, in a way only Weasyl could love.

Fixed. (dooohohohoho I'm so clever)

[Gryphoneer]

That Summercat guy stepped down over this, btw. Note how words like respect, decorum, and professionalism are mentioned in the same sentence as FA's badministration.

Welp.

[zed]

It has been, what, 7 years, and these people still have no idea how to run a website? They really, truly ought to just pack up and go home; SoFurry and InkBunny will be perfectly reasonable replacements. The FA staff have ignored all the advances in web development in the past decade, it seems. Instagram can go from startup to acquisition in two years, scaling steadily in that time to store millions and millions of pictures using cloud storage efficiently with all the trimmings of a proper web technology company. Dragoneer and yak are trying to get a RAID cluster working without any backups. No API. No mobile version. PHP.

You know what the difference is? Furries don't know how to make money. Good furry artists barely have a living wage, so they're poor. Lots of furries spend great deals of their disposable income on what are admittedly cheap art prices, so they're poor, too. And the staff running FA, InkBunny, and SoFurry don't really have business plans, so they're also poor. Even the sites that furries use for art auctions--Furbid, Furbuy, the Dealers' Den--don't have business plans either, and they are facilitating fucking financial transactions. The only smart people in the fandom have full time jobs, gee whiz. Everybody else is just perpetuation a cycle where nobody makes any money. FA runs like shit because they have no money--at least no money to hire anyone who knows a thing or two about server stacks; artists make little money because their fans can't spend anything; and the art whores make no money because they work in low wage jobs and spend their disposable income on murrs and purrs.

Maybe I'm just feeling pessimistic, but nobody here should be surprised by this state of affairs.

[Fate]

Personally, I'm starting a dead pool on the entire database. Someone will fuck this so badly that FA will simply be dead, with everything on it gone entirely.

[krinndnz]

Personally, I'm starting a dead pool on the entire database. Someone will fuck this so badly that FA will simply be dead, with everything on it gone entirely.

I've been saying for a while that that's the big determinant for a new site - whoever's in the best position when FA finally keels over and dies from the incompetence of the people running it, will be the new default Furry Art Site.

We'll see if that one turns out to be another lesson in hubris.

[magus]

That Summercat guy stepped down over this, btw. Note how words like respect, decorum, and professionalism are mentioned in the same sentence as FA's badministration.

Welp.

That seems to be about the Sonic fiasco, rather than the database issue. As much as it'd be hilarious to have someone 'stepping back  from modding for the duration of the issue' when the issue, you know, technically keeps them from modding anything.

As for the database, wasn't yak basically saying he'd turned database backups off due to the heavy load on that system?

[Pi]

this is all too 'same shit different day' for me

wake me up when it gets interesting

[greaseyote]

How about this?

http://furaffinityforums.net/

[loki]

   
Administrator notice:
Fur Affinity is currently unable to return to its functionality in full capacity. The hardware the database server is running on right now does not provide enough performance for that.
Replacement parts for the failed server should arrive on Tuesday.

I wish that said 'our software is so badly written that we require half the computing resouces of all of eastern Asia in order to stay up even 96% of the time'.

[Conan]

"Let's throw something resembling the normal site online so we can call it "up" and talk about our excellent uptime numbers at the end of the year!"

[ISleepNow]

How about this?

http://furaffinityforums.net/

huh, never knew Mitt Romney was a furfag