Oops, FA Got Hacked (Again)

[Zzyzx]

It was brought to our attention last night (May 16) that someone had obtained a copy of Fur Affinity's source code via the recent “ImageTragick” exploit in the ImageMagick library (a common server-side image processing software). This exploit was patched earlier in this month, but not before a malicious user was able to download a copy of our source code, and later actively distributed it via USB drives at a convention.

We managed to get a hold of one of the USB drives and started to analyze what was distributed. While we were investigating, somebody launched a second attack against the site using information gleaned from the source code.

This attack targeted the site’s database by deleting user information, submissions, and watches. It was stopped before any further damage could be done. Other information such as journals, notes, passwords, and personal information was not affected. We're currently in the process of doing a security audit on the existing code and closing any loopholes which may be accessible from the source code.

We are also working to restore the deleted data. Our most recent full backup is from May 11, so approximately 6 days worth of new user registrations, account watches, and new submissions have been lost due to the attack. We are still trying to evaluate the scope of the attack.

We apologize for the inconvenience to the community, and are working to rectify the issues. If anyone has any knowledge/evidence as to who perpetrated the attack, or who was distributing the USB drives containing FA’s source code, please privately contact Dragoneer on Twitter (@Dragoneer) or via email at dragoneer@furaffinity.net.

We are working to restore FA as quickly as we can, but want to make sure we take proper steps to prevent any further issues. We will keep the community updated on our progress.

http://forums.furaffinity.net/threads/5-17-site-attack.1530523/

[Spip]

Glorious. Now how long is it going to take for the leaked source code to get out into the open (or something close to it, e.g., some private Bit Torrent tracker somewhere)?

[nrr]

The exploit in question was not with FA's code but with a plugin called ImageMagick.

I was expecting FA data base have a back up similar to RAID 1, aka real time back up.

They do have backups but backing up a gigantic website will take a large amount of space and time leading to a complete halt during the backup time since it would need to basically make copies of every picture, profile, password, etc. so they backup slowly so they dont disrupt the flow of things.

On doing full database backups, depending on what SQL server you are using, you may be able to do an incremental backup of the database. *BUT* that assumes you're using MySQL and can run Percona to do the backups. Plus, Percona locks the tables, so you may be scheduling a regular Read-Only for the backups *IF* they are quick enough.

If they're not, and you're running MySQL, there's a tool called DRBD that can replicate a partition that contains a MySQL database to a second server. At night, break the DRBD link, mount the partition on the secondary server, backup the MySQL DB files, unmount, start DRBD back up again, and let it sync back up.

If you're using PostgreSQL, you got some research to do!

Oh, okay.

[ColonThree]

please privately contact Dragoneer on Twitter

Privately contact someone on Twitter? Is that a thing?

[rodox_video]

5/17 never forget

[Fate]

Pi, can we get a copy of the FA source?

[Pi]

Pi, can we get a copy of the FA source?

t fate i hope you don't take me for the kind of person who'd grab a random flash drive and plug it into my machine

[nrr]

t fate i hope you don't take me for the kind of person who'd grab a random flash drive and plug it into my machine

No, but I would take you for the kind of person to grab unsuspecting things like that and break them open later.

[graeme]

Pi, can we get a copy of the FA source?

t fate i hope you don't take me for the kind of person who'd grab a random flash drive and plug it into my machine

No, I'm taking you as the kind of person to plug in a random flash drive to someone else's machine

[Fate]

Pi, can we get a copy of the FA source?

t fate i hope you don't take me for the kind of person who'd grab a random flash drive and plug it into my machine

No, I'm taking you as the kind of person to plug in a random flash drive to someone else's machine

this. I didn't say anything about *your* machine, but even I hang onto a burner box built out of the wife's first laptop running TAILS.

[an Hoopoe]

Dragoneer, on reddit, is blaming old hardware:

A few things. We're a big target, some of the gear we're running is 8+ years old (dinosaurs in tech years). We've been phasing out the older gear and replacing it with newer, more reliable tech. We've also had some issues with hardware failures on the networking side on our host. Unfortunately, it's been a lot of various things at once.

^
https://www.reddit.com/r/furry/comments/4jtezr/fa_status_and_site_attack/d39hmfe

And he wants more hardware:

Right now, it's matter of storage and resources. We're working on upgrading those to allow more rapid backups.

^
https://www.reddit.com/r/furry/comments/4jtezr/fa_status_and_site_attack/d39os5c

Meanwhile, doing damage control on the FA forums, FA admin quoting mungo explains that luckily ' nothing has been that wrong' with the code and alas FA just doesn't have the funds or resources to fix things:

[...] unfortunately what we're stuck with is a balancing act. And experience tells us when only invisible updates happen, that breeds discontent, because from the outside it looks like nothing is being done at all (and not everyone is willing to believe us if we say "actually backend updates are happening"). So we've tried to do the best we can while keeping everyone as happy as possible.

It sucks that someone decided to take advantage of the old codebase to screw over the entire userbase. No one wishes this had never happened more than we do. And yes, vulnerabilities in our code enabled it to happen. Unfortunately, having funding and having unlimited resources are two very different things, so doing things instantly (or anywhere close) is not an option.

^
http://forums.furaffinity.net/threads/5-17-site-attack.1530523/page-46#post-5472621

[Folseh]

Dragoneer, on reddit, is blaming old hardware:

A few things. We're a big target, some of the gear we're running is 8+ years old (dinosaurs in tech years). We've been phasing out the older gear and replacing it with newer, more reliable tech. We've also had some issues with hardware failures on the networking side on our host. Unfortunately, it's been a lot of various things at once.

^
https://www.reddit.com/r/furry/comments/4jtezr/fa_status_and_site_attack/d39hmfe

And he wants more hardware:

Right now, it's matter of storage and resources. We're working on upgrading those to allow more rapid backups.

^
https://www.reddit.com/r/furry/comments/4jtezr/fa_status_and_site_attack/d39os5c

Meanwhile, doing damage control on the FA forums, FA admin quoting mungo explains that luckily ' nothing has been that wrong' with the code and alas FA just doesn't have the funds or resources to fix things:

[...] unfortunately what we're stuck with is a balancing act. And experience tells us when only invisible updates happen, that breeds discontent, because from the outside it looks like nothing is being done at all (and not everyone is willing to believe us if we say "actually backend updates are happening"). So we've tried to do the best we can while keeping everyone as happy as possible.

It sucks that someone decided to take advantage of the old codebase to screw over the entire userbase. No one wishes this had never happened more than we do. And yes, vulnerabilities in our code enabled it to happen. Unfortunately, having funding and having unlimited resources are two very different things, so doing things instantly (or anywhere close) is not an option.

^
http://forums.furaffinity.net/threads/5-17-site-attack.1530523/page-46#post-5472621

Well thank goodness FA is owned by IMVU...they can get the new hardware stuff for FA, the community cant pay for that stuff anymore Neer

[nrr]

Dragoneer, on reddit, is blaming old hardware:

To his credit, he at least understands half of the problem that he's trying to solve here. He's pretty strapped for resources. That's a fact.

Wanting newer hardware for more rapid backups, though, is a bit misguided. Really, the thing that he should be optimizing for is rapid restores. I don't give a shit how quickly you can back up data because what I ultimately care about is being able to restore service, in a timely manner, to where it was before disaster struck.

For those playing along at home: If your storage hardware is slow (hint: it's slow), find a good way to take deltas of your database, and keep a rolling log of files that get written to disk via user uploads. Write that to your backup media. Make your application and systems aware of this by both notifying them when and what things have been backed up and making them consume those notifications.

Then, after you've backed stuff up, periodically try to restore a random sample of that data. You know, just to make sure that your backups actually work.

Why we call it a backup system is beyond me; it should be called a fucking restore system. Anyway, RPO, RTO, and MTTR, motherfucker.

[Zzyzx]

To his credit, he at least understands half of the problem that he's trying to solve here. He's pretty strapped for resources. That's a fact.

Wanting newer hardware for more rapid backups, though, is a bit misguided. Really, the thing that he should be optimizing for is rapid restores. I don't give a shit how quickly you can back up data because what I ultimately care about is being able to restore service, in a timely manner, to where it was before disaster struck.

For those playing along at home: If your storage hardware is slow (hint: it's slow), find a good way to take deltas of your database, and keep a rolling log of files that get written to disk via user uploads. Write that to your backup media. Make your application and systems aware of this by both notifying them when and what things have been backed up and making them consume those notifications.

Then, after you've backed stuff up, periodically try to restore a random sample of that data. You know, just to make sure that your backups actually work.

Why we call it a backup system is beyond me; it should be called a fucking restore system. Anyway, RPO, RTO, and MTTR, motherfucker.

Incremental back-ups would certainly be nice instead of "oops, we lost six days of shit." I'm still curious how Dragoneers knows it was the ImageMagick exploit that they already patched. Or, at least, supposedly patched. Unless the person who spread the code explained how they got it on a handy NFO or something equally stupid. I really wouldn't be surprised if he's blaming ImagicMagick instead of admitting that the site is so horribly coded that there's likely a million exploits hidden in it. If the source code was spread at all. That all seems rather convenient. Too bad there haven't been any projects to, like, update FA's code or anything like that. I'm actually kind of surprised it took this long for another security breach. Though, I wonder how often people manage to slip in and steal things from the site without the staff knowing. Who knows how many leaks might've been covered up over the years.

[nrr]

Too bad there haven't been any projects to, like, update FA's code or anything like that.

You must be new here.

[Ketsuban]

The thing that surprises me is that it took two days from the ImageTragick announcement for them to patch it. I found out about it the same day, and I follow exactly one security expert on Twitter. Piche runs a major website/community hub; is it too much to expect him to follow exactly one security expert on Twitter?

[ColonThree]

is it too much to expect him to follow exactly one security expert on Twitter?

I know a rhetorical question when I see one

[Conan]

The thing that surprises me is that it took two days from the ImageTragick announcement for them to patch it. I found out about it the same day, and I follow exactly one security expert on Twitter. Piche runs a major website/community hub; is it too much to expect him to follow exactly one security expert on Twitter?

When the exploit was announced, we joked in IRC that FA wouldn't be affected because they were probably running some ancient version of ImageMagick.

Boy did they prove us wrong!

[rodox_video]

I'm still curious how Dragoneers knows it was the ImageMagick exploit that they already patched. Or, at least, supposedly patched. Unless the person who spread the code explained how they got it on a handy NFO or something equally stupid. I really wouldn't be surprised if he's blaming ImagicMagick instead of admitting that the site is so horribly coded that there's likely a million exploits hidden in it. If the source code was spread at all. That all seems rather convenient. Too bad there haven't been any projects to, like, update FA's code or anything like that. I'm actually kind of surprised it took this long for another security breach. Though, I wonder how often people manage to slip in and steal things from the site without the staff knowing. Who knows how many leaks might've been covered up over the years.

It is honestly totally plausible that this had nothing to do with ImageMagick. Piche is long past lying to cover his ass. It could have been any number of things.

IIRC 12/16 was initially blamed on reused credentials from the Gawker hack until the hacker came forward on Lulz and ruled that out.

I have to wonder if the code is being kept off the internet intentionally. Just because bitlockers are not anonymous enough to fuck with a nation-state or corporation anymore doesn't mean they aren't still more than adequate for messing with relatively insignificant places like FA if utilized correctly.

We have no idea how many drives there were and how many picked them up. Presumably some are still afraid of them being traced back, but I imagine others realize that keeping that source code off the public internet also keeps people from finding other exploits in it and forwarding them to Piche. An unlikely scenario to be sure, but weirder shit has happened.

[applesauce]

Well, it didn't take long for their database security to be cracked. Was it also from 2008? People are complaining their other sites are getting hacked.

https://twitter.com/NanukBurr/status/733597003775496194

FYI: Users are reporting unauthorized access to accounts using the same password as their FA account. Take steps NOW to secure accounts.