FA admin account compromised (yet again)

[Eevee]

Offtopic, but I noticed that their entire comment system has nothing to check for multiple HTTP POST requests

POST??  Hahahaha.

This is pure speculation, but I wouldn't be too surprised if someone from FA is reading this thread or these forums and getting a free security audit.

No, a db restore went awry and journal comments got fucked up.

[Dima]

This is pure speculation, but I wouldn't be too surprised if someone from FA is reading this thread or these forums and getting a free security audit.

I have no idea if that's actually the case but the timing of this comment thing vs. loki's post is exceedingly odd.

Impossible, this would be the smart thing for them to do.

Seriously, there should at least be ~one~ person amongst the fa staff designated to research communities of this nature to at least keep an ear to the ground and yes, get the equivalent of a free security audit, but that might be my own paranoid nature. Also, Dragoneer seems to like doing that for himself, even if he is not thorough about it.

[Jim Demintia]

I dunno, the thought of it makes me want to say less about their site's security. Dragoneer has the equivalent of about six lifetimes worth of money, toys, and reputation that he doesn't deserve, something about contributing to that even a little bit bothers me.

[Pi]

This is pure speculation, but I wouldn't be too surprised if someone from FA is reading this thread or these forums and getting a free security audit.

No, a db restore went awry and journal comments got fucked up.

More precisely:
[03:09:11] <&yak[work]> I was trying to import a database dump into a separate DB but apparently mysql decided to use FA's main db  for that. Sooooo journal comments table got nuked becore I was able to stop it. And I can restore them only till 2010-12-19, everything after that is gone
[03:11:38] <&yak[work]> I am actually doing daily backups, it's just that everything went fucked up due to all the DDoS going on, and I haven't restored everything back in operation yet
[03:12:55] <&yak[work]> Yeah. I pretty much expect a lot of people to tear me a new asshole over this.

And of course, the asspats:
[03:14:40] <@Carenath> yak[work]: You told me before.. mistakes happen.. after I went into 'oh fuck' mode two weeks back.

[threadshitter]

Seriously, there should at least be ~one~ person amongst the fa staff designated to research communities of this nature to at least keep an ear to the ground and yes, get the equivalent of a free security audit, but that might be my own paranoid nature. Also, Dragoneer seems to like doing that for himself, even if he is not thorough about it.

This is under the expectation that ANYONE on the FA staff would do something rational in the wake of all this. Which they haven't.

[Conan]

[03:09:11] <&yak[work]> I was trying to import a database dump into a separate DB but apparently mysql decided to use FA's main db  for that. Sooooo journal comments table got nuked becore I was able to stop it. And I can restore them only till 2010-12-19, everything after that is gone

THE COMPUTER DID IT!!1!!1one!!! I hope no one gets an eye poked out with all this finger pointing going on at FA.

And in other news, Inkbunny is now encrypting all data and content all the time.

[Jim Demintia]

Yeah, when the main guy whose responsible for keeping your porn site/welfare donation vehicle/excuse for being relevant operational is assigning sentient actions to the computer and random software packages, you...you've got a problem. Responsibility, pshaw.

And how much do you want to bet Yak's one of those people who doesn't really get the gist of the commands he mashes into the keyboard, and really only manages to achieve something close to the desired result because he remembers the right series of incantations to utter? Because those are the people who generally react this way (= have no clue what happened) when the magic spell doesn't work like they think it should.

And in other news, Inkbunny is now encrypting all data and content all the time.

Don't worry, there's ways to attack that too. A lot of it relies on social engineering (stupidity), so look for furries to be the first ones to fall for it. Vivisector does the same thing and every time I load it on my Android device the browser complains about the certificate not being valid. If IB has the same or a similar problem then some kinds of attacks become possible. Granted, that's not the site's fault but SSL is not the panacea it seems to be being touted as of late.

[Eevee]

Granted, that's not the site's fault but SSL is not the panacea it seems to be being touted as of late.

Sure it is!  It makes MITM attacks Someone Else's Problem.

I'm not sure if I'm being snarky or not.

[loki]

Boy, wouldn't it have been a good idea before doing ANYTHING directly on the DB to take a snapshot or 7z up the actual DB files? Especially when the site is live? Imagine the shitstorm if he nuked a table that tracks submissions to their authors or something like that...

[nrr]

Boy, wouldn't it have been a good idea before doing ANYTHING directly on the DB to take a snapshot or 7z up the actual DB files? Especially when the site is live? Imagine the shitstorm if he nuked a table that tracks submissions to their authors or something like that...

I do this literally 30-40 times per day.

Code: [Select]

svc -d /service/hurf_durf_some_j2ee_thing
svc -k /service/hurf_durf_some_j2ee_thing
su - postgres
pg_dump -O $DATABASE_NAME | gzip > ~/backups/misc/${DATE}__${DATABASE_NAME}__${EVENT}.dmp.gz
# do database shit here, maybe even drop the database!
exit
svc -u /service/hurf_durf_some_j2ee_thing && tail -f /service/hurf_durf_some_j2ee_thing/log/main/current
The incantation required for MySQL is eerily similar.

[Eevee]

Boy, wouldn't it have been a good idea before doing ANYTHING directly on the DB to take a snapshot or 7z up the actual DB files? Especially when the site is live? Imagine the shitstorm if he nuked a table that tracks submissions to their authors or something like that...

He wasn't trying to do anything to the db.  I think he was trying to load a dump into a different (testing?) database, but the dump contained CREATE DATABASE production; / USE production;.

[nrr]

He wasn't trying to do anything to the db.  I think he was trying to load a dump into a different (testing?) database, but the dump contained CREATE DATABASE production; / USE production;.

Sorry, bro, but that constitutes doing something to the database!  If you're connecting and doing anything but SELECT queries, you are potentially fucking something up.

[kayfox]

Somehow I would have imagined that he would grep the database dump for "USE" and all that, like I often do before restoring a dump from one database into another database on the same server.  But what do I know.

[loki]

He wasn't trying to do anything to the db.  I think he was trying to load a dump into a different (testing?) database, but the dump contained CREATE DATABASE production; / USE production;.

You might as well start doing DROP FA_SUPER_USER CASCADE (bye bye FA) at that point. Anyone who doesn't know that DDL commands force commits (they do in Oracle at least in most circumstances) probably shouldn't be allowed to play around with any sort of 'testing' database. Either way, you figure they would just make daily backups considering how much unused hardware they have. Even if they didn't have it, they could just keep backups for a few days back along with weekly or monthly iterations and you would be golden if something seriously bad happened.

Somehow I would have imagined that he would grep the database dump for "USE" and all that, like I often do before restoring a dump from one database into another database on the same server.  But what do I know.

You figure you would always know what you're running when doing anything like this; I'm not a DBA but it's like, holy shit, use some common sense and make sure you know what you're doing before you pull the switch. "Haha, we just lost 10 days of journals" is not acceptable and I'm [not] surprised yak is still there. :I

[ProvincialTwit]

Haha seriously who even includes the database structure itself in their mysql dumps.  It's bad enough they are using DATABASE FOR BABBIES but still.

I mean hell don't you have to add some kind of wonky flags to the mysqldump command to make it throw in CREATE DATABASE/USE commands? 

Oh wait sorry I forgot, Pi runs the show here and I don't know anything about how computers work HOW DID THESE WORDS GET HERE OH GOD

(ps fixed the ssl cert sorry about any errors or whatever)

[Conan]

Either way, you figure they would just make daily backups considering how much unused hardware they have. Even if they didn't have it, they could just keep backups for a few days back along with weekly or monthly iterations and you would be golden if something seriously bad happened.

They do have a backup server ("Neo Bahamut"), which according to WikiFur does "nightly backups of both data and DB".

Now whether it works or it's just there for show is another story. It apparently is quite a complicated process to restore old backups, since people who had their galleries wiped in the great hacking of 2010 still haven't had them restored. Presumably that's what Yak was working on when the comment thing happened.

[Jim Demintia]

according to WikiFur

Yes, well, we all know how that scene works: if you're popular and/or liked by the furry royalty, you can write (or remove) whatever you want there and it will be accepted as WikiTruth. But otherwise, expect to have your page kept around (despite your wishes to the contrary) and filled with sections like "Criticism" and "Controversy".

(cue GreenReaper in 3...2...)

[Eevee]

You might as well start doing DROP FA_SUPER_USER CASCADE (bye bye FA)

Ha!  That would imply foreign keys.  Why do you think the "thing has been deleted" inbox message exists?

Anyone who doesn't know that DDL commands force commits (they do in Oracle at least in most circumstances)

DDL is all transactional in postgres, I think save for DROP TABLE.  (And probably creating/dropping entire databases; I suspect transactions are per-database anyway.)

You figure you would always know what you're running when doing anything like this; I'm not a DBA but it's like, holy shit, use some common sense and make sure you know what you're doing before you pull the switch. "Haha, we just lost 10 days of journals" is not acceptable and I'm [not] surprised yak is still there. :I

I guess I sympathize with outright fatfingering, especially given what a horrendous nightmare mysqldumps are to work with.  This was a pretty human mistake, oh well.  github lost some activity recently, too.

What I would love to know is why he needed to load a second copy of the db on the production server in the first place, when they have a dozen machines sitting around doing fuck-all.

[Jim Demintia]

I guess I sympathize with outright fatfingering, especially given what a horrendous nightmare mysqldumps are to work with.  This was a pretty human mistake, oh well.  github lost some activity recently, too.

Sure it was, but at some point you have to say that losing user data is simply not acceptable. I don't know what happened with github (not particularly a fan of that site, but whatever), but I guess if your going to get the benefits of being perceived as professional (high traffic, being able to charge people money, etc.) you should get the responsibilities too.

"Mistakes happen" is sort of a worthless statement, imo. Especially when you're just shrugging off an incident. Sure, mistakes happen, but the professional thing to do is make sure that you aren't allowing them to (easily) happen, and if they do you figure out what happened so that whatever happened can never happen again. And really, something that can't be fixed (losing user data, as opposed to, say, downtime), is a far bigger deal than other kinds of mistakes.

I'm not saying that FA will do this, or that anyone realistically expects them to, but I guess I can't understand why you're defending the amateur hour clown show over there.

[Lazarian]

Along with the issue with multiple POST requests mentioned earlier, is there maybe an issue (judging by how many times you see "I meant to reply to the above poster") with posting replies being processed first in- first out, instead of properly linking to the parent post?

Please note: I don't know anything about site coding.