FA is Full Again

[greaseyote]

OK, what gives?

http://forums.furaffinity.net/threads/108936-10-17-2011-File-read-only-mode.

I thought this happened already, a few months ago, and I thought they went out and bought more hard discs than they could ever use. What, did they just stick them in Dragoneer's closet this whole time?

[ColonThree]

Clearly they just set up a RAID 1 with all 20 drives, just in case.

[Conan]

No. This? This is either

• They're finally moving the data over to the server they installed in June or whenever. (This makes perfect since with the recent influx of complaints that uploads were failing, just like they did a few months ago when the new server was first installed and the old one was full.)
• They did what they said they were going to do and upgraded Trogdor to serve as a second file server.

Either way, it amazes me that they didn't think that turning off the uploading art portion of an art site was something that required a mention to the userbase before it happened.

[Freehaven]

Either way, it amazes me that they didn't think that turning off the uploading art portion of an art site was something that required a mention to the userbase before it happened.

This is FurAffinity we're talking about here. Nothing about the site surprises me any more.

[Jim Demintia]

IIRC the way Alkora originally designed it it stuffed literally every fucking file into one directory, which is such a brain-dead design that they literally ran up against the limits of the file system- you can only have so many files in a given directory. And this is presumably some UNIX file system here, i.e. it's not the old MS-DOS system that's often used these days on flash media.

If you've ever looked at image-site URLs, and where they are some kind of hex-code: you will often see, say, for a33423bcdef.jpg- a/a3/a33423... for precisely this reason.

Now I think they are storing by artist name, which is no better: cf. why they can't rename or delete accounts.

[Pi]

IIRC the way Alkora originally designed it it stuffed literally every fucking file into one directory, which is such a brain-dead design

Yeah. They have /art, which contains a bunch of /art/{username} links, where the link points to /art{somenumber}/{username}.

You'd think they would realize that this is a Problem when they get up to values of "3" for "somenumber". Instead, they've extended this brain-damage to /art16.

Yak would call this a "compromise" between fixing the problem and not having the problem. I call this "the fucking problem".

[Jim Demintia]

Yak is estimating "a day, maybe more" to transfer 1.7TB. For fun, I can tell you that my MacBook Pro can transfer files over my GbE network to a suitably fast, albeit consumer-grade, hard disk, in a consumer-grade tower built in 2006, at a sustained speed approaching 60 megabytes per second. So, doing something resembling calculations on this, I get:

1.7 TB is 1740.8 GB. That's 72 GB per hour over a day (24 hours). 72 GB is 73,728 MB, which is slightly over a gig per minute (1228 MB). Divide it by sixty again and you get...20MB per second.

Even when you figure in the overhead of thousands of individual NFS transactions- which I'm sure a clever sysadmin could mitigate or at least minimize, say, using tar or rsync or something- that's pretty slow.

As an exercise for the reader, speculate as to why their five-figure equipment can't outpace a five-year-old DIY rig.

[Pi]

As an exercise for the reader, speculate as to why their five-figure equipment can't outpace a five-year-old DIY rig.

Easy: Your 5-year-old DIY rig is not also serving the production site. Remember that their template system slams the IO subsystem pretty hard (which is why they're running it off of a RAM disk (NB: freebsd allocates its RAM disks out of the swap backing store)).

[loki]

Compile pigz
Compress that 1.7TB
Transfer it

Of course this is assuming that 1.7TB figure is the noncompressed figure; not to mention you can only image formats so much.

[Conan]

It was brought up on FAF that they said they did this four months ago. Obviously it didn't happen, and Yak offered this:

Server was obtained. Had hardware issues. Looked for causes and replacement parts.

So they turned it on and it instantly broke down. Nice.

[Jim Demintia]

As an exercise for the reader, speculate as to why their five-figure equipment can't outpace a five-year-old DIY rig.

Easy: Your 5-year-old DIY rig is not also serving the production site. Remember that their template system slams the IO subsystem pretty hard (which is why they're running it off of a RAM disk (NB: freebsd allocates its RAM disks out of the swap backing store)).

I know that Linux caches reads in "free" RAM- writes are a more complex thing and are not cached in the same way, but I'd be willing to bet you could come up with a pretty fantastic way to take advantage of the gigantic memory sizes of modern computers and the relative high speed of RAM access to make worrying about I/O pretty much secondary. I am sure such a thing already exists, so why the hell aren't they doing that? I have never heard of using something like a RAM disk to run a web site, however I/O bound it might be.

Although, being I/O bound is pretty nasty- I/O isn't parallelized and involves a lot of interrupts, so it pretty easily chokes your whole system. That really does go a long way towards explaining why they need such beefy hardware for that site.

Better yet- some kind of clustering would allow a limited level of parallelization. At least, the whole system wouldn't necessarily be ground down by excessive I/O. They've got all this random hardware lying around, make use of it.

[Pi]

Basically I can think of 3 or 4 different ways to make this migration less of a clusterfuck and 3 of them even allow you to continue uploading to the site while it's in progress.

Compare and contrast with yak, who will probably accidentally blow everything up. It's amazing how often they manage to injure themselves on their software.

[ColeTrain]

So it appears that even after the whole data transfer thing, the site is still experiencing issues as many users can't post anything. As some said, "not even a 1kb text file." Changing and editing submissions doesn't work either.
And staff doesn't know what's the problem and how to fix it.

[Conan]

So it appears that even after the whole data transfer thing, the site is still experiencing issues as many users can't post anything. As some said, "not even a 1kb text file." Changing and editing submissions doesn't work either.
And staff doesn't know what's the problem and how to fix it.

That, and "thumbnail cache regeneration" has been going on for days, and it usually takes hours. It doesn't appear the thumbnails are even caching, the main page takes a good 20-30 seconds until everything loads.

Let's start taking bets as to when the new file server ignites. I say by Tuesday.

[Freehaven]

Jesus tapdancing Christ, 4chan is run more competently than FA.

[Jim Demintia]

So it appears that even after the whole data transfer thing, the site is still experiencing issues as many users can't post anything. As some said, "not even a 1kb text file." Changing and editing submissions doesn't work either.
And staff doesn't know what's the problem and how to fix it.

There's this whole thing in computer science about "formal verification" wherein you mathematically prove your program. If you do that you know everything the program can output for any given input, and what the algorithmic steps are in between. It's really not practical with most large-scale software (although I understand some contracts with the U.S. government will require it). However, we take things part of the way wherein we try to understand what the code does, and verify it, say, with unit tests. That includes what you intended for it to do, and what it might do in addition to that (side effects).

You end up chugging Aspirin smoothies trying to understand code written by another competent software engineer following good design principles. It's just hard, it doesn't seem to be something humans can do naturally. Understanding what FA is doing...well, let's just say it proves there's no God because not even God could understand that mess.

[Conan]

People are still reporting issues uploading, but Yak apparently fixed the loading issues:

<yak[work]> i was doing some sysctl tuning. had set maxsockbuf lower then what I had configured in nginx, found out about it only after I restarted the latter. appears freebsd 8.2 has performance regression for large socket buffers or something
<yak[work]> that new file server should be a tad more responsive now

Jesus tapdancing Christ, 4chan is run more competently than FA.

I can remember a time a year or so ago when Encyclopedia Dramatica went offline due to some problem with MySQL. They made a public plea for help and asked people to contact them if they could be of assistance. The site was up within a few hours.

FA would just ignore you, tell you "You don't know how FA works!" and tell you to shut up by giving you a nice fat ban.

[Freehaven]

"You don't know how FA works!"

At this point, I'm surprised that Dragoneer and Yak know how FA works. From everything I've ever read about the backend side of things, FA basically needs to be rewritten from the ground up and put on hardware built for the purpose of hosting an artdump website by someone who knows what the fuck they're actually doing.

[Conan]

"You don't know how FA works!"

At this point, I'm surprised that Dragoneer and Yak know how FA works. From everything I've ever read about the backend side of things, FA basically needs to be rewritten from the ground up and put on hardware built for the purpose of hosting an artdump website by someone who knows what the fuck they're actually doing.

No, Dragoneer does not know how FA works. That's what Yak is there for. He's the only person that "knows" how FA works, and often times he's making mistakes while claiming that. "I know how FA works and you don't! *configures everything wrong* Okay, now it works. Wait... *more fixes* Okay! No.... *another good whack* There. It's duct-taped in but at least it's working!"

 

[Jim Demintia]

No, Dragoneer does not know how FA works. That's what Yak is there for.

Yak knows nothing. Yak is able to mash buttons to the point where things look like they're working. And the theory we have (I have, anyway) is that Yak tolerates the bullshit he tolerates is because he's got unfettered access to a big pipe and powerful hardware owned by very gullible people.