Performance issues prompt "unplanned maintenance". Maybe.

[Jim Demintia]

Recently, d.facdn.net (ie. images and other artfiles, as well as thumbnails), began being served over HTTPS, for no obvious good reason. Despite the intermediate SSL certs being broken (which leads to certificate errors on certain types of popular smartphones), this adds additional load to an already overloaded infrastructure. d.facdn.net resolves to a single box in the netblock owned by Ferrox Art, LLC. This is not new, but it also means that the additional load of HTTPS is being borne by that same box, and not, say, by some kind of cloud-based caching service.

At least one Twitter user this morning said they'd been prompted to enter a reCAPTCHA challenge with the message (ostensibly from FA) that their computer had a "virus".

And now, the administrator notice, in addition to informing users about AUP changes, is also stating that the administration is aware of performance issues and that their may be "unplanned" maintenance this week to fix it. Which is progress, I guess, that they are more or less willing to admit to winging it when it comes to this stuff. Translation of "unplanned maintenance", to me, is something like "Whenever Dax feels like it/Whenever Yak is awake".

Maybe you could solve your performance issues by not using SSL when it adds no value whatsoever, and you don't have the fancy hardware and software necessary to lighten the crypto overhead? What exactly is being protected by the SSL session? There are no cookies sent to the artfile server, there is no login whatsoever necessary if you know the direct link to a mature image. All of this is as it was before. Except for the SSL.

It might make (slightly more) sense to make the main domain end-to-end HTTPS, to protect against cookie hijacking. But SSL-izing facdn.net is the one thing that makes no sense to me, and it seems to me to be the one thing that could be (maybe the only) cause of their current performance issues.

[Pi]

SSL-izing facdn.net is the one thing that makes no sense to me, and it seems to me to be the one thing that could be (maybe the only) cause of their current performance issues.

I'm pretty sure a lot of their problem also has to do with their database server being so overloaded it shits the bed. SSL certainly can't help, but they may actually be using the (again as far as i remember) standalone load balancer to do SSL offload.

It is still pretty stupid and pointless, of course.

[ColonThree]

At least one Twitter user this morning said they'd been prompted to enter a reCAPTCHA challenge with the message (ostensibly from FA) that their computer had a "virus".

That's been happening for a while, something to do with Cloudflare.

I'm pretty sure a lot of their problem also has to do with their database server being so overloaded it shits the bed.

When FA is running slow, the "Page generated" thing at the bottom shows that ~99% of the 1-2 second generation time (instead of ~0.2s) is from SQL. Presumably it's still being shit after this was fixed. Yak is blaming scrapers again.

[ProvincialTwit]

I'm waiting for the day he starts blaming specific browsers for the slowdown, claiming IE 'acts like a scraper' or some twisted logic like that.

Though really, maximum humor points would require he demand users stop browsing the site with Firefox, since that's likely around 90% of visitors.

[Conan]

Recently, d.facdn.net (ie. images and other artfiles, as well as thumbnails), began being served over HTTPS, for no obvious good reason. Despite the intermediate SSL certs being broken (which leads to certificate errors on certain types of popular smartphones), this adds additional load to an already overloaded infrastructure. d.facdn.net resolves to a single box in the netblock owned by Ferrox Art, LLC. This is not new, but it also means that the additional load of HTTPS is being borne by that same box, and not, say, by some kind of cloud-based caching service.

At least one Twitter user this morning said they'd been prompted to enter a reCAPTCHA challenge with the message (ostensibly from FA) that their computer had a "virus".

And now, the administrator notice, in addition to informing users about AUP changes, is also stating that the administration is aware of performance issues and that their may be "unplanned" maintenance this week to fix it. Which is progress, I guess, that they are more or less willing to admit to winging it when it comes to this stuff. Translation of "unplanned maintenance", to me, is something like "Whenever Dax feels like it/Whenever Yak is awake".

Maybe you could solve your performance issues by not using SSL when it adds no value whatsoever, and you don't have the fancy hardware and software necessary to lighten the crypto overhead? What exactly is being protected by the SSL session? There are no cookies sent to the artfile server, there is no login whatsoever necessary if you know the direct link to a mature image. All of this is as it was before. Except for the SSL.

It might make (slightly more) sense to make the main domain end-to-end HTTPS, to protect against cookie hijacking. But SSL-izing facdn.net is the one thing that makes no sense to me, and it seems to me to be the one thing that could be (maybe the only) cause of their current performance issues.

Dragoneer recently mentioned buying a SSL cert for facdn.net with the "income" being generated by Viglink.

They, of course, needed it because of the ridiculous "CDN" setup and the strange obsession with "Sitewide SSL" that they've had an on-again, off-again relationship with for the past year. Obviously this is a major issue and takes precedence over everything else they have on their plate.

The "scraper" thing causing slowdowns sounds made up. If they're slowing down anything, it's the database server, which again is something that they probably should be running on two machines instead of the one it's on now.

[camellia sinensis]

Doesn't Yak run scrapers for journals relating to Inkbunny and what have you

[GreenReaper]

I'm pretty sure they were just on-demand queries doing a fulltext search of the journal table.

[Jim Demintia]

They, of course, needed it because of the ridiculous "CDN" setup and the strange obsession with "Sitewide SSL" that they've had an on-again, off-again relationship with for the past year. Obviously this is a major issue and takes precedence over everything else they have on their plate.

It makes sense to make furaffinity.net end-to-end SSL, since there is a very real history of session hijacking, e.g. on open WiFi networks at conventions. End-to-end SSL makes sense on most sites because you are exchanging personal information, passwords, and the like with the site- so you get some measure of server verification (protecting against rogue DNS servers, for example, on an untrusted WiFi network) and you get confidentiality.

I will say it again- SSL on facdn.net, which is essentially an anonymous-access file server running over HTTP(S), makes zero sense. This is yet more evidence that these people read things online (end-to-end SSL was a big thing, maybe a year ago) and understand absolutely nothing about it.

I am left to wonder if they cannot do mainsite SSL due to the Cloudflare setup (edgecasted HTTPS is hard to do. Akamai, the 800 pound gorilla of the technology, only recently figured out how to do it on a massive scale. I'm sure if Cloudflare will offer it, it won't be free).

Also: if you (i.e. Yak and crew) really want to start chasing scrapers, good luck with that. You will not win.

I'm pretty sure they were just on-demand queries doing a fulltext search of the journal table.

Yes and no. I remember what happened was that someone discovered PHP pages that were essentially that- so if you want to say that they were "on demand" in that Yak set up pages where Piche could click on the link and get a list of journals that had certain keywords in them, then sure.

[Pi]

I'm pretty sure they were just on-demand queries doing a fulltext search of the journal table.

And that's really great for database performance! oh wait the opposite of that

I'm sure that with a "mysqldump --no-data" any number of people here could give them some schema/index advice. I was talking to a friend of mine who does real work doing real DBA consulting for real people and he told me about a common dumb thing that people do with their MySQL indices, for example.

[greaseyote]

Speaking of databases, does anyone remember how huge the notification database actually is?

I'm not surprised they aren't trying to "pair it down," not that it's the ideal solution but it's the kind of stopgap measures FA is known for. Consider that a lot of people just don't clear their submission notifications (I probably get to it twice a year) or journal notifications and that has to make the message database much larger than it need to be. And, think of all the people who suddenly go "fuck this shit I'm quitting furry," do their submission/journal notifications grow in perpetuity? I know people that have changed account names and still watch me under both names, so I assume they still receive constant notifications that are probably never cleared.

And what's with the fact that only so many journals are displayed. You can't see more than one page of new journals from people you watch until you clear some of them.

[Jim Demintia]

Maybe I'm just bullshitting here, but it seems to me that even having a notification table is a violation of normalization principles. That is, you are storing information about submissions twice. There are probably practical performance concerns with generating the user's inbox from queries alone, but I'm sure there's caching and other mitigation that could be done.

Even if not, why not have it so that if a user never accesses their account and notifications just pile up, those notifications aren't "generated" until the user accesses their inbox, and since they'll never do that, there's no storage problem?

[ProvincialTwit]

Haha.  Look at you.  Look at yourselves.  Suggesting practical solutions to simple problems.  To FA.

[Conan]

Just a month short of a year ago, the notification table was sitting at just under 4 million rows.

> Query OK, 392992822 rows affected (4 hours 5 min 46.16 sec)

You guys should really be cleaning out your new submissions messagecenter.

I didn't realize Cloudflare couldn't do HTTPS. I'm guessing FA hasn't either.

[Jim Demintia]

I didn't realize Cloudflare couldn't do HTTPS. I'm guessing FA hasn't either.

A lot of the big sites use the services of a company called Akamai to cache and distribute their content to the tens of millions of users they get every month. Akamai sort of invented this, called a CDN these days. It's also called edgecasting- you have data centers near major population centers and you can shove out, e.g. the Huffington Post front page to an Internet user in Chicago just that much faster, even though HuffPo is served from, say, LA.

There are a lot of tricks done with the HTTP protocol to enable this. Namely, these CDN servers, which provide frontends to a lot of major sites, have to know what the hell they're serving. This comes from the HTTP/1.1 'Host:' header sent by the browser. When everyone decided they wanted to have end-to-end SSL (Twitter, Github are two biggies that do this now), things got dicey. Not only was there the additional load on CDN networks, but SSL will scream bloody murder in most browsers if the hostname of the server isn't what's in the certificate. This is of course by design. It has only been, I'm going to guess, within the last year that SSL-enabled edgecast services have become available. I know Akamai has what they call Akamai HD, which I believe to be this SSL-enabled CDN. I don't know if anyone else has it yet, but being so new I really doubt Cloudflare offers it to their free subscribers, if they have it at all. Cloudflare has it, for paid users only. So there you go.

Of course, the 'tl;dr' in FA's case is to just get rid of the Cloudflare bullshit and set up end-to-end SSL on furaffinity.net, but my best guess as to the reason they haven't done SSL on furaffinity.net is because of Cloudflare.

I suspect they do realize this. Piche probably wanted this, but got stopped cold by the fact that Cloudflare couldn't serve up the mainsite if it was SSL-protected. So being the idiot he is, he went and bought a certificate for facdn.net and put SSL on that instead.

[Pi]

Recent versions of TLS as implemented in recent browsers can send the hostname in the TLS handshake. It's called 'server name indication'; since it's new, nobody quite supports it properly yet.

Even given SNI, for Cloudflare to do HTTPS they would have to have your private key. There is no fucking way I would give my SSL key out like that.

----* the more you know, or some shit.

[rodox_video]

Haha.  Look at you.  Look at yourselves.  Suggesting practical solutions to simple problems.  To FA.

[loki]

> Query OK, 392992822 rows affected (4 hours 5 min 46.16 sec)

You guys should really be cleaning out your new submissions messagecenter.

Honestly, 4 million rows in one table isn't that much. We've got multiple tables with double/triple the amount of data and we still can get a query to return quickly. The real question should be why he needs to update all 4 million rows to do something? I guess there's no point in asking because I doubt they will ever have decency to ask for help instead of just making excuses.

[GreenReaper]

Uh, that's four hundred million rows . . .

[ProvincialTwit]

Regardless, if they weren't using MySQL - The Pretend Database For Babbies(tm), and also had any idea how to effectively use an rdbms, it wouldn't be an issue.

[Conan]

> Query OK, 392992822 rows affected (4 hours 5 min 46.16 sec)

You guys should really be cleaning out your new submissions messagecenter.

Honestly, 4 million rows in one table isn't that much. We've got multiple tables with double/triple the amount of data and we still can get a query to return quickly. The real question should be why he needs to update all 4 million rows to do something? I guess there's no point in asking because I doubt they will ever have decency to ask for help instead of just making excuses.

They "Added the field to the messagecenter tables indicating from where/whom the message came from, to make the 'deleted' placeholders less meaningless."

That was when they still thought the "Submission has been deleted by" message was impossible to remove.

Not surprisingly, all that work was for nothing as they removed the deleted placeholder messages in November.

Uh, that's four hundred million rows . . .

oops