Performance issues prompt "unplanned maintenance". Maybe.

[Fate]

Administrator notice:
Downtime Notice: There will be a short period of downtime sometime between 9AM and 11AM UTC. Expected to last only 30 minutes.

So which is it, fuckface? Thirty minutes or two hours? 0900 and 1100 UTC on what date? Or did you mean "There will be approximately thirty minutes of downtime starting at some point between 0900 and 1100 UTC Tuesday, February 14 2012"?

[RedFlowerJigglypuff]

Administrator notice:
Downtime Notice: There will be a short period of downtime sometime between 9AM and 11AM UTC. Expected to last only 30 minutes.

So which is it, fuckface? Thirty minutes or two hours? 0900 and 1100 UTC on what date? Or did you mean "There will be approximately thirty minutes of downtime starting at some point between 0900 and 1100 UTC Tuesday, February 14 2012"?

Well, they gotta fix THEIR* 26 hour fuck-up with what will most likely be MORE than two hours to "fix." That just goes expected without even really saying at this fuckin point.

Lovin all the random slow ass page loads and 500 errors. But really, that's nothing new, with or without the them deciding something on the site needs fixing, or rather, new hardware, since that's always the answer.

EDIT: and oh yeah, the added in that it'll be the 14th.
EDIT 2: Fixed there to their because I'm a dumb cunt.

[Pi]

You mean that yak's plan of "suck it and see" didn't work out????

[Fate]

Well, they gotta fix there

"Their". They're there to fix their fucking shit.

And you'd think with their hardware budget they'd have two server sets and decent replication. But noooo, scary! Making sense as a furry!

[Conan]

after a goddamn year of asking, yak has finally provided some rationale about scrapers:

<yak[away]> Pi: scrapers load gallery pages with large numbers, very fast and sometimes in parallel; with pagination in mind this generates sql queries with `LIMIT $big_number_here, 60`. half a decade old mysql query optimizer bug interprets that as "read $big_number_here rows from disk into the buffer pool, discard them, then read 60 more and return".
<yak[away]> this both thrashes the disks and wipes a large portion of the in-memory table data cache to make room for rows that are immediately discarded.

See, again, to me this doesn't scream "upgrade the database", it screams "fix my fucking query". But that'd require, y'know, competence.

It's worth noting this is still happening and the farther back you go in something (say, Dragoneer's favorites) the longer it takes to generate a page.

300 pages back got me "Page generated in 6.127 seconds [ 0.4% PHP, 99.6% SQL ] (14 queries)"
600 pages back got me a 502 error. Another try returned "Page generated in 57.742 seconds".

It seems that anything you do this to exhibits this. Page 50 of my own favorites returned "Page generated in 23.179 seconds". I used to regularly go back farther than that and it would load in seconds. I think this update might have made it worse.

[Pi]

So, I advised yak while the upgrade was coming up on its 19th hour, or so.

<Pi> yak[away]: do i even need to say anything here?
<yak[away]> implying you didn't already. lots of data to import, takes time.
<Pi> and it's obvious that you had no idea how long this was going to take
<Pi> meaning this wasn't tested                                                                                                     
<Pi> and so you have no idea if this is going to even work or not                                                                   
<_habnabit> why didn't you start on the 11th, if you were scheduled for the 11th-12th                                               
<Pi> again: do i even need to say anything about this, or do you understand why i am frowning here                                 
<yak[away]> it's kind of difficult to predict the time it takes to import a database. I know this will work though, since i've done this many times before with smaller databases.
<yak[away]> _habnabit: because i wasn't available on 11'th                                                                         
<Pi> my god
<Pi> you really are just a cowboy                                                                                                   
<_habnabit> then why did you schedule it for the 11th                                                                               
<Pi> you're *guessing* that it'll work based on your past experience
<Pi> one of these days this is going to fuck you in the ass                                                                         
<Pi> hopefully today is not one of those days, yuri                                                                                 
<Pi> good night                                                                                                                     
---
<Pi> yak[away]: guess what your upgrade didn't resolve
<Pi> performance issues                                                                                                             
<Pi> Page generated in 3.572 seconds [ 0.4% PHP, 99.6% SQL ] (12 queries)
<Pi> yak[away]: do you want someone with a fucking clue to take a look? because you just made it worse.                             
<Pi> and before you give me that crap about the buffer cache needing to be filled, why is it that NO OTHER SITE HAS THIS PROBLEM

[Jim Demintia]

So what was previously a mild annoyance is now a can't-sweep-it-under-the-rug disaster. The site may as well be down, it's not like it's usable in this state, or that it can reasonably be sustained operating like this.

And they can't go back, not easily. And not without repeating the same process over again, with the same nontrivial risk of failure.

The "good news" is that Yak didn't irrevocably trash the database. This time.

[Pi]

It seems to be working a bit better today, after my totally 100% scientific testing of "whacking reload".

But still, I can't fathom why it takes a whole day for the filesystem buffers (as yak claims) to reach a steady-state.

[ColonThree]

Yet again, some changes to FA manages to break searching. It does not appear to be finding anything submitted after the maintenance was done. Good job!

[Fate]

man guys this new database engine

maybe we need another server with gaming-grade parts in it because those are good right

[Conan]

It seems to be working a bit better today, after my totally 100% scientific testing of "whacking reload".

But still, I can't fathom why it takes a whole day for the filesystem buffers (as yak claims) to reach a steady-state.

Not really. I can still make it 502 or give insane times (30+ seconds) by going back too far in favorites. Even after the "optimization" they did this morning.

[Pi]

Not really. I can still make it 502 or give insane times (30+ seconds) by going back too far in favorites. Even after the "optimization" they did this morning.

Gee. I wonder what would happen if i did this:

Code: [Select]

for i in `seq 1 700` ; do
    for user in $LIST_OF_USERNAMES; do
        curl http://www.furaffinity.net/favorites/$user/$i/
    done
done

[Conan]

Not really. I can still make it 502 or give insane times (30+ seconds) by going back too far in favorites. Even after the "optimization" they did this morning.

Gee. I wonder what would happen if i did this:

Code: [Select]

for i in `seq 1 700` ; do
    for user in $LIST_OF_USERNAMES; do
        curl http://www.furaffinity.net/favorites/$user/$i/
    done
done

http://www.youtube.com/watch?v=J8pIiyrPZsQ&feature=player_detailpage#t=40s

[Jim Demintia]

You know, I kinda doubt Yak's lying about the scrapers, which leads one to wonder: who the hell is scraping FA and why? You can imagine something similar to what happened with Encyclopedia Dramatica being afoot here; it's not as if FA isn't liable to go belly up for good at any given point here. We've talked about the ways that could happen before, such as a financial or medical emergency in the world of Sean Piche. It's also possible that Yak could finally really fuck things up, or an attacker could gain escalated privileges and really fuck things up (ex. DROP TABLE, or start deleting shit off of their NFS server).

Maybe someone plans to launch a website with this data; who knows. They might very well bide their time until FA really goes tits-up for good. But, if you assume that there are scrapers operating on the site to a degree that it's causing serious performance issues, you naturally wonder who and why.

[magus]

I don't know, I'd think that a site is fairly unlikely. After all, that opens you up to charges of ~art theft~, which we all know makes you about 72% worse than Hitler. Hasn't stopped people in the past, though.

I think the most likely route is regular users wanting to make some kind of backup. Furry artists do love to randomly decide they suck and delete fucking everything on a yearly-or-so basis, so people scraping their own favorites would make sense. Also, the helpdesk 'web professional' segment of Furry probably has more than a few people who would cobble together something slightly more complex than Pi's FAbomb above in order to brag to the peer group that they've got a copy of ALL THE PORN. Or, as you said, if said people are concerned about FA randomly vaporizing at some point.

Actually, if it is regular users, you'd think there'd be example code floating around. Anyone seen any?

While we're on the subject of the FA eschaton, my money's on 'lack of time/fuck-given after Sciggles gets around to oopsing Piche'.

[Pi]

scrapers operating on the site to a degree that it's causing serious performance issues

Something I've tried to drill into yak's head is that the scrapers are not the problem - the scrapers are exposing a problem, and blocking scrapers is not going to fix the underlying issue. He's come out and told me that he blocks scrapers just because he "doesn't like them", and has no good technical reason for it.

[Ketsuban]

While we're on the subject of the FA eschaton, my money's on 'lack of time/fuck-given after Sciggles gets around to oopsing Piche'.

I kinda imagine the FA eschaton as a competent network administrator wanting to set up a furry art site not run by delusional paranoids and egotistical fuckwits and being too impatient to wait for FA's userbase to jump ship.

[ISleepNow]

While we're on the subject of the FA eschaton, my money's on 'lack of time/fuck-given after Sciggles gets around to oopsing Piche'.

I think It'll happen when Sciggs turns gets through turning FA into her version of CWCipedia

[ProvincialTwit]

I'd say it's high time for someone to immanentize that eschaton.


brb changing username to Hagbard Celine

[ColonThree]

Actually, if it is regular users, you'd think there'd be example code floating around. Anyone seen any?

This one was posted on lulz a short while back. It's for IB, but based on one for FA, with minimal changes.