Author Topic: Submission Notifications Broken  (Read 1685 times)

Conan

  • Posts like Kage drinks
  • ****
  • Posts: 716
  • E-points: +39/-9
  • ¯\(°_o)/¯
    • View Profile
Submission Notifications Broken
« on: November 29, 2014, 07:24:38 am »
I noticed last night that I had not gotten a new submission notification in over a day. Sure enough, others started reporting similar problems.

Turns out that someone went and did something that completely fucked up a core aspect of the site again

We found the issue. We have a system that splits notifications into 1k notification batches if the uploader has more than 1k watchers, we stagger the sending of these batches a little to reduce load on the servers. However, they shouldn't be taking this long. Apparently some of these batches fail to send (get timed out).

The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

Basically the database table for notifications got locked by some cleanup that was being performed, and aborting that operation means the changes made so far have to be undone before it releases the table altogether. The issue does seem to affect users with larger numbers of watchers more, and I'm sure tech staff will be looking further into it as soon as they're able.

My best guess is that avoiding posting/uploading until it clears up will probably help the database catch up, but that's speculation on my part, not word from the people who actually have anything to do with the backend. (Similarly, best guess is that the white pages are related to database-related timeouts, but I'd have to confirm that with tech staff to be sure.)


Naturally, this rollback is being called "maintenance" and has had expected time windows of "Half a week", "6-8 hours" now "24-36 hours" over the course of a few hours, the final one appearing in the official notice:

Quote
FA is performing background database maintenance that may impact the process of sending out new content notifications and the removal of existing submissions and journals. The process is not expected to take longer than 24-36 hours.

Obligatory:

JTfurry

  • *
  • Posts: 34
  • E-points: +1/-0
  • Uninitiated Rube
    • View Profile
Re: Submission Notifications Broken
« Reply #1 on: November 29, 2014, 12:02:07 pm »
It took half a day last week for some of my notifications to pop up.

Saw a load of submissions on the home page before they were in my messages :P


nrr

  • Sean Piche Fan Club
  • Cabalistic Fuckhead
  • **
  • Posts: 104
  • E-points: +7/-3
  • OMG SO CUTE ^__^
    • View Profile
    • lynxies :3
Re: Submission Notifications Broken
« Reply #2 on: November 29, 2014, 08:17:48 pm »
The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

Protip: Shard your database such that your tables are close to a maximum of 1 million rows. That's roughly one quarter the keyspace you get with a 32-bit unsigned integer, and it gives you the ability to run transactions over an entire shard within a day's time. The best part is that a shard need not live on its own database cluster, but it can in the event that any one particular shard is eating up a shitton of I/O.

Also, monitor and alert on your tables' keyspaces. Once they reach (better: once you've crossed the future projection of) the 1.5 million rows mark, split that shard up into two separate shards.

This is not fucking difficult, and while there are more sophisticated schemes that use 64-bit keys and wind up reserving, like, 42 bits of it for primary keys on tables and the other 22 bits to determine which shard it's on, the naïve approach works just as well.

FA is such fucking clown college sometimes.
im glad the "I saw a furry IRL" thread is so good at bringing goons together

YOUR PARTICIPLES AREN'T THE ONLY THINGS DANGLING

ProvincialTwit

  • Abuse Dept.
  • Postcount killed Trogdor
  • *****
  • Posts: 794
  • E-points: +77/-33
    • View Profile
Re: Submission Notifications Broken
« Reply #3 on: November 30, 2014, 09:47:58 am »

FA is such fucking clown college sometimes.

Someone take a screencap from the simpsons "clown college" episode and paste in piche's head tia

Dima

  • *
  • Posts: 45
  • E-points: +7/-2
  • Inadequate
    • View Profile
Re: Submission Notifications Broken
« Reply #4 on: November 30, 2014, 10:03:44 am »
Someone take a screencap from the simpsons "clown college" episode and paste in piche's head tia


an hoopoe

  • Cabalistic Fuckhead
  • ***
  • Posts: 447
  • E-points: +36/-4
    • View Profile
Re: Submission Notifications Broken
« Reply #5 on: November 30, 2014, 06:59:05 pm »
Dragoneer's shouts ATM:




It's been like that for the past few days. Nothing but questions and complaints, no support.

Conan

  • Posts like Kage drinks
  • ****
  • Posts: 716
  • E-points: +39/-9
  • ¯\(°_o)/¯
    • View Profile
Re: Submission Notifications Broken
« Reply #6 on: December 01, 2014, 02:29:15 am »
Well, ya see, Sciggles is becoming more and more mentally unstable, and "rescued" a stray cat while she was visiting family. This cat is what Dragoneer had to drive 20 hours to pick up.

So I am in TN visiting my family for an early Thanksgiving. 3 days ago my baby sister fell and busted her chin open and had to go to the ER. When we went there was a baby kitty near the entrance crying and I went and sat next to where he was while they took my sister in. He kept crying and doing circles around me which to me seemed like he REALLY wanted help but was so scared. I tried so hard to get closer and I even got to pick it up but when I went to wrap it in my jacket it got scared and bit me then ran off. I felt so bad....

The next day we went back to see if he was still there and brought some cat food for him. We called out and he started meowing again till we found him and I made a trail of food closer to me. It was a long process but finally he came right up to me and I reached out to pet him and he jumped at ran away again. I felt SO bad...

Today we went back and it was pouring rain but we were able to call out and find him again. This time he ran into a storm drain and we were worried about him and all the rain so we called Animal Control to see if they would help. Once they came out we regretted making that call. We told him what happened and he asked if he bit anyone and we told him I was and he immediately started talking about quarantine and what not and we asked what would happen after that and he said he would kill it. Even though WE wanted to keep it after all this he said they would put it down (even if it was healthy) My anxiety went through the roof. We were determined to catch him before animal control. We left and got our own traps out and food and started setting it up. Poor thing was so hungry it went into the trap and we got it!!!

I instantly broke down crying...after three days of trying and being scared of it getting hit by a car or people being mean or animal control catching it before us it was just overwhelming that finally we got it.

Tomorrow we are taking it to the vet to find out gender and get health check. For now its in a nice warm room relaxing with plenty of food and water.

Due to all this I ended up missing my flight back home so I wont be home for another week but it was so worth it to save this baby's life.

Written by Susan Piche, age 5.

ColonThree

  • **
  • Posts: 150
  • E-points: +17/-3
  • Not a cat
    • View Profile
Re: Submission Notifications Broken
« Reply #7 on: December 01, 2014, 09:46:37 am »
Well, ya see, Sciggles is becoming more and more mentally unstable, and "rescued" a stray cat while she was visiting family. This cat is what Dragoneer had to drive 20 hours to pick up.

Is 20 hours one way or both? Either way it's ridiculous. There's a time and a place for rescuing stray cats, but being heavily in debt and 10/20 hours from home probably isn't one of them. Isn't she a bit young to be turning into a crazy cat lady?
~Witty quote~

Conan

  • Posts like Kage drinks
  • ****
  • Posts: 716
  • E-points: +39/-9
  • ¯\(°_o)/¯
    • View Profile
Re: Submission Notifications Broken
« Reply #8 on: December 02, 2014, 01:17:10 am »
From today's TL;DR announcement:

Quote
To counteract this issue, the tech team has made changes to prevent notifications from being sent to accounts that are permanently banned or deceased and is in the process of deleting millions of rows of notifications on those accounts. This process is estimated at about 1 week and may impact notification delivery during batches that time out. The work began last Friday and is estimated to be completed on Thursday, December 4, 2014. Many of you may have already noticed an improvement in the system, and when the process is complete, performance should increase greatly.

FWIW, I recall the problem of banned/deceased accounts receiving these notices being brought up on FAF a few years ago. Of course, nobody from FA cared, nobody listened, and I'm sure Yak smugly disregarded it as something "nobody knows what they're talking about".

Now it cripples the site for a week. Great fucking job, as usual.

Gourd

  • Posts: 18
  • E-points: +1/-0
  • Uninitiated Rube
    • View Profile
Re: Submission Notifications Broken
« Reply #9 on: December 04, 2014, 07:58:01 am »
Not the most noteworthy, but rather funny. Yak is on full damage control. I love that he calls the ones complaining "ungrateful bastards". Scroll a little bit further down and he even dared to say that if the Titanic were built like FA, it would not have sunk.
https://www.furaffinity.net/journal/6311970/#cid:43356820

ProvincialTwit

  • Abuse Dept.
  • Postcount killed Trogdor
  • *****
  • Posts: 794
  • E-points: +77/-33
    • View Profile
Re: Submission Notifications Broken
« Reply #10 on: December 04, 2014, 08:36:44 am »
Not the most noteworthy, but rather funny. Yak is on full damage control. I love that he calls the ones complaining "ungrateful bastards". Scroll a little bit further down and he even dared to say that if the Titanic were built like FA, it would not have sunk.
https://www.furaffinity.net/journal/6311970/#cid:43356820

Well, he's got a point; if the Titanic had been built like FA it never would've made it out of drydock.

Conan

  • Posts like Kage drinks
  • ****
  • Posts: 716
  • E-points: +39/-9
  • ¯\(°_o)/¯
    • View Profile
Re: Submission Notifications Broken
« Reply #11 on: December 05, 2014, 01:47:26 am »
Quote from: Admin message
Background database maintenance is complete. New content notifications should be going out normally.
A sizable amount of work has queued up internally and it will take up to a week for the database to catch up, possibly resulting in reduced site performance for the duration.

circus_music.wav


Gourd

  • Posts: 18
  • E-points: +1/-0
  • Uninitiated Rube
    • View Profile
Re: Submission Notifications Broken
« Reply #12 on: December 05, 2014, 02:01:52 pm »
More nonsense spouted by Yak. In an exchange between user oneandonly.
https://www.furaffinity.net/journal/6311970/#cid:43380385

Quote
This ego thing you have is really killing me. If FA died, you act like all others sites would be destroyed by the sheer number of users. At any one time, even Weasyl has only 10 times less users than FA. The best part? They have 9 Admins/Mods That's one less than how many are on the FA staff page. Staff wise, they should have plenty of time to get the job done and find other mods/admins while they're at it.

Also, if they had an issue with money because they suddenly have to pay amazon more, guess how willing I (and most others) would be to donate to them? They've proved time and time again that they are capable of running a site. Weasyl also used CloudFlare last I checked. 

Yak comes back:

Quote
Not destroyed, but would experience technical and/or financial difficulties.

To correct you on the stats, Weasyl's ones are calculated for a time span of 60 minutes, while FA's are every 15. To put this in perspective, right now there 8,683 users on FA if stats are calculated for a 15 minute time span, and 24,133 if calculated using the 60 minute time span.
Weasyl shows 893. That's 3.7% the visitors of FA, or ~27 times less.
Even if that figure was just 10 times less as you say, websites are not typically designed to handle even 5x the amount of requests on a whim. At 3x you'd likely already be seeing signs of noticeable difficulties.

The usage of Cloudflare would greatly offset their bills as the bulk of the traffic would've been offloaded to CF, but there would still be the need to finance the increase of processing power and memory required for operation. Donations are a wonderful thing, but they are typically a once-in-a-while, one-time events while the necessity to pay Amazon would have been monthly and recurring. Eventually donation money would run out and a more sustainable business plan like subscriptions, payed for features, advertisements, or integrated auction house and similar features would be required.
I am not saying they don't have them, or would be unable to obtain them. I'm just estimating on what would have likely happened from a standpoint of someone who is familiar with the topic.

Don't mistake it for a biased or heaven forbid envious opinion of a "rival" website's staff.

oneandonly rebuttals:

Quote
Don't know if you remember, but the whole zaush thing a while back actually brought in a bunch of people to the site at once. http://i.imgur.com/jX9sGbQ.png
That's the chart. They went from ~600 users at once, to 4000, a little over 6 times the amount of people. They handled it fine, along with the other furry sites.
They also handled the time FA went down for about a month fine too.

While the numbers may not have spiked that high, I do recall many shortcomings of FurAffinity resulting in high traffic surges for Weasyl. I cannot recall a time when they stressed over the influx of new/numerous users online at once. Not saying they haven't so correct if I'm wrong, but in my experience, I've yet to see it. FA on the other hand crashes every week for seemingly no reason. And as of recently, that's not even hyperbole anymore.

(Also pardon my inexperience. I'm not savvy with this place yet so I apologize for any inefficient posting)

ColonThree

  • **
  • Posts: 150
  • E-points: +17/-3
  • Not a cat
    • View Profile
Re: Submission Notifications Broken
« Reply #13 on: December 06, 2014, 05:28:40 am »
Quote
...Even if that figure was just 10 times less as you say, websites are not typically designed to handle even 5x the amount of requests on a whim. At 3x you'd likely already be seeing signs of noticeable difficulties...there would still be the need to finance the increase of processing power and memory required for operation...

This assumes that Weasyl is already running full tilt to handle the current amount of users, and also that it's a badly-coded, poorly-designed, bottle-necked piece of shit. In other words, that it's just like FA.
~Witty quote~