What was originally considered a hardware failure turned out to be:
Dec 19 03:25:40 novastorm kernel: swap_pager_getswapspace(16): failed
Dec 19 03:25:52 novastorm kernel: pid 90279 (mysqld), uid 88, was killed: out of swap space
Reconfigured MySQL to use less memory and restarted the server. Also restarted the backup process that got interrupted halfway with this crash.
FA will be on the slow side the first 15-20 minutes while the table data cache is being populated.
HARDWARE FAILURE!!!!!! Quick! Round up the donatio-.... oh it just ran out of memory.
Instead of thinking they need to have
three file servers (like plans announced earlier this summer called for), they should really be repurposing Trogdor (Or maybe the backup server-that-is-responsible-for-storing-a-single-tar-file-and-thats-it) into a second database server. They're obsessive about anything to do with files, but when it comes to the database (which has been the root cause of like 80% of site issues) they continue to let it limp along on a single box.
Also, I'm guessing now that the past week of slowness probably has to do with the backup, the times seem to match up at least...
