In the interest of full disclosure:
Yak posted this giant thing on verix's journal. It's a big long rant that we have already found a handful of misrepresentations and lies in, and includes the howlingly hilarious line "you specifically have zero reasons to complain about not being heard [... you] blew your chance by spitting in my face". Yeah, because
we did that by banning you from IRC, whereas
you're completely blameless! (asswipe).
Anyway, without further ado, the contents:
Preface.
Usually I would've replied to this, and any other of your concerns - as well as shed some light on how things are done and keep being done the way they are right now - and why - in your IRC channel.
As I always tried to do. It was the reason why I hung out there to begin with. I gave you a chance to speak to somebody - who could have made all these plans and suggestions of yours happen - directly.
But unfortunately it just happened that none of you listened o what I've said when when I was doing it. And if you listened you either chose to misinterpret what I've said, make fun of it in the tired old "lol FA sucks" way or just plain not believe it.
And you ended up banning me from it for giving you answers to your questions on the plan that was already formulated, confirmed and put in motion by the administration team.
That's fine. But from now on you specifically have zero reasons to complain about not being heard or payed attention to. You blew your chance by spitting in my face.
Hi.
A great deal of your post, and thus I assume your conclusions come from assumptions that are incorrect.
I am going to explain why, and it would be up to you to decide whether you'd want to believe them or not, and whether to adopt your point of view accordingly.
attempting to talk to this administration for years on end with no one actually listening to our advice and instead opting to go with the hardware upgrade
There was never a point where we did either.
In the beginning all the hardware upgrades that took place were completely necessary, because as the site grew so did the demand for processing power and reliable storage space that the available hardware a that point did not provide.
Increased load also exposed the instability and caused the premature fatigue of the hardware that was in operation at that time - as it wasn't designed for that sort of stress. I'm talking about nvidia chip-set based cheapo server motherboards, consumer grade hard drives and reverse engineered faulty nic drivers that cause so much grief in their time.
FA really needed stable, reliable hardware and we seeked it. That hardware was obtained after the "16k donation drive" event. At that point our hardware needs reached the point of saturation.
"on top of having software flaws, the website unfortunately also has network and infrastructure flaws"
As already explained before, the network infrastructure currently in operation evolved from a two server network.
It was only recently that we've obtained Cisco routers and switches, as well as additional servers currently in preparation of running additional services - that warrant that infrastructure being changed.
The changes are already scheduled and will take place in the next couple of months.
Novice administrators very frequently make the mistake of applying the wrong fix for what they perceive is the problem (e.g., running code in RAM to save disk I/O, as this website unfortunately does)
Cute. You don't really know anything about this other then what I've told you, yet you see fit to make a wild assumption based on that little and have the nerve to call it my flaw.
FA's code was running off an md device long before I replaced FA's eval() based template code with my own. Infact, having the entire code in a ramdisk was the reason why the new template code was written that way.
It makes use of that fact to trade off additional (nearly free) I/O for less CPU load that would have otherwise taken place to do variable substitution in templates.
On top of that having templates being php files makes them compile-able by opcode cachers like xcache or eaccelerator, which further reduces time, memory and cpu cycle requirements for template parsing. Just like Smarty... except without Smarty.
Getting back at why FA's code is running off a ramdisk.
The plan to spend the money from the "16k donation" initiative was to buy two servers: application and data. We already had a (now retired with a hardware malfunction) database server at that point.
Due to the fuckup on the hardware supplier's part the application server was delayed and in order to get the site back up and running as fast as possible we've decided to run the code on the data server. It was more then capable.
Still, since the application has a ton of small files that go into the layout - and they are very frequently requested - it has been decided by myself that putting them and the code on a 64MB ramdisk would be a worthwhile optimization.
The real I/O otherwise spent on seeking, reading and serving all those small files would then go to serving the actual user data instead. Just as it was planned.
Observing gstat, vmstat and iostat before and after the change confirmed that this action had a measurable positive effect.
This results in a building necessity toward upgrading the hardware due to a series of incorrectly applied fixes
After the 16k donation drive FA never really needed hardware upgrades to run it's primary service. We already had the quota of hardware to run the site for at least two years, assuming the hard drive space would last that long.
No money was seeked, requested or otherwise put aside for any hardware upgrades.
(e.g., buying more RAM for the server because the code is so bloated it doesn't quite fit in RAM
I am aware that it's an on-going trend to call FA's code bloated, horrible and broken. In reality the only thing FA's code is bad in right now is in it's architecture. That I will not argue.
The code itself it's very light on both the memory and the CPU, fast, streamlined, optimized and minimalistic.
FA runs off a 64 child pool of php-fastcgi processes (memory_limit=32MB) with the static request serving by nginx, which all take up maybe 1.5GB of RAM at full load with all 64 workers busy serving the heaviest of pages.
Not even close to something being bloated or slow, compared to... just about any other blog/website software that's not custom coded and many those that are.
It's got a terrible database layout that doesn't even adhere to 1NF at places (slowdowns, data duplication, problematic backups and restores, terribly inefficient queries on the browse page due to all filters being varchars).
It implements a HTTP GET based site navigation and data management.
It implements a downright broken way of storing user data files in a flat directory tree, running into 32k directory-in-a-directory filesystem limit making me use terrible symlink hacks and actively preventing such a filesystem from being periodically backed up.
Uses a "hay, lets chuck all the files in the user directory" and "hey, let's not enforce deleting some of those files after the actual submission deletion, or resizing, or just about any operation requiring temporary files" approaches, making backups unnecessarily larger and take significantly longer due to all that junk data also being backed up.
I can name many more things it does wrong. I know them all. But the one common thing among them is the fact that a proper "fix" requires long, extensive architectural/database/filestem changes and is not something you do in an evening or two, or get volunteer help on since it requires root level access and massive data manipulation.
Ferrox was supposed to do it all right. Even if simply duplicating FA's current functionality at the first release, but done right. Eevee had the chance to do all the things you speak of. An yet we're still here right now.
The issue at hand is that the administration has frequently decided to upgrade their hardware instead of attempt to fix the underlying issues, thus resulting in the waste of money
The reason why we upgraded our servers since the 16k times was not because we needed it, but because we could.
We've been donated some hardware, we put it to use. No money spent. 0$. We regret having wasted this much.
You may argue whether the hardware was put to a good use or not, that's fine; but you still have to agree that any kind of use is better then not using it at all.
Data server, for instance, could use the extra RAM for an even larger filesystem cache since it's old and tired RAID10 array of WD RE3 drives has long reached it's limit in IOPS. Secondary CDN node is already scheduled for.
Database server... I don't really need to mention that it could always use as much RAM as you throw at it, esp. considering that InnoDB, given sufficient amount of RAM, starts mirroring database contents in it, resulting in great speed even with not quite so optimized queries.
Cisco router and switch were instrumental in preventing the recent DDoS from having a lasting effect, and will play a much more significant role being key elements on the border between the internet and our internal network.
We have that hardware for free.
Yes, it's more then we currently need.
Yes, it's a bragging rights overkill.
Yes it can run five copies of FA provided sufficient bandwidth given.
And it's a good thing, because that means we have reserves the limits of which we are not likely going to reach in the next year or two.
Replying to specific points of your list:
4. The code has zero strain on the hardware. There are no code or logic related I/O strain, RAM depletion or wasted money on bandwidth.
What FA consumes now, and is running short of - are raw operational requirements. Drive capacity, RAID array IOPS and bandwidth. And of course, money to obtain it all.
There is nothing you can change that eliminate the need to serve a million of small files. Only slightly reduce that number.
Nothing you can change to make those files smaller considering that we are about to allow for higher resolution uploads (and half the code is being rewritten to make a proper use of storage space and bandwidth for it)
Nothing to change wrt: bandwidth consumption considering that image/media data takes up 98% of all the transfers.
5. Aside from Trogdor and Novastorm, no other hardware worth significant amount of money was bought by FA with it's funds. We simply don't have the money, as everything goes directly into paying monthly bills.
Most of the hardware have right now has been donated by various parties. Selling it would be a dick move.
However, it's a start. And once we get the details and the ability to start fixing the site as it is now, we can follow this guideline to actively fix the website as it currently stands today.
I want to make this perfectly clear. You will never get it, at least in the way you want.
You make wild and mostly incorrect assumptions and accusations based on false, incomplete or deliberately manipulated information. The information that you know is not true that you've chosen to believe in.
You ban a coder of this site from the ability to have an active discussion with your group, who was willing to and could have shed light on some of the details you are not aware of and was curious in. As others in the channel have done before you.
You then demand the administration of this site to hand you over the keys to the entire system based only on your honest to god promise that you will do no harm backed by zero evidence, reason to or even a plan on what you're planning to do with them. And you also expect said administration to walk over it's current technical staff.
You view the opinion of the administration with zero regard and show no signs of wanting to listen to reason, explanations or will to compromise.
You make your offer to help in the worse manner then you would have thrown 5$ in the mud in front of a dirty, homeless person in the street. Just who the hell do you think you are? Take your handouts and shove them up your ass you pretentious elitist bastard. Admittedly, the choice of words above is not what I would have chosen under normal circumstances, but reading over yours, and other people's posts on the topic has me somewhat miffed. My apologies..
Thus so far you have shown very poor objectivity and either the lack of desire - or the ability - to work with people who you do not agree with on a team. You disregard the opinion of current staff, dismiss the current plan as stupid and broken even though you know nothing of it, and expect to have been given full control over everything to have things done your "right" way. The only explanation of your way is this journal, which is built upon speculation at best.
In order to be acting in such a way and still expect to be given what you want you'd have to have some pretty significant redeeming qualities, and know them before making such an offer. What of them do you have that we can not obtain somewhere else with less a negative side to them then it is with you? People who may admittedly be less qualified then you, but can at least work in a team.
