In any company if you save $1 it goes straight to your bottom line. Meaning it’s as if you just earned another $1. The company that my wife and I have been running for about 2 years now serves over 30 Million page requests per day. We’ve invested a lot of time in getting more performance out of our hardware but about 6 months ago we started hitting pesky issues like limits on the speed of light and electrons.
So we’ve had to keep growing without going out and buying a Google-size web cluster. A lot of the wins we’ve had have been simply using every spare drop of capacity we can find. I’ve noticed a pattern during the last 6 months. It goes something like this:
Kerry (My wife, our CFO, and keeper of the graphs): Server 12 is hitting 10. [Meaning it has a load average of 10 on an 8 CPU machine which is 125% load]
Me: OK Dell has this great special on these new R410 servers that are about twice as fast as the previous generation.
Kerry: What about the other machines in the cluster?
Me: They’re already at 80%.
Kerry: OK what else do we have?
Me: Well the crawlers are maxed, the mail server’s maxed, the proxy’s maxed out, the load balancer is maxed….
Kerry: What about 25 and 26? They’re sitting at 2.
Me: Well we’d have to [technical and managerial speak explaining how complicated it’s going to be to implement]
Kerry: OK so lets do that.
Me: [More bullcrap this time rolling out the big guns desperately trying to get money for new toys]
Kerry: …[waits it out]
Me: OK so lets do that.
If you’re a CFO approving purchase decisions in your company, take it from me: Geeks and CEO’s alike love buying new stuff. I assure you there isn’t a web cluster or database cluster on this planet that you can’t squeeze a little more capacity out of without breaking things. So before you take the [technical and managerial bullcrap from your geeks and CEO] at face value, sit down with your team and have them explain all the data to you and go through all your resources with a fine tooth comb. Then, if you absolutely have to, spend some money.
And if you don’t have a CFO, nominate someone immediately!! It doesn’t matter how small you are, someone had better be the keeper of the cash-flow plan or you’re going to run out of money and wonder why.
Incidentally, this is the load decrease on one of the busiest servers in our cluster when we brought online some ‘found’ capacity earlier today.
