Scaling from 0 to 40 hits per second in 3 days

The thing about running a widget business is that you serve as many web server requests as all your users websites, combined. And if one of your users get’s Dugg or Slashdotted, you get Slashdotted too.

After I launched FEEDJIT on Thursday (5 days ago) the traffic started picking up Friday and by Saturday morning my server was groaning under the strain. Some of the highest traffic blogs were Japanese (there are more Japanese bloggers than English) and by mid-morning the Japanese were going to sleep, so that gave me a welcome reprieve.

The first thing I did was reduce Apache’s KeepAlive timeout to 2 seconds. KeepAlive’s let clients hang on to a connection which someone else could be using. If a client uses keepalive properly then it can give you a nice performance boost, but set the timeout low so slow clients don’t waste server resources.

Then I added HTML caching for the widget serving routine using Perl’s Cache::FileCache. This gave me a huge speed increase but the stats on our widgets were 1 minute delayed – and that sucked.

By Saturday night I’d rolled out the new caching code and the server was a lot faster, but I knew it wouldn’t work long term and non-realtime stats for FEEDJIT was not an option.

By Sunday I was getting 40 hits per second and rising and the server was groaning again. I had to make some fundamental changes to the way the app was architected. The old mod_perl2, MySQL and Apache2 combination wasn’t going to cut it.

So I basically redesigned the data storage routines from the ground up. I moved from mysql to a home grown data access method.
I can’t tell you how gratifying it was when I rolled out the new code last night and watched the server load average drop from 2.5 to 0.3 (unix load where 1.0 = 100%) and hold there as our traffic continued to rise.

We have several high traffic blogs now and our busiest blogs generate around 1.5 widget loads (pageviews) per second. I’m confident that if for some reason TechCrunch added us tomorrow, we’d easily handle the traffic without breaking a sweat.

12 thoughts on “Scaling from 0 to 40 hits per second in 3 days

  1. Pingback: Startup Signal - Today’s Top Blog Posts on Entrepreneurship - Powered by SocialRank

  2. Pingback: thinking in geek » Blog Archive » ERP, Financials and Large Databases

  3. I’m honestly shocked that you don’t consider “1 minute behind real time” to be real time. That’s completely within acceptable boundaries, yo.

  4. I would have gone to SQLite before a flat file. In theory, you should have the advantages of both (file and DB) with SQLite.

    If you considered it, but decided against it, I would love to hear the drawbacks you saw.

  5. Thanks Justin,

    I considered Berkeley DB as an alternative when researching this. If I was doing key/value lookups which is what many sql queries do, then I’d probably use lots of small BDB files.

    Mark.

  6. Great write up! Your widget seems well suited to direct file storage, because it’s a log basically. I wonder what you would have done had your widget required more complex queries for the data it displayed? Maybe in that situation caching and delay from real time would be necessary. Or adding more hardware.
    BTW What are the stats on your server (RAM/CPU)? It sounds like you could now handle ~ 120 hits/second @ 100%. That’s pretty impressive for one server as I see it. Nice work.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Notify me of followup comments via e-mail. You can also subscribe without commenting.