Where’s the Disruption from the Change in Startup Economics?

It’s been a year long break from blogging and getting back to writing and getting a so many new visitors this soon is cool. [Thanks HN!]


This blog runs on the smallest available Linode 512 instance for $20/month. It runs several sites including family blogs and hobby sites. I run nginx on the front end and reverse proxy to 5 Apache children which saves me having to run roughly 100 Apache children to handle the brief spikes of around 20 hits per second I saw yesterday.


Technologies like event-servers (Nginx, node.js, etc) and cheap and reliable virtualization may seem like old hat, but in 2005 Linode was charging $40/month for a 128Meg instance (it’s now $20/month for 512Megs, 88% cheaper) and Nginx was only going to hit main-stream use two years later. In fact Nginx only hit version 1.0 last month.

Five years ago many companies or bloggers would have used a physical box with 3.5 Gigabytes of memory to handle 100 apache instances and the database for this kind of traffic. About $300/month based on current pricing for physical dedicated servers from ServerBeach which hasn’t changed much since 2005.

With the move from hardware and multiprocess servers to virtualization and event-servers, hosting costs have dropped to 6% of what they were 5 years ago. A drop of 94% in a variable cost for any sector changes the economics in a way that usually causes disruption and innovation.

So where is the disruption and innovation happening now that anyone can afford a million-hits-a-month server?


Footnotes: An unstable version of Nginx was available in 2005/2006 and Lighttpd was also an alternative back then for reverse proxying. But it was for hardcore hackers who didn’t mind relatively unstable and bleeding-edge configurations. Mainstream configuration in 2005 was running big memory servers on dedicated machines with a huge number of Apache children. Sadly, much of the web is still run this way. I shudder to think of the environmental impact of all those front-end web boxes. I also don’t address the subject of Keep-Alive on Apache. Disabling Keep-Alive is a way to get a lot more bang for your hardware (specifically you need less memory because you run less apache children) while sacrificing some browser performance. The norm in 2005 was to leave keepalive enabled, but set to a short timeout. With Keepalive set to 15 seconds, my estimate of 100 apache instances for 20 hits per second is probably way too optimistic. With Keep-Alive disabled you would barely handle 20 requests per second with 100 children when taking into account latency per request for slower connections. Bandwidth cost is also a consideration, but gzip and running compressed code, using CDN versions of libs like jQuery that someone else hosts and running a stripped down site with few images helps. [Think Craigslist] With a page size of 100K, Linode’s 400GB bandwidth allowance gives you 4,194,304 pageviews.


Slow lighttpd on Ubuntu 7.10 Gutsy Server with 200+ hits/sec?

aaaah you say. Finally, after many a Google search finally I found someone who understands my pain. I know you’re in a rush and I can’t stand people who love the sound of their typing either, so here’s how you fix this little problem.

If you have a brand new super fast server and a high traffic website (200+ requests per second) and you install lighttpd and it performs like a dog, try the following:

Add this to your /etc/sysctl.conf file:

net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1

net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.wmem_default = 16777216

net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

#Only enable these if you’re dumb enough to have netfilter connection tracking enabled
#net.ipv4.netfilter.ip_conntrack_max = 1048576
#net.nf_conntrack_max = 1048576

Then run

sysctl -p

Also make darn sure you don’t have netfilter’s conntrack modules enabled in the kernel. If you’re using shorewall on your lighttpd box this will probably be enabled. You can check if conntrack is enabled by checking if the file /proc/net/nf_conntrack exists. Also run lsmod and you’ll see a ton of modules starting with nf_contrack_

To get rid of conntrack if it’s enabled I would avoid rmmodding them – rather remove the app that enabled it and reboot the box just to keep things sane.

If you must insist in using conntrack then uncomment the last two lines in the sysctl.conf sample above.

Google the individual params above and you’ll find a ton of explanation on each.