startups, technology and fishing

The thing about running a widget business is that you serve as many web server requests as all your users websites, combined. And if one of your users get’s Dugg or Slashdotted, you get Slashdotted too.

After I launched FEEDJIT on Thursday (5 days ago) the traffic started picking up Friday and by Saturday morning my server was groaning under the strain. Some of the highest traffic blogs were Japanese (there are more Japanese bloggers than English) and by mid-morning the Japanese were going to sleep, so that gave me a welcome reprieve.

The first thing I did was reduce Apache’s KeepAlive timeout to 2 seconds. KeepAlive’s let clients hang on to a connection which someone else could be using. If a client uses keepalive properly then it can give you a nice performance boost, but set the timeout low so slow clients don’t waste server resources.

Then I added HTML caching for the widget serving routine using Perl’s Cache::FileCache. This gave me a huge speed increase but the stats on our widgets were 1 minute delayed - and that sucked.

By Saturday night I’d rolled out the new caching code and the server was a lot faster, but I knew it wouldn’t work long term and non-realtime stats for FEEDJIT was not an option.

By Sunday I was getting 40 hits per second and rising and the server was groaning again. I had to make some fundamental changes to the way the app was architected. The old mod_perl2, MySQL and Apache2 combination wasn’t going to cut it.

So I basically redesigned the data storage routines from the ground up. I moved from mysql to a home grown data access method.
I can’t tell you how gratifying it was when I rolled out the new code last night and watched the server load average drop from 2.5 to 0.3 (unix load where 1.0 = 100%) and hold there as our traffic continued to rise.

We have several high traffic blogs now and our busiest blogs generate around 1.5 widget loads (pageviews) per second. I’m confident that if for some reason TechCrunch added us tomorrow, we’d easily handle the traffic without breaking a sweat.

I just launched FEEDJIT. It took me about 10.5 hours (4pm until 2:30am) from the first time my hand touched the keyboard until I fixed the last bug and went live. I got a question on the Seattle Tech Startup list about how I spent my 10.5 hours. So here’s a brief summary:

• I drew a mockup in Fireworks. It started getting complicated with user registration and so on. So I basically binned it and just wrote the software, but the mockup gave me an idea of the most basic value prop. So I made a decision to go out the door with the very very basics and see if it’s something users actually want. (1 hour)
• I designed the database schema in SQL commands using a text editor. I mentioned this CompSci quote to a friend yesterday: “Get your data structures correct first, and the rest of the program will write itself.”. Doing the data structures in the form of a schema forced me into making all the hard decisions of what features I’m keeping and what I’m not going to have time to implement. (1 hour)
• I wrote the functional app. I find that if I do graphical or UI work early on it can become very time consuming as I try to get just the right dropshadow on some element. So I just dove in and wrote the Javascript and server components. The app doesn’t require any registration so I could just write the widget and the server code to store and deliver the stats that are displayed. (roughly 6 hours)
• Then the last thing I did was create the home page (the only page on the site). After cranking out code for 6 hours I was too tired to faff with dropshadows and so on. So it became purely functional. (roughly 2 hours)

I already have a server set up at serverbeach where I host this. It’s on a 10 megabit backbone connection but doesn’t cost me much. So I basically added a virtualhosts section to the httpd.conf file and copied the source code into the proper directories. I then compressed the javascript and used my SQL text files and the mysql client to dump the schema into the database. I brought it online and couldn’t figure out why everything was appearing to be in Denver, Colorado. Then I realized I’d hard-coded my own IP address into the Geo location routines. Shows you how you can screw up when you’re rushed. I fixed that and it worked perfectly.

I hope there’s some value in that. I think the smartest thing I did was to drop everything except the features that would test whether this is a product my target market would actually find useful. That remains to be seen of course, but I’m hopeful. We’re adding quite a few new blogs per hour now.

Jake Brown fell 40 feet and landed so hard in the transition at the X-Games last night that his shoes flew off. Forward to the end of the vid and watch the slowmo version.

Brown spent Thursday night in a local hospital, where he was treated for a bleeding liver, two sprained wrists, a bruised lung, and whiplash to his back and neck. He didn’t break a single bone. Unbelievable.

If you’re a bit of a sci-fi fan, a bit of a maritime adventure fan, like all things engineering, then I have just the book for you. The Ice Limit is one of the best paperbacks I’ve read for a very very long time.

I’m busy decommissioning an old server I’ve had for almost a year. I’ve used it for dev and testing and installed a bunch of crap on it.  It has never required a reboot and has been busily chugging away. Here’s it’s uptime:

11:20:46 up 333 days,  8:13,  1 user,  load average: 1.02, 1.34, 1.21

The only reason it needed a reboot 333 days ago is because the datacenter it was hosted in was being migrated. It was up for about a year before that, again without reboot.

Douglas MacArthur - “We are not retreating - we are advancing in another direction.”

I just got back from the Naked Truth panel and party in Seattle. It was loads of fun. I met John Cook for the first time in the flesh - he’s interviewed me about 3 times and we’ve never actually met. Also met Michael Arrington briefly.

The panel was so-so. I think the general consensus is that we didn’t learn a hell of a lot that’s new, but it made a great excuse for the party afterward. There was some playful banter on the panel between Seattle PI (John Cook) and the Seattle Times (Tricia Duryee) that turned into a bit of a circulation comparison.

Michael Arrington was hilarious on the panel openly poking fun at the WSJ and Fred Vogelstein from Wired. I’ve never been a fan of Wired and glad to see I’m not alone.

Looking forward to the next one!!

Looks like my choice to use Wordpress.org on my own servers for my personal blog was a good one. Typepad has been down since 1:50pm. Unfortunately that means both Geojoey and Linebuzz corporate blogs are down.

Dear TypePad member,

The TypePad service is currently unavailable due to power issues at
our co-location facility.  This means that the TypePad application and
your TypePad blog are not reachable at this time.  This began at
approximately 1:50 pm Pacific Daylight Time today, Tuesday July 24
2007.

We are working closely with our hosting partner to bring TypePad back
online as soon as possible.  We sincerely apologize for the
inconvenience this is causing, and we appreciate your patience.  We
will send another email update with more information as soon as
possible.

Thank you,
The TypePad team

UPDATE: This article generated over 5,000 page views in under 24 hours, so I’ve posted a follow-up interview with Tony Wright, RescueTime’s Founder & CEO.

I’m participating in a closed Beta of Rescuetime.com and installed the software on Sunday. Yesterday sat down at my desk for 10 hours and then hit the site. Here are the results:

All I can say is, I was floored that after 10 hours I’d used less than half my time effectively.

Rescuetime lets you tag apps, so the graph above is a graph of apps that I’ve tagged as work, personalblog, etc. The rest of the time is distributed among random websites, apps and other distractions.
Here’s the breakdown of my top apps and websites:

I got into a flamewar with someone on news.ycombinator.com which blew away more than an hour of my day. Most of my work is in an SSH session using an app called PuTTY. I hack my hosts file when I code and use our corporate blog for testing, so blog.linebuzz.com and linebuzz.com are also tagged as work sites and grouped into the ‘work’ graph above. The time spent on markmaunder.com was writing personal blog entries. I’m a huge fan of Brad Feld’s blog, hence the time on feld.com.

So I’ve canceled two meetings this morning and have severely limited my personal blogging time today to try and get at least 7 continuous hours of REAL work in before I head to the naked truth panel and party tonight at 5pm.

I think Tony Wright and the guys at Rescuetime are on to something potentially huge. I’m watching these stats as obsessively as I watch my Google Analytics stats.

Starting a software business? Looking for a software engineering process? You can spend a month getting your head around one of these:

Agile software development
Crystal Clear
Extreme programming
Lean software development
ISO 12207
Rational Unified Process
CMM
ISO 15504

Or.. 2 seconds learning the Nike method: