Try Buying your Hardware

We took a lot of heat from the startup community when we bought $40,000 of Dell servers, a switch and a KVM and racked them ourselves in 2008. Seriously, Kerry (my wife and co-founder) and I hand-racked about 10 Dell 2950’s and a couple of 1950’s in the rack we leased at our data center. We didn’t realize the DC team could rack them for us and were so excited when the servers arrived we just dove right in.

Do you have any idea how much a DELL 2950 loaded with disks weighs? They’re heavy.

At that time the “cloud” was all the rage. Amazon services were really spinning up, Linode and SliceHost were the new ‘it’ companies, and we were derided as idiots for actually buying physical hardware: Ew!

Well turns out our business scaled very quickly and in a few short months we were pushing well over 100 megabits of bandwidth average. We were paying around $2,500 a month for that which included power to the rack, a team supporting our hardware 24/7/365 and that included the bandwidth and 5 very high quality upstream connections. We’d discovered the magic of 95th percentile billing. Most of our peers were paying by the terrabyte and getting absolutely screwed. Our business would never have survived if we didn’t use colocation.

Today we’re busy decomissioning our old Dell 1950’s and 2950’s and replacing them with amazing new Dell R630’s. Back then we were paying about $3500 per server. I just bought 4 Dell R630’s at $9250 each out the door. We’re happy to spend that kind of cash because we know these machines will pay for themselves a hundred times over (or more) by the time we’re done with them. We have a little inside joke: “Good servers go to small business heaven. Bad servers end up working for us.” We literally put our servers through hell by running them at very high CPU and IO loads. To date we haven’t had a single failure besides hard drives and redundant power supplies, all of which are hot-swappable and no big deal. No memory, chassis or controller issues. (We use PERC hardware RAID 1 or 10 usually)

So I guess I’d like to say a big Kudos to Dell for producing some kick-ass enterprise class hardware that could withstand the worst kinds of loads we could come up with. And seriously: If you’re a startup and can afford it, consider making a capital investment in your own hardware and using colocation rather than abstracting away the problem and paying more – and in some cases, a hell of a lot more.

Besides: What could be more fun that spending your Saturday night in the data center.

11 Questions Every Startup’s Money Guy Should Be Able to Answer in Her Sleep.

Every profitable business has a Money Guy. Sometimes it’s the CEO, sometimes it’s another member of the exec team. Money sticks to this persons hands for reasons unknown. They know how to get the best deals for anything they buy and they have a habit of making more money than they lose. If you don’t have someone like this in your business then you are almost certainly not profitable and never will be.

Often this persons title is Chief Financial Officer or Chief Operating Officer. They are the ones who update the cash flow plan and know how much cash the business has on hand at all times. They are the money guy. You’ll know your money guy rocks when you wake them up in the middle of the night and they answer every one of these questions as if it’s a reflex:

  1. When do we run out of money?
  2. What is our next revenue target date and amount?
  3. Are we going to make our target or are we slipping?
  4. What are the two most effective things we can do to increase revenue?
  5. How are we doing with regards to implementing those 2 most effective things?
  6. What are our top two sources of customers?
  7. Are either of those sources at risk of disappearing overnight?
  8. What are our two biggest expenses?
  9. Have we done everything we can to lower those expenses?
  10. Are we at risk of facing a large bill in the near future?
  11. Now that you’ve been woken up, is there any current or future problem in the business that will make it hard to go back to sleep?

Other awesome Money Guy attributes:

  1. Your Money Guy discovers surprising ways to save significant amounts of cash on big expenses.
  2. Your Money Guy isn’t constantly moaning or complaining, but occasionally will assemble the team and lay out hard facts that put some acid in your gut.
  3. Your Money Guy always has your important financial data on hand, often memorized for impromptu brainstorms or planning sessions.

The culture I’ve described here is unfortunately not what you find in most nascent businesses which is why most of them fail to make that critical four year mark. Also not that this does not apply, or put differently it is unable to be applied to the paralel universe of West Coast Technology Startups in the USA. But it’s something I’ve observed in businesses around the world including profitable USA businesses.

Often Money Guys are seen by tech entrepreneurs as people with green eyeshades who work under dim green lamps and aren’t fun at parties. But without the Money Guy, the parties end, entrepreneurs become employees and the innovation ends. Your Money Guy is the person who makes sure your business has enough oxygen to dive deep, take those big risks and come up for enough air to do it again.

Trial, Error and the God Complex

My new favorite economist Tim Harford did a great TED talk recently chatting about our assumption that an expert approach is needed to problem solving. He argues that instead we should rely more on trial and error, a method that has proven very effective both in nature and business.

 

If the loading animation won’t disappear then try viewing the video on this page.

Domain name search tools

Clarence from Panabee pinged me a few minutes ago mentioning Panabee.com. I hadn’t heard of it and along with nxdom.com I’m going to add it to my toolkit to brainstorm available domain names.

My attitude re names these days fluctates between the-name-is-everything and back to sanity.

A week ago I was obsessed with the domain name WordPrice.com which a friendly cybersquatter wanted to sell me for $700. I even contacted the owner of a very similar mark and kindly got the OK to use it for what I intended. Then backed off at the last minute because a) I refuse to support cybersquatting and b) names are more about creating a well loved and well remembered brand than pretty words.

Keep in mind the relative strength of different types of trademarks when you’re thinking about future brands. Make sure you do a USPTO search and at some point spend $500 with a TM attorney to get your use of your new mark on record and start the trademark clock. I also tend to screenshot a few 100-result google searches for any new potentially strong mark I’m going to use. I date them and file them. [Once you’ve had your ass handed to you in a trademark lawsuit like I have, you get paranoid]

 

Why we breathe

Free Diver in LimasolHold your breath for a moment.

In about 10 to 30 seconds you’ll be feeling a strong desire to take a breath. That’s not caused by lack of oxygen. It’s caused by excess carbon dioxide buildup in your blood.

[Ok you can breathe again.]

The trigger in mammals that causes us to want to take a breath is an excess buildup of CO2. In reptiles the trigger is lack of O2. Free divers don’t hyperventilate to get more O2 into their bloodstream. They do it to to flush out excess CO2 and remove that breathing trigger. That’s also what causes shallow water black-out as you’re surfacing, so don’t try it without a buddy.

I’ve worked in more startups than I care to count where the lack of endurance was not caused by lack of oxygen, but an excess buildup of waste. Getting a larger office, buying excess server capacity early on that isn’t needed, hiring excess people to manage that server capacity, hiring managers to manage the people, hiring an ad agency and PR firm and a small team to manage them.

Once you start down the path of waste you may still have enough oxygen in your bloodstream to surface, but the excess CO2 in your business creates a strong demand for more Oxygen which causes you to raise another round of funding, producing more CO2 and the cycle continues.

So start your business by hyperventilating to flush out all excess CO2, take a deep breath and beware of shallow water blackout as you’re approaching the surface.

[Photo credit: My good friend Bruno Stichini who hosted a free diving world record attempt in Limasol, Cyprus back in 2000]

Costs and Startups – Advice for your CFO

In any company if you save $1 it goes straight to your bottom line. Meaning it’s as if you just earned another $1. The company that my wife and I have been running for about 2 years now serves over 30 Million page requests per day. We’ve invested a lot of time in getting more performance out of our hardware but about 6 months ago we started hitting pesky issues like limits on the speed of light and electrons.

So we’ve had to keep growing without going out and buying a Google-size web cluster. A lot of the wins we’ve had have been simply using every spare drop of capacity we can find. I’ve noticed a pattern during the last 6 months. It goes something like this:

Kerry (My wife, our CFO, and keeper of the graphs): Server 12 is hitting 10. [Meaning it has a load average of 10 on an 8 CPU machine which is 125% load]

Me: OK Dell has this great special on these new R410 servers that are about twice as fast as the previous generation.

Kerry: What about the other machines in the cluster?

Me: They’re already at 80%.

Kerry: OK what else do we have?

Me: Well the crawlers are maxed, the mail server’s maxed, the proxy’s maxed out, the load balancer is maxed….

Kerry: What about 25 and 26? They’re sitting at 2.

Me: Well we’d have to [technical and managerial speak explaining how complicated it’s going to be to implement]

Kerry: OK so lets do that.

Me: [More bullcrap this time rolling out the big guns desperately trying to get money for new toys]

Kerry: …[waits it out]

Me: OK so lets do that.

If you’re a CFO approving purchase decisions in your company, take it from me: Geeks and CEO’s alike love buying new stuff. I assure you there isn’t a web cluster or database cluster on this planet that you can’t squeeze a little more capacity out of without breaking things. So before you take the [technical and managerial bullcrap from your geeks and CEO] at face value, sit down with your team and have them explain all the data to you and go through all your resources with a fine tooth comb. Then, if you absolutely have to, spend some money.

And if you don’t have a CFO, nominate someone immediately!! It doesn’t matter how small you are, someone had better be the keeper of the cash-flow plan or you’re going to run out of money and wonder why.

Incidentally, this is the load decrease on one of the busiest servers in our cluster when we brought online some ‘found’ capacity earlier today.

Screen shot 2009-11-01 at 2.29.56 PM

Posted on Hacker News.

Startup Hacks: Marketing with no money – RescueTime Interview

This is a followup post to “Think you work hard? Think again.” which generated over 5,000 pageviews in a short time and almost took down my blog server this morning. It’s an audio interview with Tony Wright, the founder and CEO of RescueTime.com.

The interview runs for just over 17 Mins. Click here to listen or right-click this link and click save-as to download.

We chat about how startups on a tight budget can market their product and business for free. Topics include getting covered by TechCrunch, linkbait and getting covered by sites like Digg or Reddit, writing great headlines for articles, getting a product to market with no money raised and ugly vs pretty websites and whether that affects your marketing success.

Full disclosure: Tony and I are friends, we both run our own tech startups (I am CEO of LineBuzz.com) and love brainstorming new ideas over a beer. We will be doing exactly that at the Stumbling Monk Pub tomorrow evening after the Seattle Tech Startup meeting at the Capitol Hill public library.