Blog

  • Does your startup pass The Sleep Test

    Having coffee at 4am after an all-nighter with my co-founder and wife a few days ago we came up with a rather obvious but interesting concept. I’ll call it The Sleep Test.

    Unless your business earns revenue while you are sleeping, it won’t scale.

    If you’re an I.T. consultant or lawyer selling your own time, you can’t scale.

    If you’re a brick-layer who employs other brick layers and also employs a sales person, driver, accountant and all the other business components so that your business runs while you’re not there, you CAN scale.

    If you’re a web developer who writes an application that earns ad revenue or that earns subscription money while you sleep, you CAN scale.

    Most businesses start off with a founder selling their time and with the maximum earnable revenue being tightly limited by the founders available time. The founder works themselves into a stupor and at some point they go through what is often a difficult transition where they “step back” from the business and employ others to take over their various jobs. Many businesses don’t make this transition and it is the subject of much discussion in MBA programs world-wide. The birth of Kinko’s is a great example of this evolution. Paul Orfalea is dyslexic and in the story of Kinko’s he mentions how this forced him to step back from the business and employ others.

    Many “Web businesess” or “Software businesses” need to employ a sales team or have components like fulfillment that don’t scale easily or cheaply. But if your business is a “Web App” that earns you money through advertising or through subscriptions and where the application is the business, it scales incredibly well.

    Web App businesses scale so well that if you “get it right”, they automatically pass the sleep test from day one and they pass the test without you having to employ additional staff.

    Two types of Web App that often pass the sleep test are:

    1. A service that attracts huge numbers of an attractive demographic that can earn you ad revenue or
    2. A service that is so valuable to a group of people that they will pay you for it, preferably on a recurring basis

    Your web app business must also:

    1. Not require additional staff time per customer
    2. Not require additional staff time per dollar earned
    3. Market itself. If it’s marketing is limited by your time, you wont’ scale.
    4. Earn you substantially more money than your business costs to run.

    And that’s it. You need to build a web application that markets itself, earns more money than it burns and that is either wildly popular or wildly valuable.

    If you have a Web App that passes The Sleep Test, congratulations because you have just bypassed one of the most difficult stages of small business evolution and one of the most common points of failure that just about every other business type is forced to navigate.

    Final caveat: I’ve written this post discussing this concept in absolutes i.e. you either do or do not pass the sleep test. Of course in reality there is not a single web app business that does not need to employ more staff as their revenue and customer base grows. Google is a fine example of a business that is designed to avoid having to employ more people as revenue or customers grow and they employ over 20,000 people today. But this test is a useful way to measure and think about how efficiently your business will scale.

  • Basic cooking for bachelor alpha geeks

    Just caught up with a good friend of mine who’s a tech geek and bachelor and needed some cooking advice. I’ve watched bachelor friends give themselves scurvy by getting home and ordering pizza every night. Being a tech geek makes it worse.

    Your $badnessOfDiet += $levelOfAlphaGeek**3

    So my wife put together the list of recipes below for him this evening. They are incredibly basic but are staples that you can make in big batches at the beginning of a week and eat the whole week. You can make a big batch on sunday, freeze half of it and keep the other half in the fridge and thaw it out halfway through the week as you run out.

    The foods below are Low GI so will help you lose weight and you can eat as much as you’d like. They also contain much of what you need in your diet. If you’re a geek looking to lose some weight, this list is definitely for you. I lost 25 pounds (200 down to 175) last year eating mostly this. I’d throw in some fruit here and there (organic apples, banannas, grapefruit, oranges, etc..) The fruit is not as low on the GI scale, so don’t go nuts.

    Also, try to drink your geek dose of caffeine AFTER you’ve eaten. You’ll notice your skin suddenly stops being so dry.

    Lentils                                    Cooking Time:  30-45 min

    Lentils are the easiest bean to cook as they don’t require presoaking.  Sort through the lentils to pick out any rocks.  Rinse them to remove dust.

    Add 2 cups water and 1 cup lentils in a pot.  Bring to a boil then set to simmer.  Cook for 30-45 minutes depending on the type of lentil.  Start checking at 30 minutes. They should be soft.  Not chewy (undercooked) or mushy (overcooked).  Add water if you need to during the cooking time.  You can drain off any excess water if they are done cooking.

    You can eat them plain or add some salt and pepper (such as ½ teaspoon salt and ½ teaspoon pepper) or other spices.  You can add spices during the cooking except salt.  Only add salt at the end or the lentils won’t soften.  You can triple/quadruple, etc the recipe to make a big batch at the beginning of the week.

    Dried Beans:                  Black Beans / Lima Beans / Kidney Beans / Pinto Beans

    Soak Time:  8 hours                  Crockpot Cooking Time:  8-12 hours

    Dried beans are also easy to make but need some planning.  You also sort through the bag of beans for rocks and give a quick rinse.  Soak overnight or at least 8 hours.

    The easiest way to cook is in a Crockpot while you are at work or overnight because you don’t have to worry about the water level getting too low.  Put the soaked beans into the Crockpot and cover with about an inch of water.  Add spices, set the Crockpot on LOW then let the beans cook for 8-12 hours.  Start checking at 8 hours or when you get home/ wake up.  Beans should be soft.  Take one out and sample it to see if it’s done.   If you get home and they aren’t quite done, you can put them on high and they cook much faster but keep an eye on them.  They can get low on water and burn.

    Some spices to add are garlic powder, onion powder, cumin seeds or cumin powder and pepper.  Again, no salt while cooking, it will prevent the beans from softening but the other spices are great to add while cooking.  You can start off adding 1 teaspoon of each spice per 2 cups of soaked beans and modify to suit your taste.

    Vegetables:                  Cooking Time: 5-10 minutes

    Go for variety when eating vegetables.  Steaming is the easiest way to cook.  Get a little metal steamer you put into the bottom of any pot.  Be sure to rinse the vegetables before cooking.  Good vegetables for steaming are broccoli, asparagus, cauliflower, green beans, mushrooms and carrots.  Put in the steamer, add water until it just touches the bottom of the steamer, put in the veggies, cover, put on high until it starts steaming then set temperature on medium.   Most vegetables are done in 5-10 minutes.  Start checking them at 5 minutes until they are the texture you like.

    You can eat them plain or drizzle with some olive oil, salt, pepper, lemon juice or even try balsamic vinegar.

    Herbed Baked Chicken:                                    Cooking 50-60 minutes

    This recipe works great with boneless breasts or thighs.  Rinse the chicken and put into a baking dish.  Over 8 pieces, drizzle a mixture of  1 tablespoon lemon juice, 1 tablespoon olive oil, 1-2 minced garlic cloves and 1 teaspoon each of dried (or 1 tablespoon fresh) of your favorite herbs such as oregano, rosemary, thyme, basil, etc.

    Cover with foil and bake at 350 for 45 minutes.  Take them out and check the internal temperature with a meat thermometer.  This is important.  The best way to not overcook or undercook is to check the internal temperature.  The chicken is done at 165.  If the chicken is already at that temperature, it’s done.  If isn’t not quite there, remove the foil and bake for another 5 minutes. Take it out and check the temperature again.  Repeat until it’s at 165.

  • If your bank doesn't like your startup's blog, they may freeze your funds

    Update: The Fabulis story had legs like I’ve never seen before. When I posted it to Hacker News it shot to number 1 in about 3 minutes and stayed there for 6 to 8 hours. A few hours later Robin Wauters from Techcrunch picked up on the story and since then it’s appeared everywhere from GigaOm and ValleyWag to FT.com. Citi has now issued a formal apology to Fabulis emphasizing their support of the LGBT community which you can read here.

    Update2: Jason posted this update about 4 hours ago with some additional rather startling detail. “Yesterday I was even instructed to come into the branch to view a print-out of the “offensive” content on our site which was in “violation” of their compliance officer’s review of our business account.”

    My original post follows:

    In an utterly bizarre move, according to the Fabulis blog, Citibank blocked Fabulis.com’s bank account a few days ago for “objectionable content on their blog”. To give you some context, Jason Goldberg the Founder & CEO is a good friend of mine and started his career working for Bill Clinton in the White House. He then went on to T-Mobile, picked up a Stanford MBA on the way, raised $50M for Jobster which ended up buying my job search engine, then founded SocialMedian (sold to Xing) and is now working on Fabulis.

    Fabulis is based in NY but Jason is still known and loved by the Seattle startup community and we may one day even forgive him for going to the east coast.

    The company is still finding their niche but it looks like they’re setting up to be a travel portal for gay men. Their blog has had a ton of hilarious videos of guys describing why they’re “Fabulis”. Zero porn, nothing even mildly suggestive or risque.

    Just to be completely clear, we’re not talking about refusing a line of credit here. This is a cash account belonging to a funded company that was blocked.

    I haven’t had a chance to speak to Jason about this yet, so I don’t have any more detail. But after Fabulis called Citi this evening they temporarily lifted the block while “a compliance officer is asked to re-review our website on Thursday”.

    I’m curious why a bank would think they have the right to block a depositor’s access to their own funds based on that banks own moral judgment. Unless I’ve misunderstood the facts, this sets a very dangerous precedent.

    Banks are highly motivated to hold on to your money as long as possible. If they have this power, it is very profitable for them to use it because they earn interest on your money every additional day it stays in your account. Ever heard of the overnight rate?

    One might speculate that this is a form of redlining and that the LGBT community is the new target.

    After their $45 billion dollar public money bailout of Citigroup in 2008 it’s ironic that they would block the bank account of a technology startup who’s goal is to create jobs.

    Whatever the reason, I hope Citi’s PR team is kicking into gear because Jason is no stranger to mainstream media (CNN video) and this seems like the sort of thing that that will get picked up.

  • How a job is created

    This morning I called in to C-SPAN’s Washington Journal to answer the question “Is it fair that the Obama administration has been called anti-business”. To give you some context, I consider myself a democrat and I voted for Obama (and attended the rallies, etc)

    I’ve been ruminating over the issue for most of the day. Here are a few more thoughts.

    At the front of every American’s mind right now is the awful job market. America has lost more than 8 million jobs since 2007. Both parties have hitched their idiological wagons to this talking point and are trying to appear to be more capable of solving the issue than the other guy.

    The issue is simple: How do we create more jobs.

    The answer is simple: You help grow existing businesses and you help create new businesses. That is how a job is created.

    Let me say that again in case your ideological lense distorted that last sentence [or in case that blue hurts your eyes too].

    You create jobs by helping grow existing businesses and helping create new businesses.

    I’m a business owner in Seattle. I own a funded Technology startup. My goal is to grow my business into something the size of Google that employs 25,000 people. I’M THE GUY TRYING TO CREATE JOBS.

    I’m trying to create jobs so hard that I work without a salary. Not only that but I invest my life savings into the cause of creating jobs. I also work harder at my job of creating jobs than most people work at their normal job. I regularly pull 80+ hour weeks. [I’m sure you medical interns are laughing at me, but you’re the exception and are clearly insane.]

    Three friends of mine in Silicon Valley have taken some of their life savings and given it to me in the hope that they might help me create 25,000 jobs the way Google did.

    But what is very very strange is that with all this Democrat and Republican talk of creating jobs, not a single democrat or republican has actually done anything to help my friends and I create jobs.

    Nothing.

    They haven’t helped startups get access to credit.

    They haven’t helped startups spend less on taxes so they can spend more on growth.

    They haven’t helped the people who fund startups. In fact there’s talk of putting them out of business because they’re “BANKERS“!

    So really the message is that the American government does not care about creating jobs. The Obama administration doesn’t care. The Democratic party doesn’t care and the Republicans don’t care either.

    The only job security every senator and congressman cares about is their own. So call or write your senator or congressman today and let them know that if they don’t do something that helps create new businesses and grow existing ones, you’re going to fire them.

    While the jockeying and posturing continues in Washington D.C., my friends and I here in Seattle and people like us all over America will continue to work our tails off to help create jobs for you. We’ll continue to burn our savings. We’ll keep working without pay. Our friends will continue to risk their savings to try and help us.

  • Crowdsourcing a real-time solution to air terrorism

    At 09:28 on the morning of September 11, 2001, a Lebanese born engineering student named Ziad Jarrah and three “muscle” hijackers started moving the passengers of flight 93 to the rear of the plane and assaulting the cockpit of flight 93.  Flights 11 and 175 had already crashed into the world trade center and flight 77 was within minutes of crashing into the Pentagon.

    Two minutes after the hijacking started, passengers and crew started making phone calls to officials and family members using GTE Airphones and cellphones. A total of 35 airphone calls and two cellphone calls were made. 27 minutes later, the passengers had determined that flight 93 was a suicide mission, had voted to rush the hijackers and went to work taking back the plane.

    On Christmass day this year, Jasper Schuringa, a Dutch passenger on Northwest flight 253, ran forward and tackled Umar Farouk Abdulmutallab as he was attempting to detonate 80 grams of PETN, the active ingredient in plastic explosives.

    The passengers on these flights are lauded as heroes, but their actions have not been acknowledged by policy changes in the TSA and other security agencies around the world. On any flight that is the victim of a terrorist attack or hijacking, the vast majority of the passengers on the flight are capable “good-guys” that are also highly motivated to prevent the attack from succeeding. Yet the TSA continues to treat passengers as self-loading cargo that may harbor a terrorist.

    In the online world we have been using the wisdom of crowds for years to determine what is good and what is bad. Google uses it to filter spam by having the millions of GMail users report spam messages and filtering out messages from sources reported frequently. My company, Feedjit, uses the wisdom of more than a million visitors monthly to filter out adult content and web spam from our site. Reddit, Slashdot.org, Digg, SumbleUpon and many other websites use crowd wisdom to expose the best of the web and to filter out malicious attempts to game the system.

    Crowds are smart and most of the people in a crowd are good and are willing to help. Airline security agencies can harness this wisdom and this willingness to help by first acknowledging that it exists. Then implementing a few simple policy changes.

    1. Give passengers a way to quickly and discreetly report suspicious activity to officials, both before boarding and during the flight. A button that simply reports suspicious activity within a 3 passenger radius may be all that’s needed.
    2. Give crew a way to constantly monitor an aggregation of these reports in real-time and develop response policies depending on the severity or intensity of the reports about an individual.
    3. Encourage passengers to engage their fellow passengers in conversation, but not in the context of preventing terrorism. For example: Include a simple statement during the pre-flight briefing of: “Northwest airlines passengers are known for their friendliness in the skies and many of us traveling to and from far off destinations. We encourage our passengers to compare travel notes and to get to know each other….etc..etc…”

    With these simple changes, you have now created an on-board distributed intelligence gathering network that is reporting back to a central authority in real-time.

    With these changes the recent Christmass day attack may have played out as follows:

    1. A passenger sitting next to Umar Farouk Abdulmutallab tries to engage him in polite conversation and discoveres he can’t speak a word of english.
    2. The passenger has the opportunity to get a good look at Umar and notices he is sweating and his hands are shaking.
    3. The passenger discreetly hits a button in their arm-rest alerting flight attendants to something suspicious.
    4. The passenger catches the eye of a woman in the isle seat and she notices it too and hits her button.
    5. Flight attendants come over and ask Umar if he’s feeling OK and would like a drink of water.
    6. Flight attendants verify that Umar is exhibiting symptoms of someone who is under extreme stress.
    7. Flight attendants start a protocol of closely monitoring the passenger’s behavior by always having someone within 15 ft.
    8. When Umar starts to rise to go to the toilet, flight attendants request that he come to the back of the plane so that they can do a cursory body search before he enters the toilet.
    9. Flight attendants discover the PETN and the story has a very different ending.

    The TSA can start by acknowledging that a vast untapped human intelligence resource exists on every flight. Then change their policy approach from “every passenger could be a terrorist” to “Only one in a million passengers are a potential terrorist and everyone else on the flight is smart, capable and highly motivated to prevent terrorism on their flight.”

  • How to handle 1000's of concurrent users on a 360MB VPS

    There has been some recent confusion about how much memory you need in a web server to handle a huge number of concurrent requests. I also made a performance claim on the STS list that got me an unusual number of private emails.

    Here’s how you run a highly concurrent website on a shoe-string budget:

    The first thing you’ll do is get a Linode server because they have the fastest CPU and disk.

    Install Apache with your web application running under mod_php, mod_perl or some other persistence engine for your language. Then you get famous and start getting emails about people not being able to access your website.

    You increase the number of Apache threads or processes (depending on which Apache MPM you’re using) until you can’t anymore because you only have 360MB of memory in your server.

    Then you’ll lower the KeepaliveTimeout and eventually disable Keepalive so that more users can access your website without tying up your Apache processes. Your users will slow down a little because they now have to re-establish a new connection for every piece of your website they want to fetch, but you’ll be able to serve more of them.

    But as you scale up you will get a few more emails about your server being down. Even though  you’ve disabled keepalive it still takes time for each Apache child to send data to users, especially if they’re on slow connections or connections with high latency. Here’s what you do next:

    Install Nginx on your new Linode box and get it to listen on Port 80. Then reconfigure Apache so that it listens on another port – say port 81 – and can only be accessed from the local machine. Configure Nginx as a reverse proxy to Apache listening on port 81 so that it sits in front of Apache like so:

    YourVisitor <—–> Nginx:Port80 <—–> Apache:Port81

    Enable Keepalive on Nginx and set the Keepalive timeout as high as you’d like. Disable Keepalive on Apache – this is just-in-case because Nginx’s proxy engine doesn’t support Keepalive to the back-end servers anyway.

    The 10 or so Apache children you’re running will be getting requests from a client (Nginx) that is running locally. Because there is zero latency and a huge amount of bandwidth (it’s a loopback request), the only time Apache takes to handle the request is the amount of CPU time it actually takes to handle the request. Apache children are no longer tied up with clients on slow connections. So each request is handled in a few microseconds, freeing up each child to do a hell of a lot more work.

    Nginx will occupy about 5 to 10 Megs of Memory. You’ll see thousands of users concurrently connected to it. If you have Munin loaded on your server check out the netstat graph. Bitchin isn’t it? You’ll also notice that Nginx uses very little CPU – almost nothing in fact. That’s because Nginx is designed using a single threaded model where one thread handles a huge number of connections. It can do this with little CPU usage because it uses a feature in the Linux kernel called epoll().

    Footnotes:

    Lack of time forced me to leave out all explanations on how to install and configure Nginx (I’m assuming you know Apache already) – but the Nginx Wiki is excellent, even if the Russain translation is a little rough.

    I’ve also purposely left out all references to solving disk bottlenecks (as I’ve left out a discussion about browser caching) because there has been a lot written about this and depending on what app or app-server you’re running, there are some very standard ways to solve IO problems already. e.g. Memcached, the InnoDB cache for MySQL, PHP’s Alternative PHP Cache, perstence engines that keep your compiled code in memory, etc..etc..

    This technique works to speed up any back-end application server that uses a one-thread-per-connection model. It doesn’t matter if it’s Ruby via FastCGI, Mod_Perl on Apache or some crappy little Bash script spitting out data on a socket.

    This is a very standard config for most high traffic websites today. It’s how they are able to leave keepalive enabled and handle a huge number of concurrent users with a relatively small app server cluster.  Lighttpd and Nginx are the two most popular free FSM/epoll web servers out there and Nginx is the fastest growing, best designed (IMHO) and the one I use to serve 400 requests per second on a small Apache cluster. It’s also what guys like WordPress.com use.

  • Fly Fishing on lake Samm

    My neighbor John Winkler just emailed this to me. He took this while I was fly fishing off our dock on Lake Sammamish earlier this year.

    meFlyfishingOnLakeSamm

  • A legend just died

    About a year ago I received two old servers from Snapvine my former neighbors (now acquired by Whitepages). Snapvine got them from another startup in Seattle, I don’t recall which. I named them Rex1 and Rex2 and put them to work immediately. The graph below shows the disk activity during the last year on Rex1:

    rex1-cpu-year

    The pink stuff in the graph is the amount of time the CPU spends waiting for the disk – showing disk activity. The application on Rex1 is one of the processes that produces the Feedjit Geoblogosphere (what people are reading in your city).

    Well because Rex1 has such a long history with Seattle Startups and has give us such a kick-ass run, I feel it’s only appropriate that I give Rex1’s first disk fail a fitting eulogy. We’ll miss you buddy!

    [Yeah that’s right, I’m killing time on this blog entry waiting for Rex1 to come back up so I can copy the data over to Rex2]

  • Spooky fun with ssh on OS X

    Want to freak out your wife/husband/kids?

    On a mac that you have access to:

    Go to ‘System Preferences’. Under ‘Internet & Networking’ there is a ‘Sharing’ icon. Run that. In the list that appears, check the ‘Remote Login’ option.

    Then ssh into your mac remotely by downloading putty if you’re on a pc or launch a terminal on another mac and run “ssh username@ip.address” without quotes to ssh to the mac while someone is working on it.

    Once you’re logged in:

    Crank up the volume by running:
    sudo osascript -e "set volume 10"

    Then run:

    sudo osascript -e 'say "I am watching you." using "Zarvox"

    Or if that doesn’t work, try:

    sudo osascript -e 'say "I am watching you." using "Cellos"'

    Make sure you have an automatic emergency defibrillator handy.

  • Why we breathe

    Free Diver in LimasolHold your breath for a moment.

    In about 10 to 30 seconds you’ll be feeling a strong desire to take a breath. That’s not caused by lack of oxygen. It’s caused by excess carbon dioxide buildup in your blood.

    [Ok you can breathe again.]

    The trigger in mammals that causes us to want to take a breath is an excess buildup of CO2. In reptiles the trigger is lack of O2. Free divers don’t hyperventilate to get more O2 into their bloodstream. They do it to to flush out excess CO2 and remove that breathing trigger. That’s also what causes shallow water black-out as you’re surfacing, so don’t try it without a buddy.

    I’ve worked in more startups than I care to count where the lack of endurance was not caused by lack of oxygen, but an excess buildup of waste. Getting a larger office, buying excess server capacity early on that isn’t needed, hiring excess people to manage that server capacity, hiring managers to manage the people, hiring an ad agency and PR firm and a small team to manage them.

    Once you start down the path of waste you may still have enough oxygen in your bloodstream to surface, but the excess CO2 in your business creates a strong demand for more Oxygen which causes you to raise another round of funding, producing more CO2 and the cycle continues.

    So start your business by hyperventilating to flush out all excess CO2, take a deep breath and beware of shallow water blackout as you’re approaching the surface.

    [Photo credit: My good friend Bruno Stichini who hosted a free diving world record attempt in Limasol, Cyprus back in 2000]