Your Vision May be Clouded

Posted on October 31, 2011 by mark

I took a lot of crap when I decided to vertically integrate our business four years ago and I invested around $40,000 with Dell to buy our own server cluster. Right then THE CLOUD was the hot new thing, and still is and I was not getting on board. I leased a rack at a respectable Seattle based hosting facility and my wife added the ability to unbox and rack Dell 2950 servers to her long list of talents. The hosting facility team would have done it for us, but we like to get our hands dirty.

That was the most work we did to set up our own server cluster. Four years later we have a 99.9% uptime record and we run a profitable company with an ad network, real-time analytics product and a free virally distributed service off our cluster of 20 machines. When we mail our customers we send over half a million emails in less than 24 hours off our own email server. We serve between 400 and 800 application requests per second all day long.

During the last four years I’ve watched friends and acquaintances get burned by the cloud either due to down time or cost. We pay $3400 per month to host our 20 dedicated machines in a single rack. We have a gigabit connection to the Net and our average bandwidth throughput is around 125 megabits per second constantly.

I’m tired of the Wired Magazine crowd giving me crap for not “being in the cloud” or “getting with the cloud” or whatever. So I’m throwing this down: During the last 4 years I’ve had 99.9% uptime and I’ve spent a total of $190,000 during those 4 years on hosting, which includes the capital investment in the servers. We’ve had a constant throughput of 80 to 120 megabits per second (increasing over time) and roughly 40% avg CPU usage on 20 dual CPU machines (with dedicated Intel E5410 CPU’s each with 4 cores). As I mentioned we do 400 to 800 app requests per second and we also have an average of 25,000 concurrent connections on our front-end server. I’ll bet anyone who reads this a beer that you won’t find a cloud provider who can do this for you for less than 3X what I’ve paid. [That works out to $3,958 per month.]

If you think having your own dedicated servers in a colocation facility ties you geographically to one place, it doesn’t. I work wherever I want. For 3 out of the last 4 years I was in Seattle. The last year I’ve been in Colorado. I spent 3 weeks in France this month and while I was there I diagnosed a failing drive in one of our servers, ordered the replacement from Dell which will arrive today and be racked by the support team at our hosting facility. We’ve done hardware replacements or upgrades like this many times, including ordering new servers, upgrading memory, upgrading Ubuntu versions and it’s no big deal. A local support person with an anti-static strap and a basic knowledge of linux shell commands can resolve 99% of issues that come up.

I encourage everyone reading this to challenge the marketing hype around THE CLOUD. Go to Dell’s site, get a feel for price/performance, call your local colo provider and get prices on a full rack with a gigabit connection. You will almost certainly be surprised at the bang you’ll get for your buck and how easy it is to manage your own physical machines.

Understand that THE CLOUD exists as a buzzword to help software companies sell more software as a service. It’s sad when software startups who should be using the buzzword to sell more service get taken in by the marketing and outsource their core infrastructure.

Working culture differences between France and the USA

Posted on October 14, 2011 by mark

I’m startled at the differences between French and USA working culture. I’m not going to wax judgemental on this because I love many aspects of both French and American culture, so I’ll share my experience as an observer:

In the USA I spent much of my time during the last year in a very small semi-rural town called Elizabeth Colorado with a population of 1500 according to Google. Elbert County that contains Elizabeth has probably around 5000 people. Elizabeth recently got a local Walmart which is basically a giant warehouse in the middle of a field surrounded by sparse houses and horses.

The Walmart in Elizabeth is open 24 hours. Being in Colorado and just under 7000 ft, Elizabeth gets some gnarly blizzards, the kind with driving snow and road cracking ice that layers up nice and thick. That Walmart stays open even in the worst weather. I’ve been down there at 3am in a blizzard with fresh laid snow covering the ground, using 4×4 to get in and out of the parking lot.

I’m currently in Bordeaux in France and I’m helping to renovate an old building that was built during the US civil war. Pretty much all buildings in this town, the town of Blaye in the Aquitane region, were built during that time.

We needed a few tools – chainsaw sharpener, wrench, curtain rail etc and headed down to the hardware store today. We spent around 40 minutes shopping, talking to folks and getting help, shopping some more. The next thing the manager pokes his head round an aisle and says they’re closing. So we start walking towards the checkout and the lights go out. All the checkouts are closed. The entire staff is heading out a side door and the front of the building has it’s shutters down. We’re almost the last people in the building. They indicate we need to drop what we’re carrying and leave and come back 2 hours later when they reopen after lunch.

So we beg a little and the manager waves to someone in the parking lot. A rather huffy woman stomps back inside and makes sighing noises as she checks us out because she counts on having her 2 hour lunchbreak to get (I’m assuming) chores done and meet with people etc. Once we’re checked out we’re hurriedly ushered out a side door and into the parking lot with our winnings.

I don’t actually have a problem with this. While we were in the store and they were open our shopping experience was great and the store attendants were very helpful considering how awful my french is. But don’t mess with the lunch break.

What ties in with the lunch break window is lunch at restaurants in France is between 12 and 2pm and I dare you to try and have lunch at any restaurant (outside a big city) before or after 12 to 2pm. I sat down at 11:45 at a sidewalk pizza joint 2 days ago while the place was open and the chef was having a beer with a friend. He came outside and told me I’m defying the laws of physics because “It’s not possible” and I need to come back later. He was very polite and now that I get it, I totally understand.

Be Right Back – with a solution to your problems

Posted on October 7, 2011 by mark

Big News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

I’m driving through Europe for the next week so am not very available online. Currently in awesome Andorra. So if I’m not very responsive to your WordPress security needs I apologize. However, I have been working on a permanent fix to help with the recent spate of WP hacks and will be making it available as soon as I’m back at work. So apologies for the delay and thanks for your patience.

Blogspot security hole lets others view unpublished blog entries

Posted on October 5, 2011 by mark

If you use the “new” Blogspot interface, try saving a draft post and then previewing it. You’ll see you’re previewing it using a link that looks something like:

http://example.blogspot.com/b/post-preview?token=<long-encoded-string>&postId=532662776851272&type=POST

One would assume that Blogspot knows you are “you” because you are signed in to Blogspot and it identifies you using a cookie. So it shouldn’t be possible for anyone else on the Net to view your unpublished blog post right? Wrong.

If anyone else sees that link with the token, they simply paste it into their browser and they can view your unpublished blog entry.

I thought that this would present a problem if you’re previewing a blog entry and you click an external link you included in your blog post. This would send your “preview” URL to that external site as a referrer, which means that external site can access your unpublished blog entry. Google seem to have anticipated this and they prevent you from clicking anything on the page by putting a DOM overlay on the preview page.

However, all widgets on the page load. That means if you’re using a Facebook Like button, a stats widget, popular pages widget or any other external javascript widget that logs the current URL (pretty much all of them) then those external sites can access your unpublished blog entry. More importantly anyone who those external sites share the URL with can access your unpublished blog entry.

I thought perhaps if I logged out of Blogspot, that security token would expire and no one would be able to access my unpublished blog entry. Nope. It’s still accessible.

Providing security through the hope that no one will discover a hidden URL in a world where URL sharing has become a deep part of everything we do is not a good design. Blogspot team please fix this. Thanks.