Jenny

I usually have these conversations with one of my two genius musician friends, but I feel like blogging about this one. If you aren’t into heavy progressive rock, tune out now.

So Octane (SiruisXM) has started playing ‘Jenny’ by Nothing More – guess they got around to releasing track 15 off their self titled 2014 album as a single.

I have a love hate relationship with this band. “This is the Time” is incredibly innovative and risky – Mr MTV is a disaster, and so we go round.

Jenny is pretty amazing. It starts soft, changes tempo during the song but stays incredibly tight throughout, has the complexity that Nothing More is known for and yet has shades of 80’s hair metal in the chorus.

What I love about this band is whenever I’m done listening to a track I feel like they’ve brought me up to date with where progressive rock (or industrial alternative prog metal if you want to nitpick) is – and needs to be headed. They get bonus points for touring the country in beat up old vans hawking their sound since 2004.

Here’s ‘Jenny’ by Nothing More:

 

 

Failure Is Not An Option

If you raise money and fail, you need to consider the opportunity cost of another entrepreneur not having had access to the investment capital you lost. If you fail, you need to be sad about your failure and also be sad about the opportunity cost of your failure.

But it’s “risk capital” you say – money that investment funds allocated to very high-risk/high-return investments. So the thinking is that it’s OK for that capital to go away because it’s expected to either succeed big and likely to fail. But what about the 1 or 10 or 50 other businesses that lost access to that capital once it was invested? Could one of them have been the next Google?

Try Buying your Hardware

We took a lot of heat from the startup community when we bought $40,000 of Dell servers, a switch and a KVM and racked them ourselves in 2008. Seriously, Kerry (my wife and co-founder) and I hand-racked about 10 Dell 2950’s and a couple of 1950’s in the rack we leased at our data center. We didn’t realize the DC team could rack them for us and were so excited when the servers arrived we just dove right in.

Do you have any idea how much a DELL 2950 loaded with disks weighs? They’re heavy.

At that time the “cloud” was all the rage. Amazon services were really spinning up, Linode and SliceHost were the new ‘it’ companies, and we were derided as idiots for actually buying physical hardware: Ew!

Well turns out our business scaled very quickly and in a few short months we were pushing well over 100 megabits of bandwidth average. We were paying around $2,500 a month for that which included power to the rack, a team supporting our hardware 24/7/365 and that included the bandwidth and 5 very high quality upstream connections. We’d discovered the magic of 95th percentile billing. Most of our peers were paying by the terrabyte and getting absolutely screwed. Our business would never have survived if we didn’t use colocation.

Today we’re busy decomissioning our old Dell 1950’s and 2950’s and replacing them with amazing new Dell R630’s. Back then we were paying about $3500 per server. I just bought 4 Dell R630’s at $9250 each out the door. We’re happy to spend that kind of cash because we know these machines will pay for themselves a hundred times over (or more) by the time we’re done with them. We have a little inside joke: “Good servers go to small business heaven. Bad servers end up working for us.” We literally put our servers through hell by running them at very high CPU and IO loads. To date we haven’t had a single failure besides hard drives and redundant power supplies, all of which are hot-swappable and no big deal. No memory, chassis or controller issues. (We use PERC hardware RAID 1 or 10 usually)

So I guess I’d like to say a big Kudos to Dell for producing some kick-ass enterprise class hardware that could withstand the worst kinds of loads we could come up with. And seriously: If you’re a startup and can afford it, consider making a capital investment in your own hardware and using colocation rather than abstracting away the problem and paying more – and in some cases, a hell of a lot more.

Besides: What could be more fun that spending your Saturday night in the data center.

Installing Ubuntu 14.10 on a Dell R630 with PERC H730 hardware RAID 10

If you arrived here, you’re probably spending your weekend doing this too, so perhaps I can save some of your weekend for you. Here’s how I did it. FYI, I’m using a PERC H730 hardware RAID controller with a 1.1TB virtual disk made up of 8 physical disks in RAID10 config. As the title says, this is a fresh DELL R630 and it has single processor and 128G of memory. See my notes below about using a 100g boot partition and creating a larger partition once you have the system up and running with grub installed in the MBR.

  • Switch the BIOS boot mode from UEFI to BIOS.
  • If you’re booting from a USB thumb drive, set that to your first boot device.
  • Boot and hit CTRL-R to go into your raid controllers bios and blow away the virtual disk. Recreate a new identical one. You’re doing this to get rid of the GPT partition.
  • Boot into ubuntu 14.10 server 64 bit.
  • Go through installation and make sure you install openssh server because you won’t be able to access the console when you first boot.
  • Also make sure that when you partition your disk, you don’t create one huge partition larger than a terabyte. Instead, you probably want to create a boot partition and then a larger partition. I use 100G boot partition and 1TB big partition which I create once I have the system up and running. When I tried to create a 1.1TB partition it has trouble installing grub into the MBR. Using UEFI or a GPT partition table might fix this but I haven’t gone down that rabbit hole and don’t really want to.
  • The grub installation onto the MBR will fail. This is because if you’re installing from thumb USB, ubuntu switches the /dev/sda and /dev/sdb devices and tries to install grub onto your thumb drive instead of your hard drive. To fix this hit CTRL-ALT-F2 open a console, then run the following:
  • chroot /target
  • grub-install /dev/sdb
  • update-grub
  • Then hit CTRL-ALT-F1 and go back to your installation.
  • Continue without installing a boot loader (because that’s what you just did).
  • Once done, when you reboot, go back into the bios and disable booting from your thumb drive (or just unplug it if you’re not doing this remotely like I am).
  • Boot into linux, except that all you’ll see is a blank screen at this point.
  • SSH into the server.
  • Edit /etc/default/grub
  • Change the value of GRUB_CMDLINE_LINUX_DEFAULT to be “vga=normal nofb nomodeset video=vesafb:off i915.modeset=0″.
  • Run update-grub2
  • Reboot and your console should now work and also won’t freeze up.

Congrats, I just saved you a few hours. Go enjoy them.

OS X 10.10 Yosemite WiFi Problems Analyzed with Wireshark

I never realized how often I google and how much I rely on sub-second response times until I upgraded my Macbook Pro to OS X Yosemite. After muddling through issues like upgrading VMWare and a few other items and fixing my terminal emulation, I couldn’t figure out why I was in such a bad mood.

Then it hit me. My Google searches while I had been doing that had been slow. I would type something in and Google’s search results page either would not appear for about 3 to 6 seconds, or it would half-appear and then the search results would only show up after 3 to 6 seconds.

There is so much garbage SEO bait out there about “what to do about Yosemite wifi problems” so I’m not going to bore you with the details of my investigation and I’m just going to cut straight to the chase:

I put a network analyzer on my wifi. It turns out that the problem appears to be duplicate packets arriving on the WiFi network card. I switched to Ethernet via the Thunderbolt adapter and the problems instantly went away.

Here’s what it looks like in Wireshark….

Screen Shot 2014-11-13 at 2.21.31 PM

What happens is the network card transmits an acknowledgement. Then there’s a 2.7 second freeze where nothing happens. And then a few packets arrive followed by a flood of duplicate packets.

The duplicates are both duplicate application data packets along with duplicate TCP acknowledgements.

Scrolling further down you can see the duplicates increase and Wireshark starts labeling them “TCP Spurious Retransmission”, implying an issue with a network interface on the network.

Screen Shot 2014-11-13 at 2.23.50 PM

Another test shows exactly the same thing. A 3.1 second delay where I’ve highlighted in blue and then a few good packets and the duplicates start.

Screen Shot 2014-11-13 at 2.30.26 PM

And then the frequency increases…

Screen Shot 2014-11-13 at 2.31.38 PM

Deleting and re-adding your wifi network or network card device does not fix this. Neither does some of the other suggestions out there like turning off bluetooth, joining a 2.4 Ghz network instead of 5Ghz, etc…etc..

To me this seems to be a driver issue where the network card freezes and when it comes out of the freeze it’s sending the OS large numbers of duplicate packets. It’s curious that the freeze is around 3 seconds each time.

This test was done on a: MacBook Pro (Retina, 15-inch, Early 2013). The only other software running while this test was being done was Chrome, Excel, X11, Wireshark, Terminal and Keyboard Maestro (a keyboard macro utility).

Writing this post after the test was done on ethernet and I can feel my sanity already returning.

Apple please fix. Thanks.

 

Startups that Move the Needle

Something that I’m becoming more cognizant of and that I see in my friends as we all get a little older is the question about whether what we’re doing is actually moving the needle for the rest of humanity. If it’s making positive change by enabling our species or improving quality of life for others.

My business is cybersecurity and the biggest positive impact I see is when we help mom and pop or small businesses keep their websites and businesses secure. But I question whether we can do more. I think Elon’s SpaceX and Tesla moves things forward for our species as a whole.

An old friend arrived in Seattle this weekend. He has a really exciting startup based in Europe and is one of the most persuasive and energetic guys I know. It’s his second or third time in Seattle, ever – he doesn’t even live in this country – and  we show up at the Black Keys concert, sold out show on Saturday night at Key Arena, he walks up to security and talks us into a sold out show without any bribes or cash changing hands.

So in between rocking out to Black Keys and then hitting a Bollywood party in Freemont, I learned about what he’s been doing for the last few years.

Oradian creates software for banks in developing countries to do what banks do. Most of their target market is either using paper or using antiquated systems that are cobbled together and run on an old PC or laptop. Oradian provides a cloud based core banking system that gives banks a way to drop in an IT solution and get up and running fast.

My first thought was skepticism that a bank in a developing country would have access to the Net. But Antonio has been on the ground selling directly into these organizations and markets for a few years (he was previously in micro-finance) and because of the heavy reliance on cellphones in these markets, the Internet is more reliable than the power grid.

They’re currently raising series A in the USA and Europe and it’s interesting hearing his perspective and seeing other companies that are raising in Seattle and the Valley. I think there are other exciting businesses out there that are moving things forward, but there are so many that are spending precious energy on attracting a few more clicks or a few more eyeballs and I’m not sure how they help make the World a better place.

It’s gotten me thinking about how we measure success and gauge whether something is a great idea or not. I’m not sure I’ve ever seen a startup appear on the West Coast that has a for-profit model that has the potential to make positive change in developing countries. I grew up in South Africa (as did Antonio, Oradian’s CEO) and we’ve seen and continue to see first hand how important it is to create a strong middle class in developing countries that is empowered with commercial opportunities and the services that surround them in the form of banking.

I’d like to see more smart people thinking about this space and if Oradian is anything to go by, my sense is that there are opportunities in the developing World that can be both profitable for investors and make significant positive change.

Edit: Found this video which gives you a better idea of what Oradian does…

4th of July Post

Posted this on Facebook today and felt like cross posting it here.

I feel obliged to post this after seen all the posts in my timeline connecting patriotism with the US military. There are ways to express love for your country without expressing a love for war or the machine that wages war.

Omitting an expression of support for your country’s military is not unpatriotic. Neither is criticizing it. The last three decades have seen the US at war in Libya, Grenada, Panama, Iraq, Somalia, Bosnia, Haiti, Kosovo, Afghanistan, Iraq (again) and Libya (again). On what’s left of this independence day weekend, consider that citizens of other countries are patriotic too. Try to remember that we’re part of a global whole and every citizen of Earth has fears, hopes and dreams and they too are proud of their history and would prefer that it remain intact.

Consider that the idea that we keep American families working on peaceful private enterprise on US soil instead of dividing them through military deployment is also a patriotic goal.

Remember that a quarter of world military spending is what we spend on our own war machine.

There will always be evil in the world and fighting evil will always create jobs and new wealth and those jobs and that wealth are missed when they’re gone. But at what cost do we go looking for new wars? At what cost do we glorify the military industrial complex as part of what makes us American?

On what remains of this fourth of July weekend, remember that old maxim: That you should treat others the way you want to be treated. And lets instead celebrate our open culture, our freedom of speech and our freedom to choose who governs us, whether they wage war and how they treat others on our behalf.

Liars and Geniuses – Thoughts on Live Jazz

There’s something about Jazz live performance that has bothers me and I think it’s the audience. It’s the beatific smiles on many of the faces that last through the entire performance – smiles that remind me of a congregation in a church that know that it’s the wanting to believe that matters most, not whether it’s true.

It’s the guy in the front row with his index finger at shoulder height pointed at the roof bouncing it back and forth to a rhythm all his own.

Jazz performance appreciation – to truly understand live jazz greatness when you see it in the flesh – is the epitome of musical achievement. To understand how a group of musicians anticipate each other’s switching from one complex time signature to another, move fluidly and rapidly between keys and throw in a little used mode to add some humor or a chromatic run which morphs into another key – or to understand when the musicians are reverting to a jazz standard or improvising something new and truly great – to understand all of this, you have to be an accomplished musician. Someone who has spent thousands of hours either studying or performing or listening.

I think those that claim live jazz appreciation are either liars or geniuses.

I listen to Rock.

Where the term “Zero Day” comes from

After seeing a FOIA request earlier today that someone created asking for FBI training documents that teach staff how to understand/communicate using hacker leet-speak, I was reminded about something I’ve wanted to put in virtual ink for a while.

Leet speak or 133+ sp34k or hacker speak did not actually originate with hackers. Neither did the term “zero day”. Back in the late 80’s and early 90’s the Internet was but a pup and most of us communicated via BBS – a dialup modem (often a bank of modems on popular BBS’s) attached to an individual’s PC that members dialed into to connect. [We also used something called Prestel or Beltel which was essentially a big government run BBS]

Phone calls were expensive in those days so if you wanted to connect to BBS’s far away – and at the time I lived in South Africa and the best BBS’s were in the USA – then you needed to become a phone phreak. So I’d fire up a piece of DOS software called Bluebeep (created by the venerable Onkel Dietelmeyer), hold a headphone to a phone mouthpiece, generate CCITT5 tones and take control of international phone trunks to get free overseas phone calls. [I’d also hack into the Post Office X.25 network to get access to an overseas modem (a DTE) which I could then control with AT commands. X.25 was a precursor to the TCP/IP Internet]

Then once you’re connected to the BBS you could upload, download, send email via fidonet and talk to anyone else online. My favorite hacker BBS was in Orange County, Californa called Digital Decay and run by a chap called Arclight. Little did I know that my future wife was also in Orange County busy being a college kid.

At the time underground BBS’s were divided into two types: Those that were hacking and phreaking related and those run by the warez crowd. The hacker/phreak BBS’s would distribute exploits, tools like Bluebeep to hack the phone lines, copies of phrack and so on.

The warez crowd would distribute pirated software and they took their job very seriously. 133+ speak originated with the warez crowd and the hacking/phreaking crowd hated it.

The term Zero Day also originated in the warez scene. On warez BBS’s software would be divided into zero day, 1 to 7 day, 8 to 14 day and so on with the lower ‘day’ being the most elite and hardest to come by. The number of ‘days’ was the days since the software was released to the public and Zero Day was software that was not commercially released yet. So someone had hacked into the company servers to grab their commercial software before they released it.

The relationship between the warez crowd and the hacking/phreaking scene was that the hackers would invent the means to get zero day warez (exploits used to hack into a company), the phreaks invented and continually reinvented the means for ‘warez couriers’ distribute the warez among BBS’s (ways to circumvent trunk seize tone filters the phone companies were using for example). Hackers and phreaks looked down on the warez crowd – even though we’d get software from them – which was a little hypocritical.

This was all around 25 years ago. At some point ‘zero day’ became something applied to vulnerabilities and the number of days a vendor has had to fix them. And at some point ‘133t sp34k’ became something hackers use. I have no idea why or when this transition occurred. 133t sp34k used to be scorned by hackers as something warez ‘pups’ did.

Times change. Like hashtags originating on Twitter where they indicated subject, which originated on IRC where they were channels – and which syntactically may have been inspired by C preprocessor directives.

Edit: Very cool discussion thread on HN about this – including some other old-schoolers checking in.

The Qantas A380 Engine Failure: The story of an ops team pulling through a crisis

[Thanks to ‘evanm’ on Hacker News, here’s a link to the full documentary if you don’t want the abbreviated versions below.]

As a low hours pilot I spend a lot of time reading NTSB reports and postmortems on flight incidents to try and avoid repeating the mistakes others have made or learn about how they successfully dealt with problems encountered during flight.

One of the most impressive stories I read about fairly recently was the Qantas A380 engine failure. I have a lot of respect for pilots who deal with crises successfully and for me Sullenberger’s landing in the Hudson has always been my favorite and most heroic story. Even though Sullenberger was facing a high workload during his engine failure, he focused on flying the plane and did a textbook water landing.

But what fascinates me about the Qantas A380 failure is it’s more about the team pulling through, especially the first officer. After the engine fails the first officer is confronted with a monster list of system failures that the avionics dumps on him and he has to very quickly work through each item. They eventually gain an understanding of the aircraft status and come up with an operational plan to put it back on the ground.

As someone who writes software for a living and used to be an operations guy, for me the Qantas A380 engine failure is the story of an ops team facing tremendous pressure with zero option for failure and incredible time pressure, and transforming a severely damaged system into something operational again in order to successfully shut it down.

If you have the time, I’d recommend watching this two part documentary which captures most of the story. You can tell from the first officer’s recounting of the incident how much pressure he was under at the time.