Month: February 2015

  • Try Buying your Hardware

    We took a lot of heat from the startup community when we bought $40,000 of Dell servers, a switch and a KVM and racked them ourselves in 2008. Seriously, Kerry (my wife and co-founder) and I hand-racked about 10 Dell 2950’s and a couple of 1950’s in the rack we leased at our data center. We didn’t realize the DC team could rack them for us and were so excited when the servers arrived we just dove right in.

    Do you have any idea how much a DELL 2950 loaded with disks weighs? They’re heavy.

    At that time the “cloud” was all the rage. Amazon services were really spinning up, Linode and SliceHost were the new ‘it’ companies, and we were derided as idiots for actually buying physical hardware: Ew!

    Well turns out our business scaled very quickly and in a few short months we were pushing well over 100 megabits of bandwidth average. We were paying around $2,500 a month for that which included power to the rack, a team supporting our hardware 24/7/365 and that included the bandwidth and 5 very high quality upstream connections. We’d discovered the magic of 95th percentile billing. Most of our peers were paying by the terrabyte and getting absolutely screwed. Our business would never have survived if we didn’t use colocation.

    Today we’re busy decomissioning our old Dell 1950’s and 2950’s and replacing them with amazing new Dell R630’s. Back then we were paying about $3500 per server. I just bought 4 Dell R630’s at $9250 each out the door. We’re happy to spend that kind of cash because we know these machines will pay for themselves a hundred times over (or more) by the time we’re done with them. We have a little inside joke: “Good servers go to small business heaven. Bad servers end up working for us.” We literally put our servers through hell by running them at very high CPU and IO loads. To date we haven’t had a single failure besides hard drives and redundant power supplies, all of which are hot-swappable and no big deal. No memory, chassis or controller issues. (We use PERC hardware RAID 1 or 10 usually)

    So I guess I’d like to say a big Kudos to Dell for producing some kick-ass enterprise class hardware that could withstand the worst kinds of loads we could come up with. And seriously: If you’re a startup and can afford it, consider making a capital investment in your own hardware and using colocation rather than abstracting away the problem and paying more – and in some cases, a hell of a lot more.

    Besides: What could be more fun that spending your Saturday night in the data center.

  • Installing Ubuntu 14.10 on a Dell R630 with PERC H730 hardware RAID 10

    If you arrived here, you’re probably spending your weekend doing this too, so perhaps I can save some of your weekend for you. Here’s how I did it. FYI, I’m using a PERC H730 hardware RAID controller with a 1.1TB virtual disk made up of 8 physical disks in RAID10 config. As the title says, this is a fresh DELL R630 and it has single processor and 128G of memory. See my notes below about using a 100g boot partition and creating a larger partition once you have the system up and running with grub installed in the MBR.

    • Switch the BIOS boot mode from UEFI to BIOS.
    • If you’re booting from a USB thumb drive, set that to your first boot device.
    • Boot and hit CTRL-R to go into your raid controllers bios and blow away the virtual disk. Recreate a new identical one. You’re doing this to get rid of the GPT partition.
    • Boot into ubuntu 14.10 server 64 bit.
    • Go through installation and make sure you install openssh server because you won’t be able to access the console when you first boot.
    • Also make sure that when you partition your disk, you don’t create one huge partition larger than a terabyte. Instead, you probably want to create a boot partition and then a larger partition. I use 100G boot partition and 1TB big partition which I create once I have the system up and running. When I tried to create a 1.1TB partition it has trouble installing grub into the MBR. Using UEFI or a GPT partition table might fix this but I haven’t gone down that rabbit hole and don’t really want to.
    • The grub installation onto the MBR will fail. This is because if you’re installing from thumb USB, ubuntu switches the /dev/sda and /dev/sdb devices and tries to install grub onto your thumb drive instead of your hard drive. To fix this hit CTRL-ALT-F2 open a console, then run the following:
    • chroot /target
    • grub-install /dev/sdb
    • update-grub
    • Then hit CTRL-ALT-F1 and go back to your installation.
    • Continue without installing a boot loader (because that’s what you just did).
    • Once done, when you reboot, go back into the bios and disable booting from your thumb drive (or just unplug it if you’re not doing this remotely like I am).
    • Boot into linux, except that all you’ll see is a blank screen at this point.
    • SSH into the server.
    • Edit /etc/default/grub
    • Change the value of GRUB_CMDLINE_LINUX_DEFAULT to be “vga=normal nofb nomodeset video=vesafb:off i915.modeset=0”.
    • Run update-grub2
    • Reboot and your console should now work and also won’t freeze up.

    Congrats, I just saved you a few hours. Go enjoy them.