Author: mark

  • Guy's house cleaned out by ad on Craigslist

    This is too much. A guy in Jacksonville, Oregon had his house cleaned out thanks to a malicious ad placed on Craigslist. Someone posted an ad saying that the house had been declared abandoned and all the belongings including a horse were free to good homes. So the entire neighborhood rocked up and started carting stuff off. When the guy arrived the looters were armed printouts of the ad and refused to hand the guys stuff over. From the Seattle Times:

    The independent contractor was at Emigrant Lake when he got a call from a woman who had stopped by his house to claim his horse.

    On his way home he stopped a truck loaded down with his work ladders, lawn mower and weed eater.

    “I informed them I was the owner, but they refused to give the stuff back,” Salisbury said. “They showed me the Craigslist printout and told me they had the right to do what they did.”

    The driver sped away after rebuking Salisbury. On his way home he spotted other cars filled with his belongings.

    Once home he was greeted by close to 30 people rummaging through his barn and front porch.

    The trespassers, armed with printouts of the ad, tried to brush him off. “They honestly thought that because it appeared on the Internet it was true,” Salisbury said. “It boggles the mind.”

    Full article here.

  • Obamagirl's latest

    Nice touch getting Bill to play the sax. šŸ™‚ [obamagirl video below] And in case you haven’t seen it, check out Hillary’s latest faux pas on youtube today. And now I need to go flagellate myself for an hour for violating my not-blogging-about-politics rule.

  • Anycasting anyone?

    [Thanks Sam for the idea for this entry] Ever heard of IP Anycasting? Thanks to my recent change from godaddy (frowny face and no link) to dnsmadeeasy (happy face and they get a link) I’m now using a DNS provider that provides anycasting. What is it and should you care?

    IP Anycasting is assigning the same IP address to multiple instances of the same service on strategic points in the network. For example, if you are a DNS provider, you might have servers in New York, London and Los Angeles with the same IP address. Then when a surfer in San Diego (about 80 Miles South of Los Angeles) makes a request to your DNS system the server in Los Angeles answers and saves the network from having to route traffic to New York or London.

    Anycasting is generally used to distribute load geographically and to mitigate the effect of distributed denial of service attacks. It’s been used by the F root server since November 2002 and has saved good ole F from getting taken down by several DDoS attacks.

    I was using dnspark.net a couple of years ago and we had a few hours of down-time while they were hit by a DDoS attack – so it’s not as uncommon as you think. [They obviously don’t use anycasting]

    Anycasting is suitable for DNS because DNS uses a connectionless session layer protocol called UDP. One packet is sent, a response is received and hey, if the response isn’t received the client just tries another DNS server. [This occurs in the vast majority of DNS queries. There are a small number of exceptions where DNS uses TCP.]

    Anycasting is not ideally suited for TCP connections like web browser-server communication because TCP is connection oriented. For example, TCP requires a 3 way handshake to establish the connection. If the network topology changes and one packet is sent to the Los Angeles server and another is sent to New York it breaks TCP because the New York server doesn’t know about the session that Los Angeles has started establishing.

    That’s the theory anyway, but if the network topology stays reasonably stable and you don’t mind a few sessions breaking when the topology does change then perhaps you’ll consider using Anycasting with your web servers. But don’t get too creative and launch a content delivery network. Akamai might sue you and they’ll probably win. They own patent No. 6,108,703 which covers a “global hosting system” in which “a base HTML document portion of a Web page is served from the Content Provider’s site while one or more embedded objects for the page are served from the hosting servers, preferably, those hosting servers near the client machine.” Akamai just won a case against competitor Limelight for violating that patent and the case is now heading to the appeal courts.

    There are other protocols that are connectionless and therefore well suited for Anycasting like SNTP and SNMP but there isn’t much demand for these because they’re network management protocols and don’t experience the massive load that more public protocols like DNS, SMTP and HTTP get.

    Deploying an anycast network is not something you’re likely to consider in the near future unless you’re eBay or Google, but outsourcing some of your services like DNS to an anycast provider is something that’s worked well for me and might work for you.

  • Very high performance web servers

    Have you ever tried to get Apache to handle 10,000 concurrent connections? For example, you have a very busy website and you enable keepalive on your web server. Then you set the timeout to something high like 300 seconds for ridiculously slow clients (sounds crazy but I think that’s Apache’s default). All of a sudden when you run netstat it tells you that you have thousands of clients with established connections to your machine.

    Apache can’t handle 10,000 connections efficiently because it uses a one-thread-per-connection model (or if you’re using prefork then one process per connection).

    If you want to allow your clients to use keepalive on your very busy website you need to use a server that uses an event notification model. That means that you have a single thread or process that manages thousands of sockets or connections. The sockets don’t block the execution of the thread but instead sit quietly until something happens and then have a way of notifying the thread that something happened and it better come take a look.

    Most of us use Linux these days – of course there are the BSD die hards but whatever. The linux kernel 2.6 introduced something called epoll that is an event notification system for applications that want to manage lots of file descriptors without blocking execution and be notified when something changes.

    Both lighttpd and nginx are two very fast web servers that use epoll and a non-blocking event notification model to manage thousands of connections with a single thread and just a few megs of ram (ram consumption is the real reason you can’t use apache for high concurrency). You can also spawn more than one thread on both servers if you’d like to have them use more than one processor or cpu core.

    I used to use lighttpd 1.4.x but it’s configuration really sucks because it’s so inflexible. I love nginx’s configuration because it’s very intuitive and very flexible. It also has some very cool modules including an experimental embedded perl module. The performance I’m getting out of it is nothing short of spectacular. I run 8 worker processes and each process consumes about 7 megs of RAM with a few modules loaded.

    So my config looks like:

    request ==> nginx_with_keepalive –> apache/appserver_nokeepalive

    If you’d like to read more about server models for handling huge numbers of clients, check out Dan Kegel’s page on the so called c10k problem where he documents a few other event models for servers and has a history lesson on event driven IO.

    Also, if you’re planning on running a high traffic server with high concurrency you should probably optimize your IP stack – here are a few suggestions I made a while back on how to do that.

  • The irrelevance of microsoft's search

    I put some cross-cluster traffic throttling in place yesterday using memcached – which rocks btw. In the last 12 hours I’ve blocked three sources – two were rogue crawlers from broadband ISP’s. The other was MSN’s live search crawler which is requesting more than 1 page per second sustained over 30 seconds. If it was Google I’d probably care, but Google has polite crawlers and unlike Google, Live search only sends me about 2% of my total search traffic.

  • How to fix munin's netstat passive connections increasing constantly

    Another thing I googled until I was all googled out and couldn’t find an answer, so for future explorers who pass by here, here’s the fix…

    If you’re running munin and you suddenly notice the number of netstat passive connections is constantly increasing in a linear fashion, rest assured it’s not your server that’s busy beating itself into oblivion. It’s a munin bug that’s easily fixed.

    If you run netstat and get something like this:

    netstat -s|grep passive
    3339672 passive connection openings
    7574 passive connections rejected because of time stamp

    …then it’s the passive connections rejected that’s confusing munin.

    To fix this edit:

    /usr/share/munin/plugins/netstat

    and change the line

    netstat -s | awk ‘/active connections/ { print “active.value ” $1 } /passive connection/ { print “passive.value ” $1 } /failed connection/ { print “failed.value ” $1 } /connection resets/ { print “resets.value ” $1 } /connections established/ { print “established.value ” $1 }’

    to

    netstat -s | awk ‘/active connections/ { print “active.value ” $1 } /passive connection openings/ { print “passive.value ” $1 } /failed connection/ { print “failed.value ” $1 } /connection resets/ { print “resets.value ” $1 } /connections established/ { print “established.value ” $1 }’

  • Sergio and Muse

    My good friend Sergio who is an extremely accomplished musician and who morphed himself from a spectacular bassist to spectacular drummer and can put most lead guiarists to shame once told me that Muse is the best rock band that has ever existed.

    Personally I don’t have the balls or the knowledge to make far reaching statements like that. And reading this I know you’re enumerating the thousands (millions?) of rock bands that have existed since African American slave communities sang their first question/answer folk songs and created the foundation for blues and then rock.

    But Serge is a smart guy and his opinion is not to be taken lightly. Go buy Muse – “Hysteria” and “Supermassive Black Hole” on iTunes and let me know what you think.

  • Why Free?

    A great article on wired about the free web economy.

    Interesting quote:

    “Anything you can consistently convert to cash is a form of currency itself, and Google plays the role of central banker for these new economies.”

  • I'm so dumb

    Don’t ever leave a website that starts to get any kind of traffic on the joke that calls itself GoDaddy. As a registrar they’re not bad but their DNS tool is very broken.

    I won’t bore you with tales of my screaming match at a manager there at 2am when a simple A record IP address change caused my image server’s address to drop in and out of their DNS at random. Or how the crankier I got the more he called me sir. Or how his colleague explained that if I choose to use their DNS service I need to know intuitively that I can’t make more than one change a day or their zone file gets corrupt – and how it’s standard procedure that you call them to do a “zone file refresh”. Or how he explained that a record I hadn’t changed at all dropped off their servers and the reason was because it’s an “Internet Thing”.

    I moved over to dnsmadeeasy.com today and so far they rock. They’re the lowest cost host that offers Anycast on their servers which gives pretty good protection against DDoS attacks – something that took out dnspark a while back when I used to use them.

  • Why I'm so glad I didn't use Rails

    I’ve been uncool for some time now. In 2000 when Java was really beginning to kick ass I grabbed a Java book and wrote some code. And I decided I was getting stuff done faster in Perl so I stuck with it. I felt like a dork who was playing with his bigwheels while the other kids had graduated to Ducati’s.

    But by and by I discovered that ModPerl kicks Java’s ass as far as performance goes and in fact loosely typed languages do rock. Not only that but anything I need has already been written and posted free in CPAN. And if you code in Java then Sun Microsystems and their friends will try to sell you stuff at every opportunity – it’s like going to the ball game where stadium forces the beer vendors to charge $10 a beer even if they make entry free and you get to play on the field.

    2 years ago at Jobster as the Java dev team was discovering the new and cool loosely typed but cleverly OO language called Ruby and it’s Rails framework I went and grabbed a Ruby book and wrote some code. I didn’t like that it didn’t have CPAN and the server model seemed clunky and immature. So I stuck with ModPerl. Again I felt like the kid left in the dust while the others went and played with the big boys.

    Turns out the big boys don’t care about you or your business. Here’s a slide from David Hansson, Rails creator:

    This is via Rob Conery’s blog which I found via Tony Wright’s blog. And here’s a quote from David:

    Iā€™m not in this world to create Rails for you. Iā€™m in this world to create Rails for me and if you happen to like that version of Rails that Iā€™m creating for me, than you are going to have a great time.

    Read Rob’s full blog entry for a lot more insight on what is scary about Rails and its community.

    In Seattle last year I spent a lot of time networking in the startup community and meeting with many entrepreneurs. When we spoke technology choices every single one of them was planning on using Rails. Eventually it became a silly question and the answer was brushed if in a “duh, like obviously I’m using Rails” fashion.

    This year on the Seattle Tech Startup list – about 2 weeks ago – there was a thread with many entrepreneurs complaining bitterly about Rails’ shortcomings.

    Startups are risky enough without adding Rails.

    Sure I wake up at night and wonder if I’m the guy who insists on using Cobol while everyone has moved on to Pascal.

    But then I get out of bed and read the recent posts in the ModPerl archives, I check on the progress of Perl6 and I log onto my servers and check mod_status and how many requests they’re serving without breaking a sweat and I realize that it takes more than a bunch of arrogant eurotrash developers to create an enthusiastic open source community churning out great products.

    It takes a lot of love for the product from the community and from its developers. It takes an inspirational leader like Larry Wall or Linus Torvalds and their lieutenants. And it takes time.