Time for a Linode downgrade

My credit card number was stolen a few days ago by someone in Palo Alto right after my site was on Hacker News’s home page. I’m going to choose to believe they are unrelated. Interesting though since I don’t live or work in California and this card has never visited there. On the positive side, Visa Signature customer service is worth every penny and 2 new cards arrived on my doorstep in France in 48 hours.

But moving on to the point of this blog entry… it forced me to look at all the recurring fees I’m paying for and either update the card number to my new card or ditch the service.

I discovered my Linode fees had crept up to $115 a month for three servers and one getting backed up. So I ditched the two dev servers and was still paying $60 for a Linode 1536 instance with backup fees.

  • So I deleted log files and brought the disk space down to 12 gigs from 50 gigs.
  • Added more aggressive log rotation to protect from running out of space.
  • Optimized Apache to only have 5 children.
  • Optimized nginx as a reverse proxy so slow clients won’t hog the apache children by setting a shorter proxy timeout.
  • Added mod_status to do real-time checks on how many apache children are busy and what site they’re serving. (This server actually runs 3 sites including skipthepie.org and the website for my sister’s amazing Cape Town restaurant.)
  • Set MaxRequestsPerChild for apache to be 100 to make sure the apache kids don’t grow if there’s a memory leak.

This of course assumes you’re running nginx in front of apache as a reverse proxy, without which you absolutely can’t run a medium traffic website on nothing but thin-air.

Once it was all done, I shrunk the disk down to 20 gigs, rebuilt the server as a 512, got my $36 prorated refund from Linode (thanks guys, very nice policy!!) and I’m now paying $25 a month for hosting instead of $115 (Or saving $1080 per year)

Not exactly rocket science or Earth Shattering, but always nice to keep things lean and mean.


Technical details and scripts of the WordPress Timthumb.php hack

Big News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Read this now.

UPDATE: WordThumb has now been merged into TimThumb and has become TimThumb 2.0. Please head over to the TimThumb site now for updates and to get the code.

As I mentioned yesterday my WordPress blog was hacked. The security hole has been picked up by hacker news and from there, The Register, ZDNet, PCWorld, and Geek.com among others. The publicity will hopefully get Theme developers to update timthumb.php or switch to a different thumbnail generator.

I’ve been contacted with requests for detailed info, so I’m going to post the technical details of how my site was hacked along with the scripts that the hacker used to get in. This is targeted at a technical audience.

The server that served you this web page is the one that was hacked. It runs Ubuntu 10.10 with all security updates installed. It is a virtual server hosted by Linode.

I also run the latest version of WordPress.org.

My WordPress root directory was writable, but making it read only would not have prevented the hack.

Timthumb.php in it’s default configuration allows site visitors to load images from a predefined set of remote websites for resizing and serving. Timthumb offers a caching mechanism so that it doesn’t have to continually re-process images. The cache directory lives under the wordpress root and is accessible by visitors to the website.

The ability for a site visitor to load content from a remote website and to make the web server write that remote content to a web accessible directory is the cause of the vulnerability in timthumb.php.

To be clear, timthumb.php does not actually execute any remote malicious code that causes this vulnerability. This was a point of confusion among some commenters in my blog post yesterday. It simply gets a remote file and places it in a web accessible directory.

Timthumb only allows remote content from a small range of websites to be loaded remotely. In it’s default configuration these included Blogger, WordPress.com and other sites that are writeable by the general public.

Timthumb’s verification that remote content was only being loaded by these domains was also broken. You could for example load content from hackersiteblogspot.com or from blogspot.com.hackersite.com.

I’ve submitted a patch that fixes the pattern matching and removed all default public hosting sites from the allowed sites list. The developer has opted to keep a small list in which I’m not in favor of.

In my case the hacker uploaded a script to my cache directory which timthumb.php stores as “external_<md5 hash>.php”. He/she then accessed this script directly in my timthumb cache directory as something like http://markmaunder.com/wp-content/themes/Memoir/scripts/cache/external_md5hash.php

The script uploaded was Alucar shell which is base64 encoded and decodes when it executes. That makes it a little harder to find it using grep or similar tool. You can see the encoded version of Alucar here and the decoded version of Alucar here (without the username and password preamble at the top).

Here’s a screenshot of the UI:

Alucar UI

This script which gives a web based shell access was then used to inject base64 code to one of my core wordpress files wp-blog-header.php which lives in the wordpress root directory. The file with injected code looked like this.

The decoded version of this base64 code is this. The code executes whenever a blog page is visited. It fetches a file from a URL and writes it to /tmp. Then it executes the php code that is contained in this file. In my case it simply echo’d some javascript code that would show ads. Here is the code contained in the file in /tmp.

Again, this file is periodically updated with new PHP code, so the attacker could have his way with my server until I found out about it. The code could be altered to instead become a spam system and work it’s way through a long list of spam emails.

The way I tracked this to conclusion was:

  • Heard audio on my blog telling me I’d won something.
  • Checked Chrome network tools and saw ad content loading and I don’t serve ads.
  • Grepped wordpress source and themes for hostname I saw in ad. Nothing.
  • Dumped mysql databases on server (all of them) and grepped for hostname. Nothing.
  • Confusion reigns.
  • Started working my way through nginx (which is my front end proxy to apache) and apache access and error logs.
  • Spotted lines in apache error log like this: “[Mon Aug 01 11:09:12 2011] [error] [client] PHP Warning: file_get_contents(http://blogger.com.zoha.vn/db/load.php): failed to open stream: HTTP request failed! in /usr/local/markmaunder/wp-content/themes/Memoir/timthumb.php on line 675”
  • Checked timthumb’s cache directory and found Alucar.
  • Realized base64 encoding is why I didn’t find anything with grep.
  • Regrepped wordpress source and database and found injection in wp-blog-header.php
  • Decoded base64 stuff and played with Alucar
  • Found tmp file in /tmp
  • Cleaned everything and fixed permissions. Ran chkrootkit and other utils on machine to see if anything else was compromised. Changed passwords, etc.


Hacker News negativity finally hurts the program it supports

I’ve posted a couple of times about the negativity on Hacker News and why I no longer post there. Well the AirBnB fiasco has been seized by the HN “nancy grace” crowd with a vengance and they’re pounding their most successful startup ever into the dirt and promoting its competitors. While you may feel the AirBnB execs could have handled the destruction of a customer’s home a little better, this seems a little over the top.

Here is a list of all recent AirBnB stories that hit #1 on Hacker News during the last 7 days starting with the oldest first. These stats are taken from an app I wrote a while ago to track HN stats.

Airbnb bags $112 million in Series B — comments guynamedloren Posted 6 days, 16 hours, 30 minutes ago. and spent 5 hours at #1


AirBnB: Crimes committed against a host — comments foxit Posted 4 days, 13 hours, 40 minutes ago. and spent 7 hours, 50 minutes at #1


The Moment Of Truth For AirBnB As User’s Home Is Utterly Trashed — comments ssclafani Posted 4 days, 30 minutes ago. and spent 6 hours at #1


On Safety: A Word From Airbnb — comments ssclafani Posted 3 days, 18 hours, 20 minutes ago. and spent 4 hours, 30 minutes at #1


Airbnb Victim Speaks Again: Homeless, Scared And Angry — comments jamesgagan Posted 2 days, 12 hours, 50 minutes ago. and spent 10 minutes at #1


Airbnb Nightmare: No End In Sight — comments moonlighter Posted 2 days, 12 hours, 30 minutes ago. and spent 5 hours at #1


Airbnb story on front page of Financial Times print edition — comments arghnoname Posted 2 days, 7 hours, 20 minutes ago. and spent 2 hours, 39 minutes at #1


Airbnb Competitor Checks IDs: ‘We Don’t Want to Trade Security for Volume’ — comments citadrianne Posted 2 days, 2 hours, 50 minutes ago. and spent 4 hours, 40 minutes at #1


Plot thickens in Airbnb vacation rental horror story — comments felipemnoa Posted 1 day, 2 hours, 20 minutes ago. and spent 1 hour, 40 minutes at #1


Another Airbnb Victim Tells His Story: “There Were Meth Pipes Everywhere” — comments jasonlbaptiste Posted 10 minutes ago. and spent 10 minutes at #1


Disappeared from Hacker News … again. So I’m done.

I get wierded out when an opinion piece I posted about the impending debt default is on the home page of Hacker News for 20 minutes and climbing – and then gets disappeared.

The article is still posted on HN, no message that it’s flagged as spam, no email. Just a little tweak that disappears me from everything except “new”.

This has happened more than once. It and an increase in comment negativity has driven me off. I’m done.

Goodbye HN, fare thee well.


MI6 to Rest of World: Cyber War is On. Anyone, Anywhere is Fair Game. Arm yourselves.

This incredibly disturbing story was posted on Hacker News 26 minutes ago.

Summary: The London Daily Telegraph (via TheAge.com.au) is reporting that British Intelligence agents from MI6 and GCHQ hacked into an AlQueda online magazine and removed instructions for making a pipe bomb. They replaced the article with a cupcake recipe. A Pentagon operation was blocked by the CIA because the website was seen as an important source of intelligence. Furthermore, both British and US intelligence have developed “a variety of cyber-weapons such as computer viruses, to use against enemy states and terrorists”.

There is no reporting on where the servers of the magazine are based, who owns the lease on them (a US or British citizen?) and under what jurisdiction these attacks were made.

The message this attack sends to the rest of the world is “Cyber war is on. Anyone, anywhere is fair game. Arm yourselves.”.

As an Internet entrepreneur this is incredibly disturbing because it makes it OK for any government agency to target our servers and the tone of the article suggests moral impunity for government agencies engaging in these attacks. If it’s OK for British intelligence to hack (most likely) US based servers then it’s OK for Chinese officials to attack an ad network based in the USA if they run an ad for a dissident website.

At first glance this looks like a cute prank. But this attack may spark the beginning of a global cyber war fought by government agencies and private contractors, the logical conclusion of which is an Iron Curtain descending on what was once an open and peaceful communication medium.