Breaking: Google starts to block hacked WordPress blogs as attack widens

Big News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

I’ve had two reports in the last 12 hours of WordPress blogs that were compromised via the Timthumb hack being listed as malware by Google. If you try to visit either site, you are confronted with the following:

 

These sites are listed with the warning that “This site may harm your computer” in Google’s search results and Google blocks access to the site with a warning forcing you to manually type the URL into your location bar if you really do want to visit the site:

One of the site owners sent me the detailed info that Google Webmaster Tools was giving her:

This malicious code is appearing intermittently on this author’s WordPress site. I’ve seen this same pattern recently in blogs I’ve repaired and the way it works is that the site is periodically downloading new PHP code from a remote site run by the attacker and re-injecting it into the wordpress code. That allows the attacker to add, remove and update whatever code he/she is executing on your blog. So they could for example update any spam links every few hours.

To prevent your site being listed as malware clean it as fast as possible

The fastest way to do this, although it doesn’t gaurantee a complete clean, is the following:

  1. Remove all old plugins and themes you aren’t using.
  2. Upgrade all your plugins and themes to the latest versions and make sure none of them use an old version of Timthumb.
  3. Clean any Timthumb cache directories.
  4. Upgrade your entire wordpress installation, even if it’s at the latest version. This overwrites all wordpress files.
  5. Search your directory tree for any remaining suspicious files that contain base64_decode wrapped in an eval() statement or URL encoded data. More info on how to do this search here. Delete any files you find. NOTE: If you don’t find any additional infected files in this step, it’s highly likely that your site is not clean. Every attack that I’ve seen so far using Timthumb gets in by uploading a file into the cache directory and then uploads an additional file into a writeable directory on the blog to ensure continued access once the cache is cleaned. Make sure you find that additional file.
  6. Make sure the only directory that is writeable in your wordpress installation is wp-content/. Directories like wp-admin and wp-includes should be read only by the web server.

If you are already listed as malware by Google, here is what to do

Clean your site using the above steps. You can find more suggestions on how to clean your site on this page.
The fastest way to get your site removed from Google’s malware list is to request a review through Google Webmaster Tools. You can find the help file on requesting a malware review on this page.
The process takes about 24 hours to get your site removed. You can find out more about Google’s Malware list and safe browsing report on this page.

 

Potential long term impact of this vulnerability

The fact that I’ve seen the same domain being used by attackers on multiple blogs suggests this attack may be partially or fully automated. The worst case scenario is that we end up with a WordPress botnet with thousands or tens of thousands of servers on high bandwidth links compromised and able to send spam emails or launch a huge DDoS attack.

Keep in mind that most botnets are compromised windows machines on relatively slow home broadband connections. Their uplink speeds are around 512kbps. These WordPress servers are on links that are a minimum of 10 Megabits per second each, so they have plenty of firepower for a coordinated attack. One WordPress server is equal to at least 20 infected PC’s in terms of pure bandwidth firepower.

 

WordPress Security: Please delete old themes and plugins

News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

I was contacted by another site owner who was hacked via vulnerable WordPress themes today. He had updated to the latest non-vulnerable version of his theme, but the WordPress theme installation or update process doesn’t remove, or remind you to remove, old themes that may be vulnerable. So while they encourage you to update everything, old versions are still lurking on your site waiting for an attacker to take advantage of them.

Remember: Delete all old unused themes and plugins.

In this case an attacker once again used an old version of timthumb to install an attack shell called Sniper_SA. The attack shell was Arabic so I’m assuming the attack came from an Arabic country. [The last 3 I've seen were english]. This one was base64 encoded inside a PHP eval.

The web host is one of the top 3 WordPress.org hosts on the web. Their default installation is to have your entire WordPress installation writeable by the web server and the server can even write to your home directory under the web root. This opens up all sorts of possibilities for a hacker to gain a remote shell. WordPress hosts, please secure your default WordPress installations so that only directories under wp-content/ are writeable. Also make sure the user’s home directory is not writeable by the web server by default.

 

DeployMint: A Staging and Deployment system for WordPress

Exec Summary: Today I’m launching a Beta open source project called DeployMint. I’m using it on WordPress installations where WordPress is being used as a CMS. It runs as a WordPress plugin and allows for staging and deployment of WordPress sites along with robust version control and zero down-time during deployments. It uses the Git version control system to store site snapshots in a safe and space efficient way. It also takes a “belt and braces” approach and provides an emergency back-out system separate to Git in case a deployment fails. You can download the latest version of DeployMint and see a video demo at the DeployMint project page on Google Code.

Full blog entry:

My company is busy moving to using WordPress as a CMS and I wanted a way to instantly deploy several new pages of content or an entire site and have dev and staging sites to test new ideas. I also wanted version control, instant deployment and an emergency back-out system.

I also needed comments to be preserved on the live site so that if I deploy a new version, the existing comments on the live site stay where they should be and only page or post content is updated.

So I created a WordPress plugin called DeployMint.

DeployMint runs under WordPress MU. You create as many subdomains as you would like, for example:

  • development.example.com
  • staging.example.com
  • example.com (your live site)
  1. Then you design  your entire site with themes, pages, content on development.example.com.
  2. Once you’re done, you take a snapshot of development.example.com and deploy that snapshot to staging.example.com.
  3. Your client reviews the new site on staging.example.com and suggests changes.
  4. You make the changes on development.example.com, take a new snapshot and deploy that snapshot to staging.example.com
  5. Once your client is happy, you take a snapshot of staging.example.com and deploy it to example.com, your live site.

Here is a video showing the basic functionality of DeployMint. DeployMint is installed on this blog and I use it to test out new themes and design changes. It works as well on a blog or when WordPress is being used as a CMS. In this video I take a snapshot of my live site (this blog) and deploy it to my staging site staging.markmaunder.com. Then I make a minor modification, I re-snapshot the staging blog and deploy that snapshot to the live site.

DeployMint is space efficient because it uses Git to store snapshots. It also makes a full copy of your entire WordPress database including all your WordPress MU sites every time you deploy. Because these require more space, you can choose how many of these full backups you want to keep. If things go awry with your database or deployment for some reason, you have an emergency backout system that will restore your WordPress MU installation to the state it was in before your previous deployment.

Behind the scenes, DeployMint (DM) works as follows:

  • To install DeployMint you need to create a data directory that is not under your web root, but is writable by your web server.
  • When you create a new project, a new Git repository is created.
  • When you take a snapshot, DM dumps all tables belonging to the blog you snapshotted into individual files.
  • Those files are checked into a ‘Git’ repository. DM uses git for storage because it’s space efficient and robust.
  • Every snapshot you create is a new branch in the repository and only the changes are stored.
  • When you deploy using DM, it simply checks out the branch you want to deploy and imports it into a temporary database.
  • In that temp database, we merge all existing comments on your site into the site we’re about to deploy.
  • DM also modifies any hostnames it needs to, to reflect the site we’re about to deploy to’s hostname.
  • Before deployment, DM takes a full backup of your entire WordPress MU database including all sites and stores this for emergencies in case you need to back-out your changes.
  • These backups take up more disk space than snapshots, so you can choose how many of them you want to keep and DM auto-deletes the oldest ones first.
  • Then a rename is done which takes a few hundredths of a second to replace your old database with the new database we’re deploying.
  • And you’re live with your new site!

Please post a comment below if you have any features suggestions or comments. Thanks.

Two techniques to scan your WordPress installation and check if you’re hacked.

News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

I just helped another target of the timthumb.php vulnerability to clean their machine. The method the hacker used to hide their tracks was a little different to what I’ve seen in the past. So I wanted to mention it here and let you know how to scan for it.

As I previously mentioned, the method I’ve seen hackers use to hide their source code is to encode it using base64 encoding and then use base64_decode and eval() in PHP to execute the code at runtime.

You can scan for base64 decoding by getting a shell on your WordPress server and running the following in the root of the WordPress installation directory:

grep -r base64_decode *

Keep in mind that some files that are not hacked will show up, like the newest version of timthumb.php which includes a base64 encoded image. But this is a good starting point to get a list of files that warrant further inspection.

The hack I saw today was different. The hacker used hexadecimal escaping to hide their tracks. They didn’t just encode hostnames and things that a security analyst would obviously search for. They also encoded individual javascript commands and strings containing HTML element names.

You can use this perl compatible regular expression to search for hex encoded data in your javascript. Again, run this in a shell in the wordpress root installation directory:

grep -rP “(?:\\\\x[A-F0-9]{2}){5}” *

This will search for strings of at least 5 sequential hex encoded digits. You may get some false positives like class-simplepie.php . But again, this will give you a list of files that require closer inspection.

The file that was infected today was wp-includes/js/l10n.js. The attacker had appended hex encoded javascript to it. You can see what a normal file looks like here.

If you’ve been hacked, or suspect you’ve been hacked, drop me an email at mmaunder at gmail. I charge a very reasonable consulting rate and it usually takes 1 to 3 hours to fix the system and harden up permissions to prevent future attacks.

Advanced WordPress: How to get Real WordPress Commenter IP Addresses behind your Nginx Proxy

News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

If you run a reasonably high traffic blog on a small Linode server like this one, it’s a really good idea to set up an Nginx front-end proxy for your Apache server. It lets you handle relatively high traffic without running out of apache children while keeping keep-alive enabled.

You can read more about how to set up Nginx and other tips on my Basic WordPress Speedup page.

If you have set up Nginx, you’ll notice that your comments no longer have the real IP address of visitors to your site. They’re all 127.0.0.1 or something similar.

The way I solved this was to edit my php.ini file. On my Ubuntu server this lives in /etc/php5/apache2/php.ini

I modifed the auto_prepend_file variable to look like this:

auto_prepend_file = /etc/php5/apache2/mdm.php

Then in the mdm.php file I put this:


<?php
$mdm_headers = apache_request_headers();
$_SERVER['REMOTE_ADDR'] = $mdm_headers["X-Forwarded-For"];
?>

 

This assumes you have the following line in your Nginx.conf to forward the real IP address:

proxy_set_header X-Forwarded-For $remote_addr;

 

TimThumb users and WordPress Theme users using TimThumb, please upgrade

News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

I’ve done a ton of work on TimThumb this weekend and there are a few great enhancements. E.g. if you have pngcrush or optipng installed, it will now use 66% less disk space and give you comparable quality images.

Please grab the latest version of TimThumb on this page. Then let me know if you have any feature requests or find any bugs by reporting them here.

Here’s the TimThumb changelog since I released 2.0 about 48 hours ago.

WordThumb is now TimThumb 2.0

News [April 24th, 2012]: I’ve launched Wordfence to permanently fix your WordPress site’s security issues. Click here to learn more.

On the suggestion of Matt Mullenweg (wordpress founder) Ben Gillbanks (timthumb author) and I have been working for the last day to merge my work on WordThumb into TimThumb 2.0.

That work is now complete and TimThumb 2.0 is now available for download from the TimThumb site.

I’m going to be working with Ben going forward to continue to have TimThumb be the easiest to use, fastest, most popular and most secure thumbnail script on the Web.

Here are a few enhancements in TimThumb 2.0:

  • Includes the ability to take website screenshots if you have Xvfb and CutyCapt installed. (Instructions included how to do this)
  • All filters and resizing can be applied to website screenshots.
  • The cache directory is now secure and is still public for flexibility across platforms.
  • TimThumb creates index files in your cache to prevent directory listings.
  • Filenames are more randomized using data that a hacker doesn’t have access to, making it very hard to guess filenames in cache and access them.
  • Cache files have a .txt extension which means the web server won’t execute them.
  • All cached files have a fixed length record at the beginning which, if a web server tries to execute them, will be interpreted as PHP code and will cause an immediate exit.
  • It includes file locking when files are created in cache to avoid conflicts.
  • The entire code base has been rewritten and refactored for better code scaleability.
  • Lots of other improvements.
So give it a whirl and if you have any suggestions or find any bugs, please file them on the TimThumb issues page. Thanks.

WordThumb now uses a secure public cache for compatibility

UPDATE: WordThumb has now been merged into TimThumb and has become TimThumb 2.0. Please head over to the TimThumb site now for updates and to get the code.

The latest version of wordthumb (just uploaded) uses a public cache again (same place timthumb does) because many system temporary folders are not writeable.

This public cache is part of what caused the timthumb vulnerability, so I’ve made it more secure as follows:

  • Using a .txt extension for all files so servers won’t execute the files when accessed.
  • Using an md5 salt to prevent hackers knowing what filenames are to make things a little harder. On a badly configured server they could still get a directory index and access files that way.
  • Added a <?php die(); ?> to the start of ever file cached. That way if a hacker manages to guess a filename and for some reason the server decides to execute a .txt file, as a last resort it will simply die.

 

WordThumb can now take screenshots of websites for you

UPDATE: WordThumb has now been merged into TimThumb and has become TimThumb 2.0. Please head over to the TimThumb site now for updates and to get the code.

Just for fun I added the ability to take screenshots of any website to WordThumb. You can even apply all the image manipulation and filters that it supports for regular images to a website screenshot.

The latest update also lets you block “hotlinking” where other websites display an image loaded from your server. That is mainly to prevent other sites using your WordThumb to generate thumbnails of websites.

Be warned, to use this you’re going to need root access to your own server. You’re also going to have to install a few basic tools, but I’ve included detailed installation instructions in the source where the configuration options are. I’ve also only tested this on Ubuntu Linux.

If you don’t have root on your machine or don’t want the feature, WordThumb is still fully backwards compatible with timthumb.php and the webshots feature is off by default. But if you like to experiment give it a whirl and let me know what you think.

The first screenshot takes a few seconds to load and then it’s cached for 24 hours (the default cache setting).

I have it running on this server, so here are a few screenshots of my favorite sites created and updated using WordThumb. You can click on one of these images and play with the URL and image width/height in the location bar to load different sites. My server is at about 80% load right now, so it will probably run faster on a less busy machine.