MarkMaunder dot com

The irrelevance of microsoft's search

I put some cross-cluster traffic throttling in place yesterday using memcached – which rocks btw. In the last 12 hours I’ve blocked three sources – two were rogue crawlers from broadband ISP’s. The other was MSN’s live search crawler which is requesting more than 1 page per second sustained over 30 seconds. If it was Google I’d probably care, but Google has polite crawlers and unlike Google, Live search only sends me about 2% of my total search traffic.

Leave a Comment

Your email address will not be published. Required fields are marked *

My name is Mark Maunder. I've been blogging since around 2003 when I started on Movable Type and ended up on WordPress which is what I use to publish today. With my wife Kerry, I'm the co-founder of Wordfence which protects over 5 million WordPress sites from hackers and is run by a talented team of 36 people. I'm an instrument rated pilot and I fly a Cessna 206 along with a 1964 Cessna 172 in the Pacific Northwest and Colorado. I'm originally from Cape Town, South Africa but live in the US these days. I code in a bunch of languages and am quite excited about our emerging AI overlords and how they're going to be putting us to work for them.