Blog

  • The Basic Ketogenic Diet

    Note: Please note that if you are interested in a Ketogenic Diet used to treat Epilepsy or Pediatric Epilepsy, please start at Johns Hopkins who are the pioneers in this field. The wikipedia page for the Ketogenic Diet diet also has information on the diet as it relates to treating epilepsy. The diet below is simply for rapid and effective weight loss and uses a 1 to 1 fat to protein ratio rather than the 4 to 1 fat to combined protein and carbs ratio of the Ketogenic Diet pioneered by Johns Hopkins used to treat epilepsy.

    [wp_ad_camp_3]

    Disclaimer: I am neither a doctor nor self proclaimed nutrition expert so please consult your doctor before starting any diet or taking any action that affects your health and wellbeing.

    After finishing Gary Taubes latest book, which seems to have rapidly become the cornerstone of a new approach to nutrition, I’ve become very interested in the Ketogenic diet. The speed of weight loss I’ve seen is incredible and my energy level has remained high. The science behind a ketogenic diet is solidly backed up by Taubes research published in “Good Calories, Bad Calories” and “Why we get fat“.  According to Taubes’ research, it may also be the only way for people who have become severely insulin resistant, to effectively lose weight.

    The Ketogenic diet has always lived on the fringes of diet lore and has been seen as extreme. But the reality is that the low glycemic index diet (Low GI Diet) is effective because it is close to, but not quite, a ketogenic diet. Other diets like the South Beach Diet are also only effective because of the reduction in carbs and consequently insulin levels.

    The science behind this diet looks solid and it is part of the massive shift in nutrition research we’ve seen in the last few years. Prominent sport physiology experts like Tim Noakes have come around to this way of thinking and Tim in particular has said that everything he wrote about “Carboloading” in Lore of Running is wrong. (Lore of Running is considered the running physiology bible by many and contains much more than nutrition advice)

    I’ve decided to put together a “Basic Ketogenic Diet” for my own reference and because the Ketogenic diets out there are either targeted at extreme bodybuilders (and usually cycle in carbs) or are peppered with pseudoscience and superstition. If you find anything technically wrong in this article please correct me as loudly as you’d like in the comments and please cite your source.

    First a summary of the science behind the Ketogenic Diet.

    Keep in mind I’m summarizing hundreds of pages of explanation and supporting data into a few paragraphs:

    [wp_ad_camp_1]

    When you eat carbs your blood glucose level is raised and your pancreas secretes insulin. This insulin puts your muscle and fat cells into “storage mode”. Your fat cells store away the glucose as triglycerides. Insulin also prevents your fat cells from breaking down those triglycerides back into fatty acids and releasing them into your blood stream for use as energy. This is important: Insulin both causes fat absorption and prevents fat from being used as energy.

    If you were to eat sugary snacks throughout the day you are keeping your insulin level high which constantly keeps your fat cells in a state of absorption and prevents the release of fat and its use as energy.

    Eating fat and protein does very little to raise your insulin level.

    So the bottom line here is, if you want your body to burn fat – meaning if you want your fat cells to break down triglycerides into fatty acids, release them into your blood stream and actually use them as energy – you need to keep your insulin level as low and absolutely avoid spikes in insulin.

    A note on calories: Counting calories or increasing exercise output, according to the research of Taubes and many others, is a bad way to try to lose weight because what usually happens is instead of your body burning fat to make up for the deficit in calories, it simply decreases the amount of energy it expends. So you end up lethargic and still fat.

    There is a lot more to this, but the science above is the rationale behind all Low GI, Low Carb and Ketogenic diets. It is why you lose weight on these diets while consuming the same number of calories.

    Low Carb and Low GI diets work because they cut out carbs that cause spikes in insulin, which you now know will cause fat absorption and prevent fat burn.

    The Ketogenic diet takes this one step further. It keeps your insulin level low which puts your body into fat-release-fat-burn mode, but also significantly increases the fat content in your diet and teaches your body to use fat as energy. Your body (your liver in particular) will enter a state of Ketosis and will break down fat into fatty acids and ketone bodies which will be used as energy.

    During the diet you can measure how much fat your body is burning by monitoring your output of acetoacetate using Ketostix. When you enter a Ketogenic state, you will see a Ketone output of 5 to 20 mg/dL (According to “Why we get fat” by Taubes and my experience). The Ketostix I have also measure blood glucose which is useful to test for Ketoacidosis which you are not at risk for unless you are diabetic.

    Note that Ketostix are used by diabetics to test whether they are entering a state known as Diabetic Ketoacidosis which is very dangerous. In this case they will see very high ketone levels accompanied by blood sugar levels of greater than 240 mg/dL. You should be seeing blood sugar levels of zero (according to your Ketostix) during your diet.

    Please note that the diet outlined below is not the “Cyclical Ketogenic Diet” used by bodybuilders. It is a basic Ketogenic diet which I’ve found to be the most effective fat burning diet available. I’ve modified this diet from the classic Ketogenic diet used to treat epilepsy which recommends a 4 to 1 ratio of protein to fat. I’ve found that I can reach a state of Ketosis with a 1 to 1 ratio of fat to protein provided I keep the carbs very low.

    Without further ado, the basic Ketogenic diet:

    What you’ll need

    The Diet

    Note, you can eat as much of the foods below as you like, but I’ve found that I quickly feel full eating this diet. Don’t overeat and more importantly, don’t under-eat and make sure you’re getting the calories you need to function. This is NOT about cutting calories, it’s about releasing fat reserves and teaching your body to burn them.

    [wp_ad_camp_2]

    Drink lots of water during the diet (but don’t go crazy). I find this helps increase my metal clarity and energy.

    Taubes and others recommend not starting an exercise program at the beginning of this diet because during the acclimatization period you will not have enough energy and it usually results in people quitting the diet.

    During the diet, measure your rate of fat burn by measuring your acetoacetate output with your Ketostix. Do this every time you visit the loo. You should see 5 to 20 mg/dL, and I’ve noticed it’s particularly high after a very fatty meal. Blood glucose (if your ketostix show this on a separate color tab) should be zero at all times. If it is not zero and above 200 mg/dL you may want to consult your doctor.

    One of the most effective tricks during a diet is routine. Eat the same meals over and over or plan your meals ahead for the whole week. I’ve found it effective to pre-make dishes on Sunday and have them ready for the week. That makes it incredibly easy to stick to the diet because I avoid thinking which means I avoid creativity and creativity requires decision making which leads to hard choices. Just avoid the hard choices and have it all planned and some of it ready to grab and go.

    The Leafy Green Salad recipe:

    This is a recipe for a basic very low carb salad with lots of flavor. It’s great to add to any high-protein, high-fat meal.

    • Spinach, lettuce and/or mustard greens.
    • Red peppers
    • Tomatoes.
    • Optional raw onions.
    • Optional green beans raw.
    • Balsamic vinegar and olive oil dressing with salt and pepper. Use lemon and lime very sparingly if you add it. The vinegar actually reduces the glycemic index of this salad (and other foods).

    Below I have outlined three days with a total of 9 meals that you can mix and match as you like. At the end I include a list of Ketogenic-safe ingredients you can add or subtract as you like. Remember the key is to have a high fat, high-protein and no-carb or very low carb diet. That means absolutely no sugar.

    • Day 1
      • Breakfast: 2 eggs done any way with yolks. Sausage containing no carbs or sugar. Bacon.
      • Lunch: Chicken with as much skin and oily gooey bits as possible. Steamed spinach or broccoli.
      • Supper: Steak with cheese. Fried mushrooms. Leafy green salad.
      • Before bed 1 glass of wine with a slice of cheese.
    • Day 2
      • Breakfast: 2 egg omlette with friend mushrooms, cheese and chopped parsley.
      • Lunch: A whole duck breast if you can get it or chicken again with as much fatty skin as possible. Leafy green salad.
      • Supper: Hamburger made with high fat mince without the bun. The mince can contain the usual paprika, chopped onions and garlic if you like, but absolutely no syrup or sugar. Add Cheese, tomatoes, lettuce and any other leafy low-carb greens you like.
      • Before bed 1 glass of wine with a slice of cheese.
    • Day 3
      • 2 eggs done any way with bacon and usual no-carb sausage.
      • Steak with cheese. Leafy green salad.
      • Salmon (or other fish) fried and served with Bok Choy and your leafy green salad.

    You can mix and match the meals above and get a good Ketogenic burn going where you will notice rapid weight loss.

    Eat as much as you want ingredient ideas:

    • Beef,
    • Steak,
    • Hamburger,
    • Prime Rib,
    • Filet Mignon,
    • Roast Beef,
    • Chicken,
    • Duck (awesome if you can get it because it is very high fat),
    • Any Fish, Tuna, Salmon, Trout, Halibut,
    • Lamb,
    • Pork,
    • Bacon,
    • Ham,
    • Eggs,
    • Shrimp,
    • Crab,
    • Lobster,
    • Butter,
    • Oils (Olive Oil, Flaxseed oil, etc.),
    • Salt, Pepper, Soy Sauce,
    • Spinach,
    • Lettuce,
    • Mustard Greens,
    • Celery,
    • Cheeses,
    • Oysters,
    • Abalone.

    Add for variety but in moderation:

    Read the ingredients (if applicable) and make darn sure they contain no sugar:

    • Avocadoes
    • Mustard (with no sugar or carbs),
    • Tea no sugar with milk
    • Coffee black no sugar
    • Heavy Cream
    • Broccoli,
    • Cabbage,
    • Bok Choy,
    • Kale,
    • Asparagus,
    • Mushrooms,
    • Cucumbers,
    • Olives,
    • Celery,
    • Green Beans,
    • Brussel Sprouts,
    • Peppers (Red, Green, Jalapeno, Habanero),
    • Onions,
    • Nuts preferably almonds,

    You absolutely must avoid all sugar on this diet because it is the highest GI carbohydrate that will very quickly spike your insulin and destroy any Ketogenic effect. Other foods to avoid roughly in order of damage they will do to the diet:

    • All sugar.
    • All Bread.
    • Did I mention avoid sugar?
    • All traditional carbs like rice, pasta, wheat, potatoes, even the low GI ones like beans and lentils.
    • Beware of sauces that contain sugar or things like corn starch.

    Effects of the diet:

    • You will see rapid weight loss of up to 6 pounds for a 200 pound person in the first 48 hours. This is your kidneys releasing water as they expel their sodium due to the absence of insulin. It’s what you’ve usually heard described as “water weight”.
    • Then you should see continued weight loss of anything from 0.25 to 2 pounds per day (an eighth to half kilo lost per day). But this varies greatly between individuals and is affected by a wide range of factors including your current weight and insulin sensitivity.
    • For the first week you may experience slightly decreased mental clarity. This clears up after a week as your brain gets used to burning ketones for energy instead of glucose.
    • According to data in “Why we get fat” your Vitamin C needs actually decrease on a low carb diet, so don’t feel the need to massively supplement.
    • Research has shown (also from Taubes) that LDL (bad) cholesterol will elevate slightly but clump size will be increased which is a net positive because larger LDL is less likely to stick to artery walls. HDL (good) cholesterol is significantly elevated with a very low carb diet like this which is a very strong net positive. This also has other great health benefits e.g. Lower insulin reduces the risk of hardening of artery walls.

    What about alcohol?

    I’ve found a glass of wine on it’s own or with a small slice of cheese before bed seems, anecdotally, to increase my fat burn rate. Anything more than a single reasonably sized glass has the opposite effect.

    Beer is the devils poison. It contains carbs in the form of maltase which raise your insulin level just like sugar does. The alcohol is turned into citrate in your liver which produces fat and that fat is efficiently stored thanks to your now raised insulin levels.

    So absolutely no drinks with sugar. That means no sweet cocktails either. I also don’t buy the idea of low-carb beers.

    I would imagine that one shot of spirits  wouldn’t be a problem provided it doesn’t contain any sugar and is something like Vodka, Whiskey or Tequila. I’ve heard they add caramel to tequila “gold” (the cheap crap) so avoid that.

    I’ve found that drinking heavily, meaning several glasses of wine followed by cocktails or shots, has a deleterious effect on my ability to burn fat and my energy level. The effect seems to last 72 hours or more.

    Conclusion

    Once again I’d like to reiterate that I’m neither a doctor nor self proclaimed nutrition guru. But it does seem that both doctors and “nutritionists” including government sources have been pulling the wool over our eyes since 1960 about what we should be eating. The ideas that “fat makes you fat” and “lower calories to lose weight” are so entrenched in our consciousness that many people find the latest research difficult if not impossible to swallow.

    If you would like to learn how this came to be, check out Planet Money’s excellent segment on “Who Killed Lard?” and the rise of Crisco and hydrogenated vegetable oil. It will give you an idea of the political forces that influence dietary advice.

    As Gary Taubes commented: If we are going see a change in the advice that governments and health authorities are handing out, it is going to take 20 years which is a lifetime. So rather than wait for those slow moving wheels to turn, take matters into your own hands.

  • 12 Surprising things you probably don't know about weight loss

    I just finished reading Gary Taubes book “Why we get fat” after hearing him interviewed on Russ Roberts excellent econtalk podcast.

    Gary wrote a book called “Good Calories, Bad Calories” a few years ago which is quite technical and includes a long history of how the state of nutrition got to where it currently is. “Why we get fat” is a well written distilation and update of his previous book.

    The following surprising facts are from “Why we get fat”:

    1. Lower testosterone or lower estrogen makes you fat because it has an inhibiting action on LDL which causes your fat cells to absorb more fat. That is why we get fat as we get older.
    2. In an experiment, mice had their ovaries removed which removed their estrogen. They ate more and got fat. When their diets were limited they got fat anyway and became sedentary to compensate for the fact that they were storing away so many calories. This illustrates a recurring concept in the book: We are not fat because we eat too much. Instead, we eat too much because we are fat.
    3. Beer is the perfect beer-belly creator because when you drink beer the alcohol is turned into citrate in your liver which aids in fat production. That’s fine, but the problem is that the carbohydrates in beer in the form of maltase cause an increase in insulin which put your fat cells into storage mode. So the new fat that the citrate produced is quickly and efficiently stored… where else but your beer belly.
    4. A high protein, high fat diet raises your LDL (bad) cholesterol slightly, but massively raises your HDL (good) cholesterol which has a strong net positive effect on your overall cholesterol.
    5. A high protein, high fat diet causes your LDL (bad) cholesterol to appear in larger clumps which are less likely to stick to your artery walls.
    6. High insulin levels caused by carbohydrate intake cause atherosclerosis (hardening of arteries).
    7. Up until 1960 the popular advice from nutrition experts was to stick to a low carb diet. Gary includes a huge amount of data and historical quotes to support this. After 1960 we somehow got screwed up and started thinking that carbs should for the base of the food pyramid.
    8. Fat does not make you fat. Carbs make you fat because they boost insulin levels which put your fat cells into absorption mode and prevent your fat cells from breaking down triglycerides into fatty acides and glycerol and releasing them to be used as energy.
    9. The reason you instantly lose 3 to 6 pounds when starting a low carb diet is because your high insulin state has been signaling your kidneys to absorb sodium which causes them to retain water. When you lower your insulin levels for the first time your kidneys release that water and you lose (in my case) 4 pounds in about 36 hours.
    10. When you eat only protein and fat, your body goes into a state of ketosis after a while which means you are using fat as energy. You can buy ketostix at your local pharmacy and perform a urine test to see if your body is in a state of ketosis and at what level. The ketostix I have include a blood glucose test on a separate color panel which is also useful data.
    11. Gary attacks the idea that you can only lose weight by decreasing calories in or increasing calories spent through exercise with the following illustration: If you accidentally over-ate 25 calories per day for 20 years you would gain 50 pounds in weight. Most of us eat outside of that narrow threshold, but our bodies manage to self regulate somehow.
    12. Most diets fail because we decrease calories consumed which simply causes us to decrease our energy expenditure and stay the same weight.

    It’s a great book and completely changed my view on nutrition and physiology as it relates to diet.

  • Enough Pretending to Ban Assault Rifles. Just Do It.

    Until January this year I lived in Elizabeth, Colorado for a year and a half which is 30 miles from where the shooting occurred 3 days ago. Many of my extended family still lives there. My brother called me in France from Cape Town to tell me it was going on in real-time, via Reddit, so I got hold of my nephew in Denver who was watching the opening of Batman, but thankfully at a different theater. It’s a connected world.

    While living in Colorado I went to a gun range in Montana and played with just about every gun they had including of course a 50 caliber handgun an AR-15 and a fully automatic assault rifle that was similar but older for legal reasons. Then did a gun course later in Colorado. I don’t own any guns.

    According to the news an AR-15 assault rifle was used in the Aurora Cinema Shooting on Thursday night along with a shotgun and two handguns. The AR-15 is an M-4 assault rifle used by the US military in places like Afghanistan and Iraq. The main difference is that it is semi-automatic. In other words it can only fire as fast as you can pull the trigger. There are a few hacks available to fix this like bump-firing where you attach a device to the stock that keeps pulling the trigger thanks to the recoil.

    I never had much of a point of view on this, but I’m beginning to strongly question why on Earth we need to be able to own AR-15 assault rifles. They tried to ban them in California but there are “California Legal” AR-15’s available. The Federal Assault Weapons Ban was another half hearted attempt – the law only banned weapons made after the law passed and it expired in 2004.

    The AR-15 is a mean weapon. It fires a .223 round at an extremely high velocity and a single shot to a human can do terrible damage. A single shot to the body can cause brain damage due to hydrostatic shock. Contrary to what they tell you in movies, there is no such thing as a “Flesh wound” and every gunshot wound is serious.

    The idea that making the AR-15 semi-automatic somehow makes it safer is absurd. On a course I attended our instructor trained us to fire in bursts of 3 to four which is the only way to stay accurate with an assault rifle. So you’re really just removing 2 to three rounds from each shot and ensuring the shooter maintains the discipline of not holding down the trigger too long.

    The AR-15 has a short barrel which makes it useless for hunting, even though a few unsporting folks use it to hunt. It’s designed for close quarters assault style combat. When you hold and point it you are hunched over the weapon in a combat stance designed to minimize recoil, present a small profile and keep you moving aggressively forward.  It’s designed to very efficiently kill multiple people in close quarters in a war setting.

    Magazine sizes of 60 rounds or more are available from sites like gunbroker.com for the AR-15. Imagine a single AR-15 with three magazines of 20 to 60 rounds. I don’t think “home defense” captures the possible uses for that configuration. [Edit: After writing this article I discovered a 100 round AR-15 magazine was found at the scene in Aurora according to this article.]

    The argument for the need to “defend ourselves from the government” is absurd because they already have the tanks and the nukes and they’re not going to let us have any.

    The argument for home defense doesn’t hold water either because your AR-15 may be up against another AR-15 or an illegal M-4 or how about 5 guys with assault rifles.

    There’s never going to be a scenario where you are “fully equipped” with legal weapons to “take all comers”. So lets stop fantasizing, lets keep weapons for sport legal and stop lying to ourselves that we are somehow empowered because we have an assault weapon stored in the place you will probably be furthest from when the boogie man comes to visit.

     

  • Time for a Linode downgrade

    My credit card number was stolen a few days ago by someone in Palo Alto right after my site was on Hacker News’s home page. I’m going to choose to believe they are unrelated. Interesting though since I don’t live or work in California and this card has never visited there. On the positive side, Visa Signature customer service is worth every penny and 2 new cards arrived on my doorstep in France in 48 hours.

    But moving on to the point of this blog entry… it forced me to look at all the recurring fees I’m paying for and either update the card number to my new card or ditch the service.

    I discovered my Linode fees had crept up to $115 a month for three servers and one getting backed up. So I ditched the two dev servers and was still paying $60 for a Linode 1536 instance with backup fees.

    • So I deleted log files and brought the disk space down to 12 gigs from 50 gigs.
    • Added more aggressive log rotation to protect from running out of space.
    • Optimized Apache to only have 5 children.
    • Optimized nginx as a reverse proxy so slow clients won’t hog the apache children by setting a shorter proxy timeout.
    • Added mod_status to do real-time checks on how many apache children are busy and what site they’re serving. (This server actually runs 3 sites including skipthepie.org and the website for my sister’s amazing Cape Town restaurant.)
    • Set MaxRequestsPerChild for apache to be 100 to make sure the apache kids don’t grow if there’s a memory leak.

    This of course assumes you’re running nginx in front of apache as a reverse proxy, without which you absolutely can’t run a medium traffic website on nothing but thin-air.

    Once it was all done, I shrunk the disk down to 20 gigs, rebuilt the server as a 512, got my $36 prorated refund from Linode (thanks guys, very nice policy!!) and I’m now paying $25 a month for hosting instead of $115 (Or saving $1080 per year)

    Not exactly rocket science or Earth Shattering, but always nice to keep things lean and mean.

     

  • Static Stretching injured my lower back

    I’m a runner and I do 3 to 5 miles 3 times a week. About 3 months ago I started doing a new static stretch in addition to my current routine. I stretch before and after my runs for about 20 minutes each time. The new stretch was sitting with legs out in front, touching my toes and putting my head on my knees.

    About a month ago my back started hurting. Not during my runs, but outside of runs. Then a few days ago I was loading groceries in the car and bang. Severe back pain, so bad that if I sneezed my legs half collapsed.

    I immediately stopped running and stuck to my stretching routine. No improvement. 2 days later I stopped stretching and within 24 hours a marked improvement and 48 hours later even more so.

    A friend’s back was severely injured in a kiteboarding accident and shared his recovery story with me earlier this year at a skiing trip. Much of the recovery was strengthening his back muscles so they could re-support his spine. Interesting part was that right after the accident his back muscles went into spasm to protect his spine. Which indicates how important those muscles are to support the spine.

    So I googled whether static stretching can weaken back muscles.

    I ran across this: http://www.nytimes.com/2008/11/02/sports/playmagazine/112pewarm.html?_r=1

    Turns out I’ve been working hard during the last 3 months to weaken my back muscles. So I’m seriously rethinking my stretching routine and will probably do the bare minimum to retain flexibility and focus on dynamic stretching as the article suggests from now on.

     

     

  • The Rise of the Data Smuggler

    I always thought the idea of physically smuggling data was absurd. Even physically transporting data seemed silly to me because if you have broadband you can simply upload or download it. For really big data I have a gigabit connection at a data center where I rent space, so sometimes I’ll do a massive download and just show up at the facility with a 1.5 terrabyte drive and hit the local Starbucks while it takes a few minutes to copy over what I’ve downloaded.

    I have either given or thrown away countless USB thumb drives I’ve been given as gifts from Google AdWords and other companies. What’s the point?

    Two things changed my mind about why physically transporting data is interesting. A conversation with Sebastian Thrun (creator of Google Street View) that I had a few years back where he told me that Fedexing data is, and probably always will be, the highest bandwidth way of moving data around. That’s why Google uses Fedex to send hard drives from their Street View vans back to headquarters.

    The second thing that changed my mind was a new law in the UK that makes it illegal to not hand over encryption keys if the police want to decrypt your data. The penalty is two to five years in prison for simply refusing to hand over the keys. The logical outcome is that a lot of energy will now be spent on hiding the existence of encrypted data.

    I think two fields will emerge. The first is the art of hiding encrypted data when transferring it across a wire. If time is not a factor then this may be the way to go. Simply altering the sequence or transmission times of TCP packets can encode data, although it will be very low bandwidth.

    The second area where I think you’ll see more activity is the physical hiding of data. The reason I think more energy will be spent in this area is because it allows for very high bandwidth. If you can hide a 2 terrabyte drive and take a 6 hour journey to get it fro A to B, your bandwidth is 776 Megabits per second. Try and get that on your cable modem or ADSL link.

    Data storage devices that self destruct aren’t interesting when it comes to solving this problem. A self destructing drive lets police know that you have data that you never allowed them to decrypt, so presumably you’ll get your 2 to 5 years. The data needs to be invisible.

    Storing data on or inside your body may be one solution. According to Scientific American:

    The human brain consists of about one billion neurons. Each neuron forms about 1,000 connections to other neurons, amounting to more than a trillion connections. If each neuron could only help store a single memory, running out of space would be a problem. You might have only a few gigabytes of storage space, similar to the space in an iPod or a USB flash drive. Yet neurons combine so that each one helps with many memories at a time, exponentially increasing the brain’s memory storage capacity to something closer to around 2.5 petabytes (or a million gigabytes). For comparison, if your brain worked like a digital video recorder in a television, 2.5 petabytes would be enough to hold three million hours of TV shows. You would have to leave the TV running continuously for more than 300 years to use up all that storage.

    I’m not sure I would want to upload data directly to my brain, lest I overwrite the breathing function. But biological data storage is clearly worth looking at if your intent is to hide data.

    So maybe Johnny Mnemonic wasn’t so absurd after all:


     

  • PHP array() is a little scary

    Push 100,000 elements onto a PHP array() where each element is a four element associative array (a hash in Perl speak). Here’s the data being pushed:

    array(
  'owner' => 100,
  'host' => 'www.example.com.co.uk',
  'path' => '/this/is/an/example/path.html',
  'hostkey' => '1111'
)

    The memory grows by over 80 megabytes.

    Pushing takes less than a second or two but shifting off the first 1000 elements takes over 17 seconds on my machine.

    Now take that same data and create a basic FIFO class that has push() and shift() methods. Use pack() and unpack() to store the data in a long string. Total time to push 100,000 and shift the first 1000 elements is around 1 second. Total memory is 7 megabytes which is less than 10% of PHP’s internal array()’s consumption.

    PHP’s splFixedArray class which is advertised as mainly having a speed advantage doesn’t fair much better. With a fixed array created of 100,000 elements and loading and unloading the same associative array() it grows by 75 megs but is very fast at half a second. Just for fun I pushed 100,000 elements on an splFixedArray which are simply the values of the test associative array concatenated into a string and it’s still weighs in at 13 megabytes.

    Here’s the FIFO class:

    class wfArray {
        private $data = "";
        private $shiftPtr = 0;
        public function __construct($keys){
                $this->keys = $keys;
        }
        public function push($val){ //associative array with keys that match those given to constructor
                foreach($this->keys as $key){
                        $this->data .= pack('N', strlen($val[$key])) . $val[$key];
                }
        }
        public function shift(){
                $arr = array();
                if(strlen($this->data) < 1){ return null; }
                foreach($this->keys as $key){
                        $len = unpack('N', substr($this->data, $this->shiftPtr, 4));
                        $len = $len[1];
                        $arr[$key] = substr($this->data, $this->shiftPtr + 4, $len);
                        $this->shiftPtr += 4 + $len;
                }
                return $arr;
        }
}

    Here’s the test script using the FIFO class with the array() tests commented out.

    require_once('wfArray.php');
error_reporting(E_ALL);
$p1 = memory_get_peak_usage();
$stime = microtime(true);
//$arr = array();
$arr = new wfArray(array('owner', 'host', 'path', 'hostkey'));
for($i = 0; $i < 100000; $i++){
        //array_push($arr, array(
        $arr->push(array(
                'owner' => 100,
                'host' => 'www.example.com.co.uk',
                'path' => '/this/is/an/example/path.html',
                'hostkey' => '1111'
                ));
        if($i % 1000 == 0){ echo $i . "\n"; }
}
$i = 0;
while($elem = $arr->shift()){
//while($elem = array_shift($arr)){
        $i++;
        if($i > 1000){ break; }
        if(! ($elem['owner'] == 100 && $elem['host'] == 'www.example.com.co.uk' && $elem['path'] == '/this/is/an/example/path.html' && $elem['hostkey'] == '1111')){
                die("Problem");
        }
}
echo "\nTotal time: " . sprintf('%.3f', microtime(true) - $stime) . "\n";
$p2 = memory_get_peak_usage();
echo "Grew: " . ($p2 - $p1) . "\n";

  • Hidden Data in The Spanish Economic Crisis

    Spain has been all over the press this weekend with a 100 Billion euro bailout agreed to by   Eurozone finance ministers. I spent the last three days in Spain and I find the coverage I’m reading somewhat disconnected with reality.

    I drove down to Madrid from where I live in Southern France and spent Thursday, Friday and Saturday morning there, then drove back home and spent Saturday evening in Pamplona where the San Fermin festival starts in a month with the running of the bulls.

    Madrid is a shining jewel in Europe. The city is immaculately clean and has a wonderful mix of new buildings like the Cuatro Torres that make for a spectacular modern skyline juxtaposed against gorgeous old buildings like the Royal Palace.

    Walking in the Parque del Oeste where the Egyptian temple of Debod was moved to save it from the Aswan Dam, the park is filled with locals who have come out at night for their evening walk. Kids playing, groups of older women or men walking together, lovers in a quiet secluded spot in the park. Everyone is happy and full of life.

    Driving around Spain there is an incredible amount of active road construction and the roads that aren’t being worked on are in great condition with many spectacular bridges.

    Pamplona was absolutely heaving with party-goers on Saturday night including a huge Spanish rock festival, packed bars and pubs and streets literally filled from wall to wall in the older part of town – and the newer part was full of locals out for their evening walk. I visited a heavy metal bar with an Iron Maiden cover band doing a terrible rendition of Maiden’s older stuff and the standing-room-only crowd loving every second of it.

    While in Madrid I got chatting to a local shopkeeper and went out on a limb and asked her about the informal or under-the-table economy in Spain. She explained that many people are employed off the books. I asked why, speculating that the tax in Spain is very high. She explained yes that’s one reason, but taxes are higher in Italy where she’s originally from. Another reason is to keep getting social benefits like a housing benefit. She also said it’s popular to pay someone only 70% of what they’re really paid into their bank account and the rest in cash to avoid tax.

    More evidence that there’s a thriving off-the-books economy is that when we stayed in Madrid, we rented self catering accommodation. The proprietor asked that we pay the roughly 200 euros bill in cash.

    All the economic indicators used to describe the “Spanish crisis” and provide rationales for bailing out spain or to predict how bad the “coming collapse” will be don’t take the informal economy into account. It also makes it difficult to understand the needs of the Spanish people, what the GDP really is, how dependent they really are on social programs and what Spain’s real ability is to service it’s debt.

    To some the informal economy in Spain may seem to be immoral because conventional wisdom holds that one should “pay your taxes” and put your money in a safe place like a bank. But the Spanish people seem to be discovering a way to live without banks and government visibility on how much they earn or what they do with their money. I suspect many of the government assistance programs are over subscribed and do little to serve their intended targets.

    It makes one wonder who the Eurozone is really bailing out.

  • Introducing Wordfence, the Ultimate WordPress security plugin.

    Exec Summary: Last year this WordPress blog was hacked which led me to discover the timthumb vulnerability you may have heard of. I fixed timthumb and worked with Ben, the author to release timthumb 2.0. Then I started work on Wordfence, what I hope will be the best security plugin in the business for WordPress. Wordfence is now completing beta testing. Install it, it’s free and it will help protect your site and keep you off Google’s malware list and in the search results. For beginners: you install Wordfence by going to your WordPress blog’s “Plugins” menu, clicking “Add New” and searching for “Wordfence”.

    Full Post:

    Last year on August 1, this WordPress blog was hacked. Thankfully I caught it quick enough to stay of Google’s malware list. I retraced the hacker’s steps and discovered a zero day vulnerability in many WordPress themes and plugins in the form of a popular image resizer called timthumb.php.

    So I rewrote timthumb.php and worked with the author of timthumb and some of the WordPress team to merge my code into timthumb and we launched it as timthumb version 2.0.

    But getting hacked made me realize that as awesome as WordPress is, it can do security better.

    So I dropped everything and spent the last few months writing what I hope will be the last word in WordPress security.

    A few days ago I quietly released Wordfence into the WordPress plugin repository. Since then I’ve been working with some amazing WordPress publishers to make Wordfence even better and I’ve been rapidly rolling out improvements, enhancements and (yes, believe it or not) a few bug fixes. I’d say Wordfence is getting close to finishing Beta testing at this point.

    Except for two (rather minor) features, Wordfence is completely free. It is also backed up by a cluster of cloud based scanning servers that do most of the heavy lifting to keep your site running super fast.

    Here are some of the more notable ways Wordfence enhances your WordPress security:

    • Scans your core files against a reference copy which I maintain in our cloud servers.
    • Lets you see what has changed, how the file has changed and even repair it.
    • Scans your comments, posts and all files including core, themes, plugins and everything else under your WordPress root directory for malware, virus signatures, vulnerabilities and (very importantly) URL’s that are known to host malware or viruses.
    • I want to re-emphasize the last point. Wordfence keeps known dangerous URL’s, including ALL URL’s that are on Googles’ safe browsing list, out of your comments, pages, posts and files. This is by far my favorite feature because it’s virtually gauranteed to keep you off the dreaded red-page-of-death-malware-list that Chrome and Google use to ban sites.
    • Wordfence comes with a complete firewall that lets you set up rules based on the type of traffic and either throttle or block offenders with an SEO safe 503 (come back later) HTTP message.
    • Another favorite feature of mine is that you can block fake Google crawlers. I actually added this after I tested Wordfence on this site because I couldn’t believe how many scrapers were pretending to be Googlebot. So now they are all instantly blocked.
    • Wordfence uses Google’s recommended reverse-forward DNS verification to sift the fake Googlebots from the real ones.
    • It includes login security against every form of brute force attack out there including abusing your lost-password form.
    • And what’s the point of having all this awesome security if you can’t see who is visiting, who’s getting blocked and what humans and robots are doing? So Wordfence includes real-time traffic that wait..for…it…
    • …Includes crawlers, scrapers, robots and all non-human traffic. Something you can’t get from Google Analytics or any other Javascript based analytics package.
    • I’ve even broken out Googlebot, other crawlers, 404 errors, humans and there’s an All Hits view.
    • And of course it includes commercial grade city-level geolocation which is another feature that comes from our cloud servers.
    • Wordfence is also built using much of the knowledge I’ve gained building Feedjit’s real-time analytics so it is careful to minimize any impact on network, website and mysql database performance and keep your website running super-fast.

    Most importantly, Wordfence comes with a commercial license if you prefer first-class support and support forums for free users including a generic WordPress security forum where I’m happy to answer general config questions.

    Improving WordPress security is going to be a marathon, not a sprint. I’m in this for the long haul. So check out Wordfence now by installing it on your blog and work with me to make the Web and WordPress more secure.

     

  • Life without privacy

    If one were to extrapolate where we will be 100 years from now, I think the most profound difference between then and now may be an almost complete absence of privacy.

    Arthur C Clarke collaborated with Stephen Baxter on a novel called “The Light of Other Days” which describes the development of a camera for consumers based on wormhole technology that allows anyone to see anywhere in 3 dimensional space, and to also move the camera backwards or forwards in time. So besides witnessing the birth of Jesus, one can see what your neighbor was doing three weeks ago in their bathroom.

    They explore how the impact of this technology modifies social behavior and accepted norms.

    We’re heading into this world at a pace that defies belief. Your cellphone contains a GPS that tells the world where you are at any moment, whether you like it or not. If you are one of the 845 million active users on Facebook, there is a record of who you are, your history and your relationships that puts to shame every national security database that ever existed. We have Google maps providing satellite coverage of most of the planet with street level views constantly updated.

    The latest development that has the potential to make Google’s coverage of the Earth real-time is that the FAA will integrate unmanned drones into United States airspace by 2015. To put this in perspective, the lowest low earth orbiting satellites are roughly 100 miles (160km) above Earth. All Google satellite imagery you see is taken from at least that distance and only on a cloud free day. Unmanned drones can reduce that to 500 feet (150 meters) or less, depending on how the FAA decides to regulate them. They can also take photos at a far more acute angle, providing images similar to Google’s street level.

    Consider the amount of street level coverage Google has provided by manually driving vans around the USA and the rest of the world, and then remove need for a human driver, increase the speed and add three dimensional space with it’s lack of traffic signals, greater space and point to point navigation.

    Privacy may become similar to music and movies. The RIAA and MPAA are trying to enforce a value system that worked before digital media became instantly reproducible and redistributable. What if we find ourselves trying to enforce a societal value system that worked before information about individuals became instantly and always available?

    Eric Schmidt’s comments back in ’09 that “If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place,” may prove to be the new social norm we live by 100 years from now.