Well the title says it all. Internet routers live at Layer 3 [the Network Layer] of the OSI model which I’ve included to the left. HTTP and HTTPS live at Layer 7 (Application layer) of the OSI model, although some may argue HTTPS lives at Layer 6.
So how is it that Layer 3 devices like routers treat HTTPS traffic differently?
Because HTTPS servers set the DF or Do Not Fragment IP flag on packets and regular HTTP servers do not.
This matters because HTTP and HTTPS usually transfer a lot of data. That means that the packets are usually quite large and are often the maximum allowed size.
So if a server sends out a very big HTTP packet and it goes through a route on the network that does not allow packets that size, then the router in question simply breaks the packet up.
But if a server sends out a big HTTPS packet and it hits a route that doesn’t allow packets that size, the routers on that route can’t break the packet up. So they drop the packet and send back an ICMP message telling the machine that sent the big packet to adjust it’s MTU (maximum transfer unit) size and resend the packet. This is called Path MTU Discovery.
This can create some interesting problems that don’t exist with plain HTTP. For example, if your ops team has gotten a little overzealous with security and decided to filter out all ICMP traffic, your web server won’t receive any of those ICMP messages I’ve described above telling it to break up it’s packets and resend them. So large secure packets that usually are sent halfway through a secure HTTPS connection will just be dropped. So visitors to your website who are across network paths that need to have their packets broken up into smaller pieces will see half-loaded pages from the secure part of your site.
If you have the problem I’ve described above there are two solutions: If you’re a webmaster, make sure your web server can receive ICMP messages [You need to allow ICMP code 4 “Fragmentation needed and DF bit set”]. If you’re a web surfer (client) and are trying to access a secure site that has ICMP disabled, adjust your network card’s MTU to be smaller than the default (usually the default is 1500 for ethernet).
But the bottom line is that if everything else is working fine and you are having a problem sending or receiving HTTPS traffic, know that the big difference with HTTPS traffic over regular web traffic is that the packets can’t be broken up.
That’s interesting… It does not seem to be any restriction (neither in the protocol, nor in technical implementation) justifying the DF option to be on for encrypted protocols…
Why implementers did so ? is there a reasonable explanation of that or is it just “feature-not-a-bug” they thought useful ?
Swoög
Commented on October 20, 2009 at 1:01 pm
@Mark:
I am not seeing the behaviour you’re describing. Take https://gmail.com for instance:
——-
# tcpdump -v src mail.google.com and tcp port 443
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:17:40.901281 IP (tos 0x0, ttl 51, id 21756, offset 0, flags [none], proto TCP (6), length 60) gy-in-f17.1e100.net.https > my.host.here.53272: S, cksum 0xf329 (correct), 616756817:616756817(0) ack 3946542441 win 5672
13:17:40.928660 IP (tos 0x0, ttl 51, id 21757, offset 0, flags [none], proto TCP (6), length 52) gy-in-f17.1e100.net.https > my.host.here.53272: ., cksum 0x36fc (correct), ack 115 win 89
13:17:40.929612 IP (tos 0x0, ttl 51, id 21758, offset 0, flags [none], proto TCP (6), length 1470) gy-in-f17.1e100.net.https > my.host.here.53272: . 1:1419(1418) ack 115 win 89
13:17:40.929671 IP (tos 0x0, ttl 51, id 21759, offset 0, flags [none], proto TCP (6), length 331) gy-in-f17.1e100.net.https > my.host.here.53272: P 1419:1698(279) ack 115 win 89
13:17:40.960292 IP (tos 0x0, ttl 51, id 21760, offset 0, flags [none], proto TCP (6), length 278) gy-in-f17.1e100.net.https > my.host.here.53272: P 1698:1924(226) ack 301 win 106
13:17:41.027614 IP (tos 0x0, ttl 51, id 21761, offset 0, flags [none], proto TCP (6), length 52) gy-in-f17.1e100.net.https > my.host.here.53272: ., cksum 0x2ae5 (correct), ack 1088 win 130
13:17:41.157382 IP (tos 0x0, ttl 51, id 21762, offset 0, flags [none], proto TCP (6), length 828) gy-in-f17.1e100.net.https > my.host.here.53272: P 1924:2700(776) ack 1088 win 130
13:17:41.158629 IP (tos 0x0, ttl 51, id 21763, offset 0, flags [none], proto TCP (6), length 397) gy-in-f17.1e100.net.https > my.host.here.53272: P 2700:3045(345) ack 1088 win 130
——-
Commented on October 21, 2009 at 11:20 am
I’d add that I _do_ see the same behaviour as you when visiting paypal.com. But this is definitely not a universal web server / https truth. 🙂
——-
# tcpdump -v src http://www.paypal.com and tcp port 443
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:25:15.536007 IP (tos 0x0, ttl 246, id 52428, offset 0, flags [DF], proto TCP (6), length 44) 66.211.169.65.https > my.host.here.59435: S, cksum 0x26b7 (correct), 1966361028:1966361028(0) ack 2475762734 win 8190
13:25:15.585680 IP (tos 0x0, ttl 246, id 19924, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 1:1381(1380) ack 114 win 40845
13:25:15.585794 IP (tos 0x0, ttl 246, id 20436, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 1381:2761(1380) ack 114 win 40845
13:25:15.585903 IP (tos 0x0, ttl 246, id 20948, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 2761:4141(1380) ack 114 win 40845
13:25:15.585932 IP (tos 0x0, ttl 246, id 21460, offset 0, flags [DF], proto TCP (6), length 243) 66.211.169.65.https > my.host.here.59435: P 4141:4344(203) ack 114 win 40845
13:25:15.837024 IP (tos 0x0, ttl 246, id 54784, offset 0, flags [DF], proto TCP (6), length 40) 66.211.169.65.https > my.host.here.59435: ., cksum 0xad2c (correct), ack 304 win 40655
13:25:15.837831 IP (tos 0x0, ttl 246, id 62464, offset 0, flags [DF], proto TCP (6), length 91) 66.211.169.65.https > my.host.here.59435: P 4344:4395(51) ack 304 win 40655
13:25:15.887928 IP (tos 0x0, ttl 246, id 59400, offset 0, flags [DF], proto TCP (6), length 40) 66.211.169.65.https > my.host.here.59435: ., cksum 0xacf9 (correct), ack 1005 win 39954
13:25:16.461813 IP (tos 0x0, ttl 246, id 22901, offset 0, flags [DF], proto TCP (6), length 1005) 66.211.169.65.https > my.host.here.59435: P 4395:5360(965) ack 1005 win 39954
13:25:16.461832 IP (tos 0x0, ttl 246, id 25973, offset 0, flags [DF], proto TCP (6), length 77) 66.211.169.65.https > my.host.here.59435: P, cksum 0xe9cc (correct), 5360:5397(37) ack 1005 win 39954
——-
Commented on October 21, 2009 at 11:27 am