Well the title says it all. Internet routers live at Layer 3 [the Network Layer] of the OSI model which I’ve included to the left. HTTP and HTTPS live at Layer 7 (Application layer) of the OSI model, although some may argue HTTPS lives at Layer 6.
So how is it that Layer 3 devices like routers treat HTTPS traffic differently?
Because HTTPS servers set the DF or Do Not Fragment IP flag on packets and regular HTTP servers do not.
This matters because HTTP and HTTPS usually transfer a lot of data. That means that the packets are usually quite large and are often the maximum allowed size.
So if a server sends out a very big HTTP packet and it goes through a route on the network that does not allow packets that size, then the router in question simply breaks the packet up.
But if a server sends out a big HTTPS packet and it hits a route that doesn’t allow packets that size, the routers on that route can’t break the packet up. So they drop the packet and send back an ICMP message telling the machine that sent the big packet to adjust it’s MTU (maximum transfer unit) size and resend the packet. This is called Path MTU Discovery.
This can create some interesting problems that don’t exist with plain HTTP. For example, if your ops team has gotten a little overzealous with security and decided to filter out all ICMP traffic, your web server won’t receive any of those ICMP messages I’ve described above telling it to break up it’s packets and resend them. So large secure packets that usually are sent halfway through a secure HTTPS connection will just be dropped. So visitors to your website who are across network paths that need to have their packets broken up into smaller pieces will see half-loaded pages from the secure part of your site.
If you have the problem I’ve described above there are two solutions: If you’re a webmaster, make sure your web server can receive ICMP messages [You need to allow ICMP code 4 "Fragmentation needed and DF bit set"]. If you’re a web surfer (client) and are trying to access a secure site that has ICMP disabled, adjust your network card’s MTU to be smaller than the default (usually the default is 1500 for ethernet).
But the bottom line is that if everything else is working fine and you are having a problem sending or receiving HTTPS traffic, know that the big difference with HTTPS traffic over regular web traffic is that the packets can’t be broken up.
I’d add that I _do_ see the same behaviour as you when visiting paypal.com. But this is definitely not a universal web server / https truth.
——-
# tcpdump -v src http://www.paypal.com and tcp port 443
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:25:15.536007 IP (tos 0×0, ttl 246, id 52428, offset 0, flags [DF], proto TCP (6), length 44) 66.211.169.65.https > my.host.here.59435: S, cksum 0x26b7 (correct), 1966361028:1966361028(0) ack 2475762734 win 8190
13:25:15.585680 IP (tos 0×0, ttl 246, id 19924, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 1:1381(1380) ack 114 win 40845
13:25:15.585794 IP (tos 0×0, ttl 246, id 20436, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 1381:2761(1380) ack 114 win 40845
13:25:15.585903 IP (tos 0×0, ttl 246, id 20948, offset 0, flags [DF], proto TCP (6), length 1420) 66.211.169.65.https > my.host.here.59435: P 2761:4141(1380) ack 114 win 40845
13:25:15.585932 IP (tos 0×0, ttl 246, id 21460, offset 0, flags [DF], proto TCP (6), length 243) 66.211.169.65.https > my.host.here.59435: P 4141:4344(203) ack 114 win 40845
13:25:15.837024 IP (tos 0×0, ttl 246, id 54784, offset 0, flags [DF], proto TCP (6), length 40) 66.211.169.65.https > my.host.here.59435: ., cksum 0xad2c (correct), ack 304 win 40655
13:25:15.837831 IP (tos 0×0, ttl 246, id 62464, offset 0, flags [DF], proto TCP (6), length 91) 66.211.169.65.https > my.host.here.59435: P 4344:4395(51) ack 304 win 40655
13:25:15.887928 IP (tos 0×0, ttl 246, id 59400, offset 0, flags [DF], proto TCP (6), length 40) 66.211.169.65.https > my.host.here.59435: ., cksum 0xacf9 (correct), ack 1005 win 39954
13:25:16.461813 IP (tos 0×0, ttl 246, id 22901, offset 0, flags [DF], proto TCP (6), length 1005) 66.211.169.65.https > my.host.here.59435: P 4395:5360(965) ack 1005 win 39954
13:25:16.461832 IP (tos 0×0, ttl 246, id 25973, offset 0, flags [DF], proto TCP (6), length 77) 66.211.169.65.https > my.host.here.59435: P, cksum 0xe9cc (correct), 5360:5397(37) ack 1005 win 39954
——-
@Mark:
I am not seeing the behaviour you’re describing. Take https://gmail.com for instance:
——-
# tcpdump -v src mail.google.com and tcp port 443
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:17:40.901281 IP (tos 0×0, ttl 51, id 21756, offset 0, flags [none], proto TCP (6), length 60) gy-in-f17.1e100.net.https > my.host.here.53272: S, cksum 0xf329 (correct), 616756817:616756817(0) ack 3946542441 win 5672
13:17:40.928660 IP (tos 0×0, ttl 51, id 21757, offset 0, flags [none], proto TCP (6), length 52) gy-in-f17.1e100.net.https > my.host.here.53272: ., cksum 0x36fc (correct), ack 115 win 89
13:17:40.929612 IP (tos 0×0, ttl 51, id 21758, offset 0, flags [none], proto TCP (6), length 1470) gy-in-f17.1e100.net.https > my.host.here.53272: . 1:1419(1418) ack 115 win 89
13:17:40.929671 IP (tos 0×0, ttl 51, id 21759, offset 0, flags [none], proto TCP (6), length 331) gy-in-f17.1e100.net.https > my.host.here.53272: P 1419:1698(279) ack 115 win 89
13:17:40.960292 IP (tos 0×0, ttl 51, id 21760, offset 0, flags [none], proto TCP (6), length 278) gy-in-f17.1e100.net.https > my.host.here.53272: P 1698:1924(226) ack 301 win 106
13:17:41.027614 IP (tos 0×0, ttl 51, id 21761, offset 0, flags [none], proto TCP (6), length 52) gy-in-f17.1e100.net.https > my.host.here.53272: ., cksum 0x2ae5 (correct), ack 1088 win 130
13:17:41.157382 IP (tos 0×0, ttl 51, id 21762, offset 0, flags [none], proto TCP (6), length 828) gy-in-f17.1e100.net.https > my.host.here.53272: P 1924:2700(776) ack 1088 win 130
13:17:41.158629 IP (tos 0×0, ttl 51, id 21763, offset 0, flags [none], proto TCP (6), length 397) gy-in-f17.1e100.net.https > my.host.here.53272: P 2700:3045(345) ack 1088 win 130
——-
Dmitriy I tried to email this to you but it bounced:
Hi Dmitriy,
Any chance you can send me a packet capture of an SSL session where you’re seeing that the DF flag isn’t set? I’ve only ever seen the DF flag set for HTTPS and I’m very interested in what you’re seeing. Just add -w filename.dump to your tcpdump options. My email is mmaunder at gmail dot com.
Thanks!!
Mark.
@Mark
You can open dump files with tcpdump on Linux with “tcpdump -r filename” and you can see whether packets have DF set or not in tcpdump by calling it with a couple of extra -v flags (tcpdump -vv). Even though yes, Wireshark rulez
I used the same methodology for my tests – and I am still not able to recreate it under Linux or Mac OS X, with wget or curl.
I am not trying to say that your findings are wrong, esp. since they are based on a real-life scenario that you were troubleshooting.
I just think there might be more to it, some factor that hasn’t been uncovered yet.
I’m not sure why DF seems to be set on all SSL traffic. You can find out how to set up PMTUD on a socket which causes the DF flag to be set here:
http://manpages.ubuntu.com/manpages/jaunty/man7/ip.7.html
Also Google the socket option IP_MTU_DISCOVER. I’ve seen a few results that bring up openssl source code.
I also saw a mention of the bonk attack being a DoS for fragmented packets and it may be related to why SSL authors set IP_MTU_DISCOVER on their servers.
Please do share what you find.
That’s interesting… It does not seem to be any restriction (neither in the protocol, nor in technical implementation) justifying the DF option to be on for encrypted protocols…
Why implementers did so ? is there a reasonable explanation of that or is it just “feature-not-a-bug” they thought useful ?
Swoög
Hi Dmitri,
Sure. On another user’s suggestion I’ve included a reference to PMTUD which is what I’m describing in the ICMP example.
The DF/non-DF data is from my own research. I was recently diagnosing a network problem that existed via HTTPS and not HTTP and discovered the difference. On Linux, most secure protocols like SCP, SSH and HTTPS set the DF flag on IP packets.
You can duplicate my research as follows:
Run the following command on your linux box to capture all SSL packets inbound and outbound. I’m assuming your ethernet card that will see the traffic is eth0:
tcpdump -w dump1.dump -s 1600 -i eth0 port 443
Then fetch a secure web document from your linux box using curl or your favorite command line web client:
curl -k –connect-timeout 20 –max-time 60 -v “https://www.paypal.com/”
Then hit CTRL-C to stop the tcpdump capture. Copy dump1.dump to your local machine. Install Wireshark (google it).
Then open the captured file with Wireshark. Click one of the data packets. Look at the IP part of the packet and you’ll see the DF flag is set.
Now repeat the process above, but modify the tcpdump port to be 80 and fetch a regular web document from a non-secure site. You’ll notice the packets do not have the DF flag set.
You can test this on a variety of protocols. Try it on SSH and SCP. Also note the size of the packets that are being captured in each case. With things like SSH the packets are small – usually single keystrokes. With HTTPS they are very large which is why MTU problems manifest themselves with HTTPS but not SSH which also has the DF flag set.
Mark.
Interesting. I just did a quick test and I saw both client and server set DF when talking either HTTP or HTTPS.
A quick grep on RFC2818 did not reveal any mentions of DF either.
Could you please describe sources based on which you reached this conclusion? Will greatly appreciate it!
Thanks.