F5 data flood

Problem: A F5 load balancer LTM sends lots of data to some clients. Sometimes this fills up all the available bandwidth with 1 Gbit or more. At the same time the input traffic does not raise. The traffic charts look like if F5 is attacking some clients (reversed DDoS :-) )

Discussion: After some time of staring at tcpdump output like in Matrix :-) I found the following behavior:

10 A client sends an HTTP request to the F5
20 The F5 forwards the HTTP request to a webserver and gets the data
30 The F5 sends the data to the client, eg. the data has 10kByte which fits into the TCP window
40 A router on the way to the client sends an "ICMP unreachable need fragmentation 1440 Byte"
50 The F5 replays the 10kByte in fragmented packets smaller than 1440 Byte
60 The router sends "need fragmentation" again
70 The F5 replays the already fragmented 10kByte at wire speed
80 goto 60 

Why does the router send “need fragmentation” although the packets are already fragmented ?
The ICMP packet shows that the fragmented packets arrive reassembled at the client side, which means some router between the F5 and the client reassembles the fragments and ignores the need fragmentation packets from the client router. “Automatic reassembly and offloading is evil!”

This is obviously a bug on the router which reassembles the packets. But on the other hand the F5 LTM should not send out the same data in a loop up to wire speed, it should either throttle the packet flow, or it should not retransmit the already fragment packets after “need fragmentation”. The normal retransmit for missing tcp packets should apply.

I filed this bug with F5 twice but they said this behavior is correct and they won’t change it.

Solution: Fix the bogus router if you can. If you can’t I think there is no other solution than don’t use F5 LTM

Update: A work arround is to disable pathmtudiscovery on F5. Without PMTU Disovery the DF IP header is not set and routers and firewalls usually fragment large packets them self and don’t send ICMP Need Frag messages.

# tmsh
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos)# sys db
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.pathmtudiscovery value disable
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.enforcepathmtu value disable

Version: tested with F5 LTM 11.2.1 and 11.5

Ping to Windows Server 2012 with Direct Access fails

Problem: Ping from Linux computers to Windows Server 2012 with Direct Access fails for hours (no response) although connectivity is working (eg. HTTPS)

Discussion: Ping from other windows machines and Cisco machines worked, but from Linux machines ping was not answered. I tried to find the difference between those ICMP Echo Request Packets, and reduced it to 16 Bits in the ICMP Packet. The ID field! Windows always uses “2”. Linux uses the pid of the ping process.

I patched the ping source to select the ID manually, and scanned through the values 0 – 65535.

Ping was answered for IDs 0 – 6000

Ping was not answered for IDs 6001 – 47000

Ping was answered for IDs 47001 – 65535

A really strange Microsoft bug.

Solution: ??? Only Microsoft can fix this. The workaround is to set the id manually with my patched ping client.

Version: any Linux with standard ping. Windows 2012 with Direct Access patched up to date (21.8.2013)