F5 data flood

Problem: A F5 load balancer LTM sends lots of data to some clients. Sometimes this fills up all the available bandwidth with 1 Gbit or more. At the same time the input traffic does not raise. The traffic charts look like if F5 is attacking some clients (reversed DDoS :-) )

Discussion: After some time of staring at tcpdump output like in Matrix :-) I found the following behavior:

10 A client sends an HTTP request to the F5
20 The F5 forwards the HTTP request to a webserver and gets the data
30 The F5 sends the data to the client, eg. the data has 10kByte which fits into the TCP window
40 A router on the way to the client sends an "ICMP unreachable need fragmentation 1440 Byte"
50 The F5 replays the 10kByte in fragmented packets smaller than 1440 Byte
60 The router sends "need fragmentation" again
70 The F5 replays the already fragmented 10kByte at wire speed
80 goto 60 

Why does the router send “need fragmentation” although the packets are already fragmented ?
The ICMP packet shows that the fragmented packets arrive reassembled at the client side, which means some router between the F5 and the client reassembles the fragments and ignores the need fragmentation packets from the client router. “Automatic reassembly and offloading is evil!”

This is obviously a bug on the router which reassembles the packets. But on the other hand the F5 LTM should not send out the same data in a loop up to wire speed, it should either throttle the packet flow, or it should not retransmit the already fragment packets after “need fragmentation”. The normal retransmit for missing tcp packets should apply.

I filed this bug with F5 twice but they said this behavior is correct and they won’t change it.

Solution: Fix the bogus router if you can. If you can’t I think there is no other solution than don’t use F5 LTM

Update: A work arround is to disable pathmtudiscovery on F5. Without PMTU Disovery the DF IP header is not set and routers and firewalls usually fragment large packets them self and don’t send ICMP Need Frag messages.

# tmsh
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos)# sys db
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.pathmtudiscovery value disable
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.enforcepathmtu value disable

Version: tested with F5 LTM 11.2.1 and 11.5

Share on Facebook0Tweet about this on TwitterShare on Google+0Email this to someone