F5 – AlexT Debugs

F5 iRule Class Match Crash

Problem: F5 iRules with “class match” crash sometimes with this message:

/Common/UA_DETECT – ambiguous option “-“: must be -all, -index, -element, -name, or -value while executing “class match [string tolower [HTTP::header User-Agent]] contains UA_STRINGS”

Discussion: the class match command has optional parameters, when the HTTP header User-Agent starts with a “-” it gets intepreted by the tcl interpreter. This is dangerous, because it’s actually a kind of code injection, with possible terrible impact.

Solution: add “‐‐” as first parameter to the class match command

class match ‐‐ [string tolower [HTTP::header User-Agent]] contains UA_STRINGS

Version: F5 LTM 12.1.2

Configure F5 TLS (SSL) Cipher String

The list of TLS ciphers is changing quite rapidly, old ciphers are considered insecure, and new ciphers are added.

When you configure a virtual server on an F5 you can add a TLS client profile, which means F5 is doing TLS to the client. I think this is a bit misleading because with “SSL client profile” you are actually configuring a TLS server.

You have to make your own SSL client profile, to add your key and certificate to the profile. You can do that at:
Local Traffic – Profiles – SSL – Client

When you change to Advanced Configuration you can change the “Ciphers” string. This setting changes the list of allowed ciphers and it’s order. You might want to change this for better security or to get a higher rating at https://ssllabs.com .

The default setting is “DEFAULT”. This translates to a longer string. For 12.1.1 it’s

!SSLv2:!EXPORT:DHE+AES-GCM:DHE+AES:DHE+3DES:RSA+AES-GCM:RSA+AES:RSA+3DES:ECDHE+AES-GCM:ECDHE+AES:ECDHE+3DES:-MD5:-SSLv3:-RC4

This list translates to the following ciphers:

 ID SUITE BITS PROT METHOD CIPHER MAC KEYX
 0: 159 DHE-RSA-AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 EDH/RSA 
 1: 158 DHE-RSA-AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 EDH/RSA 
 2: 107 DHE-RSA-AES256-SHA256 256 TLS1.2 Native AES SHA256 EDH/RSA 
 3: 57 DHE-RSA-AES256-SHA 256 TLS1 Native AES SHA EDH/RSA 
 4: 57 DHE-RSA-AES256-SHA 256 TLS1.1 Native AES SHA EDH/RSA 
 5: 57 DHE-RSA-AES256-SHA 256 TLS1.2 Native AES SHA EDH/RSA 
 6: 57 DHE-RSA-AES256-SHA 256 DTLS1 Native AES SHA EDH/RSA 
 7: 103 DHE-RSA-AES128-SHA256 128 TLS1.2 Native AES SHA256 EDH/RSA 
 8: 51 DHE-RSA-AES128-SHA 128 TLS1 Native AES SHA EDH/RSA 
 9: 51 DHE-RSA-AES128-SHA 128 TLS1.1 Native AES SHA EDH/RSA 
10: 51 DHE-RSA-AES128-SHA 128 TLS1.2 Native AES SHA EDH/RSA 
11: 51 DHE-RSA-AES128-SHA 128 DTLS1 Native AES SHA EDH/RSA 
12: 22 DHE-RSA-DES-CBC3-SHA 168 TLS1 Native DES SHA EDH/RSA 
13: 22 DHE-RSA-DES-CBC3-SHA 168 TLS1.1 Native DES SHA EDH/RSA 
14: 22 DHE-RSA-DES-CBC3-SHA 168 TLS1.2 Native DES SHA EDH/RSA 
15: 22 DHE-RSA-DES-CBC3-SHA 168 DTLS1 Native DES SHA EDH/RSA 
16: 157 AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 RSA 
17: 156 AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 RSA 
18: 61 AES256-SHA256 256 TLS1.2 Native AES SHA256 RSA 
19: 53 AES256-SHA 256 TLS1 Native AES SHA RSA 
20: 53 AES256-SHA 256 TLS1.1 Native AES SHA RSA 
21: 53 AES256-SHA 256 TLS1.2 Native AES SHA RSA 
22: 53 AES256-SHA 256 DTLS1 Native AES SHA RSA 
23: 60 AES128-SHA256 128 TLS1.2 Native AES SHA256 RSA 
24: 47 AES128-SHA 128 TLS1 Native AES SHA RSA 
25: 47 AES128-SHA 128 TLS1.1 Native AES SHA RSA 
26: 47 AES128-SHA 128 TLS1.2 Native AES SHA RSA 
27: 47 AES128-SHA 128 DTLS1 Native AES SHA RSA 
28: 10 DES-CBC3-SHA 168 TLS1 Native DES SHA RSA 
29: 10 DES-CBC3-SHA 168 TLS1.1 Native DES SHA RSA 
30: 10 DES-CBC3-SHA 168 TLS1.2 Native DES SHA RSA 
31: 10 DES-CBC3-SHA 168 DTLS1 Native DES SHA RSA 
32: 49200 ECDHE-RSA-AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 ECDHE_RSA 
33: 49199 ECDHE-RSA-AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 ECDHE_RSA 
34: 49192 ECDHE-RSA-AES256-SHA384 256 TLS1.2 Native AES SHA384 ECDHE_RSA 
35: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1 Native AES SHA ECDHE_RSA 
36: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.1 Native AES SHA ECDHE_RSA 
37: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.2 Native AES SHA ECDHE_RSA 
38: 49191 ECDHE-RSA-AES128-SHA256 128 TLS1.2 Native AES SHA256 ECDHE_RSA 
39: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1 Native AES SHA ECDHE_RSA 
40: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.1 Native AES SHA ECDHE_RSA 
41: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.2 Native AES SHA ECDHE_RSA 
42: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1 Native DES SHA ECDHE_RSA 
43: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.1 Native DES SHA ECDHE_RSA 
44: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.2 Native DES SHA ECDHE_RSA

Example:

Currently Diffie-Hellman 1024 is considered insecure so you want to change the cipher string. You can use this to disable DHE:

!DHE:DEFAULT

You can check the resulting cipher list before applying the change from the console:

 # tmm --clientciphers '!DHE:DEFAULT'
 ID SUITE BITS PROT METHOD CIPHER MAC KEYX
 0: 157 AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 RSA 
 1: 156 AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 RSA 
 2: 61 AES256-SHA256 256 TLS1.2 Native AES SHA256 RSA 
 3: 53 AES256-SHA 256 TLS1 Native AES SHA RSA 
 4: 53 AES256-SHA 256 TLS1.1 Native AES SHA RSA 
 5: 53 AES256-SHA 256 TLS1.2 Native AES SHA RSA 
 6: 53 AES256-SHA 256 DTLS1 Native AES SHA RSA 
 7: 60 AES128-SHA256 128 TLS1.2 Native AES SHA256 RSA 
 8: 47 AES128-SHA 128 TLS1 Native AES SHA RSA 
 9: 47 AES128-SHA 128 TLS1.1 Native AES SHA RSA 
10: 47 AES128-SHA 128 TLS1.2 Native AES SHA RSA 
11: 47 AES128-SHA 128 DTLS1 Native AES SHA RSA 
12: 10 DES-CBC3-SHA 168 TLS1 Native DES SHA RSA 
13: 10 DES-CBC3-SHA 168 TLS1.1 Native DES SHA RSA 
14: 10 DES-CBC3-SHA 168 TLS1.2 Native DES SHA RSA 
15: 10 DES-CBC3-SHA 168 DTLS1 Native DES SHA RSA 
16: 49200 ECDHE-RSA-AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 ECDHE_RSA 
17: 49199 ECDHE-RSA-AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 ECDHE_RSA 
18: 49192 ECDHE-RSA-AES256-SHA384 256 TLS1.2 Native AES SHA384 ECDHE_RSA 
19: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1 Native AES SHA ECDHE_RSA 
20: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.1 Native AES SHA ECDHE_RSA 
21: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.2 Native AES SHA ECDHE_RSA 
22: 49191 ECDHE-RSA-AES128-SHA256 128 TLS1.2 Native AES SHA256 ECDHE_RSA 
23: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1 Native AES SHA ECDHE_RSA 
24: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.1 Native AES SHA ECDHE_RSA 
25: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.2 Native AES SHA ECDHE_RSA 
26: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1 Native DES SHA ECDHE_RSA 
27: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.1 Native DES SHA ECDHE_RSA 
28: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.2 Native DES SHA ECDHE_RSA

The cipher string “!DHE:DEFAULT” results in A- grade at ssllabs. The reason for minus A is:

“The server does not support Forward Secrecy with the reference browsers. Grade reduced to A-”

If you want to give priority to the “Forward Secrecy” ciphers and lower priotirty to 3DES, your can change the cipher string to

-3DES:ECDHE:!DHE:DEFAULT

 # tmm --clientciphers '-3DES:ECDHE:!DHE:DEFAULT'
 ID SUITE BITS PROT METHOD CIPHER MAC KEYX
 0: 49200 ECDHE-RSA-AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 ECDHE_RSA 
 1: 49192 ECDHE-RSA-AES256-SHA384 256 TLS1.2 Native AES SHA384 ECDHE_RSA 
 2: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1 Native AES SHA ECDHE_RSA 
 3: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.1 Native AES SHA ECDHE_RSA 
 4: 49172 ECDHE-RSA-AES256-CBC-SHA 256 TLS1.2 Native AES SHA ECDHE_RSA 
 5: 49199 ECDHE-RSA-AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 ECDHE_RSA 
 6: 49191 ECDHE-RSA-AES128-SHA256 128 TLS1.2 Native AES SHA256 ECDHE_RSA 
 7: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1 Native AES SHA ECDHE_RSA 
 8: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.1 Native AES SHA ECDHE_RSA 
 9: 49171 ECDHE-RSA-AES128-CBC-SHA 128 TLS1.2 Native AES SHA ECDHE_RSA 
10: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1 Native DES SHA ECDHE_RSA 
11: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.1 Native DES SHA ECDHE_RSA 
12: 49170 ECDHE-RSA-DES-CBC3-SHA 168 TLS1.2 Native DES SHA ECDHE_RSA 
13: 157 AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 RSA 
14: 156 AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 RSA 
15: 61 AES256-SHA256 256 TLS1.2 Native AES SHA256 RSA 
16: 53 AES256-SHA 256 TLS1 Native AES SHA RSA 
17: 53 AES256-SHA 256 TLS1.1 Native AES SHA RSA 
18: 53 AES256-SHA 256 TLS1.2 Native AES SHA RSA 
19: 53 AES256-SHA 256 DTLS1 Native AES SHA RSA 
20: 60 AES128-SHA256 128 TLS1.2 Native AES SHA256 RSA 
21: 47 AES128-SHA 128 TLS1 Native AES SHA RSA 
22: 47 AES128-SHA 128 TLS1.1 Native AES SHA RSA 
23: 47 AES128-SHA 128 TLS1.2 Native AES SHA RSA 
24: 47 AES128-SHA 128 DTLS1 Native AES SHA RSA 
25: 10 DES-CBC3-SHA 168 TLS1 Native DES SHA RSA 
26: 10 DES-CBC3-SHA 168 TLS1.1 Native DES SHA RSA 
27: 10 DES-CBC3-SHA 168 TLS1.2 Native DES SHA RSA 
28: 10 DES-CBC3-SHA 168 DTLS1 Native DES SHA RSA

This string moves the ECDHE ciphers to a higher priority, and 3DES moves down. The result is “Grade A” or “A+”.

But be carefull. Better cipher often means worse performance. Depending on your hardware “ECDHE-RSA-AES128-GCM-SHA256” may be 3 times slower than “AES128-GCM-SHA256”.

Links: https://support.f5.com/csp/article/K17370 https://ssllabs.com

Annoyances while updating F5

I had to upgrade several F5 load balancers from 11.5 to 12.1 in the last weeks. Usually updating F5 is quiet easy, but there are bugs or annoyances you should know:

Sometimes F5 asks for re-activating after the first boot into the new version. It seems that you have to install the new version in a specific order to prevent this: BIGIP-Firmware, licence re-activate, BIGIP-Hotfix, Restart.
Remember the appliance has to be in stand by mode when re-activating the licence.
If the F5 asks for licence re-activation after reboot, it should be easy to re-activate. But even after licence activation, F5 is not working correctly. The SNMP MIB for LTM is not complete. You have to reboot again to activate the LTM SNMP MIB tree again.
When switching from 11.5 to 12.1 the SNMP MIB changed. Serious manufactures that special care to keep the SNMP stable and compatible. F5 doesn’t they changed data types from 11.5 to 12.1 which means you have to update the MIB database. On the other hand if you do, you cannot query those OIDs from older machines. That’s why other manufactures never change data type, The correct way is to add new OIDs wait some years and deprecate the old OID. F5 doesn’t. Here’s a diff part of mibs_f5/F5-BIGIP-LOCAL-MIB.txt:
```
- ltmNodeAddrStatCurSessions Gauge,
- ltmNodeAddrStatCurrentConnsPerSec Gauge,
- ltmNodeAddrStatDurationRateExceeded Gauge
+ ltmNodeAddrStatCurSessions CounterBasedGauge64,
+ ltmNodeAddrStatCurrentConnsPerSec CounterBasedGauge64,
+ ltmNodeAddrStatDurationRateExceeded CounterBasedGauge64
```
Beside this breaking incompatibility between 11.5 and 12.1, they also changed some value names, which breaks software that used these names. This is not a bug but still annoying. Remember: an API has to be stable and backward compatible.
```
 inband(2),
- forcedUp(3),
+ forced-up(3),
 up(4),
 down(19),
- forcedDown(20),
- iruleDown(22),
- inbandDown(23),
- downManualResume(24),
+ forced-down(20),
+ irule-down(22),
+ inband-down(23),
+ down-manual-resume(24),
 disabled(25)
```

Versions: F5 LTM 11.5.1, 11.5.3 and 12.1.1

F5 Drops WebSockets

Problem: F5 LTM is used als load balancer for multiple web servers. When the client opens a websocket connection to the web server, the connection is closed.

Discussion: F5 LTM version before 11.6.0 has a bug in the request_log module (profile). The “request_log” module crashes and drops the connection. The bug is a known issue:

https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16690.html

Solution: If you cannot update. You can apply an iRule as a work arround:

when HTTP_REQUEST {
  if { [string tolower [HTTP::header Upgrade]] contains "websocket" } {
     HTTP::disable
  }
}

This iRule disables http handling and logging for websocket requests.

F5 sending packets to wrong destination?

Problem: You have a network with two upstream routers and an F5 LTM loadbalancer. Even though the default gateway points to router R1 the F5 LTM sends packets to the mac address of R2.

Discussion: “This is a feature not a bug”. This “Feature” is called “Auto Last Hop”. Which means the F5 answers packets allways to the mac address of the received packet. This may be usefull in some cases. But from the view of standards, compliance and security this behavior is a bug. In my case R2 sent some traffic to the F5 because of BGP multihoming, and received the answer allthough R1 should have received the traffic. Unfortunatly this setting is “Enabled” by default on F5.

If a hacker manages to inject a request with a forged IP address, he will receive the answer even if the route to this IP points in a different direction.

Solution: This “Feature” can (and should) be disabled, if you don’t explicitly need it. It can be disabled globally, per VLAN, per Virtual Server or per SNAT policy.
You can find this setting globally in the web interface: System -> Configuration -> Local Traffic -> General
And for VLAN it can be disabled at: Network -> VLANs -> Configuration [ Advanced]

Version: F5 LTM 11.5.1 8.0.175

Link: https://support.f5.com/kb/en-us/solutions/public/13000/800/sol13876.html