Annoyances while updating F5

I had to upgrade several F5 load balancers from 11.5 to 12.1 in the last weeks. Usually updating F5 is quiet easy, but there are bugs or annoyances you should know:

  1. Sometimes F5 asks for re-activating after the first boot into the new version. It seems that you have to install the new version in a specific order to prevent this: BIGIP-Firmware, licence re-activate, BIGIP-Hotfix, Restart.
    Remember the appliance has to be in stand by mode when re-activating the licence.
  2. If the F5 asks for licence re-activation after reboot, it should be easy to re-activate. But even after licence activation, F5 is not working correctly. The SNMP MIB for LTM is not complete. You have to reboot again to activate the LTM SNMP MIB tree again.
  3. When switching from 11.5 to 12.1 the SNMP MIB changed. Serious manufactures that special care to keep the SNMP stable and compatible. F5 doesn’t they changed data types from 11.5 to 12.1 which means you have to update the MIB database. On the other hand if you do, you cannot query those OIDs from older machines. That’s why other manufactures never change data type, The correct way is to add new OIDs wait some years and deprecate the old OID. F5 doesn’t. Here’s a diff part of mibs_f5/F5-BIGIP-LOCAL-MIB.txt:
    - ltmNodeAddrStatCurSessions Gauge,
    - ltmNodeAddrStatCurrentConnsPerSec Gauge,
    - ltmNodeAddrStatDurationRateExceeded Gauge
    + ltmNodeAddrStatCurSessions CounterBasedGauge64,
    + ltmNodeAddrStatCurrentConnsPerSec CounterBasedGauge64,
    + ltmNodeAddrStatDurationRateExceeded CounterBasedGauge64
  4. Beside this breaking incompatibility between 11.5 and 12.1, they also changed some value names, which breaks software that used these names. This is not a bug but still annoying. Remember: an API has to be stable and backward compatible.
     inband(2),
    - forcedUp(3),
    + forced-up(3),
     up(4),
     down(19),
    - forcedDown(20),
    - iruleDown(22),
    - inbandDown(23),
    - downManualResume(24),
    + forced-down(20),
    + irule-down(22),
    + inband-down(23),
    + down-manual-resume(24),
     disabled(25)

Versions: F5 LTM 11.5.1, 11.5.3 and 12.1.1

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Rare Connection Resets after Linux Upgrade

Problem: after upgrading from Debian 6 to Debian 8 some of the machines lose their ethernet network connection under heavy load for some seconds rarely. You find lines like these in syslog:

[2333099.217735] NETDEV WATCHDOG: eth1 (tg3): transmit queue 0 timed out
[2333099.217966] tg3 0000:03:04.1 eth1: transmit timed out, resetting
[2333099.384391] tg3 0000:03:04.1 eth1: 0: Host status block [00000001:0000003c:(0000:0018:0000):(0018:01e9)]
[2333099.386091] tg3 0000:03:04.1 eth1: 0: NAPI info [00000022:00000022:(0016:01e9:01ff):019a:(0062:0000:0000:0000)]
[2333099.610954] tg3 0000:03:04.1 eth1: Link is down
[2333102.731813] tg3 0000:03:04.1 eth1: Link is up at 1000 Mbps, full duplex
[2333102.731822] tg3 0000:03:04.1 eth1: Flow control is off for TX and off for RX

The Debian upgrade changes the kernel and the new kernel seems to be not as stable as the old one which ran for years without any problem. One of the differences I found in the drivers is the ethernet acceleration mode for tg3 cards.

Workaround: after disabling some ethernet acceleration features I had no link resets. The computer is running about 9 weeks now with these settings:

/sbin/ethtool -K eth1 tso off
/sbin/ethtool -K eth1 gso off
/sbin/ethtool -K eth1 gro off

These commands disable segment offloading on eth1.

Versions: Debian 8.0 (July 2016), Kernel 3.16.0-4-amd64, Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Compiling Sendmail on Debian7

Problem: after compiling sendmail on Debian7 with “./Build” sendmail does not recognize hash .db files. You see the following error message:

readcf: map access: class hash not available

Discussion: ./Build should detect the berkley DB automatically. When devtools/bin/configure.sh finds libdb.so it adds -DNEWDB as compile option. On Debian7 the libdb.so file moved to /usr/lib/x86_64-linux-gnu/ and configure.sh fails to detect libdb.

WorkaroundLink the libdb.so and libdb.a file to /usr/lib with these commands:

cd /usr/lib/
ln -s x86_64-linux-gnu/libdb-5.1.a libdb.a
ln -s x86_64-linux-gnu/libdb-5.1.so libdb.so

Version: Debian 7, Sendmail-8.15.2

 

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Google Maps Marker on Mobile

Problem: A responsive webapp shows a google map with markers that are clickable. On desktop everything works as expected, but on mobile the markers are not clickable.

Discussion: After debugging with chrome remote inspector, I found that a div->frame with opacity:0 was lying above (explicit z-index:2) the clickable markers.

I don’t know what this frame is for, but it covers the markers and its click events.

Workaround: The frame is only loaded when the user is logged into google. You can remove this frame by removing “signed_in” from the script tag.

Version: https://maps.googleapis.com/maps/api/js on 23.2.2016. Chrome 48 on Android 5.1.1,

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

F5 Drops WebSockets

Problem: F5 LTM is used als load balancer for multiple web servers. When the client opens a websocket connection to the web server, the connection is closed.

Discussion: F5 LTM version before 11.6.0 has a bug in the request_log module (profile). The “request_log” module crashes and drops the connection. The bug is a known issue:

https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16690.html

Solution: If you cannot update. You can apply an iRule as a work arround:

when HTTP_REQUEST {
  if { [string tolower [HTTP::header Upgrade]] contains "websocket" } {
     HTTP::disable
  }
}

This iRule disables http handling and logging for websocket requests.

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Generate CSR using openssl

Browsers started to warn users about certificates with Sha1 signature. Sha256 is needed now a days.
So it’s time to renew certificates from Thawte, Godaddy, etc

You can generate a new Certificate Signing Request with openssl with this command:

openssl req -nodes -newkey rsa:2048 -keyout servername.key -out servername.csr -sha256

“servername.csr” is an ascii file you can send or paste to your certification authority’s interfaces.

Version: tested with OpenSSL 1.0.1e 11 Feb 2013 on Debian 7

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

ASR Tips’n’Tricks

ASR-1001-X and IOS-XE is sometimes different and sometimes very similar to classic IOS.

Update. You can update, the firmware as usual:

# copy http: bootflash:
# conf t
(config)# boot system flash bootflash:asr1001x-universalk9.03.16.00.S.155-3.S-ext.SPA.bin

Show SFP (transceiver) info:

# show hw-module interface tenGigabitEthernet 0/0/0 transceiver status
# show hw-module interface tenGigabitEthernet 0/0/0 transceiver idprom

.. to be continued

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

F5 sending packets to wrong destination?

Problem: You have a network with two upstream routers and an F5 LTM loadbalancer. Even though the default gateway points to router R1 the F5 LTM sends packets to the mac address of R2.

Discussion: “This is a feature not a bug”. This “Feature” is called “Auto Last Hop”. Which means the F5 answers packets allways to the mac address of the received packet. This may be usefull in some cases. But from the view of standards, compliance and security this behavior is a bug. In my case R2 sent some traffic to the F5 because of BGP multihoming, and received the answer allthough R1 should have received the traffic. Unfortunatly this setting is “Enabled” by default on F5.

If a hacker manages to inject  a request with a forged IP address, he will receive the answer even if the route to this IP points in a different direction.

Solution: This “Feature” can (and should) be disabled, if you don’t explicitly need it. It can be disabled globally, per VLAN, per Virtual Server or per SNAT policy.
You can find this setting globally in the web interface: System -> Configuration -> Local Traffic -> General
And for VLAN it can be disabled at:  Network -> VLANs -> Configuration [ Advanced]

Version: F5 LTM 11.5.1 8.0.175

Link: https://support.f5.com/kb/en-us/solutions/public/13000/800/sol13876.html

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

MLPPP over L2TP over Ethernet Channel Groups on Cisco ASR

 

Problem: After upgrading an ethernet port to a channel-group, all MLPPP connections fail on a Cisco ASR 1002-X. The log file looks like this:

Jul 31 2015 07:04:44.801 CEST: Vi4 PPP: Phase is AUTHENTICATING, Authenticated User
Jul 31 2015 07:04:44.801 CEST: Vi4 CHAP: O SUCCESS id 143 len 4
Jul 31 2015 07:04:44.801 CEST: Vi4 PPP: Phase is VIRTUALIZED
Jul 31 2015 07:04:44.802 CEST: Vi6 MLP: Added link Vi4 to bundle xxx
Jul 31 2015 07:04:44.803 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access4, changed state to up
Jul 31 2015 07:04:44.803 CEST: %LINK-3-UPDOWN: Interface Virtual-Access4, changed state to up
Jul 31 2015 07:04:44.805 CEST: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: cpp_cp:  cpp_cp encountered an error -Traceback= 1#795bed15105852c19a9ac138912d7358   errmsg:7F13FA6E0000+121D cpp_common_os:7F13FD6F1000+D8D5 cpp_common_os:7F13FD6F1000+D7D4 cpp_common_os:7F13FD6F1000+19A3E cpp_ifm:7F14106F1000+A198 cpp_mlppp_svr_lib:7F1406B63000+C351 cpp_mlppp_svr_lib:7F1406B63000+1CDC8 cpp_mlppp_svr_smc_lib:7F1406DA1000+2D28 cpp_common_os:7F13FD6F1000+11E6E cpp_common_os:7F13FD6F1000+118AA cpp_common_os:7F13FD6F1000+116EB evlib:7F13FC6D10
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP: O CONFREQ [REQsent] id 13 len 10
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP:    Address xxx
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP: Event[Timeout+] State[REQsent to REQsent]
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP: O CONFREQ [REQsent] id 14 len 10
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP:    Address xxx
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP: Event[Timeout+] State[REQsent to REQsent]

The router continues with “O CONFREQ” but never receives the “I CONFACK”.

Discussion: In this case the router is a L2TP server and handles multiple L2TP/PPP connection. Some of them are multilink PPP connections. The ASR software has a bug that leads to these tracebacks when the L2TP connections are going over an ethernet channel group. We opened a case with Cisco support. After one and a half month we received this answer:

---
Apologies for the delay. I was held up on other critical issues and hence was unable to reach out to you earlier. I was able to decode the tracebacks observed during the time of the issue and the issue points to a known software bug as the cause of the problem. Below are more details

CSCua16777 : FMFP-3-OBJ_DWNLD_TO_CPP_FAILED: SIP0: fman_fp_image: MLP bundle
Bug toolkit link : https://tools.cisco.com/bugsearch/bug/CSCua16777/?reffering_site=dumpcr

However the above bug is in closed state with the below release-notes

Symptom: FMFP-3-OBJ_DWNLD_TO_CPP_FAILED: SIP0: fman_fp_image: MLP bundle 8767, link 8766 download to CPP faile
Conditions: LNS MLPPP sessions don't stay up over port-channel
Workaround: MLPPP over port-channel is not supported on ASR1k. Don't use MLPPP over port-channel.
---

Dear Cisco! This is no solution. If you define an obvious bug as normal behaviour and the only workaround is “Don’t use..”, your customers will soon remember this:
“Cisco ? Don’t use…”

Solution: “Don’t use Cisco ?”

Version: Cisco ASR 1002-X, IOS XE Version: 03.09.02.S

Update: If I use the link of the bug report, I receive this answer:

Insufficient Permissions to View Bug
This bug contains proprietary information and is not yet publicly available.
Cisco Support Community

Unbelievable !! CSCO: STRONG SELL!

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Google Map from RSS Feed

Problem: Google had a nice feature to build google maps from rss geo information with a simple iframe tag, but this service is discontinued.

<iframe width="920" height="450" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" 
 src="https://maps.google.com/?q=http:%2F%2Ftothepin.blogspot.com%2Ffeeds%2Fposts%2Fdefault&amp;ie=UTF8&amp;t=t&amp;source=embed&amp;output=embed">
</iframe>

You could actually add ?q=rssfeed to the maps.google.com url and it produced a map from all geo data in this rss feed.

Solution: The new api for google maps is different but you can still do the same. Here some sample code:

<script src="//maps.googleapis.com/maps/api/js?v=3.exp&signed_in=true"></script>
<script>
    function initialize() {
        var myLatlng = new google.maps.LatLng(48.2084900,16.3720800);
        var mapOptions = {zoom: 4, center: myLatlng }

        var map = new google.maps.Map(document.getElementById('map-canvas'), mapOptions);

        var georssLayer = new google.maps.KmlLayer({
            url: 'http://tothepin.blogspot.com/feeds/posts/default?alt=rss'
        });
        georssLayer.setMap(map);
    }

    google.maps.event.addDomListener(window, 'load', initialize);
</script>
<style>
#map-canvas { width:800px; height:500px; }
</style>
<div id=map-canvas>Map</div>
Share on FacebookTweet about this on TwitterShare on Google+Email this to someone