PaloAlto Packet Loss of 1% and More

Problem: PaloAlto firewall is dropping packet in small burst of some seconds, and sometimes it drops TCP connections. It only happens on HA clusters on interfaces in active/passive (fail over) mode.

Solution: disable the following check box in the Ethernet interface Advanced – LLDP settings: “Enable in HA Passive State”

Discussion: Palo Alto uses only one MAC address for both machines of an HA cluster. The passive box sends LLDP packets using this MAC address. The switch learns this MAC address and sends the traffic to the passive node until the active node sends new packets. The passive node should never send packets with the MAC address of the active node, and should have its own MAC address for LLDP and possibly other services.

Version: PaloAlto current version of Nov 2016, connected to a Cisco Catalyst 6500

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

ARP is not working on Cisco ASR 1001 X

Problem: Cisco ASR router is loosing connectivity to its directly attached Ethernet neighbors. In this situation interface status is still up, packets are going in and out on both ends, even IPv6 was still working. The actual problem was that the Cisco ASR was ignoring all ARP responses from its neighbors and the ARP table to this interface was empty. Later the same happened on a second interface.

A temporary work around was to reboot the router.

Solution: Cisco support suggested a software upgrade, even though the software was only some weeks old. After the software upgrade the error didn’t happen again until now.
The old IOS version was: asr1001x-universalk9.03.16.03.S.155-3.S3-ext.SPA.bin
The new IOS version is: asr1001x-universalk9.03.16.04a.S.155-3.S4a-ext.SPA.bin

The only fix that possibly fits to the problem is:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20160804-wedge

“A remote attacker can cause an interface wedge and an eventual denial of service condition”

What’s an “interface wedge”. Cisco bug reports were more precise years ago.

 

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Cups PDF Filter Crashes

Problem: for some PDF files cups does not print the page, the web gui just tells “filter failed” and in the cups logfile you find

[Job 528] PID 19521 (gs) crashed on signal 11!

Discussion: cups on debian uses ghostscript as pdf renderer. The pdf renderer of ghostsciprt crashes on many files. You can test by simply running pdf2ps on the pdf file, and see if it is the same for you.

Solution: you can configure the pdf renderer that cups should use. I changed it to pdftops from the “poppler-utils” package.

Install “poppler-utils” using aptitude or apt-get and add the following line to /etc/cups/printers.conf in the printer section:

Option pdftops-renderer pdftops

Versions: debian-8.6 with ghostscript 9.06~dfsg-2+deb8u1 and cups 1.7.5-11+deb8u1

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Autosave for VIM

I believe that it’s a good idea that vi does not auto save during editing. Think of config files or src files that should never be in an inconsistent state. The programmer or sysadmin should decide when he wants to save data.

But sometime auto save is handy, while typing lists like todo lists or outlines etc. Every version is OK and you don’t want to save after every change and still don’t want to loos data on connection or power loss.

Use this in you current VIM session to make this current buffer autosave:

:au TextChanged,TextChangedI <buffer> silent w
Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Annoyances while updating F5

I had to upgrade several F5 load balancers from 11.5 to 12.1 in the last weeks. Usually updating F5 is quiet easy, but there are bugs or annoyances you should know:

  1. Sometimes F5 asks for re-activating after the first boot into the new version. It seems that you have to install the new version in a specific order to prevent this: BIGIP-Firmware, licence re-activate, BIGIP-Hotfix, Restart.
    Remember the appliance has to be in stand by mode when re-activating the licence.
  2. If the F5 asks for licence re-activation after reboot, it should be easy to re-activate. But even after licence activation, F5 is not working correctly. The SNMP MIB for LTM is not complete. You have to reboot again to activate the LTM SNMP MIB tree again.
  3. When switching from 11.5 to 12.1 the SNMP MIB changed. Serious manufactures that special care to keep the SNMP stable and compatible. F5 doesn’t they changed data types from 11.5 to 12.1 which means you have to update the MIB database. On the other hand if you do, you cannot query those OIDs from older machines. That’s why other manufactures never change data type, The correct way is to add new OIDs wait some years and deprecate the old OID. F5 doesn’t. Here’s a diff part of mibs_f5/F5-BIGIP-LOCAL-MIB.txt:
    - ltmNodeAddrStatCurSessions Gauge,
    - ltmNodeAddrStatCurrentConnsPerSec Gauge,
    - ltmNodeAddrStatDurationRateExceeded Gauge
    + ltmNodeAddrStatCurSessions CounterBasedGauge64,
    + ltmNodeAddrStatCurrentConnsPerSec CounterBasedGauge64,
    + ltmNodeAddrStatDurationRateExceeded CounterBasedGauge64
  4. Beside this breaking incompatibility between 11.5 and 12.1, they also changed some value names, which breaks software that used these names. This is not a bug but still annoying. Remember: an API has to be stable and backward compatible.
     inband(2),
    - forcedUp(3),
    + forced-up(3),
     up(4),
     down(19),
    - forcedDown(20),
    - iruleDown(22),
    - inbandDown(23),
    - downManualResume(24),
    + forced-down(20),
    + irule-down(22),
    + inband-down(23),
    + down-manual-resume(24),
     disabled(25)

Versions: F5 LTM 11.5.1, 11.5.3 and 12.1.1

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Rare Connection Resets after Linux Upgrade

Problem: after upgrading from Debian 6 to Debian 8 some of the machines lose their ethernet network connection under heavy load for some seconds rarely. You find lines like these in syslog:

[2333099.217735] NETDEV WATCHDOG: eth1 (tg3): transmit queue 0 timed out
[2333099.217966] tg3 0000:03:04.1 eth1: transmit timed out, resetting
[2333099.384391] tg3 0000:03:04.1 eth1: 0: Host status block [00000001:0000003c:(0000:0018:0000):(0018:01e9)]
[2333099.386091] tg3 0000:03:04.1 eth1: 0: NAPI info [00000022:00000022:(0016:01e9:01ff):019a:(0062:0000:0000:0000)]
[2333099.610954] tg3 0000:03:04.1 eth1: Link is down
[2333102.731813] tg3 0000:03:04.1 eth1: Link is up at 1000 Mbps, full duplex
[2333102.731822] tg3 0000:03:04.1 eth1: Flow control is off for TX and off for RX

The Debian upgrade changes the kernel and the new kernel seems to be not as stable as the old one which ran for years without any problem. One of the differences I found in the drivers is the ethernet acceleration mode for tg3 cards.

Workaround: after disabling some ethernet acceleration features I had no link resets. The computer is running about 9 weeks now with these settings:

/sbin/ethtool -K eth1 tso off
/sbin/ethtool -K eth1 gso off
/sbin/ethtool -K eth1 gro off

These commands disable segment offloading on eth1.

Versions: Debian 8.0 (July 2016), Kernel 3.16.0-4-amd64, Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Compiling Sendmail on Debian7

Problem: after compiling sendmail on Debian7 with “./Build” sendmail does not recognize hash .db files. You see the following error message:

readcf: map access: class hash not available

Discussion: ./Build should detect the berkley DB automatically. When devtools/bin/configure.sh finds libdb.so it adds -DNEWDB as compile option. On Debian7 the libdb.so file moved to /usr/lib/x86_64-linux-gnu/ and configure.sh fails to detect libdb.

WorkaroundLink the libdb.so and libdb.a file to /usr/lib with these commands:

cd /usr/lib/
ln -s x86_64-linux-gnu/libdb-5.1.a libdb.a
ln -s x86_64-linux-gnu/libdb-5.1.so libdb.so

Version: Debian 7, Sendmail-8.15.2

 

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Google Maps Marker on Mobile

Problem: A responsive webapp shows a google map with markers that are clickable. On desktop everything works as expected, but on mobile the markers are not clickable.

Discussion: After debugging with chrome remote inspector, I found that a div->frame with opacity:0 was lying above (explicit z-index:2) the clickable markers.

I don’t know what this frame is for, but it covers the markers and its click events.

Workaround: The frame is only loaded when the user is logged into google. You can remove this frame by removing “signed_in” from the script tag.

Version: https://maps.googleapis.com/maps/api/js on 23.2.2016. Chrome 48 on Android 5.1.1,

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

F5 Drops WebSockets

Problem: F5 LTM is used als load balancer for multiple web servers. When the client opens a websocket connection to the web server, the connection is closed.

Discussion: F5 LTM version before 11.6.0 has a bug in the request_log module (profile). The “request_log” module crashes and drops the connection. The bug is a known issue:

https://support.f5.com/kb/en-us/solutions/public/16000/600/sol16690.html

Solution: If you cannot update. You can apply an iRule as a work arround:

when HTTP_REQUEST {
  if { [string tolower [HTTP::header Upgrade]] contains "websocket" } {
     HTTP::disable
  }
}

This iRule disables http handling and logging for websocket requests.

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone

Generate CSR using openssl

Browsers started to warn users about certificates with Sha1 signature. Sha256 is needed now a days.
So it’s time to renew certificates from Thawte, Godaddy, etc

You can generate a new Certificate Signing Request with openssl with this command:

openssl req -nodes -newkey rsa:2048 -keyout servername.key -out servername.csr -sha256

“servername.csr” is an ascii file you can send or paste to your certification authority’s interfaces.

Version: tested with OpenSSL 1.0.1e 11 Feb 2013 on Debian 7

Share on FacebookTweet about this on TwitterShare on Google+Email this to someone