Cisco ASR-1001-X Update

There are at least to pieces of software you can update in a Cisco ASR-1001-X. The Rommon (Firmware) and the IOS itself.

This router uses “Cisco IOS XE Software”. Which is an IOS process on a Linux kernel as far as I know.

Cisco recommends specific Rommon releases for different generations of Software. You can look it up at: https://www.cisco.com/c/en/us/td/docs/routers/asr1000/rommon/asr1000-rommon-upg-guide.html#concept_zdm_2nx_5bb

The software versions for these ASR 1000 series routers had complicated versions numbers like these:

asr1001x-universalk9.03.16.07b.S.155-3.S7b-ext.SPA.bin which is 03.16.07b.S and actually IOS 15.5(3)S7b.

But with IOS 16 it was simplified a current software image looks like this

asr1001x-universalk9_noli.16.09.06.SPA.bin which is IOS 16.9.6.

If you have a cisco account with running service contract, you can download the Cisco software from: https://software.cisco.com/download/home

The IOS XE software, comes with or without “Payload encryption”, and with or without “Lawfull Interception”.

To check the current installed rommon and IOS version enter:

show plattform
show version

To update the rommon copy the rommon image to flash and enter:

upgrade rom-monitor filename bootflash:asr1000-romm... all

To update the IOS software copy the IOS image to the flash and enter (in config mode):

boot system flash bootflash:asr1001x-universalk9_...
no boot system flash bootflash:...old image...
no boot system flash bootflash:...older images...
boot system flash bootflash:...old image...
exit
wr

This changes the boot order to prefer the new image as default but keep the old one as fallback. After the “wr” you can check the bootvar with “show bootvar” and see if the next reboot should use the new image.

Next, reboot the router with “reload”, and check if the software has changed
after the reboot. You also might want to save a new backup of the configuration and check how it differs after the update.

DHCP Relay on Linux

DHCP relaying is used to forward DHCP requests to a DHCP server if the client and the server are not on the same network. One standard implementation of this is isc-dhcp-relay which is part of the isc-dhcp package.

Problem: The dhcrelay is forwarding the dhcp request to the dhcp server and the dhcp server is responding correctly, but the dhcrelay ignores the response.

Discussion: the option “-i” tells the dhcrelay which interface to listen on for dhcp requests. But at the same time “-i” must be set for the interface that receives the dhcp server response. In newer versions (eg 4.4.2) there are even two options “-id”, “-iu” for downstream and upstream. The problem is: Both interfaces must be ethernet interfaces and have broadcasts, even though the dhcp server response is a simple unicast packet. If the connection to your dhcp server is a “tun” interface like openvpn or something not ethernet like, isc-dhcp-relay cannot receive the dhcp server response.

Solution: Don’t use isc-dhcp-relay but dhcp-helper. dhcp-helper is much simpler and works as expected. dhcp-helper is in the main debian/devuan repositories.

Versions: Devuan 3, isc-dhcp-relay 4.4.1-2, dhcp-helper 1.1-1

LS Style After Devuan 3 or Debian 10 Update

After Devuan 3 update ls output showed characters, that are not really in the directory listing. The reason is that “ls” draws quotes around filenames with spaces. This is a bug from my point of view, ls should never change the actual filenames. If a filename has quotes or double quotes it’s even weirder . “ls” adds the other quotes or even backslash quoting and closes and opens string within the string.

alex@workstation:~/test$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 '"'\''withbothquotes'\''"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 '"withdoublequotes"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 '"withdoublequotesandquotes'\'''
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'file'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 'some file'

To fix this bug sysadmins like me have to set the following environment variable (eg. in /etc/profile, ~/.bashrc, etc)

export QUOTING_STYLE=literal

After this setting you get the real filenames back:

alex@workstation:~/ttd$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'withbothquotes'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 "withdoublequotes"
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 "withdoublequotesandquotes'
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 'file'
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 some file

Time Format after Devuan 3 and Debian 10 Update

After updating to Devuan 3 the date command shows 12hours am/pm but my days have 24 hours. The locale was always en_US.UTF8 to keep sane command and error output.

Debian 10 thinks they had to fix the correct hour display to the complicated one.

Therefor all sysadmins like me have to apply the following workaround, to keep both sane command output and reasonable time format.

update-locale LC_TIME=C.UTF-8

This changed the locale for time output to “C” in /etc/default/locale. Date looks correct again:

# date
Mon Jun  8 21:20:33 CEST 2020

Mikrotik OSPF Routing Distance Ignored

Discussion: Every routing protocol has a default distance to help the router to decide which route to use in case of multiple routes for the same destination. For Mikrotik routers these distances are listed here:
https://wiki.mikrotik.com/wiki/Manual:Route_Selection_Algorithm_in_RouterOS
If you want to configure a backup link that is only activated when the OSPF main route is missing, you can use a static route with distance 120 which is higher than the OSPF default 110.

If you enter this route in a mikrotik after the ospf route is learned it works as expected. The static 120 route is ignored until the ospf route vanishes.

But the other way arround does not work. If the 120 route is active, the OSPF route is ignored even it has the better distance of 110. And worse the Mikrotik keeps this route in its own OSPF announcement.

This is a known bug: https://forum.mikrotik.com/viewtopic.php?t=119493

Work around: You can make two smaller routes for the preferred path.
eg. two /26 should always overrule a /25 route.

Version: Mikrotik RouterOS 6.46.1

ARP and Broadcast Packets Missing

Problem: A Linux box with Debian 9 (kernel 4.9) on a HP server with Intel i40e (X710) network cards, is not reachable from neighbor machines, because ARP does not work.

Discussion: while testing with tcpdump ARP worked, but later ARP stopped working again. When tcpdump is used with “-p” (non promiscuous mode) you can see the problem. The server does not receive any broadcasts. Which means neighbors can not find the machine with ARP. Outgoing ARP does work though because ARP responses are not sent to broadcast the address (ff:ff:ff:ff:ff:ff).

Solution: A quick fix was to use “ifconfig eth0 promisc”. In this mode broadcasts are received and ARP is working. A better fix is to upgrade the Linux kernel to Debian 9 backports (4.19) or probably upgrade to Debian 10.

Versions: Debian 9 kernel 4.9 intel driver: 2.3.2-k intel firmware: 6.00 0x800034ea 18.3.6

MITMProxy and IOS 13

Problem: if you want to debug a IOS app with MITMProxy, the iPhone needs to trust the MITMProxy CA. This is done by going to http://mitm.it/ and clicking on the apple symbol. Then you have to accept the “profile” in Settings “downloaded profiles”. Then you have to trust this new CA cert in “Settings” “General” “About” “Trust Root Cert” “mitmproxy”. But then the certs generated by the MITMProxy are still not trusted.

Discussion: Starting with IOS 13, TLS server certificates must have a validity period of 825 days or fewer and MITMProxy generates certs with an expiration period of 1095 days.

Solution: I changed the py file of MITMProxy to shorten the cert validity, by changing the file /usr/lib/python2.7/dist-packages/netlib/certutils.py

# DEFAULT_EXP = 94608000  # = 24 * 60 * 60 * 365 * 3
DEFAULT_EXP = 31536000  # = 24 * 60 * 60 * 365

Versions: test with MITMProxy 0.18.2-6+deb9u2 but it looks as if current versions of MITMProxy on github still use 3 years as default expiration.

Linux Live-boot Fails after Debian/Devuan Update

Problem: after updating from Debian 8 to Devuan 2 the overlay live-boot failes with “no such device”

Discussion: I use a bootable USB stick combined with live-boot. In this case the USB stick partition 3 is a normal ext4 file system used as read only “plainroot” filesystem. Live-boot overlays this with an ramfs.
As I don’t know the /dev/sdaX file on the target system I use “root=LABEL=KROOT” to find the USB root image. This worked before but it does not any more. The reason is the following line in /lib/live/boot/9990-overlay.sh in the “plain root system” section:

mount -t $(get_fstype "${image_directory}") -o ro,noatime "${image_directory}" "${croot}/filesystem"

get_fstype “LABEL=KROOT” results in “unkown” and this mount command fails.

Solution: I removed the get_fstype part -t $(get_fstype “${image_directory}”) in /lib/live/boot/9990-overlay.sh. Mount guesses the filesystem type automatically.

After that you have to rebuild initramdisk with update-initramfs.

Version: tested with devuan 2.1, and this kernel boot options: “read-only boot=live root=LABEL=KROOT rootdelay=10 ignore_uuid plainroot”

Greenlock(-express) Letsencrypt Fails with ECONNRESET

Problem: after upgrading vom greenlock-express v2.0 to v2.5 and switching from acme-v1 to acme-v2 every attempt to register a new TLS cert with Letsencrypt fails with “ECONNRESET”

Discussion: the new version of greenlock tries to validate the .well-known/acme-challenge file before asking letsencrypt for the certificate.
If your webserver is behind a loadbalancer or firewall and the webserver can not request itself using the official public IP, this loopback request may fail. In this case only this cryptic error message is shown:

[acme-v2] handled(?) rejection as errback:
Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:200:27)
Error loading/registering certificate for 'your.webserver':
Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:200:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}

Solution: You can redirect these local loopback web requests using iptables to the local web server and bypass the loadbalancer/firewall:

iptables -t nat -I OUTPUT -d PUBLIC_WEBSERVER_IP -p tcp --dport 80 -j REDIRECT --to-port LOCAL_WEBSERVER_TCP_PORT

Apache Start Hangs during Reboot of a KVM Virtual Server

Problem: Apache needs very long to start on a virtual server running on a KVM/QEMU virtual maschine.

Solution: Apache needs a RNG (random number generator) for startup, probably because of TLS. A pure virtual maschine has no RNG device per default. If you add an RNG device to the virtual maschine configuration, apache startup is lightening fast.

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </rng>

Versions: tested with libvirt 3.0.0, qemu-kvm 2.8 on devuan 9