Devuan4/Debian11 usability annoyances

Problem: When pasting commandos to the shell, the shell no longer executes the command but shows the pasted text highlighted and shows the line feed but does not process it.

Discussion: Devuan4 and Debian11 upgraded the readline library with a new default “feature” that should prevent noobs from accidentally pasting commands into the shell. But professionals want to be able to paste commands into the shell.

Solution: Add this line to ~/.inputrc

set enable-bracketed-paste Off

Linux Connection NAT Helper not Working

Some protocols need more than one TCP or UDP connection. For NAT to work the firewall needs to open additional ports to allow client server connection automatically. Examples are FTP (port 21 handshake, additional ports for data), PPTP (port 1723 for handshake, proto GRE 47 for payload)

Since Linux kernel (~) 4.7 these helpers are not bound automatically to iptables for security reasons. The idea is to implement iptables rules to activate connection helpers explicitly. Just loading the helper module is not enough.

To change this to the old behavior you can add this to your startup (for example /etc/rc.local)

# echo 1 > proc/sys/net/netfilter/nf_conntrack_helper

Debian Curl/PHP/wget etc show an certificate error falsely

Problem: curl php wget and others show a cert error like the following since 6. Oct 2021, even though the cert has not expired:

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

The asked server uses a Letsencrypt certificate.

Discussion: Currently Letsencrypt includes two chains for validation of the certificate:

  1. Cert -> R3 -> ISRG Root X1 (in new trust store)
  2. Cert -> R3 -> ISRG Root X1 -> DST Root CA X3 (in old trust stores but expired)

Chain one works for all modern OSes and browsers, but on Debian (at least with 9.0) the cert check fails because it finds the expired CA “X3” in chain 2, and does not use the direct chain with CA “X1”. Debian 9.0 includes both CA certs.

The reason to keep the expired cert in the provided chain is that some old devices need this X3 cert, and accept it despite its expire date. e.g. Android >=2.3.6 and <7.1.1

Solution: Start the following command and deactivate the expired CA cert on the client.

# dpkg-reconfigure ca-certificates

This bug is not limited to Debian because this issue seems to be related to openssl-1.1.0, which you will find in many products. Removing the expired cert might work there as well.

Devuan / Debian Versions

Admins who prefer a Unix style operation systems and don’t like the centralized “one tool doing it all” approach of systemd, switch to Devuan. This mostly improves uptime over boot time. Admins of servers don’t care about boot time and prefer uptime.

This list keeps track of the related versions of Devuan and Debian.

Debian Devuan
7Wheezy
8Jessie1Jessie
9Stretch2ASCII
10Buster3Beowulf
11Bullseye4Chimaera (stable)
12Bookworm5Daedalus

DHCP Relay on Linux

DHCP relaying is used to forward DHCP requests to a DHCP server if the client and the server are not on the same network. One standard implementation of this is isc-dhcp-relay which is part of the isc-dhcp package.

Problem: The dhcrelay is forwarding the dhcp request to the dhcp server and the dhcp server is responding correctly, but the dhcrelay ignores the response.

Discussion: the option “-i” tells the dhcrelay which interface to listen on for dhcp requests. But at the same time “-i” must be set for the interface that receives the dhcp server response. In newer versions (eg 4.4.2) there are even two options “-id”, “-iu” for downstream and upstream. The problem is: Both interfaces must be ethernet interfaces and have broadcasts, even though the dhcp server response is a simple unicast packet. If the connection to your dhcp server is a “tun” interface like openvpn or something not ethernet like, isc-dhcp-relay cannot receive the dhcp server response.

Solution: Don’t use isc-dhcp-relay but dhcp-helper. dhcp-helper is much simpler and works as expected. dhcp-helper is in the main debian/devuan repositories.

Versions: Devuan 3, isc-dhcp-relay 4.4.1-2, dhcp-helper 1.1-1

LS Style After Devuan 3 or Debian 10 Update

After Devuan 3 update ls output showed characters, that are not really in the directory listing. The reason is that “ls” draws quotes around filenames with spaces. This is a bug from my point of view, ls should never change the actual filenames. If a filename has quotes or double quotes it’s even weirder . “ls” adds the other quotes or even backslash quoting and closes and opens string within the string.

alex@workstation:~/test$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 '"'\''withbothquotes'\''"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 '"withdoublequotes"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 '"withdoublequotesandquotes'\'''
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'file'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 'some file'

To fix this bug sysadmins like me have to set the following environment variable (eg. in /etc/profile, ~/.bashrc, etc)

export QUOTING_STYLE=literal

After this setting you get the real filenames back:

alex@workstation:~/ttd$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'withbothquotes'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 "withdoublequotes"
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 "withdoublequotesandquotes'
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 'file'
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 some file

Time Format after Devuan 3 and Debian 10 Update

After updating to Devuan 3 the date command shows 12hours am/pm but my days have 24 hours. The locale was always en_US.UTF8 to keep sane command and error output.

Debian 10 thinks they had to fix the correct hour display to the complicated one.

Therefor all sysadmins like me have to apply the following workaround, to keep both sane command output and reasonable time format.

update-locale LC_TIME=C.UTF-8

This changed the locale for time output to “C” in /etc/default/locale. Date looks correct again:

# date
Mon Jun  8 21:20:33 CEST 2020

ARP and Broadcast Packets Missing

Problem: A Linux box with Debian 9 (kernel 4.9) on a HP server with Intel i40e (X710) network cards, is not reachable from neighbor machines, because ARP does not work.

Discussion: while testing with tcpdump ARP worked, but later ARP stopped working again. When tcpdump is used with “-p” (non promiscuous mode) you can see the problem. The server does not receive any broadcasts. Which means neighbors can not find the machine with ARP. Outgoing ARP does work though because ARP responses are not sent to broadcast the address (ff:ff:ff:ff:ff:ff).

Solution: A quick fix was to use “ifconfig eth0 promisc”. In this mode broadcasts are received and ARP is working. A better fix is to upgrade the Linux kernel to Debian 9 backports (4.19) or probably upgrade to Debian 10.

Versions: Debian 9 kernel 4.9 intel driver: 2.3.2-k intel firmware: 6.00 0x800034ea 18.3.6

MITMProxy and IOS 13

Problem: if you want to debug a IOS app with MITMProxy, the iPhone needs to trust the MITMProxy CA. This is done by going to http://mitm.it/ and clicking on the apple symbol. Then you have to accept the “profile” in Settings “downloaded profiles”. Then you have to trust this new CA cert in “Settings” “General” “About” “Trust Root Cert” “mitmproxy”. But then the certs generated by the MITMProxy are still not trusted.

Discussion: Starting with IOS 13, TLS server certificates must have a validity period of 825 days or fewer and MITMProxy generates certs with an expiration period of 1095 days.

Solution: I changed the py file of MITMProxy to shorten the cert validity, by changing the file /usr/lib/python2.7/dist-packages/netlib/certutils.py

# DEFAULT_EXP = 94608000  # = 24 * 60 * 60 * 365 * 3
DEFAULT_EXP = 31536000  # = 24 * 60 * 60 * 365

Versions: test with MITMProxy 0.18.2-6+deb9u2 but it looks as if current versions of MITMProxy on github still use 3 years as default expiration.

Linux Live-boot Fails after Debian/Devuan Update

Problem: after updating from Debian 8 to Devuan 2 the overlay live-boot failes with “no such device”

Discussion: I use a bootable USB stick combined with live-boot. In this case the USB stick partition 3 is a normal ext4 file system used as read only “plainroot” filesystem. Live-boot overlays this with an ramfs.
As I don’t know the /dev/sdaX file on the target system I use “root=LABEL=KROOT” to find the USB root image. This worked before but it does not any more. The reason is the following line in /lib/live/boot/9990-overlay.sh in the “plain root system” section:

mount -t $(get_fstype "${image_directory}") -o ro,noatime "${image_directory}" "${croot}/filesystem"

get_fstype “LABEL=KROOT” results in “unkown” and this mount command fails.

Solution: I removed the get_fstype part -t $(get_fstype “${image_directory}”) in /lib/live/boot/9990-overlay.sh. Mount guesses the filesystem type automatically.

After that you have to rebuild initramdisk with update-initramfs.

Version: tested with devuan 2.1, and this kernel boot options: “read-only boot=live root=LABEL=KROOT rootdelay=10 ignore_uuid plainroot”