LS Style After Devuan 3 or Debian 10 Update

After Devuan 3 update ls output showed characters, that are not really in the directory listing. The reason is that “ls” draws quotes around filenames with spaces. This is a bug from my point of view, ls should never change the actual filenames. If a filename has quotes or double quotes it’s even weirder . “ls” adds the other quotes or even backslash quoting and closes and opens string within the string.

alex@workstation:~/test$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 '"'\''withbothquotes'\''"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 '"withdoublequotes"'
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 '"withdoublequotesandquotes'\'''
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'file'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 'some file'

To fix this bug sysadmins like me have to set the following environment variable (eg. in /etc/profile, ~/.bashrc, etc)

export QUOTING_STYLE=literal

After this setting you get the real filenames back:

alex@workstation:~/ttd$ ls -l 
total 0
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 "'withbothquotes'"
-rw-r--r-- 1 alex alex 0 Jul  9 17:53 "withdoublequotes"
-rw-r--r-- 1 alex alex 0 Jul  9 17:54 "withdoublequotesandquotes'
-rw-r--r-- 1 alex alex 0 Jul  9 17:55 'file'
-rw-r--r-- 1 alex alex 0 Jul  9 17:52 some file

Time Format after Devuan 3 and Debian 10 Update

After updating to Devuan 3 the date command shows 12hours am/pm but my days have 24 hours. The locale was always en_US.UTF8 to keep sane command and error output.

Debian 10 thinks they had to fix the correct hour display to the complicated one.

Therefor all sysadmins like me have to apply the following workaround, to keep both sane command output and reasonable time format.

update-locale LC_TIME=C.UTF-8

This changed the locale for time output to “C” in /etc/default/locale. Date looks correct again:

# date
Mon Jun  8 21:20:33 CEST 2020

MITMProxy and IOS 13

Problem: if you want to debug a IOS app with MITMProxy, the iPhone needs to trust the MITMProxy CA. This is done by going to http://mitm.it/ and clicking on the apple symbol. Then you have to accept the “profile” in Settings “downloaded profiles”. Then you have to trust this new CA cert in “Settings” “General” “About” “Trust Root Cert” “mitmproxy”. But then the certs generated by the MITMProxy are still not trusted.

Discussion: Starting with IOS 13, TLS server certificates must have a validity period of 825 days or fewer and MITMProxy generates certs with an expiration period of 1095 days.

Solution: I changed the py file of MITMProxy to shorten the cert validity, by changing the file /usr/lib/python2.7/dist-packages/netlib/certutils.py

# DEFAULT_EXP = 94608000  # = 24 * 60 * 60 * 365 * 3
DEFAULT_EXP = 31536000  # = 24 * 60 * 60 * 365

Versions: test with MITMProxy 0.18.2-6+deb9u2 but it looks as if current versions of MITMProxy on github still use 3 years as default expiration.

Linux Live-boot Fails after Debian/Devuan Update

Problem: after updating from Debian 8 to Devuan 2 the overlay live-boot failes with “no such device”

Discussion: I use a bootable USB stick combined with live-boot. In this case the USB stick partition 3 is a normal ext4 file system used as read only “plainroot” filesystem. Live-boot overlays this with an ramfs.
As I don’t know the /dev/sdaX file on the target system I use “root=LABEL=KROOT” to find the USB root image. This worked before but it does not any more. The reason is the following line in /lib/live/boot/9990-overlay.sh in the “plain root system” section:

mount -t $(get_fstype "${image_directory}") -o ro,noatime "${image_directory}" "${croot}/filesystem"

get_fstype “LABEL=KROOT” results in “unkown” and this mount command fails.

Solution: I removed the get_fstype part -t $(get_fstype “${image_directory}”) in /lib/live/boot/9990-overlay.sh. Mount guesses the filesystem type automatically.

After that you have to rebuild initramdisk with update-initramfs.

Version: tested with devuan 2.1, and this kernel boot options: “read-only boot=live root=LABEL=KROOT rootdelay=10 ignore_uuid plainroot”

Greenlock(-express) Letsencrypt Fails with ECONNRESET

Problem: after upgrading vom greenlock-express v2.0 to v2.5 and switching from acme-v1 to acme-v2 every attempt to register a new TLS cert with Letsencrypt fails with “ECONNRESET”

Discussion: the new version of greenlock tries to validate the .well-known/acme-challenge file before asking letsencrypt for the certificate.
If your webserver is behind a loadbalancer or firewall and the webserver can not request itself using the official public IP, this loopback request may fail. In this case only this cryptic error message is shown:

[acme-v2] handled(?) rejection as errback:
Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:200:27)
Error loading/registering certificate for 'your.webserver':
Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:200:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}

Solution: You can redirect these local loopback web requests using iptables to the local web server and bypass the loadbalancer/firewall:

iptables -t nat -I OUTPUT -d PUBLIC_WEBSERVER_IP -p tcp --dport 80 -j REDIRECT --to-port LOCAL_WEBSERVER_TCP_PORT

Apache Start Hangs during Reboot of a KVM Virtual Server

Problem: Apache needs very long to start on a virtual server running on a KVM/QEMU virtual maschine.

Solution: Apache needs a RNG (random number generator) for startup, probably because of TLS. A pure virtual maschine has no RNG device per default. If you add an RNG device to the virtual maschine configuration, apache startup is lightening fast.

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </rng>

Versions: tested with libvirt 3.0.0, qemu-kvm 2.8 on devuan 9

Sparse Files Howto

Unix file systems like ext3/4 can store files which are partly empty more efficiently by not storing blocks with all zeros. These files are called sparse files. When reading these files every things works as normal but “all zero” blocks don’t wast space on the drive.

This can be useful for different application. For example a database can make a big file for random access, without using the space on the drive. The actual size on the disk grows with every used block. Another example are raw disk images for virtualization like KVM. You can make a 10GB disk image which uses almost no space, and grows only when used.

Usefull sysadmin commands:

"du -h --apparent-size FILE" shows the full file size including sparse areas
"du -h FILE" shows the actual space used on the file system
"ls -lh FILE" show the full file size
"ls -sh FILE" shows the actual space used on the file system
"fallocate -d FILE" make a file sparse, which means "digs holes" for "all zero" blocks
"rsync -S ..." the -S option makes rsync sparse file aware and produces sparse files at the receiver
"truncate -s 1G FILE" makes a sparse file with 1GB that uses no file system space

Remove ID3 Tags from Flac Files

Problem: Some flac players refuse to play some flac files, and even tools like an old ffmpeg can’t handle some flac files

Solution: These flac files might have id3v2 tags which they realy should not, because flac uses vorbis style tags and not id3. Remove those id3v2 tags with this command:

> id3v2 --delete-all song.flac

This removes the id3v2 tags from the flac file in place

Discussion: Those flac files were made with EAC. In the encoder settings “Add ID3 Tags” was checked, and EAC added ID3 tags even though flac files don’t need and must not have ID3 tags. If you like to know, whether your flac files have these false ID3 tags your can run “id3v2 -l song.flac” or look into the files with hexdump.

hexdump -C song-with-id3.flac | head
00000000  49 44 33 03 00 00 00 06  44 0b 54 49 54 32 00 00  |ID3.....D.TIT2..|
00000010  00 27 00 00 01 ff fe 54  00 68 00 65 00 20 00 46  |.'.....T.h.e. .F|
00000020  00 69 00 72 00 65 00 20  00 54 00 68 00 69 00 73  |.i.r.e. .T.h.i.s|
00000030  00 20 00 54 00 69 00 6d  00 65 00 54 50 45 31 00  |. .T.i.m.e.TPE1.|

hexdump -C song-without-id3.flac | head
00000000  66 4c 61 43 00 00 00 22  10 00 10 00 00 00 10 00  |fLaC..."........|
00000010  2b 54 0a c4 42 f0 00 bf  39 4c 3d e0 59 d1 58 72  |+T..B...9L=.Y.Xr|
00000020  49 b7 d4 56 99 08 c4 ae  45 b5 03 00 02 0a 00 00  |I..V....E.......|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 10 00  |................|

Correct flac files start with “fLaC” and not “ID3”

Version: EAC “Exact Audio Copy” Sept. 2019

APT sources list

Problem: when debian goes from “testing” to “stable” to “oldstable” the package sources change. eg. jessie-updates are remove, same happened to jessie-backports

The current file /etc/apt/sources.list for jessie (currently oldstable) could look like this

deb http://ftp.debian.org/debian/ jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free