Debugging Akamai

Akamai just works, … most of the time. But sometimes you have to check what’s going on, and Akamai gives you a handy tool for this.

There is an HTTP request header that tells Akamai to respond with some internal information.

Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id

With this request header Akamai includes this in the response header

X-Cache: TCP_MISS from a84-53-161-127.deploy.akamaitechnologies.com (AkamaiGHost/9.6.2.0.1-25325260) (-)
X-Cache-Key: S/L/16382/612780/0s/www.yourdomain.de/ cid=what_TOKEN=dings_
X-Cache-Key-Extended-Internal-Use-Only: S/L/16382/612780/0s/www.yourdomain.de/ vcd=1948 cid=what_TOKEN=dings_
X-True-Cache-Key: /L/www.yourdomain.de/ vcd=1948 cid=what_TOKEN=dings_
X-Akamai-SSL-Client-Sid: lZWwRTj17XXXXXXXXXU5Cw==
X-Check-Cacheable: NO
X-Akamai-Request-ID: f82516c

Some important parts:

  • TCP_MISS shows that Akamai didn’t use it’s cache for this request, but the origin
  • X-Cache-Key shows what Akamai used to reference the cache position. In this case the url was http://www.yourdomain.de/?what and a cookie named TOKEN was included in the cacheID (“cid=…”)

MikroTik Automatic IPSec Failover

Problem: Mikrotik allows only one ipsec policy per network-to-network pair. If you want to have redundant tunnels between two locations with two upstreams you cannot configure ipsec redundancy on Mikrotik because one ipsec policy is always marked as “invalid” by the OS.

Solution: I made a Mikrotik script that checks the status and reachabilty of the ipsec tunnel and endpoint, and switches between a primary and secondary tunnel policy and peer. You can add this script to the scheduler, for automatic failover. (Source: “/system script run 0” if this script is script “0”)

{
:local PrimaryPolicy 2
:local SecondaryPolicy 3
:local PrimaryPeer 0
:local SecondaryPeer 1

:local PrimaryOK [:ping count=3 src-address=localAip remoteAip];
:local SecondaryOK [:ping count=3 src-address=localBip remoteBip];
:local PrimaryActive [/ip ipsec policy get $PrimaryPolicy active];

# :log info "Status: $PrimaryOK $SecondaryOK $PrimaryActive";
# Test case: set $PrimaryOK 0;

:if ($PrimaryOK < 1 && $SecondaryOK > 1 && $PrimaryActive) do={
:log warn "switch to failover";
/ip ipsec policy disable $PrimaryPolicy;
/ip ipsec policy enable $SecondaryPolicy;
/ip ipsec peer disable $PrimaryPeer;
/ip ipsec peer enable $SecondaryPeer;
}
:if ($PrimaryOK = 3 && !$PrimaryActive) do={
:log warn "switch to primary";
/ip ipsec policy disable $SecondaryPolicy;
/ip ipsec policy enable $PrimaryPolicy;
/ip ipsec peer disable $SecondaryPeer;
/ip ipsec peer enable $PrimaryPeer;
}
}

Version: tested with RouterOS 6.44.1

FortiGate HA Synchronization Fail

Problem: Two FortiGate firewall show “not synchonized” in the HA status.

Discussion: the problem with this is, that FortiGate does not show why it fails. I found no log file with a reasonable error message. So I tried to synchronize the config myself, which is exactly what should NOT be necessary when using HA synchronization.

Solution: When an ipsec-phase1 setting in the master is removed while the slave is not online, the ipsec-phase1 removal fails during synchronization. Why Fortinet, doesn’t your box log this? Removing the phase1-section by hand did not work ether:

FortiGate-Master # execute ha manage 1

FortiGate-Slave $ config vpn ipsec phase1-interface
FortiGate-Slave (phase1-interface) $ delete VPN-PEER
This phase1-interface is currently used
command_cli_delete:5937 delete table entry VPN-PEER unset oper error ret=-23
Command fail. Return code -23

Like with most cheap software I had to reboot the slave, and then I could remove the phase1-interface section, and then the synchronization worked again.

I don’t remember if I ever had to reboot a Cisco or Linux box to fix a bug.

Version: FortiGate 6.0.4

My Contact was Shared with Facebook

Facebook gives a list of companies who run Facebook ads, and sent my contact information (tel, email) to Facebook.

This list for my account lists some obvious companies, some companies I never heard of, and some companies who really should never have forwarded my contact information to Facebook!

Look up your own list by:

  • Click the three dots on any ad in facebook
  • Click “Why am I seeing this?”
  • Click “Manage Your ad Preferences”
  • Click “Advertisers”

Annoyances While Setting Up FortiGate 100E

After unboxing and starting two Fortigate 100E firewalls the following things failed. Fortinet should really work on QA I think:

  • When starting the Webgui for the first time the box asks to register or login to activate the box. I tried to register and the box asks so many questions, doesn’t Fortinet know GDPR ? Don’t force users to enter data that you don’t need to provide your service. We are not talking about a free service that tries do sell ads. This is a paid product , I don’t want to give you information about my company the number of employs and so on.
  • The registration on the box itself does not work, after filling out the form and clicking “OK” nothing happens, no error message. no response.
  • After registering on the Fortinet webpage, I entered login/password, the system complained about “username password wrong”. This error message was wrong, because it actually activated the box, despite the error message.
  • Then I tried to update the software (was 6.02). First the System Firmware page said there is no update. I downloaded a new version. When coming back some minutes later the “firmware” page says, that there is an update. (6.03). I installed it with a mouse click.
  • Then I tried the update on the second box. Some behavior “there is no update” wait some minutes “there is an update”. But now 15 minutes after the first update it says 6.04!
  • Start over with the first box. Which says there is not update for 6.03. This time I used the downloaded update from the Fortinet website.
  • Then I tried to connect them to a cluster. I tried several settings double, tripple checked cluster name, password, percentages. No success no error messages.
  • While checking back and forth, I was connected to one mgmt port directly and to the WAN side of the other box using the local notebooks WLAN. I recognized that the box connected to WAN often lost the connection. The reason: both boxes seem to have the same MAC and therefor the same IP! Two boxes not connected to a cluster have the same MAC? Fortinet, really ?
  • The cluster was still not connected. No error message, no hint, until I recognized that HA1 was connected to HA2 of the second box, and the other way round. After swapping cables the cluster nodes saw each other. Why does this matter ? And if the developers think I is ok to let the sysadmins swap the cable for them, why is there no reasonable logfile information. I didn’t find any.
  • Cluster was still not in sync, I had to enter a CLI execute command to activate synchronization

I work a lot with hardware from different vendors like Cisco, F5, Juniper, HP Flexfabric, Ubiquiti, Mikrotik. This user experience was the worst in the last years.

F5 iRule Class Match Crash

Problem: F5 iRules with “class match” crash sometimes with this message:

/Common/UA_DETECT – ambiguous option “-“: must be -all, -index, -element, -name, or -value while executing “class match [string tolower [HTTP::header User-Agent]] contains UA_STRINGS”

Discussion: the class match command has optional parameters, when the HTTP header User-Agent starts with a “-” it gets intepreted by the tcl interpreter. This is dangerous, because it’s actually a kind of code injection, with possible terrible impact.

Solution: add “‐‐” as first parameter to the class match command

class match ‐‐ [string tolower [HTTP::header User-Agent]] contains UA_STRINGS

Version: F5 LTM 12.1.2

No Text Console After NVME Storage Upgrade

Problem: After upgrading a system to NVME, Linux boots without textconsole, or sends the text console to the wrong graphics card.

Discussion: The NVME upgrade needed to change from BIOS boot to UEFI boot. UEFI boot disables the pure text console, and the kernel uses whatever frame buffer is available, or without text console if no frame buffer is available. That means the kernel config item “CONFIG_VGA_CONSOLE” does not work any longer in UEFI, at least on my hardware.

Solution: I activated a framebuffer console driver with this kernel config item: CONFIG_FB_EFI

Versions: Debian9, Linux 4.9, ASUS Z170-A, Nvidia GPU

Juniper MX204 Upgrade

    1. Check the current version.
    2. > show version
      Hostname: Router
      Model: mx204
      Junos: 18.2R1.9
      JUNOS OS Kernel 64-bit [20180614.6c3f819_builder_stable_11]

    3. go to https://support.juniper.net/support/downloads/ and select MX204, and search for newer version
    4. Select “Install Package” -> “VMHOST 64-BIT” -> “tgz”
    5. After login you will see a URL. Copy this URL
    6. Download the file onto your Juniper MX204, with this command

      > file copy “URL YOU COPIED”  /var/tmp/image-name.tgz

    7. You may validate the image with:

      > request system software validate /var/tmp/image-name.tgz

    8. Install the software:

      > request vmhost software add /var/tmp/image-name.tgz
      > request vmhost reboot

Source: https://www.juniper.net/documentation/en_US/junos/topics/concept/installation_upgrade.html