MikroTik Automatic IPSec Failover

Problem: Mikrotik allows only one ipsec policy per network-to-network pair. If you want to have redundant tunnels between two locations with two upstreams you cannot configure ipsec redundancy on Mikrotik because one ipsec policy is always marked as “invalid” by the OS.

Solution: I made a Mikrotik script that checks the status and reachabilty of the ipsec tunnel and endpoint, and switches between a primary and secondary tunnel policy and peer. You can add this script to the scheduler, for automatic failover. (Source: “/system script run 0” if this script is script “0”)

{
:local PrimaryPolicy 2
:local SecondaryPolicy 3
:local PrimaryPeer 0
:local SecondaryPeer 1

:local PrimaryOK [:ping count=3 src-address=localAip remoteAip];
:local SecondaryOK [:ping count=3 src-address=localBip remoteBip];
:local PrimaryActive [/ip ipsec policy get $PrimaryPolicy active];

# :log info "Status: $PrimaryOK $SecondaryOK $PrimaryActive";
# Test case: set $PrimaryOK 0;

:if ($PrimaryOK < 1 && $SecondaryOK > 1 && $PrimaryActive) do={
:log warn "switch to failover";
/ip ipsec policy disable $PrimaryPolicy;
/ip ipsec policy enable $SecondaryPolicy;
/ip ipsec peer disable $PrimaryPeer;
/ip ipsec peer enable $SecondaryPeer;
}
:if ($PrimaryOK = 3 && !$PrimaryActive) do={
:log warn "switch to primary";
/ip ipsec policy disable $SecondaryPolicy;
/ip ipsec policy enable $PrimaryPolicy;
/ip ipsec peer disable $SecondaryPeer;
/ip ipsec peer enable $PrimaryPeer;
}
}

Version: tested with RouterOS 6.44.1

Juniper MX204 Setup Guide

Juniper MX204 is router from Juniper running Junipers own operation system Junos.

The MX204 has 4x 40Gb and 8x 10Gb. The 40Gb ports can be split into 4x 10Gb.

After unboxing it has no configuration. Connect a standard RS232 console cable with a Cisco style RJ45 connector, set your terminal to 9600 8N1 and power it up.

The Junos console welcomes you with a standard FreeBSD login.

Login in with “root” and no password.

Start the Junos CLI with “cli”

roo@:# cli
root>

This is the standard mode that you will reach later when configuring such device over a network connection (Telnet/SSH)

Like Cisco, Junos has two modes “standard” mode and “configure” mode:

root> configure
[edit]
root#

Different to Cisco, on Junos configuration changes are not active immediatly. You can configure different things in config mode and when you finished type “commit” to active changes or “exit” to discard your changes.

Here are some settings for the first setup:

# change root password
set system root-authentication plain-text-password
# add another user
set system login user USERNAME authentication plain-text-password
set system login user USERNAME class super-user
# set host name
set system host-name HOSTNAME
# set the managment IP for the “mgmt” port
set interfaces fxp0 unit 0 family inet address ADDRESS/PREFIX_LENGTH
# On Junos 18.2 the default setting for fxp0 is dhcp, deactivate dhcp
delete interfaces fxp0 unit 0 family inet dhcp
# Starting in Junos OS 17.3R1 you can seperate the mgmt interface from the default routing table
set system management-instance
set routing-instances mgmt_junos routing-option static route 0.0.0.0/0 next-hop MGMT_LAN_ROUTER
# activate ssh (and/or telnet)
set system services telnet
set system services ssh
# Junos 18.2 has auto software upgrade, deactivate it
delete chassis auto-image-upgrade
# active and save all changes
commit

You can list you current configuration with “show” inside and “show configuration” outside of configuration mode

Starting with 18R1 Juniper MX204 can mix 10G and 1G ports on the 8 SFP+ Ports.

You have to set the pic to per port mode with

set chassis fpc 0 pic 1 port 0 speed 10g
set chassis fpc 0 pic 1 port 1 speed 10g
set chassis fpc 0 pic 1 port 2 speed 10g
set chassis fpc 0 pic 1 port 3 speed 10g
set chassis fpc 0 pic 1 port 4 speed 10g
set chassis fpc 0 pic 1 port 5 speed 10g
set chassis fpc 0 pic 1 port 6 speed 10g
set chassis fpc 0 pic 1 port 7 speed 10g

# and
set interface xe-0/1/7 gigether-options speed 1g
# or
set interface xe-0/1/7 gigether-options speed 10g

If you like to backup the config to a server you can copy the config using ssh:

file copy /config/juniper.conf.gz configs@80.245.199.199:router.conf.gz

ToBeContinued

Source IP Address Based on User

If you want to use different source IP addresses based on the logged in user or running service on a Linux computer you can use these simple commands:

/sbin/ifconfig eth0:1 NEW-IP-ADDRESS netmask YOUR-NORMAL-NETMASK
/sbin/iptables -t nat -A POSTROUTING -m owner --uid-owner USERNAME -j SNAT --to-source NEW-IP-ADDRESS

You can use this if the source IP is necessary for remote firewall filter lists, or to separate IP traffic from services that don’t allow to configure the outgoing source IP.

Add these lines to /etc/rc.local to make it permanent.

Version: Should work on every Linux kernel of the last 10 years, tested on Linux 4.11.1

Ubiquiti UniFi the Next Botnet ?

I tested a Ubiquiti access point today. UAP-AC-Lite seems to be a very good and cheap access-point.

When you take it out of the box and connect it to the network it gets an IP address using DHCP and waits for a configuration. In this mode it sends broadcasts to find a controller and listens on port 22 (ssh) with standard login/password of ubnt/ubnt.
That’s not best practice but very usual for devices of this kind.

I tried two configuration modes:

    1. MobileApp based using my Android Phone:
      This App looks good, and works great, if you need just one SSID and not VLAN. Thumbs up, well done ubiquiti.
      But I guess this method will not work if this is you first access point in the network, because you will end with a chicken and egg problem.
    2. UniFi Controller based:
      UniFi runs on Win/Mac and Linux. The Debian package is far to big but it installs cleanly (Why does this webapp need 27MB of fonts?).
      With this webapp you can configure everything and it works good. But then I checked the security…

First I checked what new ports are open on my server:

tcp6 0 0 :::8443            :::* LISTEN 1373/java 
tcp6 0 0 :::6789            :::* LISTEN 1373/java 
tcp6 0 0 :::8843            :::* LISTEN 1373/java 
tcp6 0 0 :::8880            :::* LISTEN 1373/java 
tcp6 0 0 :::8080            :::* LISTEN 1373/java 
udp6 0 0 MYPUBLICIP:50880   :::* 1373/java 
udp6 0 0 :::10001           :::* 1373/java 
udp6 0 0 :::3478            :::* 1373/java 
udp6 0 0 MYINTERNALIP:58426 :::* 1373/java 

That’s to much for a Linux box with a public IP interface.
The documentation tells a little bit what these ports are used for, but some are not explained or not needed for normal operation.
I tried to strip down the open ports for security reasons, but I found no way to disable unused services or at least bind only to one IP. My minimum requirement would be to bind only to an internal interface and block the public interface.

But no way (officially: https://community.ubnt.com/t5/UniFi-Feature-Requests/Allow-Controller-to-run-on-a-Single-IP-Address/idi-p/959213 )

Shure I could write an iptables filterlist to block these ports, but that’s risky. Today they use these 9 ports, but what will happen on the next update ?

Then I checked what services are actually running on these ports. It’s a tomcat server !
A java/tomcat server that listens in all directions IPv4/6 and no easy way to limit this access. What can possibly go wrong?

Most people will never update this controller software, and tomcat had and will have security problems.
http://www.cvedetails.com/product/887/Apache-Tomcat.html?vendor_id=45

Hopefully ubiquiti will provide a smaller footprint configuration tool, with a bit more settings than the app, and add some security settings to the controller software.
Then I would really recommend this nice piece of hardware: Vendor Link  ,  Amazon Link

Version: UAP-AC-Lite, unifi 5.4.11-9184

PaloAlto Packet Loss of 1% and More

Problem: PaloAlto firewall is dropping packets in small bursts of some seconds, and sometimes it drops TCP connections. It only happens on HA clusters on interfaces in active/passive (fail over) mode.

Solution: disable the following check box in the Ethernet interface Advanced – LLDP settings: “Enable in HA Passive State”

Discussion: Palo Alto uses only one MAC address for both machines of an HA cluster. The passive box sends LLDP packets using this MAC address. The switch learns this MAC address and sends the traffic to the passive node until the active node sends new packets. The passive node should never send packets with the MAC address of the active node, and should have its own MAC address for LLDP and possibly other services.

Version: PaloAlto current version of Nov 2016, connected to a Cisco Catalyst 6500

ARP is not working on Cisco ASR 1001 X

Problem: Cisco ASR router is loosing connectivity to its directly attached Ethernet neighbors. In this situation interface status is still up, packets are going in and out on both ends, even IPv6 was still working. The actual problem was that the Cisco ASR was ignoring all ARP responses from its neighbors and the ARP table to this interface was empty. Later the same happened on a second interface.

A temporary work around was to reboot the router.

Solution: Cisco support suggested a software upgrade, even though the software was only some weeks old. After the software upgrade the error didn’t happen again until now.
The old IOS version was: asr1001x-universalk9.03.16.03.S.155-3.S3-ext.SPA.bin
The new IOS version is: asr1001x-universalk9.03.16.04a.S.155-3.S4a-ext.SPA.bin

The only fix that possibly fits to the problem is:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20160804-wedge

“A remote attacker can cause an interface wedge and an eventual denial of service condition”

What’s an “interface wedge”. Cisco bug reports were more precise years ago.

 

ASR Tips’n’Tricks

ASR-1001-X and IOS-XE is sometimes different and sometimes very similar to classic IOS.

Update. You can update, the firmware as usual:

# copy http: bootflash:
# conf t
(config)# boot system flash bootflash:asr1001x-universalk9.03.16.00.S.155-3.S-ext.SPA.bin

Show SFP (transceiver) info:

# show hw-module interface tenGigabitEthernet 0/0/0 transceiver status
# show hw-module interface tenGigabitEthernet 0/0/0 transceiver idprom

.. to be continued

MLPPP over L2TP over Ethernet Channel Groups on Cisco ASR

 

Problem: After upgrading an ethernet port to a channel-group, all MLPPP connections fail on a Cisco ASR 1002-X. The log file looks like this:

Jul 31 2015 07:04:44.801 CEST: Vi4 PPP: Phase is AUTHENTICATING, Authenticated User
Jul 31 2015 07:04:44.801 CEST: Vi4 CHAP: O SUCCESS id 143 len 4
Jul 31 2015 07:04:44.801 CEST: Vi4 PPP: Phase is VIRTUALIZED
Jul 31 2015 07:04:44.802 CEST: Vi6 MLP: Added link Vi4 to bundle xxx
Jul 31 2015 07:04:44.803 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access4, changed state to up
Jul 31 2015 07:04:44.803 CEST: %LINK-3-UPDOWN: Interface Virtual-Access4, changed state to up
Jul 31 2015 07:04:44.805 CEST: %CPPOSLIB-3-ERROR_NOTIFY: SIP0: cpp_cp:  cpp_cp encountered an error -Traceback= 1#795bed15105852c19a9ac138912d7358   errmsg:7F13FA6E0000+121D cpp_common_os:7F13FD6F1000+D8D5 cpp_common_os:7F13FD6F1000+D7D4 cpp_common_os:7F13FD6F1000+19A3E cpp_ifm:7F14106F1000+A198 cpp_mlppp_svr_lib:7F1406B63000+C351 cpp_mlppp_svr_lib:7F1406B63000+1CDC8 cpp_mlppp_svr_smc_lib:7F1406DA1000+2D28 cpp_common_os:7F13FD6F1000+11E6E cpp_common_os:7F13FD6F1000+118AA cpp_common_os:7F13FD6F1000+116EB evlib:7F13FC6D10
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP: O CONFREQ [REQsent] id 13 len 10
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP:    Address xxx
Jul 31 2015 07:04:45.152 CEST: Vi6 IPCP: Event[Timeout+] State[REQsent to REQsent]
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP: O CONFREQ [REQsent] id 14 len 10
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP:    Address xxx
Jul 31 2015 07:04:47.168 CEST: Vi6 IPCP: Event[Timeout+] State[REQsent to REQsent]

The router continues with “O CONFREQ” but never receives the “I CONFACK”.

Discussion: In this case the router is a L2TP server and handles multiple L2TP/PPP connection. Some of them are multilink PPP connections. The ASR software has a bug that leads to these tracebacks when the L2TP connections are going over an ethernet channel group. We opened a case with Cisco support. After one and a half month we received this answer:

---
Apologies for the delay. I was held up on other critical issues and hence was unable to reach out to you earlier. I was able to decode the tracebacks observed during the time of the issue and the issue points to a known software bug as the cause of the problem. Below are more details

CSCua16777 : FMFP-3-OBJ_DWNLD_TO_CPP_FAILED: SIP0: fman_fp_image: MLP bundle
Bug toolkit link : https://tools.cisco.com/bugsearch/bug/CSCua16777/?reffering_site=dumpcr

However the above bug is in closed state with the below release-notes

Symptom: FMFP-3-OBJ_DWNLD_TO_CPP_FAILED: SIP0: fman_fp_image: MLP bundle 8767, link 8766 download to CPP faile
Conditions: LNS MLPPP sessions don't stay up over port-channel
Workaround: MLPPP over port-channel is not supported on ASR1k. Don't use MLPPP over port-channel.
---

Dear Cisco! This is no solution. If you define an obvious bug as normal behaviour and the only workaround is “Don’t use..”, your customers will soon remember this:
“Cisco ? Don’t use…”

Solution: “Don’t use Cisco ?”

Version: Cisco ASR 1002-X, IOS XE Version: 03.09.02.S

Update: If I use the link of the bug report, I receive this answer:

Insufficient Permissions to View Bug
This bug contains proprietary information and is not yet publicly available.
Cisco Support Community

Unbelievable !! CSCO: STRONG SELL!

F5 data flood

Problem: A F5 load balancer LTM sends lots of data to some clients. Sometimes this fills up all the available bandwidth with 1 Gbit or more. At the same time the input traffic does not raise. The traffic charts look like if F5 is attacking some clients (reversed DDoS :-) )

Discussion: After some time of staring at tcpdump output like in Matrix :-) I found the following behavior:

10 A client sends an HTTP request to the F5
20 The F5 forwards the HTTP request to a webserver and gets the data
30 The F5 sends the data to the client, eg. the data has 10kByte which fits into the TCP window
40 A router on the way to the client sends an "ICMP unreachable need fragmentation 1440 Byte"
50 The F5 replays the 10kByte in fragmented packets smaller than 1440 Byte
60 The router sends "need fragmentation" again
70 The F5 replays the already fragmented 10kByte at wire speed
80 goto 60 

Why does the router send “need fragmentation” although the packets are already fragmented ?
The ICMP packet shows that the fragmented packets arrive reassembled at the client side, which means some router between the F5 and the client reassembles the fragments and ignores the need fragmentation packets from the client router. “Automatic reassembly and offloading is evil!”

This is obviously a bug on the router which reassembles the packets. But on the other hand the F5 LTM should not send out the same data in a loop up to wire speed, it should either throttle the packet flow, or it should not retransmit the already fragment packets after “need fragmentation”. The normal retransmit for missing tcp packets should apply.

I filed this bug with F5 twice but they said this behavior is correct and they won’t change it.

Solution: Fix the bogus router if you can. If you can’t I think there is no other solution than don’t use F5 LTM

Update: A work arround is to disable pathmtudiscovery on F5. Without PMTU Disovery the DF IP header is not set and routers and firewalls usually fragment large packets them self and don’t send ICMP Need Frag messages.

# tmsh
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos)# sys db
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.pathmtudiscovery value disable
root@(F5)(cfg-sync Changes Pending)(Active)(/Common)(tmos.sys.db)# modify tm.enforcepathmtu value disable

Version: tested with F5 LTM 11.2.1 and 11.5

Cisco ASR 1002-X and PPTP

Problem: PPTP from any client to an ASR1002-X Cisco does not work. PPTP Connections starts but in PPP LCP phase the connection fails.

Solution: Cisco ASR1002-X with Software IOS-XE 15.3(2)S2 has no PPTP support. You have to take a different Router!

Discussion: The weird thing is, that most of the PPTP stack is still configureable and working, but all packets coming from the client inside the PPTP tunnel are dropped!

Some examples:

.) #show vpdn tunnel
%No active L2TP tunnels
%No active PPTP tunnels

.) in vpdn-group you can set protocol any

.) the router is answering PPTP (TCP 1723)

.) the router starts the PPP layer when a connection is coming in

.) the router even sends LCP O CONFREQ packets to the client

But! The Cisco ASR Router drops every LCP I CONFACK coming from the client.

Cisco was always a reliable piece of hardware for me, but this looks like they removed a feature without removing the code and their QA department worked like this: “it compiles, ship it”.