ios – AlexT Debugs

Cisco ASR-1001-X Update

There are at least two pieces of software you can update in a Cisco ASR-1001-X. The ROMMON (firmware) and the IOS itself.

This router uses “Cisco IOS XE Software”. Which is an IOS process on a Linux kernel as far as I know.

Cisco recommends specific ROMMON releases for different generations of software. You can look it up at: https://www.cisco.com/c/en/us/td/docs/routers/asr1000/rommon/asr1000-rommon-upg-guide.html#concept_zdm_2nx_5bb

The software versions for these ASR 1000 series routers had complicated versions numbers like these:

asr1001x-universalk9.03.16.07b.S.155-3.S7b-ext.SPA.bin which is 03.16.07b.S and actually IOS 15.5(3)S7b.

But with IOS 16 it was simplified a current software image looks like this

asr1001x-universalk9_noli.16.09.06.SPA.bin which is IOS 16.9.6.

If you have a cisco account with running service contract, you can download the Cisco software from: https://software.cisco.com/download/home

The IOS XE software, comes with or without “Payload encryption”, and with or without “Lawfull Interception”.

To check the current installed rommon and IOS version enter:

show plattform
show version

To update the rommon copy the rommon image to flash and enter:

upgrade rom-monitor filename bootflash:asr1000-romm... all

To update the IOS software copy the IOS image to the flash and enter (in config mode):

boot system flash bootflash:asr1001x-universalk9_...
no boot system flash bootflash:...old image...
no boot system flash bootflash:...older images...
boot system flash bootflash:...old image...
exit
wr

This changes the boot order to prefer the new image as default but keep the old one as fallback. After the “wr” you can check the bootvar with “show bootvar” and see if the next reboot should use the new image.

Next, reboot the router with “reload”, and check if the software has changed
after the reboot. You also might want to save a new backup of the configuration and check how it differs after the update.

MITMProxy and IOS 13

Problem: if you want to debug a IOS app with MITMProxy, the iPhone needs to trust the MITMProxy CA. This is done by going to http://mitm.it/ and clicking on the apple symbol. Then you have to accept the “profile” in Settings “downloaded profiles”. Then you have to trust this new CA cert in “Settings” “General” “About” “Trust Root Cert” “mitmproxy”. But then the certs generated by the MITMProxy are still not trusted.

Discussion: Starting with IOS 13, TLS server certificates must have a validity period of 825 days or fewer and MITMProxy generates certs with an expiration period of 1095 days.

Solution: I changed the py file of MITMProxy to shorten the cert validity, by changing the file /usr/lib/python2.7/dist-packages/netlib/certutils.py

# DEFAULT_EXP = 94608000  # = 24 * 60 * 60 * 365 * 3
DEFAULT_EXP = 31536000  # = 24 * 60 * 60 * 365

Versions: test with MITMProxy 0.18.2-6+deb9u2 but it looks as if current versions of MITMProxy on github still use 3 years as default expiration.

ARP is not working on Cisco ASR 1001 X

Problem: Cisco ASR router is loosing connectivity to its directly attached Ethernet neighbors. In this situation interface status is still up, packets are going in and out on both ends, even IPv6 was still working. The actual problem was that the Cisco ASR was ignoring all ARP responses from its neighbors and the ARP table to this interface was empty. Later the same happened on a second interface.

A temporary work around was to reboot the router.

Solution: Cisco support suggested a software upgrade, even though the software was only some weeks old. After the software upgrade the error didn’t happen again until now.
The old IOS version was: asr1001x-universalk9.03.16.03.S.155-3.S3-ext.SPA.bin
The new IOS version is: asr1001x-universalk9.03.16.04a.S.155-3.S4a-ext.SPA.bin

The only fix that possibly fits to the problem is:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20160804-wedge

“A remote attacker can cause an interface wedge and an eventual denial of service condition”

What’s an “interface wedge”. Cisco bug reports were more precise years ago.

Update Cisco Catalyst Software

I had to update the software of a new Cisco Catalyst 4948 yesterday.
As usual I did:

copy tftp://<hostname>/<filename> bootflash:
conf t
boot system flash bootflash:<filename>
exit
reload

But the switch ignored the new software image.
During boot it said:

Booting first image from bootflash

Solution: The config-register was set to 0x2101 right out of the box. I had to change this to 0x2102. With this value the switch honors the bootvar and boots into the configured system image.

conf t
config-register 0x2102
exit
reload

The config-register has different meanings on different cisco plattforms.
For the Catalyst 4948 the config-register contains a bit field explained at:

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/configuration/guide/supcfg.html#wp1051990

Windows Network Load Balancing NLB and Cisco Routers/Switches

Problem: Windows NLB IPs are are not reachable through and from Cisco routers and switches. NLB services could be IIS arrays, Exchange CAS arrays, etc.

Solution: NLB mode was set to Multicast. In this mode Windows incorrectly uses multicast mac addresses. Set the NLB mode to Unicast and configure static mac address table entries on your switch to prevent broadcast flooding.

Discussion: Windows NLB does only work if all members of an array get all packets for a balanced service. To achieve this, Windows knows 3 modes of load balancing: Unicast, Multicast, and Multicast IGMP. And all of them have problems:

Unicast: Windows uses a normal mac address for his virtual IP address, but never sends any packet from this mac. The switch never learns a mac address entry and has to broadcast all packets for this mac. (Broadcast Flooding)

Multicast (IGMP): Windows uses mutlicast macs for the virtual IP. This seams correct, because this way the switch could learn which ports are part of the array and which are not (IGMP), but the problem is: an arp request for an unicast address (virtual IP) must not resolve to an multicast mac address. Cisco switches simply ignore this arp responses. Multicast mac addresses start with the lsb bit of the first bit set, typically 01:XX:XX:XX:XX:XX or 03:XX:XX:XX:XX:XX.

My solution was to use Unicast mode and don’t use IGMP. The other solution would be to statically set arp and the mac-address-table on the cisco switch, and force it to use the incorrect mac address.

BTW: A Network Load Balancing mechanism where every array member receives all traffic, in no real “Network” load balancing, because you dont’t reduce the traffic per server, it just adds additional computers and no additional network capacity.

More info: http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

Cisco has weird naming conventions for groups of interfaces

If you want to bind some ethernet interfaces into one logical interface between two ciscos switches you have to configure an interface first:

interface Port-channel 1

then add some interfaces into this new interface:

interface GigabitEthernet 1/0/1
channel-group 1 mode active

and you can check the status with:

show etherchannel 1 summary

Three names for the same thing. Does anybody know, why ?

L2TP tunnel between two cisco routers

Problem: a Cisco 1941 [15.1(2)T2] (Client) tries to connect to an Cisco7300 [12.3(22)] (Server) using L2TP, but after successful authentication of the client the server drops the connection.

Solution: The client tried to authenticate the server, but the server had no password for authentication. Normally only the client authenticates to the server in ISP environments. Server sends “O CHALLENGE” client sends “I RESPONSE”. In this case the Client sent “O CHALLENGE” back. I debugged this problem by removing the following line from the clients interface Virtual-PPP section:

ppp authentication chap pap callin optional

Usually this means “authenticate a client (optionally) using chap or pap when it calls in”, but on this router/ios-release the client tried to authenticate the server when this line is present.

Cisco Access Point blocking PPTP

Vienna: 10. August 2011

Problem: A network of some Cisco Access Points AIR-AP1142N-E-K9 connected to an small router works good as expected but when the user tries to connect to en PPTP VPN the connection fails with a time-out.

Solution: it was no NAT problem in this case (I fixed this before). Cisco IOS 12.4(21)?? has officially a bug not sending GRE (Proto:47) packets from the network to the client. IOS Version 12.4(25d)JA fixed this bug.

Strange! An access point in bridging mode filters depending on protocol numbers. So I guess it’s not really a bridge that Cisco implemented in this IOS device.