Ping to Windows Server 2012 with Direct Access fails

Problem: Ping from Linux computers to Windows Server 2012 with Direct Access fails for hours (no response) although connectivity is working (eg. HTTPS)

Discussion: Ping from other windows machines and Cisco machines worked, but from Linux machines ping was not answered. I tried to find the difference between those ICMP Echo Request Packets, and reduced it to 16 Bits in the ICMP Packet. The ID field! Windows always uses “2”. Linux uses the pid of the ping process.

I patched the ping source to select the ID manually, and scanned through the values 0 – 65535.

Ping was answered for IDs 0 – 6000

Ping was not answered for IDs 6001 – 47000

Ping was answered for IDs 47001 – 65535

A really strange Microsoft bug.

Solution: ??? Only Microsoft can fix this. The workaround is to set the id manually with my patched ping client.

Version: any Linux with standard ping. Windows 2012 with Direct Access patched up to date (21.8.2013)

WScale and Cisco Content Switch CSS.

Problem: Windows 7 clients wait 5 seconds to send simple HTTP requests to a web server behind a cisco CSS content switch. But there is no delay when the client connects directly to a web server behind the CSS.

Description: I found one difference between CSS and non-CSS connects: the TCP WSCALE option.
When the CSS is in layer 7 HTTP switching mode (e.g. url “/*”), it does the tcp handshake and the http-request parsing on it’s own, and afterwards it decides which service is receiving the request. For this initial connect the CSS uses WSCALE=0. Which means window scaling is welcome, but the css sets it’s inbound scaling to 1 (2^0). After the request is sent from the client to the css, the css connects to the server, which negotiates WSCALE=8 (scale 256, seams to be the Windows 2008 R2 standard value).
Now the client believes WSCALE=0 for outgoing data and the server believes WSCALE=8 for incoming data.
After the first HTTP request (Keep-alive: active) the Windows server sets it’s window to 256, which is 256*256 byte for the server and 256*1 byte for the client. When the client
requests a second url it has to send the new request in 256 byte pieces and waits for the ack from the server for each packet. And both Linux and Window clients do exactly that. This is a performance penalty of course, but still it should work, but windows sends this 256 byte data pieces after a timeout of 5 seconds. I don’t know why the windows client waits, but it does. Perhaps MS thinks the window will raise magically in the meantime.

Solution: We have 2 bugs here. CSS is doing the WSCALE wrong in layer 7 switching mode, and windows waits 5 seconds before it sends  > 256 byte data into a connection with a 256 byte window. For the first bug I have two work arounds:

  1. Use only layer 3/4 switching, if the url or header based switching is not necessary
  2. Apply the following setting on your css: flow tcp-window-scale disabled

The second work around disables wscale at all. The CSS sends no wscale option, which disables wscale in both directions (RFC 1323). Without the TCP WScale option the maximum window size is 64k, which should be enough. (except for huge file downloads, over far network distances perhaps)

Windows Network Load Balancing NLB and Cisco Routers/Switches

Problem: Windows NLB IPs are are not reachable through and from Cisco routers and switches. NLB services could be IIS arrays, Exchange CAS arrays, etc.

Solution: NLB mode was set to Multicast. In this mode Windows incorrectly uses multicast mac addresses. Set the NLB mode to Unicast and configure static mac address table entries on your switch to prevent broadcast flooding.

Discussion: Windows NLB does only work if all members of an array get all packets for a balanced service. To achieve this, Windows knows 3 modes of load balancing: Unicast, Multicast, and Multicast IGMP. And all of them have problems:

Unicast: Windows uses a normal mac address for his virtual IP address, but never sends any packet from this mac. The switch never learns a mac address entry and has to broadcast all packets for this mac. (Broadcast Flooding)

Multicast (IGMP): Windows uses mutlicast macs for the virtual IP. This seams correct, because this way the switch could learn which ports are part of the array and which are not (IGMP), but the problem is: an arp request for an unicast address (virtual IP) must not resolve to an multicast mac address. Cisco switches simply ignore this arp responses. Multicast mac addresses start with the lsb bit of the first bit set, typically 01:XX:XX:XX:XX:XX or 03:XX:XX:XX:XX:XX.

My solution was to use Unicast mode and don’t use IGMP. The other solution would be to statically set arp and the mac-address-table on the cisco switch, and force it to use the incorrect mac address.

BTW: A Network Load Balancing mechanism where every array member receives all traffic, in no real “Network” load balancing, because you dont’t reduce the traffic per server, it just adds additional computers and no additional network capacity.

More info: http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml