Prime AP Migration Tool - Part 3

In the third part of our series of articles on how to seamlessly upgrade your Cisco WLC infrastructure, we'll examine the important thing to think about - the caveats, hints and tips for a smooth upgrade. If you've not see the other two articles in this series, you can catch up with the two links below: Prime AP Migration Tool Part 1: Explore the background with HA and prerequisites Prime AP Migration Tool Part 2: Details on how to upgrade, using the free IPTel supplied tool

Caveats, Hints and Tips 

A code upgrade is a MAJOR change in any wireless network. The following notes have been collated on the back of real world experience upgrading large sites with thousands of APs. You must work VERY cautiously on a large or critical site upgrade.

1. Firstly, and most importantly – save and BACK UP the config on your WLC prior to starting any major work.
2. Take care if you have a Primary and Secondary WLC already allocated for your APs – the IPTel script will overwrite this. Run some tests prior to undertaking any changes to confirm if the script works as you expect. The tool is offered without warranty - TEST IT IN THE LAB FIRST!
3. As well as the possibility of a failure during the upgrade requiring a rollback, you might hit new bugs. Make sure you have a well thought out change control process and ideally test a subset area on the new code before rolling out across the site.
4. Lab test your upgrade and new code. If you have any sort of unusual end devcices, they may not operate or roam correctly on the new code. Do some lab testing.
5. Read the release notes. Yes, it is boring, but you'll at least know if any of the bugs might be an issue for you.
6. Make sure your smartness support contract is up to date. Just done an upgrade and its failed? No smartnet and you're on your own - make sure its up to date first!
7. We have come across many new Cisco bugs on the larger sites we work on – treat a code upgrade with great care and be sure to thoroughly test all services on the new code once the upgrade is complete.
8. Only do a batch of APs at a time – or you will take all APs offline at once, which defeats the purpose.
9. Run the script only on one AP first as a test – confirm it moves the AP between the desired WLCs correctly
10. Test all your services on the AP once its on your spare WLC – the spare WLC probably doesn’t get used much, so confirm its working as expected before moving a lot of APs across!
11. There’s a major caveat to note with the use of the spare controller – the last digit of the MAC address of each SSID (i.e. the BSSID) is determined based on the order it was added to the AP group on the WLC (the order the SSIDs were created in the case of the default AP group). If you don’t correctly configure all the WLCs AP groups with the SSIDs in the same order as each other, you will find the last digit on the SSIDs will change when you move APs to the spare WLC. This is not a problem with most applications – but will affect RTLS applications such as Ekahau that rely on the MAC address of the AP.

Liability Disclaimer 

We supply the tools on this website free of charge for the wireless community use. Note with all our tools the liability disclaimer – we do our best to make these tools as useful as we can, but accept no liability for their use or misuse:

https://www.iptel.com.au/terms-and-conditions-for-usage-of-provided-tools.html

Prime AP Migration Tool - Part 2

In this second article on how to seamlessly upgrade your Cisco WLC, we explore the migration process and how to use the free tool IPTel have built for the purpose.

If you've not see the other two articles in this series, you can catch up with the two links below - check the caveats, hints and tips before you start any upgrades: Prime AP Migration Tool Part 1: Exploration of HA and pre-requisites to seamless code upgrades Prime AP Migration Tool Part 3: Caveats, Hints and Tips on WLC code upgrades

AP Migration Process

There are caveats later in this blog – familiarise yourself with these – if you are unsure of what you are doing do not run this script on a live network. Practice in your lab environment until you are happy with the process.

Access the tool from the IPTel website – here’s the link: https://www.iptel.com.au/ap-migration-tool.html

You will need to make a choice when moving APs – do you upgrade the spare WLC to the new code and upgrade APs on their way over, or just do a straight move then upgrade the primary WLC and upgrade them on the way back. There’s pros and cons to each method.

At some point you need to upload the new WLC code to your controller – this can be done at any stage prior to, or after offloading all the APs.

A screen shot of the tool is shown below:

Prime AP Migration Tool – Entry Screen

The process is as follows:

• Select the ‘Move AP Script’ radio button
• Enter the ‘From’ details in the Controllers section (hostname and IP address of your current Primary WLC)
• Enter the ‘To’ details for the spare / holding platform WLC to move the APs to
• In the Access Points box, paste in a list of all your APs
• Click the generate button

This will generate a list of commands to be pasted to the WLC GUI. When pasted into the GUI, APs will have the order of their Primary controller changed – the commands will then reload the AP and it will move to the secondary platform.

Once all APs are moved, you can upgrade the primary WLC (or HA if its in HA mode) without affecting any of the APs.

When you’re ready to move the APs back to the original WLC, select the ‘Reset AP Script’ button and click the Generate button – this will generate the reverse script to return the APs back to the Primary WLC.

Pre-download

To reduce the impact of the upgrade (speed up the process), pre-download the new AP image to the APs before moving them to a WLC that uses a different code version. Note that without a pre-download, APs will have to download the correct image then reload to boot that image when you move them between WLCs with different code versions! To do a pre-download, you need to download the new code onto the WLC but don't reboot onto it. You will then be able to pre-download that code to all APs (or a single AP at a time if you want to test) either via the GUI or the CLI.

Liability Disclaimer 

We supply the tools on this website free of charge for the wireless community use. Note with all our tools the liability disclaimer – we do our best to make these tools as useful as we can, but accept no liability for their use or misuse:

https://www.iptel.com.au/terms-and-conditions-for-usage-of-provided-tools.html

Prime AP Migration Tool - Part 1

Mark at IPTel is sharing some of his scripts to help with controller upgrades! Check it out and let us know what you think!  

This article is the first in a series dedicated to how to upgrade your Cisco WLCs within the minimum of client impact. This article provides the background and pre-requisites, with further articles detailing the process. In addition, IPTel have provided the AP migration tool to make life easy to transfer APs from one WLC to the next.

If you've not see the other two articles in this series, you can catch up with the two links below - check the caveats, hints and tips before you start any upgrades:

• Prime AP Migration Tool Part 2: Details on how to upgrade, using the free IPTel supplied tool
• Prime AP Migration Tool Part 3: Caveats, Hints and Tips on WLC code upgrades

The tool can be found on this link: https://www.iptel.com.au/ap-migration-tool.html

If you don't like having all your eggs in one basket, you'll have deployed your Cisco WLAN controllers in SSO mode and now discovered when you do an upgrade that they both reload, one after another. In this article we explore what you need to do to perform a seamless upgrade, with virtually no client impact on a Cisco WLAN Controller. Firstly, you need to take into account a couple of pre-requisite factors: Ensure you have a spare controller Use the Prime AP Migration Tool to move APs around

The bad news is that you need some spare hardware - if you have only a single controller or only a HA pair (in SSO mode) then you can’t do a seamless upgrade. 

When you upgrade the HA pair it copies the new code between the WLCs and reloads each in turn – you’re in for a 15 – 20 minute outage.

The process is even worse if you've not done a pre-upload of code to your APs prior to the WLC code upgrade – once the WLCs are operational on the new code, the APs will join, download, reload onto the new code and join again. The network is going to be unstable while this process occurs.

High Availability (HA)

Firstly, let's have a quick segway onto the issue of High Availability. If you've got one WLC and it goes offline, you're, well offline. If you happen to have a spare WLC handy AND a copy of your config, all good. You can restore the failed controller and maybe have 3 - 4 hours of downtime.

Don't have a spare onsite, and oops, don't have the recent config? You're in real trouble - it could be days or weeks until you return to service.

The HA part codes from Cisco are much cheaper than the fully licenced controllers and inherit the licence once connected - you now have completely seamless fail over (depending on the code version you're running).

That's the good news. The bad news is that you essentially have one single unit, just operating across two pieces of hardware. Make a config error on one and you've just made it on both. There's an outside chance of a bug taking both out at once, but in any case, when you do a code upgrade they will both reload.

The answer to this is a spare controller.

Spare Controller

To overcome this for sites which cannot have any interruption in service, we have developed designs and techniques. The first stage is ensuring you have a spare controller in the network. Fortunately due to Cisco licencing, you can buy the minimum priced HA unit and use this as an HA secondary; once in HA secondary mode (with SSO disabled) it will licence itself for the full number of APs – but only for 90 days (after which there’s a ‘nag’ message).

In addition to providing a holding platform during code upgrades, adding a Spare controller also adds to the overall resilience of the network. We normally configure this as the secondary controller on each AP; it can act as a secondary to multiple primary HA pairs of controllers (you need to configure this for each AP though).

Holding Platform

Once the Spare WLC is in place as the holding platform, you’re all set to be able to do seamless upgrades. The basic premise is to gradually move all APs from the current WLC (or HA pair) to the spare platform. Gradually moving the APs mean any clients will just roam when the AP disappears during the move. To reduce the impact even further, randomise the order in which APs are moved, so you only move one AP at a time in a particular area.

The Prime AP Migration tool is designed to perform just this task – work out a randomised list of APs to change their primary controller to be the spare – they are then rebooted and the AP reconnects on the spare WLC.

Liability Disclaimer 

We supply the tools on this website free of charge for the wireless community use. Note with all our tools the liability disclaimer – we do our best to make these tools as useful as we can, but accept no liability for their use or misuse:

https://www.iptel.com.au/terms-and-conditions-for-usage-of-provided-tools.html

802.11 Packet Capture Skillz To Pay The Bills

Digging deep into the Stefanick archives of real world 802.11 issues. I challenge YOU with 4 real world examples. Keep in mind sometimes the obvious is not so obvious. While frames don’t lie understanding 802.11 is important to see the truth. 

These are real customer issues on real networks with real problems.

PROBLEM #1: 

Customer complained of slow WiFi performance in a specific part of the warehouse. It's always been slow said one worker. It's never really preformed right since it was installed. 

During my packet capture I observed a lot of frames with a similar “bit" being marked. What “bit" could be a clue that might contribute to a slow network ?

 

ANSWER #1

If you answered retry bit you would be right. The retry counter was above 30% for channel 6. While the noise reference on channel was within reason the packet capture was a "bit" misleading displaying a -92. No pun intended. I turned on WiSpy, low and behold layer 1 interference. There were old security cameras operating on 2.4 no longer in use but still powered. The cameras were causing interference across channels 1 - 6, causing high retry rates. 

WiSpy 

PROBLEM #2: 

After a recent firmware update a number of Cisco 7925 phones exhibited an odd behavior. They would connect to the wifi network and then disconnect and display Locating Network Services. This happen repeatable.

I open my sniffer and see frames much like this one. 

ANSWER #2

If you answered duration timer you would be correct. The duration value caught my attention during troubleshooting. In the end it was a firmware bug on the handset due to an interoperability with a specific configuration and 802.11n access points. Note when a client sends a duration value, clients who can demodulate this frame will use this value and reset their clocks to busy. This was impacting the entire cell and not just the phones. 

Read this blog post in its entirety:

http://community.arubanetworks.com/t5/Technology-Blog/802-11-Packet-Capture-Skillz-To-Pay-The-Bills/ba-p/149414

30 Random Technical Thoughts by a WiFi Engineer

1) CRC is cyclic redundancy check. This means a radio received a frame and failed the checksum. A normal communication the intended receiver will not ACK and the sender will retransmit the frame. What’s important to understand when sniffing just because you have a high CRC rate in your sniffer window doesn't mean the actual client communication is experiencing the same. In fact while sniffing, if you experience a high CRC rate moving closer to the transmitting radios often solves the problem. It simply means your radio can't interpret the frame. If you want to see the actual client CRC rate, you would need to visit the actual radios.

2) When a client on channel transmits a frame ALL radios on the channel must synchronize to the preamble and demodulate the pending frame. The receiving radios peek at the mac address to see who the intended frame is for. If it doesn't match their mac address they look at the NAV timer to set their clocks and discards the frame. Idle clients are very busy processing frames! 

3) Noise calculations done by an 802.11 radio knows nothing about layer 1 spectrum. They determine the noise floor by various methods. Including retry rate, channel assessment, and energy detect.

4) Placing access points in a hallway, also called a hallway design is so 2007. Hallway designs contribute to excessive CCI (co channel interference). As client density increases and sensitive applications are added these designs fail miserably. Consider room placement during your survey. 

5) One way speech can be caused by a poor link budget. Imagine your on a call and you can hear them but they can't hear you ? If your access point transmit power is at 100mW and your client is at 20mW this imbalance can cause data retires. Your frames don’t have the punch to travel back to the access point. Always consider the lowest client in your wifi design and match their power on the access points.

6) Walls are your friends. Design using walls as attenuation points. Letting RF run amuck and leak into areas cause unnecessary CCI.

7) If you’re a player in WiFi, you better bring the tools and know how to use them. The three S’s. Spectrum, Sniffer and Survey tools. Know them. Know them very well.

8) Channel 165 / UNII2 - 2E  support. While most infrastructure devices support channel 165. Most clients do not. Allowing 165 in your design can cause outages. Same is true for UNII 2 and UNII2E. 

9) UNII2 - 2E DFS is real folks. It can disrupt communications. I’ve been the victim of weather radar and my connection dropped.  Pick your channels wisely my friend! 

10) The WiFi client is the biggest cowboy of them all! There is one thing which is consistent, it’s your wifi network. Your access points should be configured the same. They should be on the same code. You should expect a certainly level of performance from your infrastructure. Your clients on the other hand. What a hand bag of dysfunctional little peeps. Having an understanding of your clients is important. Know that clients aren't created equal. Like humans they all hear, talk and behave differently. 

Read this blog post in its entirety:

http://community.arubanetworks.com/t5/Technology-Blog/30-Random-Technical-Thoughts-by-a-WiFi-Engineer/ba-p/137033 

802.11 - Reason Codes and Status Codes

802.11 - Reason Codes and Status Codes 

The 802.11 standard section 8.4 comments on reason codes and status codes. I’ve used these myself when troubleshooting frame captures. These codes provide insight to Wi-Fi related problems like stations connecting and disconnecting. Lets dive in and see what the standard says about reason and status code fields. Then lets look at real world frame captures and see these codes at work.

802.11 Standard Overview

8.4.1.7 Reason Code field 

This Reason Code field is used to indicate the reason that an unsolicited notification management frame of type Disassociation, Deauthentication, DELTS, DELBA, DLS Teardown, or Mesh Peering Close was generated. It is contained in the Mesh Channel Switch Parameters element to indicate the reason for the channel switch. It is contained in the PERR element to indicate the reason for the path error. The length of the Reason Code field is 2 octets. The Reason Code field is illustrated in Figure 8-41. 

8.4.1.9 Status Code field 

The Status Code field is used in a response management frame to indicate the success or failure of a requested operation. The length of the Status Code field is 2 octets. The Status Code field is illustrated in Figure 8-43.

Reason Code Field 

When conducting frame captures you can find the reason code in some of the management frames like the response and disassociation frames. I like how the 802.11 standard comments:  “unsolicited notification”. 

It’s unsolicited information whereby radios can provide connection information. 

Example: Disassociation frame with reason code 1. This radio is informing the other radio it’s disassociating for unspecified reasons.

Read this blog post in its entirety here:

http://community.arubanetworks.com/t5/Technology-Blog/802-11-Reason-Codes-and-Status-Codes/ba-p/257893

"Wi-Fi doesn't stand for anything. It is not an acronym. There is no meaning.”

Let's start 2016 with a blog post that will surely get some of you thinking. As a professional who focuses on Wi-Fi communication I’m asked from time to time what does Wi-Fi mean?

The conversation usually goes something like this: What does Wi-Fi stand for?

Is Wi-Fi an acronym for something? Who came up with the term Wi-Fi? Who owns the name Wi-Fi? Is it WiFi or Wi-Fi?"

When I respond that Wi-Fi is a made up word I get the stare, usually followed by, "really?"

I think the biggest misunderstanding or assumption is many folks think Wi-Fi means “Wireless Fidelity”. This is almost always the response I get when I ask, "what do you think it means?"

Another point of interest is the proper term is Wi-Fi with the hyphen. While many of us, myself included, use the term WiFi that would not be the correct registered trademark. Wi-Fi is a registered trademark of the Wi-Fi Alliance. Here is a link to their brands.

http://www.wi-fi.org/who-we-are/our-brands

Read the entire blog post here: 
http://community.arubanetworks.com/t5/Technology-Blog/What-does-Wi-Fi-stand-for/ba-p/256914 

802.11 - TIM and DTIM Information Elements

In this blog post I investigate 802.11 TIM and DTIM.

Read the entire blog post here: 

http://community.arubanetworks.com/t5/Technology-Blog/802-11-TIM-and-DTIM-Information-Elements/ba-p/256997 

Traffic Indication Map (TIM) - 

After reviewing what the 802.11 standard says about TIM. Lets discuss in real world terms what a TIM is and how it works. 

You will specifically find TIM in a management frame called a beacon. A beacon is triggered by default on an access point every 102us. Think of a beacon as a network advertisement. The beacon advertises specific <BSSID> wireless network information such as supported PHY rates, security protocols, supported QoS/WMM, vendor specific information and much much more. Included in the beacon is a TIM information element. 

Example: BEACON WITH TIM

Delivery Traffic Indication Map (DTIM) - 

After reviewing what the 802.11 standard says about DTIM. Lets discuss in real world terms what a DTIM is and how it works. 

You will specifically find DTIM in a management frame called a beacon under the TIM information element. DTIM is to broadcast / multicast traffic as TIM is to unicast traffic.

Under the TIM you will see DTIM count and DTIM period. 

Example: DTIM COUNT / DTIM PERIOD 

 

Cisco 8.0 MR3 Beta Open to Public

Cisco announced public participation in 8.0 M3 beta testing. If you're interested visit Cisco Support Community. 

https://supportforums.cisco.com/discussion/12755726/80mr3-beta-availability 

8.0.122.x Available - 8.0MR3 Beta

 

We are pleased to announce the availability of 80MR3 beta (Upcoming 8.0.130.0) for general testing

If you are interested in participating on the beta program, please send email to wnbu-mrbeta@cisco.com with your CCO username, network size and planned usage scenario

Resolved List - 8.0.122.39

CSCtl96208	"capwap ap hostname" CLI returns "ERROR!!! Command is disabled."
CSCuf77488	wips alarm detection time stamp is ahead of AP clock
CSCui95938	fast Switching SSDi and IPAD Issue
CSCul42704	WIPS-Rogue APs are mistaken as infrastructure devices
CSCun06894	duplicate mac address issue of Ap Rcv image
CSCun52472	"show dtls connection" shows blank in AP Name column for Capwap_Data
CSCuo16301	HA:-Unable to pair up the active/Standby wlc due to config sync failure.
CSCuo19601	Flex AP in Standalone mode not triggering ap-primed-join-timeout timer
CSCuo48442	Stale old DTLS data_encryption session histories are left on WLC
CSCup13091	Local eap, local user, created for specific WLAN works for diferent wlan
CSCup64468	WLC device sends invalid format "#" in front of syslog message
CSCup68372	Stats are carried over when session timeout occurs
CSCup72502	5500 on 7.6 does not deauth client when Flex ACL is not present on AP
CSCup75446	Default interface takes precedence over foreign VLAN mapping with CWA
CSCup80403	Low iMac Tput -supported rate IE in association response has ZERO length
CSCuq50069	SHA1 key cipher not working between WLC 80 and MSE 80 CCO versions.
CSCuq68753	5500 anchor running 7.6.122.21 crashed on osapiBsnTimer
CSCuq73590	WLC adds incorrect class attribute in accounting stop
CSCuq88748	Rogue APs wrong classification from malicious to unclassified
CSCur13400	DHCP Option 82 and Sub Option 5 issue in WLC 8.0
CSCur25239	Controller crash on mping command over telnet/ssh
CSCur32475	NewMobility Web-Auth on MacFilter Failure always send client to web-auth
CSCur43124	WSSI module stops working after Upgrade from 7.4.121.0 to 7.6 MR3
CSCur48612	8.1 emWeb crash when adding devices to mDNS policy
CSCur48944	Problem in Client Stats Reports and Optimized Roaming
CSCur49165	WiSM2 system crash radiusTransportThread aaaRadiusAuth
CSCur57909	Client misses to override vlan after shifting wlan.
CSCur60218	New mobility web auth on mac filter failure Export Anchor request fails
CSCur74208	Name/OID: cLMobilityExtMgrAddress.0; Returning in IP in Reverse Order
CSCur80935	8.0.100.0:AAA overridden acl is not applied on Guest access controller
CSCur88307	AP name unknown in dissoc messages (Intermittent)
CSCur90555	T8.0 WLC keeps ghost client entry
CSCur91936	MDNS discovery issue with WLC 8.0.100
CSCur95365	Controller crashes when issuing command show ap config general
CSCus03406	Dataplane crash on 8500 WLC with 7.6
CSCus07013	Adding mac filter check when client is changing SSID for webauth
CSCus20991	Radius NAC Client auth issues for 7.6.130.0
CSCus30429	OEAP600 not giving ip on remote LAN port in 8.0
CSCus33759	Local Policies not working after OUI Update
CSCus51456	WLC - Radius multiple UDP source port support for radius protocol
CSCus53635	Add 802.11a Philipines country support for 1532I Aps joined to 5760.
CSCus61445	DNS ACL on wlc is not working - AP not Send DTLS to WLC
CSCus68340	Standby keeps auto rebooting and stays in "STANDBY COLD" state
CSCus73932	Multicast configuration issue on 8510 WLC OS 8.0.110.4
CSCus74299	New mobility:Client not deleted on 5508 when it roams at webauth state
CSCus74362	controller crashed with task radiusTransportThread
CSCus76833	CT5508 crashes at sisfSwitcherTask
CSCus77368	WLC: Crash ewaFormSubmit_cell_edit
CSCus80478	CAP1530 not forward, send packets to wired side after bootup
CSCus80685	AP sends few frames with previous security association's packet number
CSCus89468	Need to add AP802 to list of APs that support Flex+Bridge mode
CSCus91439	WLC - Memory leak - k_mib_cisco_lwapp_dot11_client.c
CSCut02524	default NAS-ID value at the AP-Groups should be empty or "none"
CSCut09821	Unused Data DTLS session is remained on WLC running 7.6.130.19
CSCut14459	session ID changes for an intercontroller client roam using EAPFAST
CSCut16170	Mobility tunnel down after switchover on 7.6
CSCut25670	CSCuwAP: %DTLS-5-SEND_ALERT: Send FATAL; join failure loop
CSCut27598	Client unable to get IP when switching wlan on New mobility.
CSCut31468	local profile showing wrong stats under Manufacturer Stats
CSCut39118	WLC 8510 Failure to collect feature MobilityExtGroupMember on PI 2.2
CSCut42926	WLC crash on SNMPtask after doing config audit from PI
CSCut43770	PMIPv6 Client Traffic is Sent to the Wrong LMA
CSCut62319	Broadcast Key Rotation won't occur after MAC Filtering enabled
CSCut70403	Jian WLC crashed with task name 'HAConfigSyncTask'
CSCut74263	MAG on AP:AP does not clear bindings after session/user timeout & deauth
CSCut76481	WLC sends 1499 bytes MTU switchover
CSCut87326	WLC generates SNMP traps to PI 2.2 for AIR-3702 PoE+ getting low power
CSCut93569	MSE - NMSP inactive with WLC
CSCut93712	AP not send RM IE for 11k in association response; no 11k for iOS > 8.1
CSCut97683	WLC crash on spamApTask2 8.0.110.0
CSCut99150	2702 AP requesting as a Type 1 power device instead of Type 2
CSCuu06047	Packet drops on 2702 AP in flex local auth/local switch mode
CSCuu07700	EAP Packet does not get encrypted in Re-auth request from client
CSCuu08752	SXP Crash when running Trust Sec clients on Talwar
CSCuu20097	Token Bucket leak when QoS Roles setup and when working with WebAuth
CSCuu37437	WLC8510 crashing while NMSP polling in progress.
CSCuu45186	dot11 arp-cache does not works well
CSCuu47016	Cisco Application Visibility and Control UDP Vulnerability
CSCuu49291	7925 decrypt errors with AP1131 running 8.0 code
CSCuu54100	Switching between SSIDs fails with FAST SSID enabled on PMIPv6 WLANs
CSCuu59589	False positive AP sourced AP impersonation on corrupted beacon
CSCuu66675	Lock crash on radiusTransportThread during CMCC external auth
CSCuu68490	duplicate radius-acct update message sent while roaming
CSCuu72366	Silent Crash 8.0.110.11 mmListen process
CSCuu77304	Clients deauthenticated from OEAP 600 LAN ports
CSCuu82416	Evaluation of wlc for OpenSSL June 2015
CSCuu83748	WLC sends bsnRogueAPRemoved Trap when notify configured none
CSCuu83941	8510: Error enabling global multicast with capwap mode unicast
CSCuu91001	Netflow record sent without client IP address
CSCuu93296	EAP-TLS loosing device certificate in standalone mode after reboot
CSCuu98792	AP 1570, antenna enable config is lost on reboot
CSCuu99344	WLC crash - DHCP packet content while on new mobility
CSCuv00107	PMIPv6 Client MAC Address shows up on the mac address table of Switch
CSCuv00598	Optimized Roaming per WLAN feature
CSCuv04474	Sanity: AP1700 crashed during multicast client traffic(cont.CSCuu89311)
CSCuv09655	Anchor crash on 8.0.110.14 New Mobility apf_msDeleteTblEntry
CSCuv13731	3702 AP sends burst traffic - AMPU/MSDU/Off-channel/RRM disabled
CSCuv22052	Link local multicast control traffic sent by APs, IGMP Snooping Enabled
CSCuv27320	Wired clients in 702w AP leaking traffic across ports/vlans
CSCuv28555	3702 - Voice Queue stuck, with no new clients able to associate.
CSCuv31162	WLC 5500 Crashes continously in HA Setup@task: apfRogueTask_2 and 3
CSCuv34277	Wireless Client not able to get IP address on 3650 MA from 5508 anchor
CSCuv34946	EOGRE and PMIPv6 client fails to move to Run state
CSCuv36505	8.0 WLC messages flooding cli after debug client
CSCuv37613	Apple devices failing 802.11r FT roam
CSCuv51521	Active WLC should send GARPs when HA Re-Paring after Active-Active state
CSCuv53952	SSID still broadcasted by the AP after the wlan is deleted from wlc
CSCuv61271	Window DHCP BAD_ADDRESS for Access Points
CSCuv67144	Need to re-evaludate Algeria if in -E or -I
CSCuv69967	OEAP600 wired 802.1x remote LAN forward traffic in 802.1x Required State
CSCuv82110	vWLC: Decrypt errors occurred for client using WPA2 key on 802.11a intf
CSCuv85747	Mobility Member entries going stale
CSCuv86494	WLC clears AP MAC before deleting client, sends netflow with Zero AP MAC
CSCuv87657	8.0/8.1 WLC's fail to send FRAMED-IP attribute to AAA server
CSCuv87839	Wired clients in 702w AP getting put in mgmt vlan
CSCuv90333	Pineridge - afpmsConntask flood when running client console debug
CSCuv96333	readonly user able to change "Telnet Capability" setting
CSCuv97793	WiSM2 crash AP_DB_CREATE_ERR Message queue MFP-Q is nearing full
CSCuw03414	WLC crash: "Software Failed while accessing the data"
CSCuw06127	Silent crash 8.0.120 due to memory leak in CDP Main
CSCuw06153	unauthorized configuration change for web management
CSCuw10610	Non authenticated HTTP page allows to logout any connected client
CSCuw12544	Rate-limiting is causing 500ms gap of traffic when roaming
CSCuw13264	702w missing interface information on controller after HA failover
CSCuw15008	Mobility Task Hogs CPU - Reaper Reset in SpamApTask
CSCuw24476	Increased Ping latency & Reduced traffic on 8510 with QOS rate limiting
CSCuw26377	Crash due to invalid form field validation on switch_cfg_rw.html
CSCuw29419	Cisco Wireless LAN Controller Radius Packet of Disconnect Vulnerability
CSCuw29564	APs show 0 neighbors on 5GHz band and client 802.11 packets are ignored
CSCuw35341	IP address lost on AAA override+muiltiple subnetworks per vlan+DHCP req
CSCuw35349	DHCP registration failing when mask from WLC intf does not match client received mask
CSCuw44480	802.11r client fails auth if self reset before user idle timeout expires
CSCuw50324	Crash on high CPU for bonjour
CSCuw57588	C3600 AP crash on am_xml_GetChildCount
CSCuw87468	Rogue containment not working on 8.0.120.0 for AP3700 with WSM module
CSCuw89581	WLC System Crash on apfReceiveTask
CSCuw90625	Rogue rules not applied correctly after upgrade to 7.6.130.30
CSCuw91763	Feature "AES Key Wrap" does not work
CSCux03108	8510 crashed on Task Name:portalMsgTask.
CSCux18259	PI 3.0 - Sync Issue on Flexconnect Native VLAN Configuration
CSCux41354	Evaluation of wlc for OpenSSL December 2015 vulnerabilities
CSCux47470	8.0.110.14 controller crash at openssl_cert_hash_algo_check_callback
CSCux22935	HA+802.11r:Post SSO FT PSK/EAP Apple clients fails to connect.
CSCux22620	8510 WLC crash in radiusTransportThread system task
CSCut23325	1700 AP not encrypting icmp and arp sent from the client over the air
CSCus92667	GET on Ap groups Table after set - response missing
CSCus39396	8.0.100.0 QoS Bronze Profile not marking traffic to AF11 on Flex
CSCut06502	WLC crash due to task name RRM-CLNT-5_0
CSCut48172	LSC AP provisioning happening after MAP is disconnected for long time
CSCtu45614	Spectrum Management Bit Should be set to 1 all the time

802.11 - Action Frames

The 802.11 standard section 8.5 comments on action frames. Action frames are interesting. Action frames can be triggered by access points or client stations. The action frame provides information and direction as in what to do. The 802.11 standard comments about action frames in 17 different sections of subsection 8.5. While many of these aren't used by vendors today some important ones are. Lets review some comments about action frames and then head to the frame captures.

<continue.....>

Example: DFS event is under way. The access point is sending an action frame to the cell to announce a channel change.

Category - 0 Spectrum Management
Action - 4 Channel Switch Announcement
Element - New Channel 64

Example: In this example TSPEC. Where a client is requesting a TS <traffic stream>.

Category - 17 WNM
Action - 0 ADDTS Reuquest
Status Code: 0 Admission Accepted

 * Note I believe Omnipeek is decoding this wrong. I believe the category code should read WNM.

Click here for the entire blog post:
http://community.arubanetworks.com/t5/Technology-Blog/802-11-Action-Frames/ba-p/256811 

Field Notice: FN - 64003 - AIR-ANT2568VG-N - Potential Moisture Intrusion to Radome - Replace Antenna

Important Cisco Field Notice for antenna AIR-ANT2568VG-N

http://www.cisco.com/c/en/us/support/docs/field-notices/640/fn64003.html?emailclick=CNSemail

More Power More Problems! When Excessive RF Power Degrades your WiFi Performance!

In the wireless world we often think more power is good. The louder the signal surely higher the performance gain. I’m sorry to say that’s not  true in most cases. RF power is like a delicate flower and should be treated with respect. Simply choosing a higher power output and not properly tuning your radios could cause you more pain than you really know. In this quick blog post, I share a pair of static bridges being bench tested 70 feet apart. The only difference in configuration is simply changing the RF power. While I only share the capacity values, the throughput values have been excluded to keep the focus on power.

Example #1 - (HOTTEST)

In this example we pump up the power @ 30 dBm.

(1) Link @ -17 dBm
(2) Modulation at 16 / 64 QAM
(3) TX Power 30 dBm
(4) Capacity Link TX 205, RX 200

Example #2 - (HOT)

In this example we power down to @ 24 dBm.

(1) Link @ -22 dBm
(2) Modulation at 256 / 256 QAM
(3) TX Power 24 dBm
(4) Capacity Link TX 396, RX 391

Example #3 - (PEACHY)

In this example we power down to @ 18 dBm.

(1) Link @ -27 dBm
(2) Modulation at 1024 / 1024 QAM
(3) TX Power 18 dBm
(4) Capacity Link TX 482, RX 469

Modulate Gain: 16 vs 1024 and 64 vs 1024
Capacity Link Gain: TX 205 vs 481, RX 200 vs 469

Why excessive power gain is bad is because it increases noise and distortion at the receivers radio. In Example #1, both radios can hear each other at -17 dBm! Think of it this way, imagine having someone in your ear with a megaphone yelling today’s lunch specials at you. You can’t hear so well, can you ? Take away the megaphone and step back a few feet and all is peachy.

My quick less-techy blog post for today! 

Cheers!

Which antenna gets deactivated when you provide less than full power to a Cisco 3700 ?

A question was asked on Cisco Support Community (CSC) enquiring about what antenna is deactivated when a Cisco 3700 access point doesn't receive a full 16.1 Watts. 

We have purchased 3702e and some of these access points can only get PoE (802.3af). Which antenna will be activated in this case?

802.3at                 4x4:3 on 2.4/5 GHz                         16,1W
802.3af                 3x3:3 on 2.4/5 GHz                         15,4W

Thats a good question and it had me thinking. So I tapped my Cisco CSE, Carlos. BTW Carlos is one of the best CSE’s you’ll find. I’m very fortunate to have him as our CSE. The guy has memory recall with such precision it’s scary. Not to mention he is a CCIE R/S and W. 

When an access point isn't provided full power it can deactivate some combination of radio chains and spatial streams. Manufactures can dial back the access points performance while still providing reliable WiFi communications. This allows flexibility with power at the switch power level (PoE).

We’ll focus on the Cisco 3700. The data sheet shows 802.3at and 802.3af power combinations. Less power, less chains and streams. More power, more chains and streams.

EXAMPLES

From a Cisco 3700 access point do:  show controllers dot11Radio X.

802.3at POWER PROVIDED TO CISCO 3700

In this example you will see the access point is fully powered. We can tell this because of the the number of antennas used for RX and TX. A,B,C and D.

Antenna:                        Rx[a b c d ]
                                    Tx[a b c d  ofdm all]

802.3af POWER PROVIDED TO CISCO 3700

In this example you will see the access point is not fully powered. The access point was provided .af power. We can tell this because of the the number of antennas used for RX and TX. A,B, and C and the mention “Radio on Low Power Mode due to PoE, restricted to 3 antennas”

Antenna:                        Rx[a b c ]
                                     Tx[a b c  ofdm all]

A,B,C, and D

You might be wondering which antenna port is D. On a Cisco 3700E look closely at the antenna bulk head. Each one is identified with A,B,C, and D. In this case the D antenna, it is located in the lower left of the 3700 access point. 

Field Notice: FN - 63697 - Protective Boot on Certain Network Cables Might Push the Mode Button and Cause an Unexpected Reset on the 48-Port Models of Cisco Catalyst 3650 and 3850 Series Switches

Sorry, but I had to really LOL when I read this field notice! 

Picture is worth a thousand words! 

http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html 

Client Debug Macro Change - Cisco code: 7.6.130.0 - 8.0.110.0

A quick blog post on an observation I made while debugging in the lab. The command debug client enables a set variable of commands which enable muliple debugs. You can see what these commands are with the “show debug” command. 

Notice the change in the commands enabled between 7.6 and 8.0. 

Revolution WiFi Capacity Planner

Did you miss Andrew von Nagy's Capacity Planner webinar ? No worries because the links are below. It's one of those sessions you will want to watch a few times and let it soak in. This session will be a guaranteed "classic". A staple of sorts for WiFi engineers to use in the future. 

What makes this even more special. Andy didn't get paid to create this calculator. It's his way of giving back to the community. Andy is a true master at his craft. Really honored to call him a friend. Someone who always answers your emails, takes your calls and willing to explain a subject 10 different ways till you understand it.

If you're new to WiFi or a veteran this session is packed with nuggets! 

Revolution WiFi Capacity Planner 

http://www.revolutionwifi.net/capacity-planner?mkt_tok=3RkMMJWWfF9wsRonsq3Neu%2FhmjTEU5z16O0rXaC2hokz2EFye%2BLIHETpodcMTsJqMbrYDBceEJhqyQJxPr3FLNkNyMBvRhfnDw%3D%3D 

Revolution WiFi Recorded Webinar Capacity Planner 

http://page.arubanetworks.com/05.19_Revolution_Wi-Fi_On_Demand.html?mkt_tok=3RkMMJWWfF9wsRonsq3Neu%2FhmjTEU5z16O0rXaC2hokz2EFye%2BLIHETpodcMTsJqMbrYDBceEJhqyQJxPr3FLNkNyMBvRhfnDw%3D%3D 

Cheers!

Top 10 Sessions From Interop Las Vegas 2015

Interop Las Vegas 2015 was a blast! Few conferences bring together a rich mix of vendors, products, solutions and attendees all in one place. I was particularly interested in Cisco's Hyperlocation, which just so happen to win Best of Interop Award - 2015 Mobility. Interop was a gathering of old friends and meeting new ones. I thought the mobility track was exceptional this year. 

Cisco Hyperlocation:
http://www.interop.com/lasvegas/special-events/best-of-interop-awards.php?itc=we_ilv_le_ilv_drp_text 

I was also a panel guest at Cisco's Mobility lunch where WiFi Mobility, 802.11ac and our AWO (All Wireless Office) was topic of discussion. It was 60 minutes of great discussion and guest interaction. I would like to thank Cisco's Bill Rubino for the invite. 

I spoke at my own session "Designing Todays WiFi Network for Tomorrow's Applications". I always enjoy sharing my real world hands on experience with others. WiFi is still black magic to many IT folks in the industry. The goal in my session, take 2 things away that you didn't know before. I think the attendees agreed. My session made Interop's Top 10 Sessions and ranked #6 in the standings as voted by attendees. I would like to thank Andrew Murray for the invite and having me back at Interop. 

Interop Top 10

http://www.informationweek.com/interop/top-10-sessions-from-interop-las-vegas/d/d-id/1320459? 

In closing three articles were published from my Interop session. 

Remember The Restroom When Deploying Wireless

http://www.informationweek.com/mobile/mobile-devices/remember-the-restroom-when-deploying-wireless/d/d-id/1320230 

What happens if you remove an acceptable use policy from guest Wi-Fi?

http://searchnetworking.techtarget.com/news/4500245600/What-happens-if-you-remove-an-acceptable-use-policy-from-guest-Wi-Fi 

Diversity of connected devices in hospitals poses unique challenge for going fully wireless

http://www.fiercemobileit.com/story/diversity-connected-devices-hospitals-poses-unique-challenge-going-fully-wi/2015-05-04   

Cheers!

TAC Recommended AireOS 7.6 and 8.0 - 2Q CY15

TAC code recommendations for AireOS 7.6 and 8.0 customers. 

Folks on 7.6 and 8.0 or who might be thinking about going to these versions Cisco is recommending the following releases:

7.6 - 7.6.130.24

8.0 - 8.0.110.9 

https://supportforums.cisco.com/document/12481821/tac-recommended-aireos-76-and-80-2q-cy15 

NOTE: Above links to release notes for 7.6.130.24 and 8.0.110.9

Field Notice: FN - 63916 AireOS 8.0.100.0 or Cisco IOS-XE 3.6.0E - AP Unable to Join WLC or AP Stuck in Downloading State - Software Update Required

NOTICE: 

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Date	Comment
1.0	12-JAN-2015	Initial Public Release

Products Affected

Products Affected
Cisco Aironet 1530 Series
Cisco Aironet 1550 Series
Cisco Aironet 1600 Series
Cisco Aironet 1700 Series
Cisco Aironet 2600 Series
Cisco Aironet 2700 Series
Cisco Aironet 3500 Series
Cisco Aironet 3600 Series
Cisco Aironet 3700 Series

Problem Description

Some Wireless Access Points (APs) manufactured between August 2014 and October 2014 might have an incorrectly programmed SHA-2 certificate.

The affected product families are:

• Cisco Aironet 1530 Series
• Cisco Aironet 1550 Series
• Cisco Aironet 1600 Series
• Cisco Aironet 1700 Series
• Cisco Aironet 2600 Series
• Cisco Aironet 2700 Series
• Cisco Aironet 3500 Series
• Cisco Aironet 3600 Series
• Cisco Aironet 3700 Series

Issue 1

After you upgrade a Wireless LAN Controller (WLC) to software version 8.0.100.0 or 3.6.0E

AND

after the Wireless APs download the new software version, any Wireless AP with an incorrectly programmed SHA-2 certificate disconnects from the WLC and is not able to rejoin the WLC if the WLC has a SHA-2 certificate.

Issue 2

Any new Wireless AP with software version 8.0.100.0 and with an incorrectly programmed SHA-2 certificate fails to validate the image downloaded from the WLC. The result is that the AP is unable to establish a connection to a WLC with version 8.0.100.0 software.

If the AP has an incorrectly programmed SHA-2 certificate and the WLC has version 8.0.100.0 or 3.6.0E, the likelihood of this issue being observed is 100%.

Background

Between August and October 2014, a manufacturing change was added to support SHA-2 certificates. In the certificate chain transition, some APs were manufactured with incorrect certificate information. Prior to this change, the APs only had a SHA-1 device ID certificate. After the change the APs had both SHA-1 and SHA-2, but the SHA-2 was incorrectly programmed on the affected units.

The available fixed code ensures that the APs continue to function as APs that were manufactured prior to August 2014.

The affected APs are fully functional and equivalent to APs manufactured prior to August 2014.

In the future, Cisco will provide support for SHA-2 authentication between APs and more recently manufactured WLCs.

Problem Symptoms

New Aironet APs with factory installed recovery Cisco IOS^® are able to join the controller that runs software version 8.0.100.0 or 3.6.0E and download version 15.3(3)JA or 15.3(3)JN IOS. However after the AP reload, the APs are unable to join the controller. On the AP, logs similar to these are seen:

*Oct 16 12:39:06.231: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Oct 16 13:14:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: ***.***.***.*** peer_port: 5246Peer certificate verification failed FFFFFFFF

*Oct 16 13:14:56.127: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
*Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to ***.***.***.***:5246
*Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to ***.***.***.***:5246

Another symptom of this issue is that the AP might be able to join the software version 8.0.100.0 controller, download a new Cisco IOS code, and boot up and join the controller correctly; however when it goes to upgrade to the newer 8.x code it gets stuck in a loop and fails the download.

*Nov 11 10:13:53.003: Currently running a Release Image
*Nov 11 10:13:53.027: Using SHA-2 signed certificate for image signing validation.
*Nov 11 10:13:53.091: Image signing certificate validation failed (FFFFFFFF).
*Nov 11 10:13:53.091: Failed to validate signature
*Nov 11 10:13:53.091: Digital Signature Failed Validation (flash:/update/ap3g2-k9w8-mx.v153_80mr.201410311616/final_hash)
*Nov 11 10:13:53.091: AP image integrity check FAILED Aborting Image Download
Download image failed, notify controller!!! From:8.0.100.0 to 8.0.102.34, FailureCode:3 archive download: takes 339 seconds
*Nov 11 10:14:02.399: capwap_image_proc: problem extracting tar file

Workaround/Solution

AireOS

In order to avoid this issue, if the WLC runs software version 7.6 or earlier and you have APs affected by this issue, do not upgrade to version 8.0.100.x train. Wait for the next Cisco Connection Online (CCO) release.

Workaround for AireOS

If the WLC has been upgraded to version 8.0.100.x and the APs are supported in AireOS 7.6, downgrade to this version.

Solution for AireOS

If the WLC has software version 7.6 or earlier, upgrade the WLC to version 8.0.110.0.

If the WLC has software version 8.0.100.x, follow these steps:

1. Upgrade the WLC to software version 8.0.104.0:
   • All controllers
   • WISM2
2. Allow all APs to join the WLC and upgrade to software version 8.0.104.0.
3. Upgrade the WLC to software version 8.0.110.0.
   Note: Step 2 is required to push the 8.0.104.0 special software version onto the APs in order to allow all future upgrades.

Cisco IOS-XE

In order to avoid this issue, if the WLC has software version 3.3.x or earlier and you have APs affected by this issue, do not upgrade to version 3.6.0E.

Workaround for Cisco IOS-XE

If the WLC has been upgraded to version 3.6.0E and APs are supported in Cisco IOS-XE Version 3.3.x, downgrade to this version.

Solution for Cisco IOS-XE

If the WLC has software version 3.6.0E, follow these steps:

1. Upgrade to version 3.6.1 or 3.7.0 or later.
2. Enter the wireless security certificate force-sha1-cert command from the prompt.

CDETS

To follow the bug ID link below and see detailed bug information, you must be a registered customer and you must be logged in.

CDETS	Description
CSCur43050 (registered customers only)	APs mfg in Aug./Sept./Oct. 2014 unable to join an AireOS controller
CSCur50946 (registered customers only)	APs mfg in Aug./Sept./Oct. 2014 unable to join an IOS-XE controller

How To Identify Hardware Levels

From the AP CLI, enter the show version command and look for the "Top Assembly Serial Number". An example of a Top Assembly Serial Number is FTX1613GJGA.

If the AP is joined to an AireOS controller:

• From the CLI, enter the  show ap inventory APNAME command.
• From the GUI, select  Wireless > All APs > APNAME > Inventory in order to view the serial number.

If the AP is joined to a Cisco IOS-XE controller:

• From the controller CLI , enter the show ap name APNAME inventory command and look for the "Cisco AP" serial number.
• From the GUI, select Configuration > Wireless > Access Points > All APs > APNAME > Inventory in order to view the serial number.

Alternately, the serial number can be found on the back/bottom of the AP:

Confirm that your serial number is affected with the Serial Number Validation Tool.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods: 

• Open a service request on Cisco.com
• By email
• By telephone

Cisco Access Point Models Not Supported On 8.1 Code

If you're like me you may have hundreds or even thousands of Cisco 1131, 1242 and 1250 access points deployed in your wireless network today. 

Take special care and attention to the information below. A number of legacy access point models will no longer be supported past 8.0 code. 

This is reminiscent of 1000 series access points. I can recall the horror stories, people upgrading to 5.0 only to realize that the 1000 series would not join the WLC. #DontBeThatGuy!

Ask your Cisco rep about buy back programs and bundle purchases; buy X and get 5-10 free access points!