Normally when you troubleshoot roaming issues, you have to go on-site with a multi-channel packet-capture device (for every channel you want to capture on, you need an individual NIC, since you can only capture on one concurrent channel per NIC), get a hold on the mobile device with said issues, and then walk with both of them around the area where they reported the issue and do your captures.
Wouldn’t it be great to be able to just have these possibilities at the end of your fingertips? Sitting at your desk, logging in to your controller, and just start capturing all those frames needed remotely? You don’t have to travel, and you can just wait for the customer to give you a call saying “it just happened again!”, and all the information you need for trobleshooting are already there.
WiNG allows you to do this with its powerful “remote-debug live-pktcap”-toolkit.
It lets you start a distributed packet-capture from a controller, pushes the command with all its parameters and filters out to the defined APs – either to a single one, a set of defined multiple ones , or even a whole RF-Domain – and merges the captured frames from all APs back centrally into a single capture file.
The command I usually use to troubleshoot authentication and roaming issues is as followed:
remote-debug live-pktcap rf-domain my-rf-domain write tzsp 192.168.1.101 radio all count 10000 filter ether host 11-22-33-44-55-66 and ((ether proto 34958) or dot11 mgmt and (dot11 stype 0 or dot11 stype 1 or dot11 stype 2 or dot11 stype 3 or dot11 stype 4 or dot11 stype 5 or dot11 stype 10 or dot11 stype 11 or dot11 stype 12))Breakdown of Parameters
- rf-domain my-rf-domain: capture on every AP that is part of the RF-Domain my-rf-domain
- write tzsp 192.168.1.101: write the capture results via TZSP (TaZmen Sniffer Protocol) in realtime to the host 192.168.1.101 *
- radio all: capture on all wireless radios (2.4GHz & 5GHz)
- count 10000: capture up to 10’000 frames per AP **
- filter: use the following capture filters ***
- ether host 11-22-33-44-55-66: frame has to come from or to the client MAC address 11-22-33-44-55-66 (watch out, this is case sensitive and needs the hyphen) and is either a….
- ether proto 34958: …IEEE 802.1X / EAP frame or a…
- dot11 mgmt: …management frame with the subtype…:
- dot11 stype 0: …Association Request or…
- dot11 stype 1: …Association Response or…
- dot11 stype 2: …Reassociation Request or…
- dot11 stype 3: …Reassociation Response or…
- dot11 stype 4: …Probe Request or…
- dot11 stype 5: …Probe Response or…
- dot11 stype 10: …Disassociation or…
- dot11 stype 11: …Authentication or…
- dot11 stype 12: …Deauthentication
*to be able to stream your captures directly to a host via TZSP, you need to make sure that Port 37008 UDP is open between your controller and the host you’re writing the frames to.
To make the capture as lightweight as possible, I recommend starting Wireshark with the capture filter port 37008, so you only capture the TZSP frames:
Please keep in mind that when you use TZSP, the 802.11 frame is encapsulated and sent to your computer via UDP, so there is a IP- and UDP-Header. In addition, instead of the usual Radiotap-Header, there is a TZSP header with certain Wi-Fi related information, so you might need to create an extra Wireshark profile to have your views that you’re comfortable with:
**per default, every AP stops capturing frames after the 50th frame. To be able to capture over a longer period of time, I like to use a higher count, and then just manually end the capture when I have the frames I was looking for. To end a capture, simply type show remote-debug on your controller to find out the name of your debug-session, and then end the appropriate one with remote-debug end-session %name-of-session%
***As in every coding language or even in Wireshark itself: the correct use of brackets is imperative when you mix different filter options with a combination of and/or to get the results you’re looking for. This goes for the capture filters too (see example-command above)!
Alterations
- If you do not want to capture on the whole RF-Domain, and want to limit the capture to one or multiple APs, just use “hosts ap1 ap2 ap3” instead of “rf-domain %rf-domain%“
- If you can’t stream TZSP (UDP Port 37008) frames directly to your computer, you can write a PCAP-file to a FTP or TFTP server (write tftp://10.10.10.10/test.pcap)
- If you want to capture every authentication and roam of every client, you can just erase the “ether host 11-22-33-44-55-66” filter. But watch out – there will be a lot of Probe Requests and Probe Responses, so maybe consider removing the filters “dot11 stype 4 or dot11 stype 5” too.
Considerations
Despite the awesome troubleshooting capabilities WiNG offers you with this command, there are a few caveats that you need to keep in mind. When capturing from the distribution system, you capture the situation as seen from the APs site of view. Spotting for example excessive retransmissions from Client -> AP is harder, and you might still have to consider going on-site if the PCAPs you get do not show you the information you were looking for.
In addition, some WiNG 7.x versions have issues with packet-capturing, which might or might not have been solved when you read this post (the Release Notes for WiNG 7.3.1.3-004R mention some fixes for issues with packet captures). I did my tests in WiNG 5.9.4.1-004R where everything worked perfectly.
Examples
Enough with the foreplay – now I’d like to show you a few examples I’ve captured in my lab environment. Please note that in order to keep the screenshots cleaner, I have excluded the Probe Requests and Probe Responses.
In example 1, you can see the Authentication/Association process (purple), an 802.1X/EAP authentication (grey) and the 4 way handshake (yellow) during the initial authentication. After that, every roam is just an exchange of 4 frames (with 802.11r, the 4 way handshake is wrapped in the authentication- and reassociation frames):
In example 2, you can see the Authentication/Association process (purple), an 802.1X/EAP authentication (grey) and the 4 way handshake (yellow) during the initial authentication. After that, every roam consists of the authentication- and reassociation frames, as well as the 4 way handshake Thanks to OKC and PMK-Caching, no 802.1X/EAP frames are needed during roaming:
In example 3, you can see the Authentication/Association process (purple), an 802.1X/EAP authentication (grey) and the 4 way handshake (yellow) during the initial authentication. After that, every roam has to go through the whole authentication process again due to the lack of any roaming optimization being present:
