Tcpdump with Snat

Hi Mario,

A nice way to filter your tcpdumps on an F5 is: tcpdump -vnni 0.0:nnn -s0 '(host Client_IP and host Virtual_IP) or (host SNAT_IP and host Pool_member1) or (host SNAT_IP and host Pool_member2)' -w /var/tmp/capture.pcap

The "-v" in front of "nni" will enable you to see how many packets are being captured every second, in real-time, even when the output is going to a file. Also, this way of filtering restricts your SNAT communication only with the pool members belonging to the Virtual Server that you're trying to troubleshoot, so you should have a reasonable amount of traffic to work with.

you can always filter on either the client or server IP. so if you want to tcpdump traffic "behind" the F5 and can't use the client IP due to SNAT you can filter on the Node IP address(es). this still will get you some more traffic then when you could isolate the client, but better then with all the SNATs.

i understand we have no choice unless you configure specific snat ip for the client (i.e. using irule).

normally i use client ip (or virtual server ip) or pool member ip and not non-floating selfip as a tcpdump filter.

The BIG-IP will also use the client source port as the source port for SNAT connections if the port is available. This can be used to correlate client->BIG-IP and SNAT->pool member captures.

Even better, starting in v11.2, there is an awesome undocumented feature that can help. It's a new "-p" flag to dump on "peer" flows.

Instructions:

"

Log on to CLI via SSH and start packet capture:

tcpdump -ni 0.0:nnnp -s 0 host client-ip -w /var/tmp/traffic_from_client.pcap

Replacing client-ip with IP of failing client.

ex. from bash

tcpdump -ni 0.0:nnnp -s 0 host 10.100.100.102 -w /var/tmp/traffic_from_client.pcap

Note, with the “-p” flag, you can narrow down by all traffic to that VIP as well if you put

tcpdump -ni 0.0:nnnp -s 0 host and port -w /var/tmp/traffic_to_vip.pcap

Ex.

tcpdump -ni 0.0:nnnp -s 0 host 1.1.1.1 and port 443 -w /var/tmp/traffic_to_vip.pcap

Now reproduce Issue with Failing Client:

After reproduction completes, type Ctl-C to stop the packet capture.

Note: Above capture takes advantage of new tcpdump flag "-p" that captures peer sides of the connection which is useful when traffic is snatted on the serverside. It requires a little workaround to reset/clear the filter internally ( running a different capture without the -p flag that won't match original filter )

tcpdump -ni 0.0:nnn -s 0 port 1

Type Ctl -C to stop the capture immediately after it started.

"

Voila! No more capturing an insane amount of traffic for that needle in a haystack on the serverside!

I wrote this (ugly way how we had to do it before):

SOL11555: Gathering data in preparation for a traffic impacting change to the BIG-IP system

http://support.f5.com/kb/en-us/solutions/public/11000/500/sol11555.html?sr=29863417

I'll send a solution update request to add this to the mix.

Even better, starting in v11.2, there is an awesome undocumented feature that can help. It's a new "-p" flag to dump on "peer" flows.

Instructions:

"

Log on to CLI via SSH and start packet capture:

tcpdump -ni 0.0:nnnp -s 0 host client-ip -w /var/tmp/traffic_from_client.pcap

Replacing client-ip with IP of failing client.

ex. from bash

tcpdump -ni 0.0:nnnp -s 0 host 10.100.100.102 -w /var/tmp/traffic_from_client.pcap

Note, with the “-p” flag, you can narrow down by all traffic to that VIP as well if you put

tcpdump -ni 0.0:nnnp -s 0 host and port -w /var/tmp/traffic_to_vip.pcap

Ex.

tcpdump -ni 0.0:nnnp -s 0 host 1.1.1.1 and port 443 -w /var/tmp/traffic_to_vip.pcap

Now reproduce Issue with Failing Client:

After reproduction completes, type Ctl-C to stop the packet capture.

Note: Above capture takes advantage of new tcpdump flag "-p" that captures peer sides of the connection which is useful when traffic is snatted on the serverside. It requires a little workaround to reset/clear the filter internally ( running a different capture without the -p flag that won't match original filter )

tcpdump -ni 0.0:nnn -s 0 port 1

Type Ctl -C to stop the capture immediately after it started.

"

Voila! No more capturing an insane amount of traffic for that needle in a haystack on the serverside!

I wrote this (ugly way how we had to do it before):

SOL11555: Gathering data in preparation for a traffic impacting change to the BIG-IP system

http://support.f5.com/kb/en-us/solutions/public/11000/500/sol11555.html?sr=29863417

I'll send a solution update request to add this to the mix.

tcpdump -vvnni 0.0:nnnp -s0 host ip_address -w /path/file.pcap

This command captures end-to-end packets that can be traced using the F5 plugin for Wireshark.

This is also included in a script I published today.