This is a small guide on how to collect PCAPs for network traffic from/to a virtual machine running on VMware ESXi. Scripts have been provided by VMware GSS and can be found here and used at your own risk.
(I get it, who would want to use Python scripts downloaded from a random website and run them on your ESXi, feel free to look into the code after download, or else request them from VMware)
The scripts can help to collect on multiple PortIDs, divide the PCAPs into smaller files, and rotate the logs for each x minutes and it can help to monitor the capture that is ongoing.
If you don’t want to use the scripts, then there is also a more manual way to run the collection here.
Finding the VM port IDs
Connect to the ESXi host with SSH and list the running VMs.
[root@dc1esxedge1-02:~] esxcli network vm list
World ID Name Num Ports Networks
-------- ---------------- --------- --------
68018705 nsxedgenode1 3 dvportgroup-914135, dvportgroup-914135
Find the VM that you want to collect from, let’s take WorldID 68018705 and check the switch port IDs for it
[root@esxedge1-02:~] esxcli network vm port list -w 68018092
Port ID: 67108942
vSwitch: DSVNSX
Portgroup: dvportgroup-914135
DVPort ID: 109
MAC Address: 00:50:56:ab:82:59
IP Address: 0.0.0.0
Team Uplink: vmnic2
Uplink Port ID: 2214592556
Active Filters:
Port ID: 67108943
vSwitch: DSVNSX
Portgroup: dvportgroup-914135
DVPort ID: 108
MAC Address: 00:50:56:ab:06:b0
IP Address: 0.0.0.0
Team Uplink: vmnic2
Uplink Port ID: 2214592556
Active Filters:
Port ID: 67108944
vSwitch: DSVNSX
Portgroup: dvportgroup-914134
DVPort ID: 98
MAC Address: 00:50:56:ab:93:ab
IP Address: 0.0.0.0
Team Uplink: vmnic1
Uplink Port ID: 2214592558
Active Filters:
In this case, we note down the following port IDs and save them for later
67108942
67108943
67108945
67108946
Start the collection with scripts
Upload the Python script to the /tmp/ on the ESXi host. You also need a folder on the local datastore of the ESXi host where the pcap logs can be stored.
Prepare and run the first script where we provide the PortIDs and where to store the PCAPs
[root@esxedge1-02:~] python /tmp/prepare_pktcap.py -p 67108942 -p 67108943 -p 67108945 -p67108946 -u -d /vmfs/volu
mes/prd.dc1esxedge1-02/capture/42434546 -o /tmp/rotating_cap.sh -G 15m -r 3600
The current vmci heap is 99% free and so far there have been 0 allocation failures
[root@esxedge1-02:~]Next, we will need to start the rotating log script, this will start the PCAPs collection threads and rotate the logs each 15 min as we defined in the prepare_pktcap script. The script should be left running in the SSH connection. if you stop it, the collection will also stop.
[root@esxedge1-02:~] /tmp/rotating_cap.py
Dump: 294272, broken : 0, drop: 0, file err: 0.
Dump: 175296, broken : 0, drop: 0, file err: 0.
Dump: 371584, broken : 0, drop: 0, file err: 0.
Dump: 317248, broken : 0, drop: 0, file err: 0.
Dump: 371648, broken : 0, drop: 0, file err: 0.
Dump: 317312, broken : 0, drop: 0, file err: 0.Monitor the collection
The last script is for doing the monitoring of the sessions. You will need to open a new SSH session to the host to run this script.
[root@esxedge1-02:~] /tmp/pktcap_sessions.py -l
The vmci heap is 58% free
session portID devName
579 67108942
580 67108942
581 67108945
582 67108946
583 67108943
584 2214592556 vmnic2
585 67108946
586 2214592556 vmnic2
587 67108943
588 67108945
589 2214592558 vmnic1
590 2214592558 vmnic1From the output, we can see that it collects the PortIDs that we have defined, but it also collects PCAP for the VMNics. This is valuable to us if we need to compare traffic on what is going in on the host and to the VM and visa versa.
Looking at the output that is stored on the datastore
[root@esxedge1-02:/vmfs/volumes/5fd89aca-1f6f344a-f65f-043f72c0064a/capture/42434546] ls
dc1esxedge1-02_2023-11-29T07_42_11_p67108942_d0_sna.pcap
....
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d0_sna.pcap.log dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d0_sna.pcap_rot.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d0_sna.pcap_rot.log dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap.log dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap_rot.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap_rot.log
killfileWe can see the PCAPs for each of the PortIDs and VMnics, and it will rotate every 15 minutes.
Stopping the collection
You might have noticed the “killfile” from above. Remote this file and the collection will stop.
[root@esxedge1-02:~] rm -rf /vmfs/volumes/5fd89aca-1f6f344a-f65f-043f72c0064a/capture/42434546/killfileConclusion
The scripts are handy because they rotate the logs and have a way to monitor and kill the collections. This way we don’t have to manually kill processes on the ESXi host.
When the collection is done, you can copy out the logs for further analysis, I found the Filezilla SFTP client to be the fastest way of copying out the data.
If out want to merge the PCAPs afterward, on MacOS, you can do it with
mergecap -w merged.pcap *.pcap
Troubleshooting
If you find that you can’t start the rotating logs script it might be because you have tried to start it before and it somehow stall. You can find the process IDs for it and kill it manually.
[root@esxedge1-02:~] ps -Tcjstv | grep -i rotating_cap
[root@esxedge1-02:~] kill 
