Author: Jesper Ramsgaard

I recently ran into an interesting issue with a Lenovo DE4200 connected directly to VMware ESXi hosts using SAS. Everything worked normally until one of the storage controllers was restarted.

During the controller restart the SAS paths disappeared as expected and I/O continued through the remaining controller. The problem was that when the controller came back online, the SAS paths did not return. A reboot of the ESXi host was required before all paths were available again.

The environment

The storage is a Lenovo DE4200 connected using SAS, and the ESXi hosts are using the lsi_msgpt35 driver.

The Lenovo DE-Series devices were already using NMP with VMW_SATP_ALUA and VMW_PSP_RR. This is also the configuration Lenovo documents for DE-Series storage connected to VMware.

Storage Array Type: VMW_SATP_ALUA
Path Selection Policy: VMW_PSP_RR
Plugin: NMP
Transport: sas

Under normal conditions, both SAS paths could be seen from ESXi:

vmhba3:C0:T1:L2
Plugin: NMP
State: active
Transport: sas

vmhba3:C0:T2:L2
Plugin: NMP
State: active
Transport: sas

What happened during a controller restart?

When restarting one of the DE4200 controllers, ESXi correctly detected that the SAS device had disappeared. The problem happened when the controller came back online.

The lsi_msgpt35 driver could see the SAS device again and attempted a TEST_UNIT_READY, but the device was not successfully brought back into a ready state.

lsi_msgpt35_0: _scsih_test_unit_ready: TEST_UNIT_READY: handle(0x0009), lun(0)
lsi_msgpt35_0: _scsih_add_device: device not ready: slot(5)
lsi_msgpt35_0: _scsih_add_device: device not ready: enclosure level(0x0000), connector name(C1)

After a long support case with Lenovo on the problem, they found that it’s a known bug, and I needed to update the firmware on the HBA.

Updating the SAS HBA firmware to the latest supported firmware did not change the behavior.

After some digging I found Broadcom KB 312116 describing problems with SAS paths and the lsi_msgpt3/lsi_msgpt35 drivers. One of the examples in the KB is losing access to SAS front-end volumes during a controller failover.

The interesting part was the following lsi_msgpt35 module parameter:

issue_scsi_cmd_to_bringup_drive

The parameter controls whether the host driver issues SCSI commands to bring a drive into the READY state. The default value is 1.

[root@esxi:~] esxcli system module parameters list -m lsi_msgpt35

issue_scsi_cmd_to_bringup_drive  bool    Enable/Disable host driver SCSI command issuing capability to bring
                                        the drive to READY state(default=1)
                                        0: Disable host driver SCSI command issuing capability
                                        1: Enable host driver SCSI command issuing capability

Broadcom recommends setting this value to 0 for the SAS path issue described in KB 312116. I therefore tested the same setting on one of the affected ESXi hosts.

esxcli system module parameters set -m lsi_msgpt35 -p issue_scsi_cmd_to_bringup_drive=0

The configured value can be verified with:

esxcli system module parameters list -m lsi_msgpt35 | grep issue_scsi_cmd

The ESXi host needs a reboot before the module parameter takes effect.

Testing the controller failover again

After rebooting the ESXi host I restarted the same DE4200 controller again while monitoring vmkernel.log.

tail -f /var/log/vmkernel.log | grep -E "lsi_msgpt35|ScsiClaimrule|ScsiPath|NMP|HPP|TEST_UNIT|REPORT_LUNS|device not ready"

This time the behavior was different. ESXi removed the paths when the controller disappeared, but when the controller came back the lsi_msgpt35 driver successfully discovered the SAS device and its LUNs again.

lsi_msgpt35_0: _scsih_sas_device_add: handle(0x0009)
lsi_msgpt35_0: _scsih_discover: Found Device C0:T1:L0
lsi_msgpt35_0: _scsih_discover: Found Device C0:T1:L1
lsi_msgpt35_0: _scsih_discover: Found Device C0:T1:L2

Most importantly, the previous “device not ready” messages were gone and the SAS path returned without a reboot or manual rescan of the ESXi host.

The test was repeated with another controller reset and the path recovered correctly again.

vmhba3:C0:T1:L2
Plugin: NMP
State: active
Transport: sas

vmhba3:C0:T2:L2
Plugin: NMP
State: active
Transport: sas

Conclusion

Changing the lsi_msgpt35 issue_scsi_cmd_to_bringup_drive parameter from the default value of 1 to 0 changed the behavior completely in this environment.

• With the default value, the SAS path did not recover after a DE4200 controller restart.
• Updating the HBA firmware did not solve the problem.
• With issue_scsi_cmd_to_bringup_drive=0, the SAS path automatically recovered.
• The controller restart was tested twice after the change with successful path recovery both times.
• No ESXi reboot or manual storage rescan was required for the paths to return.

Broadcom KB 312116 describes the lsi_msgpt35 module parameter and similar SAS controller failover problems on ESXi 7.x and 8.x. As always with storage driver parameters, verify that the configuration is supported for your specific ESXi, HBA and storage combination before applying it to production hosts.

I’m sure out of my comfort zone on this post. I picked up an old Huawei cmr-al09 Android tablet. This device, from 2018, runs Android 9 and is no longer supported. I have never used Android, but the thought of it running a Frankenstein Linux is kind of compelling, so I was curious whether this was really e-waste or if it could get a second life.

I quickly found an XDA Developers forum thread where the community has created a custom rom/os for the device, LineageOS. But I have never tried to unlock, root, or install custom firmware on a phone or tablet before. So this was surely an interesting experience.

LineageOS

LineageOS is an open‑source Android distribution that replaces the manufacturer’s firmware. It is built by the community and is often used to break free of OEM bloatware and regain control of your device. In my case, it can extend the life of an outdated Android tablet.

Hardware overview:

The Huawei MediaPad M5 (model CMR‑AL09) was released in early 2018, which makes it old by mobile standards. But the hardware is still very capable.

• SoC: Octa-core (4×2.1 GHz Cortex-A73 & 4×1.8 GHz Cortex-A53)
• GPU: Mali‑G71 MP8
• Memory: 4 GB DDR4
• Storage: 64 GB internal + microSD
• Display: 10.8″ IPS, 2560×1600 (16:10)
• Connectivity: Wi‑Fi ac, Bluetooth 4.2, LTE (AL09)

Tools

On macOS, the simplest way to install the adb and fastboot tools is with Homebrew, which is more or less all you need. LineageOS also have a good guide on common use of the tools here.

brew install --cask android-platform-tools

Verify installation:

jeram@QL4QJP2F4N ~ % adb version       
Android Debug Bridge version 1.0.41
Version 36.0.2-14143358
Installed as /opt/homebrew/bin/adb
Running on Darwin 25.2.0 (arm64)

jeram@QL4QJP2F4N ~ % fastboot --version
fastboot version 36.0.2-14143358
Installed as /opt/homebrew/bin/fastboot

Prepare the Android device

For ADB and fastboot to communicate with the tablet, you need to enable Developer Mode.

1. Open Settings → About tablet
2. Tap Build number 7 times
3. Go to Settings → Developer options
4. Enable:
   • USB debugging
   • OEM unlocking

Connect the device to the Mac via USB, you will get a pop-up on the tablet where you need to allow the Mac or PC to connect to it. Verify connectivity:

jeram@QL4QJP2F4N ~ % adb devices
List of devices attached
8FXDU18924000194	device

Unlock the bootloader

OEMs protect devices by locking the bootloader so it can only run the vendor-designated OS. This means that we can’t install another OS on it before the bootloader is unlocked, because security? There are different procedures for getting the unlock codes. Some vendors have a service where you can request the unlock key, others simply just dont allow it and third party services have surfaced where you can pay a fee and get it done. If you search for “ministryofsolutions” you will find one of them.

I found a tool called PortatoNV. This tool can find unlock codes for some older Android devices. But it’s not supposed to be easy, you need to short a test pin on the motherboard to unlock a special serial port for the tool to use.

Opening the cmr-al09 device was fairly simple. I have an iFixit repair kit, and using the small, thin blue plastic prying tools, I slipped them in along the side of the screen to open the device. The adhesive was not as strong as I would expect.

Searched online for “cmr-al09 test point” and compared the motherboard revision to what I had. There were plenty of people showing where to find it.

You will need to power off the tablet, unplug the USB cable, short to ground for 3 seconds, plug the USB cable back in, and hold it for 2 more seconds. The tablet won’t power on, but will get into a mode where you will see a

Unfortunately, the PortatoNV tool only runs on Windows, so I found my old Windows machine, installed the tool, shortened the testpoint, and got an unknown device in Device Manager showing “USB SER”. Finding the drivers for this is another challenge…

I found the drivers online, but they’re so old they won’t run on Windows 10/11 without disabling Microsoft Defender Memory Integrity. After a lot of searching, I gave up and disabled Memory Integrity. I could then get the PortatoNV tool to find the unlock code, which only took a couple of minutes.

I could now boot the tablet again and, using ADB, put it into fastboot mode, where the firmware can be flashed.

adb reboot bootloader

Rom flashing

The XDA post offers a short, useful list of commands for flashing. Download all the files and flash them to the device.

fastboot oem unlock <UNLOCKCODE>

The unlock command will display a prompt on the tablet, where you must confirm you want to unlock the bootloader.

fastboot flash kernel boot.img
fastboot flash recovery_ramdisk recovery.img
fastboot flash recovery_vendor recovery_vendor.img

Now you need to boot into recovery mode. I tried to do this with the power and volume+ buttons. But I could only get into the eRecovery menu. With ADB, you can also reboot it into recovery mode

adb reboot recovery

This way, I got the LineageOS recovery screen, which showed that I was corrupted and needed to be restored. There was an option to factory reset, so I did. After this, I could choose “Apply Update” in the LineageOS recovery menu. From there, you can sideload the main LineageOS zip file from the XDA forum.

Factory reset -> Format data/factory reset
Apply update -> Apply from ADB
adb sideload lineage-20.0-20240718-RELEASE-mediapadm5wifi/lte.zip

LineageOS working

The OS is not installed and booted up – success. The next step would be to install the Google Apps suite on it to enable an app store. Root it and get SSH running so it can serve as a web server or something else useful.

As an Apple user, I wish that Apple had kind of the same tools

After Broadcom’s new license policy, where only edge clusters doing stateful services need to be licensed, Cloud Director implemented the feature to honor a stateless firewall setup in 10.6.1.

This gave you the possibility to choose clusters from the feature set that were needed. https://knowledge.broadcom.com/external/article/420629/creating-stateful-firewall-rule-with-clo.html

It also introduced a bug if you directly created an Edge gateway within a Datacenter Group, it would then make the Edge gateway with stateless firewall rules. If you decided to create an Edge gateway and then afterward import it into the Datacenter Group, it would be stateful.

This is all good, features are always welcome. But this also introduced a bug, where I need

The fix: instead of recreating the Edge with all the rules and config in Cloud Director, we can clone the rules in NSX Manager and then make the new policy, where the rules live, stateful.

Choosing a clone on the policy that Cloud Director has created. Marking the policy will also mark all the rules.

After it’s been cloned, you need to edit the cloned policy and make it stateful. The NSX ID for the policy will now become “copy_xxxxxx”.

For Cloud Director to be able to read and edit the policy in the future, we will have to delete the old non-stateful policy, then publish, so that it can be cloned again, but this time without the “copy of xxxx” in front of the policy, so that the stateful policy will get the original name and it can take its place from the non-statefull rule set.

For it to look the same in Cloud Director, we also need to remove all the prefixed rules that now say “Copy of Copy of”. After that, we can also mark the first “copy of” policy for deletion, then publish the config.

Conclusion

This is a workaround so you don’t have to create all the rules manually again, if they have been created as stateless firewall rules, due to a bug where Edge gateways created directly in a Cloud Director Datacenter Group, it would become stateless instead of stateful

More and more customers seem to be requesting ERSPAN for Network Trafic Analyser appliances. Companies promise to analyze your VM network traffic and give security tips on what’s happening in your environment.

NSX supports multiple ways of doing SPAN(Switched Port Analyser) or port mirroring. I have listed the two useful variants first. The two last are very limited and not very useful.

1. Logical SPAN—The source and destination must be in the same overlay segment. The source will be VMs, and each SPAN entry will be limited to 5 VMs.
2. Remote L3 SPAN – The source can be an overlay segment and the destination must be L3 reachable from all eligible host VMKernel ports.

There are two more, accessible from a policy view, but I don’t find them useful.

1. Local SPAN – The source and destination must be on the same host.
2. Remote SPAN – The Source and destination must be L2 reachable

For this article, we will look at how we can implement ERSPAN. With this setup the hosts holding the VM will be the ones to create the port mirror, encapsulate it in GRE, and send it off to its destination This means we will need to create an overlay segment where we can connect VMKernel from all hosts to.

What is ERSPAN?

Packets are encapsulated in GRE and additional headers are added to the original packet. Details below

Original Ethernet Frame: The packet being mirrored (source traffic).

1. ERSPAN Header: metadata, including session ID, VLAN ID, timestamp, and sequence number.
2. GRE (Generic Routing Encapsulation) Header: Wraps the ERSPAN traffic to enable transport over an IP network.
3. Outer IP Header: Specifies the source and destination IP addresses of the ERSPAN session, enabling routing across the network.

Environment:

Quick drawing of what we are doing. The ERSPAN will mirror customer network traffic and source it from the hosts VMKernal ports into the mirror segment. NSX will be told to ship off the ERSPAN traffic to the collector IP.

First, we need to create a VMKkernel port for the mirror network. In my case, I created an overlay segment dedicated to the mirror traffic. On the two hosts in my setup, I made a VMKernal port with an IP in the mirror segment subnet.

My collector VM then has a nic and IP in the mirror segment and this is where we are going to send the ERSPAN traffic.

In the source I have the customer network segment and the destination is the IP of the collector VM.

On the collector VM a dedicated nic is used for the ERSPAN traffic. Using Wireshark on the interface with the option “ip proto 0x2f” decapsulated the GRE and showed the original packets.

In the example above I’m following a TCP stream where a client is accessing an IIS server on 10.44.44.10. Here we can see all the communication packets of between the client and server.

Performance

let’s say, it’s not free in terms of the use of resources. For the host to duplicate and encapsulate the traffic it takes up some CPU resources. DPUs can offload this if you want, but that also comes with a cost.

I haven’t been able to find out the real impact of ERSPAN in terms of resources. If you know a way to see how many CPU cycles etc are used to create ERSPAN, please let me in on this secret 🙂

Performance test

Here two VMs on the customer network segment have an iperf running. The hosts are connected with a 10Gbit interface, and with the iperf setup below we saturate this interface.

iperf3.exe -c 10.44.44.10 -P2 -t 1000

In Wireshark we can see that we received all the packages, but for bandwidth i would expect more. I only see about 9 Mbit of traffic on the Windows task manager. My expectation would be that Wireshark to receive the complete package. ERSPAN traffic should match the iperf traffic usage.

Again, if you know the ERSPAN traffic does not match the monitored segments bandwidth usage, let me know 🙂

Conclusion

NSX can indeed deliver mirror port data. VMware documentation is stating that its a feature for troubleshoting and not permanent usage. Talked to a VMware employee at Explore 2024 that said that this might be an obsolete statement in the documentation, but that they have had cases where network got complete exhausted due to enable port mirror on the networks that where also carrying the ERSPAN traffic, which ofcause will make a loop in traffic.

Have been running this setup for couple of month now and it seems to be stable. A way to help ensure that ERSPAN traffic will not flood your network would be to implement QoS on the overlay segment carrieing the ERPSAN. This way it the traffic will be droped