XenServer iscsi MultiPath

Posted on by Jesper Ramsgaard

During a recent ONTAP upgrade, I ran into an issue with XenServer and iSCSI multipathing. I’m not an expert in Xen, so this was my first time.

The Xen UI was showing multi-pathing 2/4. This needed some digging, normally used to working with VMware products and you can get a bit blind since its handled so very nice compared to how Xen does it.

mpathutil status could show there were two paths active, sda and sdb, backing dm-0

[root@xen1 ~]# mpathutil status
show topology
3600a098038314a74433f5546696e626b dm-0 NETAPP  ,LUN C-Mode
size=100G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 8:0:0:0 sda 8:0  active ready running
  `- 9:0:0:0 sdb 8:16 active ready running

iscsiadm -m session showed a bit more of where the paths were connected to target IPs. The setup is a NetApp A220, so it has two nodes. Each node has one IP iscsi IP on two VLANs or subnets. NetApp node1 holding .10 addresses and node2 holding .11 addresses. So, in this view, you can see that the Xen server is only connected to one NetApp node, which would have been a problem for our Ontap upgrade.

[root@lxen1 ~]# iscsiadm -m session
tcp: [1] 172.16.28.11:3260,1033 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)
tcp: [2] 172.16.29.11:3260,1035 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)

The fix was to try to discover the .10 addresses. Here we got the return on all targets from the NetApp

[root@xen1 ~]# iscsiadm -m discovery -t sendtargets -p 172.16.28.10
172.16.28.10:3260,1032 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4
172.16.29.11:3260,1035 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4
172.16.28.11:3260,1033 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4
172.16.29.10:3260,1034 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4

So now we can log in and add targets using the iscsiadm commands.

[root@xen1 ~]# iscsiadm -m node -T iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 -p 172.16.28.10 -l
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4, portal: 172.16.28.10,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4, portal: 172.16.28.10,3260] successful.

[root@ltp1xen1 ~]# iscsiadm -m node -T iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 -p 172.16.29.10 -l
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4, portal: 172.16.29.10,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4, portal: 172.16.29.10,3260] successful.

Listing the iSCSI sessions, and we can now see sessions to all targets

[root@xen1 ~]# iscsiadm -m session
tcp: [1] 172.16.28.11:3260,1033 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)
tcp: [2] 172.16.29.11:3260,1035 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)
tcp: [3] 172.16.28.10:3260,1032 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)
tcp: [4] 172.16.29.10:3260,1034 iqn.1992-08.com.netapp:sn.b8b89e3f660411edb07dd039ea9d8cc2:vs.4 (non-flash)

showing paths again and we can see two extra paths

[root@xen1 ~]# mpathutil status
show topology
3600a098038314a74433f5546696e626a dm-0 NETAPP  ,LUN C-Mode
size=100G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 8:0:0:0  sda 8:0  active ready running
| `- 9:0:0:0  sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 10:0:0:0 sdc 8:32 active ready running
  `- 11:0:0:0 sdd 8:48 active ready running

Testing

To be sure that the new iscsi paths are working we can offline the old paths and we should be able to see storage trafic move over on the .10 IPs on the NetApp system, whiched is the unoptimzed paths.

echo offline > /sys/block/sda/device/state
echo offline > /sys/block/sdb/device/state

You could now see the paths as “Faulty” in the mpathutil status output.

[root@xen4 tmp]# mpathutil status
show topology
3600a098038314a74433f5546696e626e dm-0 NETAPP  ,LUN C-Mode
size=100G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 10:0:0:0 sda 8:0  failed faulty offline
| `- 11:0:0:0 sdb 8:16 failed faulty offline
`-+- policy='service-time 0' prio=10 status=active
  |- 12:0:0:0 sdc 8:32 active ready  running
  `- 13:0:0:0 sdd 8:48 active ready  running

Next step is to check storage is still working, if it didnt you would by now probally see the system become slow and inresponsible. In my case it worked, and i could run a dd test and see trafic on the .10 interfaces from the NetApp UI.

cd /tmp or /run/sr-mount/<VOLUMEUUID>
time dd if=/dev/zero of=testfile bs=1M count=512 oflag=direct
rm -f testfile

Lastly we can enable the other paths again

echo running > /sys/block/sda/device/state
echo running > /sys/block/sdb/device/state

Conslusion

Surely learned something new today. Found the most of the documentation for how to do in on Citrix own side, one link was very helpfull https://support.citrix.com/external/article/CTX140118/helpful-iscsi-commands.html

Cloud Director NSX-V Cleanup

Posted on by Jesper Ramsgaard

Over the past 4 years, I have been in a constant battle to move all workloads to NSX-T. And I say “battle” because it certainly felt like it. A larger entity acquired the company I work for. Although we completed the NSX-V>T project for our vSphere environments, the larger entity still had a small NSX-V workload for me to take on.

But happy to announce that the wait is over, all NSX-V bits have been uninstalled and removed.

One of the last errors during NSX-V uninstall was cleaning up in Cloud Director. Many of the logical switches were becoming stranded items due to issues Cloud Director encountered during cleanup. Usually, we can remove the stranded items, but here they were stuck. This was a blocking issue preventing the removal of the NSX-V manager from Cloud Director.

Before proceeding, take a backup with the cell tool, and a VM snapshot on the cells wouldn’t hurt either. If you are not sure about messing around with your vCloud database, I strongly recommend that you create a Broadcom support case and let them assist.

1. Finding the Network pool id

vcloud=# select * from network_pool where name = 'VXLAN-NP';
-[ RECORD 1 ]-----+-------------------------------------
id                | 39a4461a-0b90-435f-84d4-6388fc8830cd
name              | dc1compute01-VXLAN-NP
description       |
dvswitch_id       |
network_pool_type | 3
vdn_scope_id      | vdnscope-6
vlan_id           | 0
promiscuous_mode  | f
vc_id             | d0b752ac-be4c-4939-9245-d5e191806430
version_number    | 4
creation_status   | READY
universal_id      |
nsxt_manager_id   |

2. Listing networks backed by the NSX-V network pool

Use the network pool id from the query above

vcloud=# select
    real_network.id as real_network_id,
    real_network.name as real_network_name,
    real_network_backing.id as real_network_backing_id,
    logical_network.id as logical_network_id,
    logical_network.scope_type as logical_network_scope,
    logical_network.name as logical_network_name
from network_pool
left join real_network on real_network.pool_id = network_pool.id
left join real_network_backing on real_network_backing.rnet_id = real_network.id
left join logical_network on logical_network.rnet_id = real_network.id
where real_network.pool_id = '39a4461a....'

real_network_id            | real_network_name                                      | real_network_backing_id        | logical_network_id | logical_network_scope | logical_network_name
---------------------------+--------------------------------------------------------+--------------------------------+--------------------+-----------------------+----------------------
9e2459ec-1013-4a85-b0b7-   | dvs.193a-4fa4-90ba-327429e479b3              | 598522d5-da6d-4fd2-b1fc-       |                    |                       |
cfa130cd-7d95-46b5-a994-   | dvs.e83ba4a6-ad0e-4d6f-a07b-b4874156883b     | ffd658bb-e9a0-4120-b847-       |                    |                       |
9096c757-d42c-43a6-9a8d-   | dvs.a2554d6f-dd83-4073-8200-425914618b00     | 492561e3-e116-477b-b9db-       |                    |                       |
81c4962c-bd44-4fc9-aaa8-   | dvs.e2142ca8-a5c7-48ab-9c87-6fbb9c8d41f8     | 18538223-f184-4990-831c-       |                    |                       |
(4 rows)

3. Review each of the networks

To confirm, it will not be listed twice to provide a before-and-after view.

vcloud=# select * from real_network_backing where rnet_id = '9e2459ec-1013-4a85-b0b7'
                  id                  |               rnet_id                |                vc_id                 |     dvpg_moref
--------------------------------------+--------------------------------------+--------------------------------------+--------------------
ffd658bb-e9a0-4120-b847-c230890b89fd | 9e2459ec-1013-4a85-b0b7-              | ef4e323d-fdc2-4a0d-87e2-fefed1b281e1 | dvportgroup-121351
(1 row)

4. Removing the network from the Network Pool

vcloud=# delete from real_network_backing where rnet_id = '9e2459ec-1013-4a85-b0b7-';
DELETE 1

After deleting all networks, rerun step 2 and verify that there are no longer any network dependencies on the network pool. If it returns an empty response, you are good to go and can delete the network pool in Cloud Director.

Conslusion

After 200+ customers have been migrated to NSX-T, I’m happy and hope to never be in the crossfire of VMware changing its network virtualisation product again in a way that would require us to migrate all of the customers’ networks.

NSX Route Aggregation

Posted on by Jesper Ramsgaard

TL;DR: If your upstream peers are drowning in /32s from NATs and VIPs—or you just want faster convergenc. NSX‑T Route Aggregation lets you advertise neat summaries (e.g., 10.10.0.0/16) northbound and suppress the noise.

When you announce a /32 to your core routers, it’s likely they won’t get accepted. The smallest internet routed ipv4 is a /24 scope. Internally in your network, you can have as many /32 routes as you or your route table will allow.

By default, NSX-T won’t do any route aggregation, so it’s all /32 routed addresses. An example could look like this:

This picture is more to show you have it actually works, but let’s go with the route table was full of /32 addresses and we wanted to make it simpler. We could introduce the route aggregation without out summary.

Now we can see both the /24 and the /32 routes for the 185.66.12.0 scope. To make the routetable even simpler, we can choose to summarize. That would put all the /32 routes under the larger /24 scope.

Failed to get dvportgroup properties of source Org VDC network

Posted on by Jesper Ramsgaard

Not everyone has finished their NSX V>T migrations yet… I have had a couple of problems when using the VCDNSXMigration 1.4.2.1 tool, in the bridging process, it calls to get the dvpg settings. The problem is not the migration tool itself, but Cloud Director referring to a dvpg not in vCenter.

GET /cloudapi/1.0.0/orgVdcNetworks/urn:vcloud:network:1f5a73c5-7d39-4d5f-bee3-1652ce224b9e/dvpgProperties HTTP/1.1" 500 None.
Exception: Failed to get dvportgroup properties of source Org VDC network xxx

Emulating the same API call in Postman and you get a response that the requested portgroup does not exist.

Looking for the dvpg in the Cloud Director database shows us that VCD thinks it exists.

vcloud=#  select * from real_network_backing where dvpg_moref = 'dvportgroup-5945' and vc_id = (select id from virtual_center where name = 'vcsa');
                  id                  |               rnet_id                |                vc_id                 |    dvpg_moref
--------------------------------------+--------------------------------------+--------------------------------------+------------------
 1182fbdb-7ad4-4480-9773-8b926eddb2cc | 9e17dddc-5b75-49d2-8442-5f63d01a0862 | d0b752ac-be4c-4939-9245-d5e191806430 | dvportgroup-5945
(1 row)

To check that vCenter agress that it does not exist, we can use MOB to see if tere is a entry. If it gives back 404 we are good to delete the entry in the VCD database.

If we get a 404, it means that vCenter does not know about it, and you can proceed with the Cloud Director database cleanup.

delete from real_network_backing where dvpg_moref = 'dvportgroup-5945' and vc_id = (select id from virtual_center where name = 'vcsa');
DELETE 1
vcloud=#

The stall entry has now been cleaned, and when you now the postman call you get a valid JSON in return.

{
    "dvpgProperties": [
        {
            "dvpgMoref": "dvportgroup-5945",
            "vlanId": null,
            "promiscuousMode": true,
            "forgedTransmit": true,
            "dvSwitch": {
                "backingRef": {
                    "id": "dvs-20",
                    "name": null
                },
                "virtualCenter": {
                    "name": "vcsa",
                    "id": "urn:vcloud:vimserver:d0b752ac-be4c-4939-9245-d5e191806430"
                }
            }
        }
    ]
}

Conclusion

It’s a bug in VCD for NSX-V logical wires. Hopefully, the usage for this will soon be over because all are finishing up their NSX-V migrations. But until then there is a fix for it with the above process.

ERSPAN with NSX

Posted on by Jesper Ramsgaard

More and more customers seem to be requesting ERSPAN for Network Trafic Analyser appliances. Companies promise to analyze your VM network traffic and give security tips on what’s happening in your environment.

NSX supports multiple ways of doing SPAN(Switched Port Analyser) or port mirroring. I have listed the two useful variants first. The two last are very limited and not very useful.

  1. Logical SPAN—The source and destination must be in the same overlay segment. The source will be VMs, and each SPAN entry will be limited to 5 VMs.
  2. Remote L3 SPAN – The source can be an overlay segment and the destination must be L3 reachable from all eligible host VMKernel ports.

There are two more, accessible from a policy view, but I don’t find them useful.

  1. Local SPAN – The source and destination must be on the same host.
  2. Remote SPAN – The Source and destination must be L2 reachable

For this article, we will look at how we can implement ERSPAN. With this setup the hosts holding the VM will be the ones to create the port mirror, encapsulate it in GRE, and send it off to its destination This means we will need to create an overlay segment where we can connect VMKernel from all hosts to.

What is ERSPAN?

Packets are encapsulated in GRE and additional headers are added to the original packet. Details below

Original Ethernet Frame: The packet being mirrored (source traffic).

  1. ERSPAN Header: metadata, including session ID, VLAN ID, timestamp, and sequence number.
  2. GRE (Generic Routing Encapsulation) Header: Wraps the ERSPAN traffic to enable transport over an IP network.
  3. Outer IP Header: Specifies the source and destination IP addresses of the ERSPAN session, enabling routing across the network.

Environment:

Quick drawing of what we are doing. The ERSPAN will mirror customer network traffic and source it from the hosts VMKernal ports into the mirror segment. NSX will be told to ship off the ERSPAN traffic to the collector IP.

dc1
dc1
VDS

VDS
vmnic1
vmnic1
vmnic1
vmnic1
vmk – mgmt
vmk – m…
vmnic1
vmnic1
vmnic1
vmnic1
vmk – mirror
vmk – m…
segment – customer network
segment – customer network
app2
app2
vmk – vmotion
vmk – vm…
vmk – mirror
vmk – m…
vmk – mgmt
vmk – m…
vmk – vmotion
vmk – vm…
segment – mirror network
segment – mirror network
collector
colle…Text is not SVG – cannot display

First, we need to create a VMKkernel port for the mirror network. In my case, I created an overlay segment dedicated to the mirror traffic. On the two hosts in my setup, I made a VMKernal port with an IP in the mirror segment subnet.

My collector VM then has a nic and IP in the mirror segment and this is where we are going to send the ERSPAN traffic.

In the source I have the customer network segment and the destination is the IP of the collector VM.

On the collector VM a dedicated nic is used for the ERSPAN traffic. Using Wireshark on the interface with the option “ip proto 0x2f” decapsulated the GRE and showed the original packets.

In the example above I’m following a TCP stream where a client is accessing an IIS server on 10.44.44.10. Here we can see all the communication packets of between the client and server.

Performance

let’s say, it’s not free in terms of the use of resources. For the host to duplicate and encapsulate the traffic it takes up some CPU resources. DPUs can offload this if you want, but that also comes with a cost.

I haven’t been able to find out the real impact of ERSPAN in terms of resources. If you know a way to see how many CPU cycles etc are used to create ERSPAN, please let me in on this secret 🙂

Performance test

Here two VMs on the customer network segment have an iperf running. The hosts are connected with a 10Gbit interface, and with the iperf setup below we saturate this interface. 

iperf3.exe -c 10.44.44.10 -P2 -t 1000

In Wireshark we can see that we received all the packages, but for bandwidth i would expect more. I only see about 9 Mbit of traffic on the Windows task manager. My expectation would be that Wireshark to receive the complete package. ERSPAN traffic should match the iperf traffic usage.

Again, if you know the ERSPAN traffic does not match the monitored segments bandwidth usage, let me know 🙂

Conclusion

NSX can indeed deliver mirror port data. VMware documentation is stating that its a feature for troubleshoting and not permanent usage. Talked to a VMware employee at Explore 2024 that said that this might be an obsolete statement in the documentation, but that they have had cases where network got complete exhausted due to enable port mirror on the networks that where also carrying the ERSPAN traffic, which ofcause will make a loop in traffic.

Have been running this setup for couple of month now and it seems to be stable. A way to help ensure that ERSPAN traffic will not flood your network would be to implement QoS on the overlay segment carrieing the ERPSAN. This way it the traffic will be droped

VMware NSX BGP Route maps

Posted on by Jesper Ramsgaard

BGP Routemaps within VMware NSX allow administrators to manipulate BGP routing decisions based on specified conditions. They enable administrators to influence route selection, adjust attributes, and filter routes to meet the needs of their network architecture.

Key Components of BGP Route maps:

Before we delve into practical examples, let’s review the fundamental components of BGP Route maps in VMware NSX:

  1. Match Conditions: These define the criteria against which routes are evaluated. Match conditions include route prefixes, AS paths, route origins, etc.
  2. Actions: Actions specify what should happen to routes that match the specified conditions. Common actions include setting attributes like local preference, AS path prepend, and route filtering.

In each route map, multiple things can be set. In this section, we will look at two very common ways to control incoming and outgoing traffic.

AS Prepend

AS prepend is a technique that influences inbound traffic flow by manipulating the AS path attribute of outgoing BGP advertisements. Adding your AS number multiple times to the AS path can make certain routes less preferable or expensive to incoming traffic.

Local Preference

Local preference is an attribute used to influence outgoing traffic flow by indicating the preferred exit point for traffic leaving your network.

The setup

We already got our T0 setup with BGP Peers to our upstream routers.

We also have a VM behind a T1 that is connected to the T0. From here we can trace routes so we can see what path it will take.

A tracert from the VM shows us that we are getting out of the .185 peer.

How to

Create the prefix lists

If not already, then we need to define some prefix lists. It can be done with 0.0.0.0/0 or a more specific prefix if we want to specify what the route map will hit IPs. Head into the NSX manager, Networking, Tier-0 Gateways, and edit the T0. Under its Routing tab, you will see prefixes.

In this example, there is both a specific and an any. We will use the 0.0.0.0/0 here

Create the route maps

Next, we need to prepare the route maps. They can also be found under the tier-0 routing section. Here we will define two routemaps. one for prepend and one for local pref.

Apply the route maps

Now we have set together their route map with its criteria and prefixes. We now need to apply the route maps to the BGP Peers. Under the BGP section of the T0 press the “BGP Neighbors”.

Now we will add the local pref for the .177 peer. This will manipulate the use of this peer for the outgoing traffic for the specified prefixes. 

Caution: i have seen that if you choose local pref on one bgp peer and the other peers dont have as prepend setup, then traffic will drop. Not acaltly sure why, but i assume that i could be due to asymetric route. eggress on on the peer with local pref and incoming on the other peer. But when setting as prepend on the other peers it works again.

Traffic from our test VM will now flow through the .177 peer instead of the .185 peer that was before. Looking at the incoming traffic it also goes to the peer with the best route, hence the peer that doesn’t have any prepend on its AS number.

Conclusion

With BGP Routemaps in VMware NSX, administrators can control traffic flow precisely, optimizing network performance, resource utilization, or migration. Experiment with different match conditions and actions to tailor route maps to the unique requirements of your NSX environment.

Cloud Director – LB Content Switching

Posted on by Jesper Ramsgaard

Content switching is used to direct incoming traffic to different pools/servers based on the content of the requests. By looking at the application layer (L7 in the OSI model) we can inspect the client request content, such as URL, header, cookies, queries, etc.

ALB has the concept of policies on virtual services. Each policy rule can have a match and action to it. So we can match on a host header http request and then do a content switch action afterward.

This is, unfortunately, a feature that is not yet supported by Cloud Director(10.5.1). But it can be done in the ALB manager, behind the back of VCD. Rules are in the VCD UI, but not the content of the policy.

My guess is therefore that when it comes to VCD UI in the future, it will be able to adopt the existing content-switching policies. 

If you also think this feature should be implemented in VCD, please do a feature request on this site.

Workaround

In Cloud Director you setup the pools you want to do content switching on. From the screenshot below you can see port80 and port443 are the ones being used to day. Creating the extra pools here will have VCD aware of the pool, so we only need to change a bit on existing objects in the ALB manager.

In the ALB manager, find the virtual service, edit it, and navigate to the policy section. Here we can see the different policy types. Find “HTTP request” and add to new rules.

Each rule will have the match and action point as mentioned before. When doing the action content switch get to choose a specific pool.

Save it all and you are ready to test the content switching. From the screenshot below, you can see how it will look inside the VCD UI.

NSX V2T migration

The reason for me to research how to do content switching was primarily that some tenants are still using NSX-V because they use haproxy application rules to do content switching. NSX migration for Cloud Director tool does support basic load balancing migrations. But not when there are application rules applied.

Now tenant can schedule the service window, remove the application rules, and have the migration tool migrate all nat, firewall, routes, and basic load balancing.

After the migrations are done the content switching rules can be created manually, from what the application rules in haproxy specified.

Conclusion

Even if it’s not supported by VCD yet, we can still do it. Ofcause the tenants can’t do it themself but will need to log a support case with you until the feature is introduced. And when they do, then let’s hope it will adopt the rules being done behind its back.

Cloud Director PowerShell deploy

Posted on by Jesper Ramsgaard

Having to deploy multiple Cloud Director environments can be tedious, especially when OVF deployment in vCenter fails with some obsure error all the time. But using powershell to deploy is smooth sailing.

Here you will get the small piece of PowerShell to do so. Its very basic and but works everytime.

Connect-VIServer -server vcsa.home.lab

$OVA = "C:\teknik\VMware_Cloud_Director-10.4.1.9057-20912720_OVF10.ova"

# vCenter Server used to deploy to
$VMCluster = "mgmt01"
$VMDatastore = "vSan"

# VCD Configuration
$VMName = "vcd1-03"

$VMNetwork1 = "vcd_vCloud_External_Perimeter"
$VMNetwork2 = "vcd_vCloud_Internal_Perimeter"

$VMNetworkIP1 = "10.66.66.31"
$VMNetworkIP2 = "10.55.55.31"
$VMNetmask = "255.255.255.0"
$VMGateway = "10.66.66.1"

$VMDNS = "10.44.44.10"
$VMNTP = "ntp.home.lab"
$VMSearchPath = "home.lab"

$eth0Routes = ""
$eth1Routes = "10.55.55.1 10.44.44.0/24, 10.55.55.1 10.1.94.0/24"

$DeploymentSize = "standby-medium"
$RootPassword = ''

$cluster = Get-Cluster $VMCluster
$vmhost = $cluster | Get-VMHost | Get-Random
$datastore = $vmhost | Get-Datastore $VMDatastore

# Setup ovfconfig
$OvfConfig = Get-OvfConfiguration $OVA

$OvfConfig.DeploymentOption.Value = $DeploymentSize

$OvfConfig.NetworkMapping.eth0_Network.Value = $VMNetwork1
$OvfConfig.NetworkMapping.eth1_Network.Value = $VMNetwork2

$OvfConfig.vami.VMware_vCloud_Director.ip0.Value =  $VMNetworkIP1
$OvfConfig.vami.VMware_vCloud_Director.ip1.Value =  $VMNetworkIP2
$OvfConfig.vami.VMware_vCloud_Director.netmask0.Value = $VMNetmask
$OvfConfig.vami.VMware_vCloud_Director.netmask1.Value = $VMNetmask
$OvfConfig.vami.VMware_vCloud_Director.gateway.Value = $VMGateway
$OvfConfig.vami.VMware_vCloud_Director.DNS.Value = $VMDns
$OvfConfig.vami.VMware_vCloud_Director.domain.Value = $VMName
$OvfConfig.vami.VMware_vCloud_Director.searchpath.Value = $VMSearchPath

$OvfConfig.vcloudapp.VMware_vCloud_Director.enable_ssh.Value = $true
$OvfConfig.vcloudapp.VMware_vCloud_Director.expire_root_password.Value = $false
$OvfConfig.vcloudapp.VMware_vCloud_Director.ntp_server.Value = $VMNTP
$OvfConfig.vcloudapp.VMware_vCloud_Director.varoot_password.Value = $RootPassword

$OvfConfig.vcloudnet.VMware_vCloud_Director.routes0.Value = $eth0Routes
$OvfConfig.vcloudnet.VMware_vCloud_Director.routes1.Value = $eth1Routes

# Deploy
Import-VApp -Source $OVA -OvfConfiguration $OvfConfig -Name $VMName -Location $cluster -VMHost $vmhost -Datastore $datastore -DiskStorageFormat thin

ESXi VM PCAP collection

Posted on by Jesper Ramsgaard

This is a small guide on how to collect PCAPs for network traffic from/to a virtual machine running on VMware ESXi. Scripts have been provided by VMware GSS and can be found here and used at your own risk.

(I get it, who would want to use Python scripts downloaded from a random website and run them on your ESXi, feel free to look into the code after download, or else request them from VMware)

The scripts can help to collect on multiple PortIDs, divide the PCAPs into smaller files, and rotate the logs for each x minutes and it can help to monitor the capture that is ongoing.

If you don’t want to use the scripts, then there is also a more manual way to run the collection here.

Finding the VM port IDs

Connect to the ESXi host with SSH and list the running VMs.

[root@dc1esxedge1-02:~] esxcli network vm list
World ID  Name              Num Ports  Networks
--------  ----------------  ---------  --------
68018705  nsxedgenode1             3  dvportgroup-914135, dvportgroup-914135

Find the VM that you want to collect from, let’s take WorldID 68018705 and check the switch port IDs for it

[root@esxedge1-02:~] esxcli network vm port list -w 68018092
   Port ID: 67108942
   vSwitch: DSVNSX
   Portgroup: dvportgroup-914135
   DVPort ID: 109
   MAC Address: 00:50:56:ab:82:59
   IP Address: 0.0.0.0
   Team Uplink: vmnic2
   Uplink Port ID: 2214592556
   Active Filters:

   Port ID: 67108943
   vSwitch: DSVNSX
   Portgroup: dvportgroup-914135
   DVPort ID: 108
   MAC Address: 00:50:56:ab:06:b0
   IP Address: 0.0.0.0
   Team Uplink: vmnic2
   Uplink Port ID: 2214592556
   Active Filters:

   Port ID: 67108944
   vSwitch: DSVNSX
   Portgroup: dvportgroup-914134
   DVPort ID: 98
   MAC Address: 00:50:56:ab:93:ab
   IP Address: 0.0.0.0
   Team Uplink: vmnic1
   Uplink Port ID: 2214592558
   Active Filters:

In this case, we note down the following port IDs and save them for later

67108942
67108943
67108945
67108946

Start the collection with scripts

Upload the Python script to the /tmp/ on the ESXi host. You also need a folder on the local datastore of the ESXi host where the pcap logs can be stored.

Prepare and run the first script where we provide the PortIDs and where to store the PCAPs

[root@esxedge1-02:~] python /tmp/prepare_pktcap.py -p 67108942 -p 67108943 -p 67108945 -p67108946 -u -d /vmfs/volu
mes/prd.dc1esxedge1-02/capture/42434546 -o /tmp/rotating_cap.sh -G 15m -r 3600
The current vmci heap is 99% free and so far there have been 0 allocation failures
[root@esxedge1-02:~]

Next, we will need to start the rotating log script, this will start the PCAPs collection threads and rotate the logs each 15 min as we defined in the prepare_pktcap script. The script should be left running in the SSH connection. if you stop it, the collection will also stop.

[root@esxedge1-02:~] /tmp/rotating_cap.py
Dump: 294272, broken : 0, drop: 0, file err: 0.
Dump: 175296, broken : 0, drop: 0, file err: 0.
Dump: 371584, broken : 0, drop: 0, file err: 0.
Dump: 317248, broken : 0, drop: 0, file err: 0.
Dump: 371648, broken : 0, drop: 0, file err: 0.
Dump: 317312, broken : 0, drop: 0, file err: 0.

Monitor the collection

The last script is for doing the monitoring of the sessions. You will need to open a new SSH session to the host to run this script.

[root@esxedge1-02:~] /tmp/pktcap_sessions.py -l
The vmci heap is 58% free
session    portID       devName
579        67108942
580        67108942
581        67108945
582        67108946
583        67108943
584        2214592556   vmnic2
585        67108946
586        2214592556   vmnic2
587        67108943
588        67108945
589        2214592558   vmnic1
590        2214592558   vmnic1

From the output, we can see that it collects the PortIDs that we have defined, but it also collects PCAP for the VMNics. This is valuable to us if we need to compare traffic on what is going in on the host and to the VM and visa versa.

Looking at the output that is stored on the datastore

[root@esxedge1-02:/vmfs/volumes/5fd89aca-1f6f344a-f65f-043f72c0064a/capture/42434546] ls
dc1esxedge1-02_2023-11-29T07_42_11_p67108942_d0_sna.pcap
          ....
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d0_sna.pcap.log      dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d0_sna.pcap_rot.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d0_sna.pcap_rot.log  dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap          dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap.log      dc1esxedge1-02_2023-11-29T07_42_11_pvmnic2_d1_sna.pcap_rot.log
dc1esxedge1-02_2023-11-29T07_42_11_p67108945_d1_sna.pcap_rot.log  
killfile

We can see the PCAPs for each of the PortIDs and VMnics, and it will rotate every 15 minutes.

Stopping the collection

You might have noticed the “killfile” from above. Remote this file and the collection will stop.

[root@esxedge1-02:~] rm -rf /vmfs/volumes/5fd89aca-1f6f344a-f65f-043f72c0064a/capture/42434546/killfile

Conclusion

The scripts are handy because they rotate the logs and have a way to monitor and kill the collections. This way we don’t have to manually kill processes on the ESXi host.

When the collection is done, you can copy out the logs for further analysis, I found the Filezilla SFTP client to be the fastest way of copying out the data.

If out want to merge the PCAPs afterward, on MacOS, you can do it with 

mergecap -w merged.pcap *.pcap

Troubleshooting

If you find that you can’t start the rotating logs script it might be because you have tried to start it before and it somehow stall. You can find the process IDs for it and kill it manually.

[root@esxedge1-02:~] ps -Tcjstv | grep -i rotating_cap

[root@esxedge1-02:~] kill

Download

prepare_pktcapDownload

Just Enoght vSphere rights for replication

Posted on by Jesper Ramsgaard

Just enough rights for a user to see and manage VMs in order to setup migration replication with either Azure Migrate or VMware Availability.

We are seeing more and more moving VMs around between providers, just a few years back this was not something that anybody wanted to go into, but the market is in a transition where offboarding is as important as onboarding. Good for the customer.

To ensure that the rights are just enough so that Azure Migrate or VMware Availability Onprem can’t see all VMs in your datacenter you need to limit the rights that the appliance is given.

To help out I have created a small permission script that can help with the setup of permissions.

It will create two roles, one global and one for the tenant resource group. Then it will setup the permissions so it’s just enough for the replication to work.

If you do not want the VCDA plugin into vCenter then you can remove the lines that define “Extension”

$roleGlobal = "vcda_repl_global"
$roleTenant = "vcda_repl_tenant"
$viserver = "vcsa1.home.lab"
$tenantRespool = "tenant1.comp1.01 (6284cdf1-cb7f-43bb-8e0f-09f439e09555)"
$vsphereUser = "tenant1_vcda"

Connect-VIServer -server $viserver

$roleIds = @()
$roleIds += "System.Anonymous"
$roleIds += "System.View"
$roleIds += "System.Read"
### Cryptographic Operations
$roleIds += "Cryptographer.ManageKeys"
$roleIds += "Cryptographer.RegisterHost"
### Datastore Privileges
$roleIds += "Datastore.Browse"
$roleIds += "Datastore.Config"
$roleIds += "Datastore.FileManagement"
### Extension Privileges - Not needed if you dont want plugin to vcenter 
$roleIds += "Extension.Register"
$roleIds += "Extension.Unregister"
$roleIds += "Extension.Update"
### Host Configuration Privileges
$roleIds += "Host.Config.Connection"
### Profile-driven Storage Privileges
$roleIds += "StorageProfile.View"
### Storage Views Privileges
$roleIds += "StorageViews"
### Host.Hbr.HbrManagement
$roleIds += "Host.Hbr.HbrManagement"

$roleIdsTenant = @()
### Resource Privileges
$roleIdsTenant += "Resource.AssignVMToPool"
### Virtual Machine Configuration Privileges
$roleIdsTenant += "VirtualMachine.Config.AddExistingDisk"
$roleIdsTenant += "VirtualMachine.Config.Settings"
$roleIdsTenant += "VirtualMachine.Config.RemoveDisk"
### Virtual Machine Inventory Privileges
$roleIdsTenant += "VirtualMachine.Inventory.register"
$roleIdsTenant += "VirtualMachine.Inventory.Unregister"
### Virtual Machine Interaction
$roleIdsTenant += "VirtualMachine.Interact.PowerOn"
$roleIdsTenant += "VirtualMachine.Interact.PowerOff"
### Virtual Machine State Privileges
$roleIdsTenant += "VirtualMachine.State.CreateSnapshot"
$roleIdsTenant += "VirtualMachine.State.RemoveSnapshot"
### Host.Hbr.HbrManagement
$roleIdsTenant += "VirtualMachine.Hbr.ConfigureReplication"
$roleIdsTenant += "VirtualMachine.Hbr.ReplicaManagement"
$roleIdsTenant += "VirtualMachine.Hbr.MonitorReplication"


New-VIRole -name $roleGlobal -Privilege (Get-VIPrivilege -Server $viserver -id $roleIds) -Server $viserver
Set-VIRole -Role $roleGlobal -AddPrivilege (Get-VIPrivilege -Server $viserver -id $roleIds) -Server $viserver

New-VIRole -name $roleTenant -Privilege (Get-VIPrivilege -Server $viserver -id $roleIdsTenant) -Server $viserver
Set-VIRole -Role $roleTenant -AddPrivilege (Get-VIPrivilege -Server $viserver -id $roleIdsTenant) -Server $viserver

$globalPrivileges = Get-VIPrivilege -Role $roleGlobal

$rootFolder = Get-Folder -NoRecursion
$permission1 = New-VIPermission -Entity $rootFolder -Principal (Get-VIAccount -Domain vsphere.local -User $vsphereUser ) -Role $roleGlobal -Propagate:$false

$tenantRespool = Get-ResourcePool -Name $tenantRespool
$permission1 = New-VIPermission -Entity  $tenantRespool -Principal (Get-VIAccount -Domain vsphere.local -User $vsphereUser ) -Role $roleTenant -Propagate:$true

Cloud Director – Tenant SAML SSO with AzureAD

Posted on by Jesper Ramsgaard

I have done this post before, but it was back in the day when we called VCD for vCloud, miss the name a bit but not the flash GUI. In the future Cloud Director will not support local users in organizations. Therefore, if the customer needs to have access to their organization it needs to be with an IDP identity provider for SAML or OAuth.

Many customers today have Microsoft 365 already and therefore also AzureAD which is a convenient choice for SAML integration For the non-AzureAD IDP admin, you should also be able to use this guide since it’s only claims and metadata.

This guide will focus on access assigned based on group membership, but you can also use roles. The main difference between roles and groups would be that roles can be more granular than group memberships. For example, you can have a role that gives access to expand a disk or delete a VM, the roles translate better than groups.

Microsoft have changed the name of AzureAD to Entra, this post will refer to both AzureAD but this also means Entra

Process

  • Create Enterprise app in desired resource AzureAD
  • Add entity ID and claims
  • Import AzureAD enterprise app federation metadata to Cloud Director
  • Assign allowed users/groups to roles in Cloud Director

Claims

Cloud Director is very picky about claims, VMware documentation gives a list of how the claims should look.

  • email address = “EmailAddress”
  • user name = “UserName”
  • full name = “FullName”
  • user’s groups = “Groups”
  • user’s roles = “Roles”

If you want the claim names with the full namespace, then you can map it in the Cloud Director SAML attribute mapper. Attribute mapper

AzureAD(Entra) setup

Enterprise Application creation

The basic setup of Entra is shown in this short story below. We will end up with an Enterprise App that is ready to setup.

Site icon
Cloud Director – Tenant SAML SSO with AzureAD
In Entra, find the Enterprise application and create a new one.

Enterprise app configuration

SAML Claims

Set up the claims according to the screenshot below. If you make claim names as in the screenshot then it will match Cloud Director so the SSO should work out of the box.

The only special setup is for the group claim where you should do the following

  • Choose “Groups assigned to the application”. If you have many groups in Entra SAML setup is only returning some of them. And if the group Cloud Director is looking for is not returned, then you are not logged in.
  • Source attribute of “Group ID”. Haven’t gotten it
  • Under Advanced options, choose “Customize the name of the group claim” and set it to “Groups” This will make sure Cloud Director knows that its groups.
If you are using a differnt IDP wher you cant alter the claim names, then you the attribut mapper in Cloud Diretor to translate the claimnames over to something that Cloud Director understands.

Cloud Director configuration

Go to tenant context and navigate to “Administration” > “SAML”. You might be hit with a prompt that the certificate is expired.

Hit the “Regenerate certificate” and afterward the “Configure” button and a popup show.

Fill out the wanted entity ID. I usually just use the URL for the tenant vcd. The important thing is just that the entity ID matches between Entra Enterprise App and Cloud Director.

Download the federation metadata XML from the AzureAD Enterprise application.

choose the “Federation Metadata XML” link

Upload the metadata to Cloud Director SAML configuration, enable the SAML identity provider, and press save.

We are now ready to upload the metadata from Cloud Director.

This can be found under “Administration” > “SAML” and “Retrieve Metadata”.

Now upload the metadata to the Enterprise Application

After the upload the Entity ID and reply URL will be filled out.

Save the uploaded metadata

Cloud Director SAML groups

For the Cloud Director to know what role the login user should have we need to tell it what group ID is associated with what role. We do that by fetching the ID of the group in AzureAD.

Copy the ID and go over to Cloud Director under “Administrator” > “Groups” > “Import Groups”

Paste in the ID and select the role followed by save. Using the Group ID helps if someone decides to change the name of the groups in Entra. For easier to see what group is behind the ID I tend to insert the group name(s) into the description after it has first been created.

Troubleshooting

There can be multiple places where problems can happen. Most of the time I start to look into a saml response, if that contains the groups that I expect then you will need to go on and look in the VCD logs.

SAML response

In your favorite browser, find developer mode and have it capture the traffic. Then log on to VCD have the error occur and then stop the capture again.

Capture the base64 encoded text and decode it in your favorite tool
<samlp:Response Destination="https://vcd.ramsgaard.dk/login/org/system/saml/SSO/alias/vcd" ID="_684262e1-e528-422f-8bf7-dd6d78f3ab49" InResponseTo="af3h19802aa811a5824j5b18aidee2" IssueInstant="2023-09-18T11:52:08.837Z" Version="2.0" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol">
	<Issuer xmlns="urn:oasis:names:tc:SAML:2.0:assertion">https://sts.windows.net/c5c123a4-e828-43bd-84d6-05cbfe1efae5/</Issuer>
	<samlp:Status>
		<samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/>
	</samlp:Status>
	<Assertion ID="_e96104c4-681e-4624-111e-a6e9a2bf6900" IssueInstant="2023-09-18T11:52:08.833Z" Version="2.0" xmlns="urn:oasis:names:tc:SAML:2.0:assertion">

We want to see a success for the SAML authentication. If we got that then it can be due to the claims being sent with it. 

			<Attribute Name="name">
				<AttributeValue>user@domain.tld</AttributeValue>
			</Attribute>
			<Attribute Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/UserName">
				<AttributeValue>user@domain.tld</AttributeValue>
			</Attribute>
			<Attribute Name="Groups">
				<AttributeValue>93547461-c174-4cab-961f-6f21c43051bd</AttributeValue>
				<AttributeValue>3c86569c-9e03-43f8-87fa-30b566ebb829</AttributeValue>
			</Attribute>
		</AttributeStatement>

In the snip above you can see that I got two group IDs back in the claim, check if these are the group IDs VCD is looking for. If not, then you need to investigate if the user is a member of the correct groups.

Conclusion

Since VCD local users are deprecated and will be removed in the future we need to convert over to use an IDP instead. AzureAD is a nice way to implement MFA and has a single identity for accessing corporate resources.

VCD – Force delete network

Posted on by Jesper Ramsgaard

In our v2t conversion, the NSX for Cloud Director migration tool has had some issues when doing cleanup. One of them is that it cant delete the old NSX-V backed network even though there is nothing left in VCD using it. The error message can be seen below. 

2023-05-22 10:54:28,551 [connectionpool]:[_make_request]:452 [DEBUG] [tenant.01] | https://vcd.ramsgaard.me:443 "DELETE /cloudapi/1.0.0/orgVdcNetworks/urn:vcloud:network:ce108a33-fa5c-4cae-8c16-60edd536ad20 HTTP/1.1" 400 None
2023-05-22 10:54:28,556 [vcdOperations]:[deleteOrgVDCNetworks]:1090 [DEBUG] [tenant.01] | Failed to delete Organization VDC Network lan.[ 1ca6fd03-de82-4835-b12e-58c5c043b2bc ] Network lan cannot be deleted, because it is in use by the following vApp Networks: lan.
2023-05-22 10:54:28,556 [vcdNSXMigratorCleanup]:[run]:230 [ERROR] [tenant.01] | Failed to delete Org VDC networks ['lan'] - as it is in use
Traceback (most recent call last):
  File "src\vcdNSXMigratorCleanup.py", line 218, in run
  File "<string>", line 1, in <module>
  File "src\core\vcd\vcdValidations.py", line 53, in inner
  File "src\core\vcd\vcdOperations.py", line 1094, in deleteOrgVDCNetworks
Exception: Failed to delete Org VDC networks ['lan'] - as it is in use

I found someone else having this problem, where they discovered a forceful way to delete the network. I have used this but wrapped it in Powershell instead. In my case, it can get the network URN from the log of the migration tools. Else you can also easily see the URN from the GUI URL when in the context of the network. 

### Variables
$vcdUrl = "https://vcd.ramsgaard.me"
$apiusername = "@system"
$password = ''
$networkUrn = "urn:vcloud:network:ce108a33-fa5c-4cae-8c16-60edd536ad20"

### Auth against API and enable TLS1.2 for PowerShell
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $apiusername,$password)))
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
$auth =Invoke-WebRequest -Uri "$vcdUrl/api/sessions" -Headers @{Accept = "application/*;version=36.0";Authorization="Basic $base64AuthInfo"} -Method Post

### Get VirtualWire
$virtualWire = Invoke-RestMethod -Uri "$vcdUrl/cloudapi/1.0.0/orgVdcNetworks/$($networkUrn)" -Headers @{Accept = "application/json;version=36.0";Authorization="Bearer $($auth.Headers.'X-VMWARE-VCLOUD-ACCESS-TOKEN')"} -Method GET
$virtualWire

### Delete VirtualWire
$deleteStatus = Invoke-RestMethod -Uri "https://vcd.hostcenter.dk/cloudapi/1.0.0/orgVdcNetworks/$($networkUrn)?force=true" -Headers @{Accept = "application/json;version=36.0";Authorization="Bearer $($auth.Headers.'X-VMWARE-VCLOUD-ACCESS-TOKEN')"} -Method DELETE
$deleteStatus

Above PowerShell is used at your own risk 🙂

Basic VMware PhotonOS config

Posted on by Jesper Ramsgaard

Still a bit new to PhotonOS. But it’s getting more and more that I use PhotonOS for deploying things with, like CSE for VCD.

I always struggle with the basics, even though I know a bit about Linux and Unix. So here is come quick tips on

  • First login
  • Network
  • SSH
  • Passwords
  • NTP

First login

When the VM is deployed you can now login to it with root and password “changeme”. It will then require you to change the password to something else.

Network

You can view network interfaces with networkctrl and IP a to show what is already configured.

If you want to set it to a static ip you have to create a new config file. 

cat > /etc/systemd/network/10-static-eth0.network << "EOF"
 
[Match]
Name=eth0
 
[Network]
Address=172.16.4.225/24
Gateway=172.16.4.1
DNS=172.16.4.10
Domains=home.lab
EOF

After the file has been created you can set the correct permissions and restart the network and resolver. If you skip the chmod you will probably see a fail in network reboot due to the system not being able to read the new file

chmod 644 /etc/systemd/network/10-static-eth0.network
systemctl restart systemd-networkd
systemdctl restart systemd-resolved

SSH config

When you want to connect to PhotonOS with a password but also have either your pageant or native ssh console running you will try to authenticate with a public/private key. If your key is not added to the server you will get a “Too many failed authentications”. To get around this you can use a parameter.

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no <hostname>

This will then make authentication with a password.

Forgot root password

If you forgot the root password and need to reset it you can do the following.

  1. reboot and press “e” to enter the grub boot loader. add the line rw init=/bin/bash shown on the screenshot. Press F10 to boot

2. You should now be in a shell on the system, from here you can run passwd

3. Unmount / and reboot the system

umount /
reboot -f

Disable password history

If you encounter the problem of trying to reset the password but the password you want to set says: Password has been already used. Choose another.

You can disable this by editing the /etc/pam.d/system-password.

By changing ‘remember ‘ from 5 to 0 we can disable the remember password count and reset the root password.

There are many parameters for password quality or history. One of the qualities is to check the password against dictionary attacks. This can be disabled with ‘dictcheck’ to 0.

Disable password expiration

chage -M 99999 root

Verify the change (optional): You can check the password settings by running chage -l root to confirm that the password expiration is set to "never". 

Paste into PhotonOS PuTTy session

When using vi there is a “bug” that prevents you from normal paste with right-click. To workaround this you need to write :set mouse= in vi command mode. After done you can not use paste with right-click.

Shift+Ins might be also used to paste on PuTTY.

NTP settings

Some apps running in PhotonOS are very time-sensitive, vcda is one of them. use watch -n 0,1 date on each of the appliances that need to communicate and verify that time is not skewed.

If you need to set up NTP post-deployment you can do as follows. Edit the timesyncd.conf file with a text editor such as vi:

vi /etc/systemd/timesyncd.conf

In the [Time] section edit the NTP entry with the correct NTP server address:

[Time]
#FallbackNTP=time1.google.com time2.google.com time3.google.com time4.google.com
NTP=ntpAddress

After having put in the NTP server you want to use restart the network and time sync service

systemctl restart systemd-networkd
systemctl restart systemd-timesyncd

Verify that the time on the appliance is now synchronizing with the NTP server.

Further troubleshooting is to see if the NTP service is running

systemctl status systemd-timesyncd

NSX 4.0.1 > 4.1.0 upgrade problems

Posted on by Jesper Ramsgaard

Precheck gave warnings back for all the edge nodes. Where it stated the problem below.

Edge node 4006d386-a394-43a4-6b04b242f8b3 vmId is not found on NSX Manager. Please refer to https://kb.vmware.com/s/article/90072

The KB article states that NSX managers are missing the VM_ID for the edge nodes and gave an example of how to manually find the Edge VM moref and post it to the NSX API.

Using PowerShell to update the VM_ID

Instead of the manual procedure from the KB, I made a small script.

### Login details
$nsxUsername = "admin"
$nsxPassword = "Yi....kes12!"
$nsxmanager = "nsxt.home.lab"

### Connect to vcenter so that we can fetch moref
Connect-VIServer vcsa1.home.lab

### NSX Manager auth header
$Type = "application/json;charset=UTF-8"
$Header = @{"Authorization" = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($nsxUsername + ":" + $nsxPassword)) }
$nsxUri = "https://$($nsxmanager)"

### Edge Vm moref update
$edgenodes =  (Invoke-RestMethod -Uri "$nsxUri/api/v1/transport-nodes" -Headers $Header -Method GET -ContentType $Type).results | Where-Object {$_.node_deployment_info.deployment_type -eq "VIRTUAL_MACHINE"}

### Loop through the edge nodes
foreach($edgenode in $edgenodes){
write-host "Updating edge node - $($edgenode.display_name)"
$specEdge =  (Invoke-RestMethod -Uri "$nsxUri/api/v1/transport-nodes/$($edgenode.id)" -Headers $Header -Method GET -ContentType $Type).node_deployment_info.deployment_config

$vmid = ((get-vm $edgenode.display_name).Id).Split("-")[-1]
write-host "Found edge node moref in vcenter - vm-$vmid)"

write-host "Removing form factore and adding vm_id to object)"
$specEdge | Add-Member -NotePropertyName vm_id -NotePropertyValue "vm-$vmid"
$specEdge = $specEdge | Select-Object -Property * -ExcludeProperty form_factor

try {
    write-host "Updating against NSX API"
    Invoke-RestMethod -Uri "$nsxUri/api/v1/transport-nodes/$($edgenode.id)?action=addOrUpdatePlacementReferences" -Headers $Header -Method POST -ContentType "application/json" -Body $($specEdge|ConvertTo-Json -Depth 10)
}
catch {
   $streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
   $ErrResp = $streamReader.ReadToEnd() | ConvertFrom-Json
   $streamReader.Close()
  }
if($ErrResp){
   write-host "$($ErrResp.error_message) - $($edgenode.display_name) not updated with success)" -ForegroundColor Red }
   else{
   write-host "$($edgenode.display_name) updated with success)"
   }
}

Unfortunately, even after updating vm_id the precheck still failed, with the same error. NSX API accepted with code 200 the post nothing happened behind the scenes

VMware support:

Opening a VMware SR, they got logs and the info that we already tried to update the vm_id. They asked us to do a couple of other things.

Refreshing the edge node config data

VMware SR stated us to try and do an API call for refreshing the edge node config data. VMware code NSX API info. With the info from the previous script, we can do a POST against NSX API. 

Invoke-RestMethod -Uri "$nsxUri/api/v1/transport-nodes/$($edgenode.id)?action=refresh_node_configuration&resource_type=EdgeNode&read_only=true" -Headers $Header -Method POST

Reboot of edge nodes

Put an Edge node into NSX maintenance mode and afterward do a reboot of the node, initiated from vCenter with a “Restart Guest OS”. The reboot went fine and the Edge node was put into production again.

Reboot of NSX Managers

Rebooting the NSX Managers, one at a time, and of cause waiting for the rebooted node to come back online with no errors before continuing with the next one.

Result

Unfortunately, non of the above helped, precheck still gives the same error.

Further troubleshooting:

Looking at the API guide I stumbled over an edge node redeployment call. I have redeployed the edge nodes manual before, and I have to say, it’s a pain! It’s not hard, but it takes a lot of time. But this call will help to redeploy in a way that doesn’t affect the data plane.

  • Edge is being put into NSX maintenance mode
  • Edge is then deleted and a new one is deployed, with the same naming as the old one.
  • After the edge node is deployed and registered in the manager it exits maintenance mode and goes into production again

Executing the API call

First, we get the config for the specific edge. Afterward, we post to the redeploy API with the body that we got from the config get request.

The try-catch is helping you get a better error description, if something goes wrong.

$redeployBody = (Invoke-RestMethod -Uri $nsxUri/api/v1/transport-nodes/$($edgenode.id) -Headers $Header -Method GET) 

try {
    Invoke-RestMethod -Uri "$nsxUri/api/v1/transport-nodes/$($edgenode.id)?action=redeploy" -Headers $Header -Method POST -ContentType "application/json" -body $redeployBody
    }
catch {
    $streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
    $ErrResp = $streamReader.ReadToEnd() | ConvertFrom-Json
    $streamReader.Close()
    $ErrResp
}

After redeployment, more problems….

After the redeployment of one edge node, the error with vm_id was no more for the redeployed node. Great! but now the upgrade coordinator gave a new precheck error… I can’t exactly remember, but it said something like “The Host Upgrade Unit Groups are not suitable for a T0.”

Google results pointed in the direction of VMware KB to reset the upgrade plan. But was not successful. (I then later found out that on the next page of the upgrade wizard, there is a “Reset plan” button)

Continued to redeploy all the remaining edge nodes, this helped clear all the errors from the upgrade precheck. The upgrade could then begin 😀

And another problem…

After the edge nodes that held the T0 gateway, one of the edge nodes was not negotiating up BGP to physical fabric. Tried a lot of things.

  • Redeployment of the edge node with the redeploy API action didn’t help.
  • Migrating the edge node to the same hosts as the working edge node was on, didn’t help.
  • Ping from working edge node to non-working on the VLAN uplink IPs worked.
  • Ping from non-working to physical fabric didn’t work.

Then tried to remove and add the interfaces of the T0 on the non-working edge node. Made everything work again. A quite random bug?

Conclusion

NSX upgrades compared to NSX-V upgrades seem to be quite troublesome or let’s say that there is room for improvements. The good thing is that when edge nodes are upgraded then all tenants are also. And the upgrade happens with zero downtime. The T0/T1 robustness is amazing, no drops, no IPSec go down.

VCD CPI/CCM – Load balancer

Posted on by Jesper Ramsgaard

When a TKG cluster is deployed by Container Service Extension(CSE) it means that it lives within VMware Cloud Director(VCD).

Inside this TKG cluster, you will find a Cloud Controller Manager(CCM) pod under kube-system called “vmware-cloud-director-ccm”. CCM pod is part of Cloud Provider Interface(CPI) that gives you some capabilities on how for example to add a Persistent Volume(PV) or do load balancing with NSX Advanced Loadbalancer(ALB). Basically, the CCM will contact the VCD CPI API and from there orchestrate what you requested in your Kubernetes YAML files.

At the time of writing, L4 load balancing(LB) features through ALB are the only option available. This is because the CPI is not yet completely featured to create L7 LB with ALB. It’s on the roadmap though.

One-arm vs. two-arm…

I found that there are two conspects that are with knowing of, one-arm and two-arm load balanceres.

The two LB methods are described here. Since it’s an old article, AVI/ALB was not in the VMware portfolio back then. And also NSX-T has migrated away from the LB service where it lived as haproxy within tier1 and over to AVI/ALB service engines.

The default setting of load balancing with VCD CPI is two arms, meaning that it will tell VCD to create a DNAT rule towards a 192.168.8.x internal subnet used to create ALB VIPs. 

WAN > T1 DNAT(185.139.232.x:80)> 192.168.8.x(LB internal subnet) > ALB SE > LB Pool members

Since L7 LB features in VCD are not yet available AND also it will become very costly. Most customers will probably choose to have an Nginx or Apache ingress controller inside their own Kubernetes cluster.

Since VCD 10.4 the two-arm config has been working and therefore it’s more desirable since you can use multiple ports on a single public IP. Where one-arm config would allocate one public IP pr Kubernetes service(correct me if I’m wrong).

If you are running your own ingress controller then some find the one-arm approach more desirable since ALB will then hold the public IP address.

WAN > T1 Static Route to ALB (185.139.232.x) > ALB SE > LB Pool members

How to change to one arm LB?

Hugo Phan has done a good write-up on this blog.

Basically, it’s downloading the existing config and changing the config map of VCD CPI removing the part where it 192.168.8.x subnet is defined. After this, you delete the existing CPI and then add it from the yaml file that you edited.

Snip from Hugo Phan blog – vmwire.com

How can I use/test this with my VCD TKG cluster?

If you don’t have a demo app that you prefer, then I can recommend either yelp or retrogames. Here I will do it with yelp. William Lam has done a good write-up and also hosts deployment files for yelp.

Step 1 – Deploy the application

kubectl create ns yelb
kubectl apply -f https://raw.githubusercontent.com/lamw/vmware-k8s-app-demo/master/yelb-lb.yaml

Step 2 – Check that all pods are running

jeram@QL4QJP2F4N ~ % kubectl -n yelb get pods
NAME                             READY   STATUS    RESTARTS   AGE
redis-server-74556bbcb7-f8c8f    1/1     Running   0          6s
yelb-appserver-d584bb889-6f2gr   1/1     Running   0          6s
yelb-db-694586cd78-27hl5         1/1     Running   0          6s
yelb-ui-8f54fd88c-cdvqq          1/1     Running   0          6s
jeram@QL4QJP2F4N ~ %

The deployment file is asking k8s for a service of a load balancer, CCM picks this up and asks VCD CPI to have ALB creating the L4 load balancing.

Task view from VCD

Step 3 – Get the IP and go check out the yelb site 

jeram@QL4QJP2F4N ~ % kubectl -n yelb get svc/yelb-ui
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)        AGE
yelb-ui   LoadBalancer   100.64.53.23   185.177.x.x   80:32047/TCP   5m52s

Step 4 – Scale the UI

Let’s see how many instances have from the initial deployment. 

jeram@QL4QJP2F4N ~ % kubectl get rs --namespace yelb
NAME                       DESIRED   CURRENT   READY   AGE
redis-server-74556bbcb7    1         1         1       8m11s
yelb-appserver-d584bb889   1         1         1       8m11s
yelb-db-694586cd78         1         1         1       8m11s
yelb-ui-8f54fd88c          1         1         1       8m11s
jeram@QL4QJP2F4N ~ %

We can then scale the UI to 3 and check again to see if that happens. 

jeram@QL4QJP2F4N ~ % kubectl scale deployment yelb-ui --replicas=3 --namespace yelb
deployment.apps/yelb-ui scaled

jeram@QL4QJP2F4N ~ % kubectl get rs --namespace yelb
NAME                       DESIRED   CURRENT   READY   AGE
redis-server-74556bbcb7    1         1         1       9m45s
yelb-appserver-d584bb889   1         1         1       9m45s
yelb-db-694586cd78         1         1         1       9m45s
yelb-ui-8f54fd88c          3         3         3       9m45s
jeram@QL4QJP2F4N ~ %

UI is now scaled to replicates of 3. Seen from the Load Balancer view I VCD it will only show the worker nodes. Since k8s is doing its own loadbalancing arose the pod instances.

Step 5 – Cleanup 

jeram@QL4QJP2F4N ~ % kubectl -n yelb delete pod,svc --all && kubectl delete namespace yelb
pod "redis-server-74556bbcb7-f8c8f" deleted
pod "yelb-appserver-d584bb889-6f2gr" deleted
pod "yelb-db-694586cd78-27hl5" deleted
pod "yelb-ui-8f54fd88c-6llf7" deleted
pod "yelb-ui-8f54fd88c-9r6wf" deleted
pod "yelb-ui-8f54fd88c-cdvqq" deleted
service "redis-server" deleted
service "yelb-appserver" deleted
service "yelb-db" deleted
service "yelb-ui" deleted
namespace "yelb" deleted
jeram@QL4QJP2F4N ~ %

Again, CCM will instruct VCD CPI to clean up. NICE!

Conclusion

We now have a good idea of how load balancing works with VCD TKG deployd K8s clusters. Off cause we are looking forward to the L7 features. But it’s a good start and VMware is working hard to help in making k8s deployment and day2 operations easier.

NSX-T – Topology view troubleshooting

Posted on by Jesper Ramsgaard

Ever seen the beneath picture when trying to see the cool topology view in NSX-T?

If so, there is an API call that can help you resync the topology view so you again can see it. 

# Login details
$nsxUsername = "admin",
$nsxPassword = "",
$nsxmanager = "nsxt.home.lab"

# NSX Manager auth header
$Type = "application/json;charset=UTF-8"
$Header = @{"Authorization" = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($nsxUsername + ":" + $nsxPassword)) }
$nsxUri = "https://$($nsxmanager)"

# Request topology resync
Invoke-RestMethod -Uri "$nsxUri/policy/api/v1/ui/network-topology/resync" -Headers $Header -Method POST -ContentType $Type

You don’t get any feedback from the API call, but after a few minutes, you will again be able to see the topology again.

NSX-V Edge – Disable ALG

Posted on by Jesper Ramsgaard

In NSX-V 6.4.11 and 6.4.12 there was introduced a bug to the Application Layer Gateway where it would cause package drop.

From the release notes of NSX-V 6.4.13, it states:

Fixed Issue 2886595: ALG-based services are not working in NSX Data Center for vSphere 6.4.11 and 6.4.12.:

Edge Firewall drops packets for ALG-based services FTP/SFTP/SNMP when using NSX Data Center for vSphere 6.4.11 and 6.4.12.

A temporary workaround is to disable ALG on the affected edge until the NSX-V installation can be upgraded to 6.4.13 or 6.4.14.

Procedure

  • Connect to the NSX Manager as admin and enter enable mode by typing: enable
    Enter engineering mode by typing: st en
  • Enter the NSX Manager root password: IAmOnThePhoneWithTechSupport
    Get the password for the Edge by typing:
    /home/secureall/secureall/sem/WEB-INF/classes/GetSpockEdgePassword.sh edge-ID
[root@nsxvmanager ~]# /home/secureall/secureall/sem/WEB-INF/classes/GetSpockEdgePassword.sh edge-1269
Edge root password:
        edge-1269       -> u8ORKdfFIM$hZ
[root@nsxvmanager ~]#
  • Access the Edge VM console, log in as the admin user and enter enable mode by typing: enable
  • Enable engineering mode by typing: debug engineeringmode enable
  • Enter the root shell on the Edge by typing the password obtained from the NSX manager: st en
  • Run commands as followings to make the workaround reboot permanent and to disable the ALG in the kernel without a reboot.
    echo “net.netfilter.nf_conntrack_helper = 1” >> /etc/sysctl.conf
    sysctl net.netfilter.nf_conntrack_helper=1

Conclusion

Since the procedure above is done directly on the edge it will not survive an edge redeploy. This is because a redeploy will take its configuration from the NSX manager and not look at what is done directly on the edge itself.

Off cause, the correct solution would be to have the NSX manager upgraded to the latest version and afterward upgrade the edge version.

NSX-T Edge node password management

Posted on by Jesper Ramsgaard

The default NSX-T password expiration time is set to 90 days. But in a lab environment, this is not required. So here is a bit on how to disable the timer but also how to recover from an expired or forgotten password.

Reset exipred password

If the password for either of the users, audit, root, or admin, has expired you will see it when you try to log in with SSH. It will then prompt you to enter the current password followed by the new two times. Since this is only for home lab, and would like the previous password, I set a new and quick-to-remember password. Fimmer_old_password1. The SSH session then disconnects and you start a new connection with the new password. 

nsx-edge> set user admin password My-New_VMware1!_Password old-password Fimmer_old_password1

After the reset and re-reset you now have 90 days of password again. or you could disable the password expiration…

If you find yourself in a situation of a forgotten admin password.

You will most likely be able to log in with the root account. Even if expired using the console of the Edge VM will always work. From there you can use the normal Linux password reset command to reset the admin account password.

passwd admin

And if you have tried the wrong password too much you can unlock the account with pam tally.

pam_tally2 --user admin --reset

Another note when you are logged in with root, users can still use nsxcli, just wrap your nsxcli commands with su admin -c ”

su admin '-c clear user audit password-expiration'

If your find yourself completely locked out of NSX-T

VMware has some good documentation on this. Basically it is

  • Connect to the console of the appliance and reboot the system. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. Press e to edit the menu. Press e to edit the selected option.
  • Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line. Press Ctrl-X to boot.

Set password to never expire

SSH to the edge node with the admin account. Using the nsxcli we can adjust the expiration to a maximum of 90 days. The commands below will set the password expiration to 9999 days and clear the expiration if already happened. VMware has it in their documentation here

nsx-edge> set user admin password-expiration 9999
nsx-edge> set user root password-expiration 9999
nsx-edge> set user audit password-expiration 9999
nsx-edge> clear user admin password-expiration
nsx-edge> clear user root password-expiration
nsx-edge> clear user audit password-expiration

SSH-keys

Something that is always better than passwords is SSH Keys. You can add multiple ssh-keys to the same users in NSX-T. The cool thing is that you have a label for the key so multiple users can have access with their own SSH key, this way you avoid some of the hassles of having to use passwords in with your SSH connections

nsx-edge> set user admin ssh-keys label jr type ssh-rsa value AAAAB3NzaC1yc2EAAAADAQABAAACAQC/VPq30qzyJHr8v6qh1vF1CVY8R9U09iCkqnIs9H6d9hBOeDu/e52rPj2BOQUfHwBmGRPVqZUyuOO20hDgT/BzP0QxISv9l2OpFariz8AmHu9m4kUwAdrBDvplw8fFeafppUwQF/aFsIF+t1PtFluz0Bp3N/sp3NQGWfkez7myctGc9X3eMc6oUAYrPPJeDZz1x5JoGdwdH/w6wjr3uK03kRx6TX1kNqxSypIQQ/8lYg1TG7yAuF5DhX4fJrPjpiLau1H6z0vChVpqY1q8oMntzHHtYtByFMrNtWFfAvG94BT27h/Lkmz5JM5d41TbL0YdZT8zCTrXzUG87wdEaRiB5ZeKy9LENgfxKO66scSU2gjiXwpyJTrHKZYz9g5EERH/41w+qMT90HAM3ArSIvk7pROoKhZy0IeOwfWbmMlQvKQFjS7OtKnFEeVUYRqnLvi3XeUiFbLxmW3ID8IqQy3iDNuESiVNRcp/PoN7lxL9cfGJdXBuJ3PBcaQZx/vQpePRqW9eBSmhhS1beIUlLV0UOFdRGTMMMjOlp7m7jaw5EnvztbInfPOdMPoUuSL9iGut7M0SVMgEzo0MiJDHNdLQYK0EKO8qrWMz76UHhpdnhOQNdi3/wtVVzxVUR/D9zBa1q2oL8ml7jKVubVbBd6Vm0lEEquDEN3I9Dan/Ev0j9w==

Conclusion

Having an expired password will cause you all sorts of trouble.

If you don’t have a PAM solution that can help you to automatically change the password, then setting the expiration to 9999 days will for sure help your manageability.

Putting your SSH key onto the nodes and managers will help you in the long run, and is in my opinion also a more secure solution than having passwords.

ESXi network routes

Posted on by Jesper Ramsgaard

I honestly don’t know why this is still a problem. Support for routed vmotion traffic was added back at vSphere6. Here we are vSphere7 and still have to set your gateway/routes for the vmotion stack through esxcli.

Either way, here is how it works

vMotion stack

Each tcp/ip stack can only have one gateway, that makes sense. And if you want to keep your management and vMotion traffic separated you need two tcp/ip stacks.

It’s nicely done through vSphere vCenter GUI and there is a KB for it. And you even have the option to override the default gateway and specify the right one for your vMotion stack. 

[root@dc1esxcompx-xx:~] esxcli network ip route ipv4 list -N vmotion
Network     Netmask        Gateway  Interface  Source
----------  -------------  -------  ---------  ------
10.1.115.0  255.255.255.0  0.0.0.0  vmk1       MANUAL

But when looking at the routing table from esxcli it is not set. If you know, feel free to give me a kick and enlighten me.

ESXCLI add a static route

So for me to set an actually default route I have to do it as shown below

[root@dc1esxcompx-xx:~] esxcli network ip route ipv4 add -g 10.1.115.1 -n 0.0.0.0/0 -N vmotion

[root@dc1esxcompx-xx:~] esxcli network ip route ipv4 list -N vmotion
Network     Netmask        Gateway     Interface  Source
----------  -------------  ----------  ---------  ------
default     0.0.0.0        10.1.115.1  vmk1       MANUAL
10.1.115.0  255.255.255.0  0.0.0.0     vmk1       MANUAL

PowerCLI

Need to do it on a cluster with multiple hosts? No problem LucD from VMware community got you covered. I only did a little customization and it works for my needs. 

connect-viserver -Server 

$stackName = 'mirror'
$ipGateway = '10.1.115.1'
$ipDevice = 'vmk3'
$cluster = "computexx"
$vmhosts = get-cluster $cluster | get-vmhost

foreach($vmhost in $vmhosts)
{
$esx = Get-VMHost -Name $vmhost
$netSys = Get-View -Id $esx.ExtensionData.ConfigManager.NetworkSystem
$stack = $esx.ExtensionData.Config.Network.NetStackInstance | where{$_.Key -eq 'mirror'}
$stack.IpRouteConfig.DefaultGateway

if($($stack.IpRouteConfig.DefaultGateway) -eq "10.1.146.1" ){
    write-host -ForegroundColor Green "Host $($vmhost.name) is ok"
}
else{
   write-host -ForegroundColor red "Host $($vmhost.name) is not ok, adding the route" 

   $config = New-Object VMware.Vim.HostNetworkConfig
   $spec = New-Object VMware.Vim.HostNetworkConfigNetStackSpec
   $spec.Operation = [VMware.Vim.ConfigSpecOperation]::edit
   $spec.NetStackInstance = $stack
   $spec.NetStackInstance.ipRouteConfig.defaultGateway = $ipGateway
   $spec.NetStackInstance.ipRouteConfig.gatewayDevice = $ipDevice
   $config.NetStackSpec += $spec
   $netsys.UpdateNetworkConfig($config,[VMware.Vim.HostConfigChangeMode]::modify)



   $stack = $esx.ExtensionData.Config.Network.NetStackInstance | where{$_.Key -eq 'mirror'}
   $stack.IpRouteConfig.DefaultGateway
   if($($stack.IpRouteConfig.DefaultGateway) -eq "10.1.146.1" ){
   write-host -ForegroundColor green "Host $($vmhost.name) is now ok" 
   }
   else{
        write-host -ForegroundColor red "Host $($vmhost.name) is still not ok"
   }
}
}

Conclusion

Manipulating the vmotion stack route table with either esxcli or PowerCLI is working great.

Need to know more? there are plenty of good bloggers and KBs out here.

ssacli – CLI configuring SmartArray

Posted on by Jesper Ramsgaard

When you install an HPE server with the VMware custom image for HPE servers you automatically get all the HPE tools for configuring the hardware. Neat.

Here is a small guide on how to clean and setup the array. The ssacli is located in /opt/smartstorageadmin/ssacli/bin on the ESXi serve – start here and follow the next commands

Show the existing config:

[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli ctrl slot=0 ld all show
Smart Array P440ar in Slot 0 (Embedded)
   Array A
      logicaldrive 1 (279.37 GB, RAID 1, Failed)
   Array B
      logicaldrive 2 (3.27 TB, RAID 1+0, OK)

Delete the existing logical drives:

[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli ctrl slot=0 ld 1 delete
Warning: Deleting an array can cause other array letters to become renamed.
         E.g. Deleting array A from arrays A,B,C will result in two remaining
         arrays A,B ... not B,C 
Warning: Deleting the specified device(s) will result in data being lost.
         Continue? (y/n)  y
[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli ctrl slot=0 ld 2 delete
Warning: Deleting the specified device(s) will result in data being lost.
         Continue? (y/n)  y
[root@esx2:/opt/smartstorageadmin/ssacli/bin]

Show all physical drives in the server:

[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
   Array A
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:14 (port 1I:box 1:bay 14, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:15 (port 1I:box 1:bay 15, SAS HDD, 900 GB, OK)
      physicaldrive 1I:1:16 (port 1I:box 1:bay 16, SAS HDD, 900 GB, OK)

Create a new volume with available physical drives:

[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli ctrl slot=0 create type=ld drives=1I:1:9,1I:1:10,1I:1:11,1I:1:12,1I:1:13,1I:1:14,1I:1:15,1I:1:16 raid=6

Warning: Controller cache is disabled. Enabling logical drive cache will not take effect until this has been resolved.
[root@esx2:/opt/smartstorageadmin/ssacli/bin]

Conclusion

Nice and easy – Give the HPE ssacli manual a read for more commands, starting page 57 and forward or use ssacli help.

[root@esx2:/opt/smartstorageadmin/ssacli/bin] ./ssacli help

CLI Syntax
   A typical SSACLI command line consists of three parts: a target device, 
   a command, and a parameter with values if necessary. Using angle brackets to
   denote a required variable and plain brackets to denote an optional 
   variable, the structure of a typical SSACLI command line is as follows:

      <target> <command> [parameter=value]

   <target> is of format:
      [controller all|slot=#|serialnumber=#]
      [array all|<id>]
...........