VCD – remove org and its items

Having a nice VRO job to create the VCD tenants with its VDCS, Edges and networks are nice. When having to clean up after testing its a pain to click through the GUI to first remote networks, then edges, then disable VDC, delete it, and final delete the org. A bit of PowerShell fu can help with the task, this is a quick and dirty script set of commands, but it works as intended.

Cloud Director 10.1 released

Been using vCloud since version 5.1. After a brief love affair with something called “Azure Pack” we put all our focus into vCloud.

8.20 was the first sign of heartbeats coming from VCD. We got confirmation that vCloud was for sure the platform that we were and had been looking for. Now we see the 10.1 released and from my point of view it’s a big one, may things change in GUI as in infrastructure. This release is also the final farewell to the old flex GUI.

First off we have to address the naming, I always liked the vCloud term, for me a strong brand. So a bit sad to see that go and now we have to get used to the Cloud Director instead. Thankfully we can still use the acronym VCD for VMware Cloud Director. #LongLiveVCD.

In the next few points, I will address some of the major things within this release.

APIs

We use a lot of the functionality of the APIs of VCD. Since we see that the development of VCD is changing into higher gear, so is the deprecation of the older API versions. For a small service provider, it’s always hard to revisit automation already working with existing APIs. When going on board 10.1 we have to go through a couple of workflows to update the to use the new 34.0 API. But on the other side, it’s also a good chance to refactor and optimize.

  • VMware Cloud Director API version 29 and below are not supported.
  • VMware Cloud Director API version 30.0 is deprecated and will become unsupported after VMware Cloud Director 10.1
  • VMware Cloud Director API version 31.0 is deprecated.

NSX-T feature improvements

More of the core NSX-T features is now available through VCD.

  • IPSec VPN
  • Dedicated External Network
  • BGP and Route Advertisement

We have been looking from the side for NSX-T development to reach an acceptable level for some time. NSX-V is still doing a good job. As someone who right now is standing up a new 16 node VMware cluster as a new provider VDC, I would have wished for it to be 6 months later so that all NSX-T functionality was ready and we could hopefully solo use NSX-T.

But we have to look into maybe having two 8 node clusters for NSX-V and on for NSX-T so we can already now start to transition to NSX-T…

But the good thing about being a VMware customer is that you are not left in the dust. There have been already been created migration tools for NSX-V > NSX-T, NSX-T Data Center Migration Coordinator, but it had no integration to VCD. which bring me to the next point!

NSX-V to NSX-T VCD Migration Tool

This is a way of helping us transition from NSX-V to NSX-T as we are seeing NSX-V lacking to the end of support in January 2021.

Before we could still do a new provider VDC that was backed by NSX-T controller and then start to move workloads over to the new cluster and at the time had to use NSX-T functionality, but all in a manual process.

There is now an automated way to do it, which is VCD aware. The approach will require a new cluster since NSX-V and NSX-T can’t coexist in the same cluster. From the Whats New in 10.1 it stats that the workflow will help with following

  • Automates migration of vCD metadata and workloads from NSX-V to NSX-T
  • Migrate per Org VDC migration to reduce maintenance window to single tenant
  • Minimize network downtime with bridged networks during migration
  • Live migrate with vMotion to ensure non-disruption to user workloads
  • Keep source VDC configuration and environment as-is to allow rollback
Before live migration
After live migration

Tomas did a good discussion on this subject

SSL and Certificate Management

This seems like something to read up on carefully. In short, VCD does not trust endpoint certificates unless they have been imported to the trust store.

There is a tool helping with the import, trust-infra-certs, that automatically connect to the endpoint, grabbing and importing the certificate. If this is not done successfully you will not be able to talk to those endpoints after upgrading to VCD 10.1.

App Launchpad

A new feature to help introduce a marketplace with the help of the content from Bitnami. From there we can now offer customers to easily find, deploy and manage new workloads. Not just as VMs but also as containers.

Daniel did an excellent write up on this subject.

Conclusion

There is still a lot more in this release to talk about, CSE2.6, OSE1.5, Terraform 2.7 provider, etc. read more from the official release notes.

Might have had to write a disclaimer for the length of this post and the lack of interesting pictures, will try to improve for next time.

I love to see VCD take flight. We are looking forward being part of the future journey where things like Bitnami and App Launchpad together with more NSX-T functionality and a whole lot of other features helps us Cloud Providers to help other business to there digital transformation .

Big shout out to VMware and the VCD team!

vCloud SAML authentication – Automation

Cant say that I did everything by my self in this post, I had a great great help from my college and friend Kasper Hansen. Also gotten a great help from the vExpert community, especially Tom Fojta.

In my last post I found out how to setup vCloud SAML against AzureAD. Now we are gonna look on how to automate each tenant to use the same AzureAD. In these days everybody have either a Microsoft og AzureAD account, so this way its easy to invite them as guest users and this way have controlled access but also ensure that vCloud users have MFA enabled.

We use VRO for the creation of vCloud tenants, in this flow we are now going to introduce a new workflow that will do following. Although the workflow is just a restcall to trigger an event in Azure Automation.

  • Create AzureAD Groups for admin and viewer
  • Post federation metadata to vCloud tenant
  • Post federation groups to vCloud tenant

Enable SAML

Fojta have some very good articles on his blog on the basic setup of SAML to different IDP systems. I also did a piece on it where Azure where the IDP provider.

Because we want to have all organisations linked to the same SAML app in Azure we need to have the same SAML certificate on all organisations. You can only do this with the API, but what the documentation did not say was that the certificate needs to be trusted by the keystore, the java keystore of the cells.

Create a self-signed certificate and make vCloud trust it

These commands will help you create a certificate and a private key in the needed pkcs8 format and certificate in the x509 format.

When you have done the new self-signed certificate you need to import it to each and one of your cells. After import you will need to restart the cells. One of the errors I did here way that I tried to import a .pem where the private key and certificate where combines, that won’t work. Only import the certificate.

Publish federation settings to API

We where having a lot of trial and error in this step, because that vCloud did not trust the certificate. Each time a put where done to the API the log complained. /opt/vmware/vcloud-director/logs/vcloud-container-debug.log it showed “Failed to generate keystore | requestId=<id>,request=PUT.”

Following is a example done in PowerShell, insert your self-signed certificate where it says —–END/BEGIN CERTIFICATE—–

Publish SAML groups to API

Now that the SAML metadata/certificate is uploaded and in place we need to add groups to tenant. You can read more about what groups/users should be imported in my other SAML blog post.

Each tenant have its own role ids, so when doing automation with group import we need to query the vCloud API and get the role ids. There is a specific query API to get data. When using a system account we need to specify a “VCLOUD-TENANT-CONTEXT” in the header of the request. This we we can query a tenant context from a system account.

After we got the role id we can now send up the group together with the role id and this was be able to authenticate based on a SAML group from AzureAD.

Conclusion

Now we have all the pieces for making automation where we can enable a tenant for SAML authentication and afterwards import f.eks. a viewer and admin group. External users will then be invited to the AzureAD, imported into the right group and now they have access to the their tenant. We can help the organisation secure the access to their virtual datacenter with MFA and they will have single sign-on with there own user that originates from their own AzureAD or Microsoft account.

A service library will be made where users can be invited to the tenant organisation. So that when one user have been invited that user will be able to invite its colleagues.

vCloud – Changing SSL certificate

In this post, I will explain how to install a public certificate into vCloud Director cell(s). This exact environment has a public signed cert that is up for renewal. A new certificate has been bought and signed and is ready to import.

vcd cells have 2 IP addresses that allow support for 2 different SSL endpoints (http and consoleproxy). Each endpoint requires its own SSL certificate. vCloud Director uses a java keystore to read its SSL certificates from.  In a multi-cell environment, you need to create 2 certificates for each cell and import the certificates into the vcd java keystore. But since we hare here using a wildcard certificate the same certificate will be used to but endpoints.

The new certificate have been created with a CSR that was not generated from the vCloud cells, so we need to import both private and public key from an export of the certificate. In this case it’s a .PFX.

Certificate is a wildcard. If you are using a UCC SAN certificate with the exact names then be sure that the names in certificate are matching accordingly to vCloud settings.

I assume you got

  • Already working/configured vCloud environment
  • New public signed certificate exported to a .PFX format (contains both public and private key)

We will

  • Connect to cell with winscp and transfer the .PFX to /tmp/
  • Connect to cell with Putty
    • Create a new keystore with the new certificate
    • Stop vcd service
    • Swap old keystore with new
    • Start vcd service

Initialize certificate change…

winscp copy the .pfx to the cell tmp directory

The commands for creating the new keystore and importing the cert is below. Change the STOREPASS and KEYPASS to something meaningful for your environment. It is also important to notice that the alias of each certificate must be “http” and “consoleproxy”. Else vcd won’t find the certs.

### Stop vCloud Director service
service vmware-vcd stop

### Add the certificate to a new created certificates.ks keystore.
/opt/vmware/vcloud-director/jre/bin/keytool \
-keystore /tmp/certificates.ks \
-alias http \
-storepass STOREPASS \
-keypass KEYPASS \
-storetype JCEKS \
-importkeystore \
-srckeystore /tmp/wildcard2020.pfx

### List certificate alias and rename
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-storepass passwd \
-keystore /tmp/certificates.ks \
-list | grep -i alias

/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-changealias \-alias "te-d487d1c7-2c76-482a-8e61-69107ee3027f" \
-destalias http -keystore /tmp/certificates.ks

### Add the Remote Console Proxy certificate to a new created certificates.ks keystore.
/opt/vmware/vcloud-director/jre/bin/keytool \
-keystore /tmp/certificates.ks \
-storepass STOREPASS \
-keypass KEYPASS \
-storetype JCEKS \
-importkeystore \
-srckeystore /tmp/wildcard2020.pfx

### List certificate alias and rename
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-storepass passwd \
-keystore /tmp/certificates.ks -list | grep -i alias

/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-changealias \
-alias "te-d487d1c7-2c76-482a-8e61-69107ee3027f" \
-destalias consoleproxy \
-keystore certificates.ks

### Make a backup of the existing keystore 
cd /opt/vmware/vcloud-director
cp certificates.ks certificates.ks_old 

### Copy the new keystore file to the vCloud Director environment
cp /tmp/certificates.ks /opt/vmware/vcloud-director/certificates.ks

### Set correct permissions to the keystore file
chown vcloud:vcloud /opt/vmware/vcloud-director/certificates.ks
chmod 600 /opt/vmware/vcloud-director/certificates.ks

### Make cells generate proxy console certs based on new keystore.
./cell-management-tool certificates -p -k /opt/vmware/vcloud-director/certificates.ks -w KEYSTORE_PASSWD

### Start vCloud Director service
service vmware-vcd start

To see if the cell have booted correctly you can tail the cell log. It will give you a “startup completed in x”.

 tail -f /opt/vmware/vcloud-director/logs/cell.log

I got more than one cell…

That’s awesome – me too. You can scp the certificate from the cell with the new cert to the other cells. So let’s get that newly created keystore over the other cells.

### SSH to next cell and stop the vCloud Director service
service vmware-vcd stop

### Go to keystore path
cd /opt/vmware/vcloud-director/

### Move the existing to new filename with .old surfix.
mv certificate.ks certificate.ks.old

### Copy the new certificate into place
scp root@dc1svcdcell01:/opt/vmware/vcloud-director/certificates.ks .

### Make cells generate proxy console certs based on new keystore.
./cell-management-tool certificates -p -k /opt/vmware/vcloud-director/certificates.ks -w KEYSTORE_PASSWD

### Start vCloud Director service
service vmware-vcd start

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.