Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here:
Cookie Policy
Content switching is used to direct incoming traffic to different pools/servers based on the content of the requests. By looking at the application layer (L7 in the OSI model) we can inspect the client request content, such as URL, header, cookies, queries, etc.
ALB has the concept of policies on virtual services. Each policy rule can have a match and action to it. So we can match on a host headerhttp request and then do a content switchaction afterward.
This is, unfortunately, a feature that is not yet supported by Cloud Director(10.5.1). But it can be done in the ALB manager, behind the back of VCD. Rules are in the VCD UI, but not the content of the policy.
My guess is therefore that when it comes to VCD UI in the future, it will be able to adopt the existing content-switching policies.
If you also think this feature should be implemented in VCD, please do a feature request on this site.
Workaround
In Cloud Director you setup the pools you want to do content switching on. From the screenshot below you can see port80 and port443 are the ones being used to day. Creating the extra pools here will have VCD aware of the pool, so we only need to change a bit on existing objects in the ALB manager.
In the ALB manager, find the virtual service, edit it, and navigate to the policy section. Here we can see the different policy types. Find “HTTP request” and add to new rules.
Each rule will have the match and action point as mentioned before. When doing the action content switch get to choose a specific pool.
Save it all and you are ready to test the content switching. From the screenshot below, you can see how it will look inside the VCD UI.
NSX V2T migration
The reason for me to research how to do content switching was primarily that some tenants are still using NSX-V because they use haproxy application rules to do content switching. NSX migration for Cloud Director tool does support basic load balancing migrations. But not when there are application rules applied.
Now tenant can schedule the service window, remove the application rules, and have the migration tool migrate all nat, firewall, routes, and basic load balancing.
After the migrations are done the content switching rules can be created manually, from what the application rules in haproxy specified.
Conclusion
Even if it’s not supported by VCD yet, we can still do it. Ofcause the tenants can’t do it themself but will need to log a support case with you until the feature is introduced. And when they do, then let’s hope it will adopt the rules being done behind its back.
Having to deploy multiple Cloud Director environments can be tedious, especially when OVF deployment in vCenter fails with some obsure error all the time. But using powershell to deploy is smooth sailing.
Here you will get the small piece of PowerShell to do so. Its very basic and but works everytime.
Just enough rights for a user to see and manage VMs in order to setup migration replication with either Azure Migrate or VMware Availability.
We are seeing more and more moving VMs around between providers, just a few years back this was not something that anybody wanted to go into, but the market is in a transition where offboarding is as important as onboarding. Good for the customer.
To ensure that the rights are just enough so that Azure Migrate or VMware Availability Onprem can’t see all VMs in your datacenter you need to limit the rights that the appliance is given.
To help out I have created a small permission script that can help with the setup of permissions.
It will create two roles, one global and one for the tenant resource group. Then it will setup the permissions so it’s just enough for the replication to work.
If you do not want the VCDA plugin into vCenter then you can remove the lines that define “Extension”
I have done this post before, but it was back in the day when we called VCD for vCloud, miss the name a bit but not the flash GUI. In the future Cloud Director will not support local users in organizations. Therefore, if the customer needs to have access to their organization it needs to be with an IDP identity provider for SAML or OAuth.
Many customers today have Microsoft 365 already and therefore also AzureAD which is a convenient choice for SAML integration For the non-AzureAD IDP admin, you should also be able to use this guide since it’s only claims and metadata.
This guide will focus on access assigned based on group membership, but you can also use roles. The main difference between roles and groups would be that roles can be more granular than group memberships. For example, you can have a role that gives access to expand a disk or delete a VM, the roles translate better than groups.
Microsoft have changed the name of AzureAD to Entra, this post will refer to both AzureAD but this also means Entra
Process
Create Enterprise app in desired resource AzureAD
Add entity ID and claims
Import AzureAD enterprise app federation metadata to Cloud Director
Assign allowed users/groups to roles in Cloud Director
Claims
Cloud Director is very picky about claims, VMware documentation gives a list of how the claims should look.
email address = “EmailAddress”
user name = “UserName”
full name = “FullName”
user’s groups = “Groups”
user’s roles = “Roles”
If you want the claim names with the full namespace, then you can map it in the Cloud Director SAML attribute mapper. Attribute mapper
AzureAD(Entra) setup
Enterprise Application creation
The basic setup of Entra is shown in this short story below. We will end up with an Enterprise App that is ready to setup.
Set up the claims according to the screenshot below. If you make claim names as in the screenshot then it will match Cloud Director so the SSO should work out of the box.
The only special setup is for the group claim where you should do the following
Choose “Groups assigned to the application”. If you have many groups in Entra SAML setup is only returning some of them. And if the group Cloud Director is looking for is not returned, then you are not logged in.
Source attribute of “Group ID”. Haven’t gotten it
Under Advanced options, choose “Customize the name of the group claim” and set it to “Groups” This will make sure Cloud Director knows that its groups.
If you are using a differnt IDP wher you cant alter the claim names, then you the attribut mapper in Cloud Diretor to translate the claimnames over to something that Cloud Director understands.
Cloud Director configuration
Go to tenant context and navigate to “Administration” > “SAML”. You might be hit with a prompt that the certificate is expired.
Fill out the wanted entity ID. I usually just use the URL for the tenant vcd. The important thing is just that the entity ID matches between Entra Enterprise App and Cloud Director.
Download the federation metadata XML from the AzureAD Enterprise application.
Upload the metadata to Cloud Director SAML configuration, enable the SAML identity provider, and press save.
We are now ready to upload the metadata from Cloud Director.
Now upload the metadata to the Enterprise Application
After the upload the Entity ID and reply URL will be filled out.
Cloud Director SAML groups
For the Cloud Director to know what role the login user should have we need to tell it what group ID is associated with what role. We do that by fetching the ID of the group in AzureAD.
Copy the ID and go over to Cloud Director under “Administrator” > “Groups” > “Import Groups”
Paste in the ID and select the role followed by save. Using the Group ID helps if someone decides to change the name of the groups in Entra. For easier to see what group is behind the ID I tend to insert the group name(s) into the description after it has first been created.
Troubleshooting
There can be multiple places where problems can happen. Most of the time I start to look into a saml response, if that contains the groups that I expect then you will need to go on and look in the VCD logs.
SAML response
In your favorite browser, find developer mode and have it capture the traffic. Then log on to VCD have the error occur and then stop the capture again.
In the snip above you can see that I got two group IDs back in the claim, check if these are the group IDs VCD is looking for. If not, then you need to investigate if the user is a member of the correct groups.
Conclusion
Since VCD local users are deprecated and will be removed in the future we need to convert over to use an IDP instead. AzureAD is a nice way to implement MFA and has a single identity for accessing corporate resources.
Since my last article on how to update Cloud Director SSL certificates, there has been a major change. No more binary java truststore – jaaay.
Cloud Director has changed over too, what I think, is a better and more normal way of storing the private and public keys, which is in PEM format. From release notes, the change actually happened in 10.2, but the certificate path changed again in 10.3. If you are in doubt of where the certificate path is then look inside global.properties
/opt/vmware/vcloud-director/etc/global.properties
VMware’s own documentation state that we can now just swap the .pem files, use the cell-management tool to import and restart the cell.
What we will do and what is needed
Get a new public signed certificate
Either in PEM format as .key and .pem(certificate including intermediate)
Or in PFX so it can be exported
Backup existing certificates
Replace existing certificates with your new certificate
Run VCD tool to import and define the private key encryption password
Restart cell(s)
Process
If you have a pfx you can use this article to extract the key and cert. If you already have the two files, .key end .pem then you can proceed.
We will follow VMware documentation and create a backup of the existing files.
Now we can wither SCP in our key and certificate or edit and replace the content of the files on the server by copying and pasting in content from the files you have. Whatever you find to be the easiest.
Forgot your root password for the Cloud Director appliance, off cause not. But anyway, here is a link to reset it....
After the “user.http.pem/key” and “user.consoleproxy.pem/key” files have been updated with the new certificate data we can tell Cloud Dictor to update its config with the commands below. This is done to update the encryption password for the private key.
If you don’t care about security you can also update without –key-password, then off cause your private key will need to be in an unencrypted format in the .key files.
VMware has made it much easier to change a certificate in Cloud Director. The new way of storing certificates is a warm welcome change.
I did see a few different placements for the .key and .pem files depending on versions or if the cells have been created with raw Linux or an appliance, but you can always look in the conflig file placed in the same folder as the certificates.
Finding free public IPs in Cloud Director backed by NSX-V is not as easy as it should be. Some people will tell you to ping the scope and see what’s responding. But pinging is not reliable was of finding free IPs. Not every device is responding to ICMP messages.
Somewhere along the line, I found a guy on the VMware forum posting a script for finding available IPs in Cloud Director using the PowerCLI module for querying VCD and getting back IPs that are not allocated by an Edge. I have been using the script quite a bit since. His blog is not available today, but the code is still on the forum.
Now it’s also available here on the site with a bit more explanation on how to connect and use the function. I have been using it with NSX-V as backend, haven’t tried it with NSX-T at the network backend yet.
You can help yourself by copy and pasting the code snip into either PowerShell ISE or VisualCode. And since you need to install a cmdlet you need to run it with elevated rights. If you get a red message with importing the module it’s probably because of execution rights, you then need to run to command beneath. This is for allowing remote signed cmdlets to be executed.
Having a nice VRO job to create the VCD tenants with its VDCS, Edges and networks are nice. When having to clean up after testing its a pain to click through the GUI to first remote networks, then edges, then disable VDC, delete it, and final delete the org. A bit of PowerShell fu can help with the task, this is a quick and dirty script set of commands, but it works as intended.
Been using vCloud since version 5.1. After a brief love affair with something called “Azure Pack” we put all our focus into vCloud.
8.20 was the first sign of heartbeats coming from VCD. We got confirmation that vCloud was for sure the platform that we were and had been looking for. Now we see the 10.1 released and from my point of view it’s a big one, may things change in GUI as in infrastructure. This release is also the final farewell to the old flex GUI.
First off we have to address the naming, I always liked the vCloud term, for me a strong brand. So a bit sad to see that go and now we have to get used to the Cloud Director instead. Thankfully we can still use the acronym VCD for VMware Cloud Director. #LongLiveVCD.
In the next few points, I will address some of the major things within this release.
APIs
We use a lot of the functionality of the APIs of VCD. Since we see that the development of VCD is changing into higher gear, so is the deprecation of the older API versions. For a small service provider, it’s always hard to revisit automation already working with existing APIs. When going on board 10.1 we have to go through a couple of workflows to update the to use the new 34.0 API. But on the other side, it’s also a good chance to refactor and optimize.
VMware Cloud Director API version 29 and below are not supported.
VMware Cloud Director API version 30.0 is deprecated and will become unsupported after VMware Cloud Director 10.1
VMware Cloud Director API version 31.0 is deprecated.
NSX-T feature improvements
More of the core NSX-T features is now available through VCD.
IPSec VPN
Dedicated External Network
BGP and Route Advertisement
We have been looking from the side for NSX-T development to reach an acceptable level for some time. NSX-V is still doing a good job. As someone who right now is standing up a new 16 node VMware cluster as a new provider VDC, I would have wished for it to be 6 months later so that all NSX-T functionality was ready and we could hopefully solo use NSX-T.
But we have to look into maybe having two 8 node clusters for NSX-V and on for NSX-T so we can already now start to transition to NSX-T…
But the good thing about being a VMware customer is that you are not left in the dust. There have been already been created migration tools for NSX-V > NSX-T, NSX-T Data Center Migration Coordinator, but it had no integration to VCD. which bring me to the next point!
NSX-V to NSX-T VCD Migration Tool
This is a way of helping us transition from NSX-V to NSX-T as we are seeing NSX-V lacking to the end of support in January 2021.
Before we could still do a new provider VDC that was backed by NSX-T controller and then start to move workloads over to the new cluster and at the time had to use NSX-T functionality, but all in a manual process.
There is now an automated way to do it, which is VCD aware. The approach will require a new cluster since NSX-V and NSX-T can’t coexist in the same cluster. From the Whats New in 10.1 it stats that the workflow will help with following
Automates migration of vCD metadata and workloads from NSX-V to NSX-T
Migrate per Org VDC migration to reduce maintenance window to single tenant
Minimize network downtime with bridged networks during migration
Live migrate with vMotion to ensure non-disruption to user workloads
Keep source VDC configuration and environment as-is to allow rollback
This seems like something to read up on carefully. In short, VCD does not trust endpoint certificates unless they have been imported to the trust store.
There is a tool helping with the import, trust-infra-certs, that automatically connect to the endpoint, grabbing and importing the certificate. If this is not done successfully you will not be able to talk to those endpoints after upgrading to VCD 10.1.
App Launchpad
A new feature to help introduce a marketplace with the help of the content from Bitnami. From there we can now offer customers to easily find, deploy and manage new workloads. Not just as VMs but also as containers.
There is still a lot more in this release to talk about, CSE2.6, OSE1.5, Terraform 2.7 provider, etc. read more from the official release notes.
Might have had to write a disclaimer for the length of this post and the lack of interesting pictures, will try to improve for next time.
I love to see VCD take flight. We are looking forward being part of the future journey where things like Bitnami and App Launchpad together with more NSX-T functionality and a whole lot of other features helps us Cloud Providers to help other business to there digital transformation .
Cant say that I did everything by my self in this post, I had a great great help from my college and friend Kasper Hansen. Also gotten a great help from the vExpert community, especially Tom Fojta.
In my last post I found out how to setup vCloud SAML against AzureAD. Now we are gonna look on how to automate each tenant to use the same AzureAD. In these days everybody have either a Microsoft og AzureAD account, so this way its easy to invite them as guest users and this way have controlled access but also ensure that vCloud users have MFA enabled.
We use VRO for the creation of vCloud tenants, in this flow we are now going to introduce a new workflow that will do following. Although the workflow is just a restcall to trigger an event in Azure Automation.
Create AzureAD Groups for admin and viewer
Post federation metadata to vCloud tenant
Post federation groups to vCloud tenant
Enable SAML
Fojta have some very good articles on his blog on the basic setup of SAML to different IDP systems. I also did a piece on it where Azure where the IDP provider.
Because we want to have all organisations linked to the same SAML app in Azure we need to have the same SAML certificate on all organisations. You can only do this with the API, but what the documentation did not say was that the certificate needs to be trusted by the keystore, the java keystore of the cells.
Create a self-signed certificate and make vCloud trust it
These commands will help you create a certificate and a private key in the needed pkcs8 format and certificate in the x509 format.
### Create the self-signed private key and certificate
jr@mbp:~ jr$ openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout selfsigned.key.pem -out selfsigned-x509.crt
### Convert the private key to pkcs8 format
jr@mbp:~ jr$ openssl pkcs8 -topk8 -inform PEM -outform PEM -in
selfsigned.key.pem -out selfsigned-pkcs8.key -nocrypt
When you have done the new self-signed certificate you need to import it to each and one of your cells. After import you will need to restart the cells. One of the errors I did here way that I tried to import a .pem where the private key and certificate where combines, that won’t work. Only import the certificate.
We where having a lot of trial and error in this step, because that vCloud did not trust the certificate. Each time a put where done to the API the log complained. /opt/vmware/vcloud-director/logs/vcloud-container-debug.log it showed “Failed to generate keystore | requestId=<id>,request=PUT.”
Following is a example done in PowerShell, insert your self-signed certificate where it says —–END/BEGIN CERTIFICATE—–
Now that the SAML metadata/certificate is uploaded and in place we need to add groups to tenant. You can read more about what groups/users should be imported in my other SAML blog post.
Each tenant have its own role ids, so when doing automation with group import we need to query the vCloud API and get the role ids. There is a specific query API to get data. When using a system account we need to specify a “VCLOUD-TENANT-CONTEXT” in the header of the request. This we we can query a tenant context from a system account.
### Retrieve roles from tenant
[xml]$xml = Invoke-RestMethod -UseBasicParsing -Uri 'https://<VCD_URI>/api/query?type=role&page=1&pageSize=20&links=true' -Method get -Headers @{'x-vcloud-authorization'= $vCDAuthorizationToken ; Accept = 'application/*+xml;version=31.0'; "Content-type" = "application/*+xml;version=31.0"; "X-VMWARE-VCLOUD-TENANT-CONTEXT" = "$orgId"}'
### Find the real is for x
$RoleHref = ($xml.QueryResultRecords.RoleRecord | where {$_.Name -eq "$role"}).href
After we got the role id we can now send up the group together with the role id and this was be able to authenticate based on a SAML group from AzureAD.
### Define XML with role and groupid
$xmlBody =
@'
<Group xmlns="http://www.vmware.com/vcloud/v1.5"
xmlns:ns9="http://www.vmware.com/vcloud/versions" name="{1}"
type="application/vnd.vmware.admin.group+xml">
<ProviderType>SAML</ProviderType>
<Role href="{0}" type="application/vnd.vmware.admin.role+xml"/>
</Group>
'@ -f $RoleHref , $GroupId
### Post xml to vcd
Invoke-RestMethod -UseBasicParsing -Uri "https://<VCD_URI>/api/admin/org/$orgId/groups" -Method Post -Headers @{'x-vcloud-authorization'= $vCDAuthorizationToken ; Accept = 'application/*+xml;version=31.0'; "Content-type" = "application/*+xml;version=31.0"} -Body ([System.Text.Encoding]::UTF8.GetBytes(($xmlBody)))
Conclusion
Now we have all the pieces for making automation where we can enable a tenant for SAML authentication and afterwards import f.eks. a viewer and admin group. External users will then be invited to the AzureAD, imported into the right group and now they have access to the their tenant. We can help the organisation secure the access to their virtual datacenter with MFA and they will have single sign-on with there own user that originates from their own AzureAD or Microsoft account.
A service library will be made where users can be invited to the tenant organisation. So that when one user have been invited that user will be able to invite its colleagues.
In this post, I will explain how to install a public certificate into vCloud Director cell(s). This exact environment has a public signed cert that is up for renewal. A new certificate has been bought and signed and is ready to import.
vcd cells have 2 IP addresses that allow support for 2 different SSL endpoints (http and consoleproxy). Each endpoint requires its own SSL certificate. vCloud Director uses a java keystore to read its SSL certificates from. In a multi-cell environment, you need to create 2 certificates for each cell and import the certificates into the vcd java keystore. But since we hare here using a wildcard certificate the same certificate will be used to but endpoints.
The new certificate have been created with a CSR that was not generated from the vCloud cells, so we need to import both private and public key from an export of the certificate. In this case it’s a .PFX.
Certificate is a wildcard. If you are using a UCC SAN certificate with the exact names then be sure that the names in certificate are matching accordingly to vCloud settings.
I assume you got
Already working/configured vCloud environment
New public signed certificate exported to a .PFX format (contains both public and private key)
We will
Connect to cell with winscp and transfer the .PFX to /tmp/
Connect to cell with Putty
Create a new keystore with the new certificate
Stop vcd service
Swap old keystore with new
Start vcd service
Initialize certificate change…
The commands for creating the new keystore and importing the cert is below. Change the STOREPASS and KEYPASS to something meaningful for your environment. It is also important to notice that the alias of each certificate must be “http” and “consoleproxy”. Else vcd won’t find the certs.
A note about the alias, I have seen it generate GUID but also just numbers. So if your list command is showing “1” then you need to change alias 1 to respectively http or consoleproxy.
### Stop vCloud Director service
service vmware-vcd stop
### make passwords variable in unix
STOREPASS= <pass>
KEYPASS= <pass>
### Add the certificate to a new created certificates.ks keystore.
/opt/vmware/vcloud-director/jre/bin/keytool \
-keystore /tmp/certificates.ks \
-storepass STOREPASS \
-keypass KEYPASS \
-storetype JCEKS \
-importkeystore \
-srckeystore /tmp/wildcard2020.pfx
### List certificate alias
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-storepass STOREPASS \
-keystore /tmp/certificates.ks \
-list | grep -i alias
### Rename certificate random alias to http
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-changealias \-alias "te-d487d1c7-2c76-482a-8e61-69107ee3027f" \
-destalias http -keystore /tmp/certificates.ks
### Add the Remote Console Proxy certificate to a new created certificates.ks keystore.
/opt/vmware/vcloud-director/jre/bin/keytool \
-keystore /tmp/certificates.ks \
-storepass STOREPASS \
-keypass KEYPASS \
-storetype JCEKS \
-importkeystore \
-srckeystore /tmp/wildcard2020.pfx
### List certificate alias
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-storepass STOREPASS \
-keystore /tmp/certificates.ks -list | grep -i alias
### Rename certificate random alias to consoleproxy
/opt/vmware/vcloud-director/jre/bin/keytool \
-storetype JCEKS \
-changealias \
-alias "te-d487d1c7-2c76-482a-8e61-69107ee3027f" \
-destalias consoleproxy \
-keystore certificates.ks
### Make a backup of the existing keystore
cp /opt/vmware/vcloud-director/certificates.ks /opt/vmware/vcloud-director/certificates.ks_old
### Copy the new keystore file to the vCloud Director environment
cp /tmp/certificates.ks /opt/vmware/vcloud-director/certificates.ks
### Set correct permissions to the keystore file
chown vcloud:vcloud /opt/vmware/vcloud-director/certificates.ks
chmod 600 /opt/vmware/vcloud-director/certificates.ks
### Make cells generate proxy console certs based on new keystore.
cd /opt/vmware/vcloud-director/bin
./cell-management-tool certificates -p -k /opt/vmware/vcloud-director/certificates.ks -w STOREPASS
### Start vCloud Director service
service vmware-vcd start
To see if the cell have booted correctly you can tail the cell log. It will give you a “startup completed in x”.
tail -f /opt/vmware/vcloud-director/logs/cell.log
I got more than one cell…
That’s awesome – me too. You can scp the certificate from the cell with the new cert to the other cells. So let’s get that newly created keystore over the other cells.
### SSH to next cell and stop the vCloud Director service
service vmware-vcd stop
### Go to keystore path
cd /opt/vmware/vcloud-director/
### Move the existing to new filename with .old surfix.
mv certificate.ks certificate.ks.old
### Copy the new certificate into place
scp root@dc1svcdcell01:/opt/vmware/vcloud-director/certificates.ks .
### Make cells generate proxy console certs based on new keystore.
./cell-management-tool certificates -p -k /opt/vmware/vcloud-director/certificates.ks -w KEYSTORE_PASSWD
### Start vCloud Director service
service vmware-vcd start
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing.