Veeam – ramsgaard.me

Cloud Director – Tenant SAML SSO with AzureAD

Posted on September 19, 2023 by Jesper Ramsgaard

I have done this post before, but it was back in the day when we called VCD for vCloud, miss the name a bit but not the flash GUI. In the future Cloud Director will not support local users in organizations. Therefore, if the customer needs to have access to their organization it needs to be with an IDP identity provider for SAML or OAuth.

Many customers today have Microsoft 365 already and therefore also AzureAD which is a convenient choice for SAML integration For the non-AzureAD IDP admin, you should also be able to use this guide since it’s only claims and metadata.

This guide will focus on access assigned based on group membership, but you can also use roles. The main difference between roles and groups would be that roles can be more granular than group memberships. For example, you can have a role that gives access to expand a disk or delete a VM, the roles translate better than groups.

Microsoft have changed the name of AzureAD to Entra, this post will refer to both AzureAD but this also means Entra

Process

Create Enterprise app in desired resource AzureAD
Add entity ID and claims
Import AzureAD enterprise app federation metadata to Cloud Director
Assign allowed users/groups to roles in Cloud Director

Claims

Cloud Director is very picky about claims, VMware documentation gives a list of how the claims should look.

email address = “EmailAddress”
user name = “UserName”
full name = “FullName”
user’s groups = “Groups”
user’s roles = “Roles”

If you want the claim names with the full namespace, then you can map it in the Cloud Director SAML attribute mapper. Attribute mapper

AzureAD(Entra) setup

Enterprise Application creation

The basic setup of Entra is shown in this short story below. We will end up with an Enterprise App that is ready to setup.

Cloud Director – Tenant SAML SSO with AzureAD

In Entra, find the Enterprise application and create a new one.

Enterprise app configuration

SAML Claims

Set up the claims according to the screenshot below. If you make claim names as in the screenshot then it will match Cloud Director so the SSO should work out of the box.

The only special setup is for the group claim where you should do the following

Choose “Groups assigned to the application”. If you have many groups in Entra SAML setup is only returning some of them. And if the group Cloud Director is looking for is not returned, then you are not logged in.
Source attribute of “Group ID”. Haven’t gotten it
Under Advanced options, choose “Customize the name of the group claim” and set it to “Groups” This will make sure Cloud Director knows that its groups.

If you are using a differnt IDP wher you cant alter the claim names, then you the attribut mapper in Cloud Diretor to translate the claimnames over to something that Cloud Director understands.

Cloud Director configuration

Go to tenant context and navigate to “Administration” > “SAML”. You might be hit with a prompt that the certificate is expired.

Hit the “Regenerate certificate” and afterward the “Configure” button and a popup show.

Fill out the wanted entity ID. I usually just use the URL for the tenant vcd. The important thing is just that the entity ID matches between Entra Enterprise App and Cloud Director.

Download the federation metadata XML from the AzureAD Enterprise application.

choose the “Federation Metadata XML” link

Upload the metadata to Cloud Director SAML configuration, enable the SAML identity provider, and press save.

We are now ready to upload the metadata from Cloud Director.

This can be found under “Administration” > “SAML” and “Retrieve Metadata”.

Now upload the metadata to the Enterprise Application

After the upload the Entity ID and reply URL will be filled out.

Cloud Director SAML groups

For the Cloud Director to know what role the login user should have we need to tell it what group ID is associated with what role. We do that by fetching the ID of the group in AzureAD.

Copy the ID and go over to Cloud Director under “Administrator” > “Groups” > “Import Groups”

Paste in the ID and select the role followed by save. Using the Group ID helps if someone decides to change the name of the groups in Entra. For easier to see what group is behind the ID I tend to insert the group name(s) into the description after it has first been created.

Troubleshooting

There can be multiple places where problems can happen. Most of the time I start to look into a saml response, if that contains the groups that I expect then you will need to go on and look in the VCD logs.

SAML response

In your favorite browser, find developer mode and have it capture the traffic. Then log on to VCD have the error occur and then stop the capture again.

Capture the base64 encoded text and decode it in your favorite tool

<samlp:Response Destination="https://vcd.ramsgaard.dk/login/org/system/saml/SSO/alias/vcd" ID="_684262e1-e528-422f-8bf7-dd6d78f3ab49" InResponseTo="af3h19802aa811a5824j5b18aidee2" IssueInstant="2023-09-18T11:52:08.837Z" Version="2.0" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol">
	<Issuer xmlns="urn:oasis:names:tc:SAML:2.0:assertion">https://sts.windows.net/c5c123a4-e828-43bd-84d6-05cbfe1efae5/</Issuer>
	<samlp:Status>
		<samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/>
	</samlp:Status>
	<Assertion ID="_e96104c4-681e-4624-111e-a6e9a2bf6900" IssueInstant="2023-09-18T11:52:08.833Z" Version="2.0" xmlns="urn:oasis:names:tc:SAML:2.0:assertion">

We want to see a success for the SAML authentication. If we got that then it can be due to the claims being sent with it.

			<Attribute Name="name">
				<AttributeValue>user@domain.tld</AttributeValue>
			</Attribute>
			<Attribute Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/UserName">
				<AttributeValue>user@domain.tld</AttributeValue>
			</Attribute>
			<Attribute Name="Groups">
				<AttributeValue>93547461-c174-4cab-961f-6f21c43051bd</AttributeValue>
				<AttributeValue>3c86569c-9e03-43f8-87fa-30b566ebb829</AttributeValue>
			</Attribute>
		</AttributeStatement>

In the snip above you can see that I got two group IDs back in the claim, check if these are the group IDs VCD is looking for. If not, then you need to investigate if the user is a member of the correct groups.

Conclusion

Since VCD local users are deprecated and will be removed in the future we need to convert over to use an IDP instead. AzureAD is a nice way to implement MFA and has a single identity for accessing corporate resources.

Veeam – retrive saved passwords from VBR

Posted on January 17, 2022 by Jesper Ramsgaard

Ever needed to retrieve a saved Veeam password? I did – Found the process for it on the Veeam forum.

Open SQL Studio as administrator and connect to the Veeam DB instance
Run query from below on the VeeamBackup database

SELECT TOP (1000) [id]
,[user_name]
,[password]
,[usn]
,[description]
,[visible]
,[change_time_utc]
FROM [VeeamBackup].[dbo].[Credentials]

Query the Veeam DB for all stored credentials to backup infrastructure components

Get the password hash from the results (match the description to the one you need). Then run PowerShell below with the hash you grabbed.

Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$encoded = 'AQAAANCM....RhQ'
[Veeam.Backup.Common.ProtectedStorage]::GetLocalString($encoded)

Conclusion:

Is this a security problem? Depends, but it will give you a reminder of how important it is to keep your Veeam VBR server safe. Never domain join and have the firewall closed as much as possible. If a malicious person comes by your Veeam server they can grab the keys for the rest of your infrastructure, including your backup of cause. In most cases that would mean game over.

Faster and more scripted way:

$instance = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlInstanceName).SqlInstanceName
$server = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlServerName).SqlServerName
$result = Invoke-Sqlcmd -Query "SELECT TOP (1000) [user_name],[password],[description] FROM [VeeamBackup].[dbo].[Credentials]" -ServerInstance "$server\$instance"
Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$result | ForEach-Object { [Veeam.Backup.Common.ProtectedStorage]::GetLocalString($($_.password))}

Veeam – Network Extention Appliance performance

Posted on November 10, 2021 by Jesper Ramsgaard

This post will give a brief write-up on what to expect from a network perspective when using the Veeam Network Extention.

Since you found this post I don’t think an introduction is needed. But anyway. A quick write-up of the network so you can visualize how the test is performed.

Greenline indicated the L2 VPN made from both NEA to CloudGateway
The on-prem environment with 10gbit internet uplink
Service provider with multiple 10gbit internet uplinks
4ms between on-prem and service provider

Tests:

So when a replica VM has been failover and the NEA L2 is running. What to expect? Veeam does not give you any info on the performance of the NEA. Veeam support is not either able to give out a performance chart. So here a the results from ping and iperf test.

Test 1 – ping of latency over the L2 tunnel:

Ping to 185.177.120.140 showing the latency over the internet. Ping to 192.168.12.151 showing the latency over the L2 VPN.

So from a latency perspective, it seems good. Only adding 1ms to the internet latency. Which is pretty good.

Test 2 – iperf over tunnel

iperf test from a VM in service provider side to an on-prem server.

About 110Mbit, not very good compared to the internet being able of doing 10Gbit.

iperf test from VM in service provider side to an on-prem server. This time with -P 8 for 8 parallel threads.

8 threads are not giving any further bandwidth.

Test 3 – Multiple VLAN with multiple NEA

It’s always interesting to find where the bottleneck could be. Since iperf over the internet is giving a completely different result. then it must be within NEA. When I tried to do multiple VLAN bridges to the cloud resources in Cloud Director I get the same results pr NEA. Meaning it could be something in NEA or its components making the bottleneck.

The good news is off cause that you will see the same result pr NEA even when doing iperf test to the same target in the other end. So NEA will scale linearly.

A look from the Veeam Cloud Connect gateway being the broker of the L2 VPN connections.

View from the iperf server – showing the connections from the servers in the other end of L2 VPN.

Conclusion

NEA is a very helpful solution, especially when it comes to large migrations where L2 between datacenters is required meanwhile migrating. Bandwidth using this solution is not great, but I would say is ok. L2 connection should only be used shortly when doing actually migrations.

In numbers, it seems NEA will add +1ms to the latency seen over the internet between the two environments. Bandwith is between 110 to 140mbit pr sec.

Change VM MoRef in VBR database

Posted on August 26, 2021 by Jesper Ramsgaard

This information can be found in many other places on the big internet, but since I can never find it myself, I will make a post more about the procedure.

When you switch ESXi host, vCenter, or remove and add from inventory your VM will get a new ID. In the world of VMware, it’s called MoRef ID.

When this happens Veeam will lose its coupling to the VM and backup will fail with:
– Virtual Machine <> is unavailable and will be skipped from processing.
– Nothing to process. All machines were excluded from task list.

How to verify there is a MoRef mismatch:

From a VMware perspective it’s easy:

connect-viserver <vcenter> -Credential $cred
Get-VM | select name, id

This will give you something like:

PS C:\Windows\system32> Get-VM | select name, id
Name Id
---- --
VirtualMachine-vm-71326

From Veeam perspective it’s a bit harder since you will need to query the MS SQL database that Veeam uses. So download the SQL Studio Manager from Microsoft.

Open the SQL Studio Manager as administrator on the server to gain access to the Veeam database. You can use the following query to find the MoRef that is in the Veeam database:

SELECT [dbo].[BObjects].id, [dbo].[BObjects].object_id, [dbo].[BObjects].host_id, [dbo].[BObjectsSensitiveInfo].object_name, [dbo].[BObjectsSensitiveInfo].path
FROM [dbo].[BObjects]
INNER JOIN [dbo].[BObjectsSensitiveInfo] ON [dbo].[BObjectsSensitiveInfo].bObject_id=[dbo].[BObjects].id  
WHERE object_name = '<vmname>'

Verify:

So we can now see that the VM in VMware has MoRef “vm-71326”. But Veeam database has “vm-992”. From here on you know what’s wrong and you need to open a Veeam support case to get the supported procedure.

If you don’t care about supported procedures you can update the database with VMware VM new MoRef ID and your VBR job should be running again. The SQL query would look like this:

UPDATE [dbo].[BOobjects]
SET [object_id] = 'new-id'
WHERE [object_id] = 'old-id'

Conclusion

It’s not that had to change the MoRef in the VBR database. But remember, if you care about having a supported installation. Then you need to create a Veeam support case and have them help you. Something could have changed in the VBR database schema since this post.