veeam

Veeam – retrive saved passwords from VBR

Posted on by Jesper Ramsgaard

Ever needed to retrieve a saved Veeam password? I did – Found the process for it on the Veeam forum.

  • Open SQL Studio as administrator and connect to the Veeam DB instance
  • Run query from below on the VeeamBackup database
SELECT TOP (1000) [id]
,[user_name]
,[password]
,[usn]
,[description]
,[visible]
,[change_time_utc]
FROM [VeeamBackup].[dbo].[Credentials]
Query the Veeam DB for all stored credentials to backup infrastructure components

Get the password hash from the results (match the description to the one you need). Then run PowerShell below with the hash you grabbed.

Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$encoded = 'AQAAANCM....RhQ'
[Veeam.Backup.Common.ProtectedStorage]::GetLocalString($encoded)
Password revealed and ready to use

Conclusion:

Is this a security problem? Depends, but it will give you a reminder of how important it is to keep your Veeam VBR server safe. Never domain join and have the firewall closed as much as possible. If a malicious person comes by your Veeam server they can grab the keys for the rest of your infrastructure, including your backup of cause. In most cases that would mean game over.

Faster and more scripted way:

$instance = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlInstanceName).SqlInstanceName
$server = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlServerName).SqlServerName
$result = Invoke-Sqlcmd -Query "SELECT TOP (1000) [user_name],[password],[description] FROM [VeeamBackup].[dbo].[Credentials]" -ServerInstance "$server\$instance"
Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$result | ForEach-Object { [Veeam.Backup.Common.ProtectedStorage]::GetLocalString($($_.password))}

Veeam – Network Extention Appliance performance

Posted on by Jesper Ramsgaard

This post will give a brief write-up on what to expect from a network perspective when using the Veeam Network Extention.

Since you found this post I don’t think an introduction is needed. But anyway. A quick write-up of the network so you can visualize how the test is performed.

  • Greenline indicated the L2 VPN made from both NEA to CloudGateway
  • The on-prem environment with 10gbit internet uplink
  • Service provider with multiple 10gbit internet uplinks
  • 4ms between on-prem and service provider

Tests:

So when a replica VM has been failover and the NEA L2 is running. What to expect? Veeam does not give you any info on the performance of the NEA. Veeam support is not either able to give out a performance chart. So here a the results from ping and iperf test.

Test 1 – ping of latency over the L2 tunnel:

Ping to 185.177.120.140 showing the latency over the internet. Ping to 192.168.12.151 showing the latency over the L2 VPN.

So from a latency perspective, it seems good. Only adding 1ms to the internet latency. Which is pretty good.

Test 2 – iperf over tunnel

iperf test from a VM in service provider side to an on-prem server.

About 110Mbit, not very good compared to the internet being able of doing 10Gbit.

iperf test from VM in service provider side to an on-prem server. This time with -P 8 for 8 parallel threads.

8 threads are not giving any further bandwidth.

Test 3 – Multiple VLAN with multiple NEA

It’s always interesting to find where the bottleneck could be. Since iperf over the internet is giving a completely different result. then it must be within NEA. When I tried to do multiple VLAN bridges to the cloud resources in Cloud Director I get the same results pr NEA. Meaning it could be something in NEA or its components making the bottleneck.

The good news is off cause that you will see the same result pr NEA even when doing iperf test to the same target in the other end. So NEA will scale linearly.

A look from the Veeam Cloud Connect gateway being the broker of the L2 VPN connections.
View from the iperf server – showing the connections from the servers in the other end of L2 VPN.

Conclusion

NEA is a very helpful solution, especially when it comes to large migrations where L2 between datacenters is required meanwhile migrating. Bandwidth using this solution is not great, but I would say is ok. L2 connection should only be used shortly when doing actually migrations.

In numbers, it seems NEA will add +1ms to the latency seen over the internet between the two environments. Bandwith is between 110 to 140mbit pr sec.

Change VM MoRef in VBR database

Posted on by Jesper Ramsgaard

This information can be found in many other places on the big internet, but since I can never find it myself, I will make a post more about the procedure.

When you switch ESXi host, vCenter, or remove and add from inventory your VM will get a new ID. In the world of VMware, it’s called MoRef ID.

When this happens Veeam will lose its coupling to the VM and backup will fail with:
– Virtual Machine <> is unavailable and will be skipped from processing.
– Nothing to process. All machines were excluded from task list.

How to verify there is a MoRef mismatch:

From a VMware perspective it’s easy:

connect-viserver <vcenter> -Credential $cred
Get-VM | select name, id

This will give you something like:

PS C:\Windows\system32> Get-VM | select name, id
Name Id
---- --
VirtualMachine-vm-71326

From Veeam perspective it’s a bit harder since you will need to query the MS SQL database that Veeam uses. So download the SQL Studio Manager from Microsoft.

Open the SQL Studio Manager as administrator on the server to gain access to the Veeam database. You can use the following query to find the MoRef that is in the Veeam database:

SELECT [dbo].[BObjects].id, [dbo].[BObjects].object_id, [dbo].[BObjects].host_id, [dbo].[BObjectsSensitiveInfo].object_name, [dbo].[BObjectsSensitiveInfo].path
FROM [dbo].[BObjects]
INNER JOIN [dbo].[BObjectsSensitiveInfo] ON [dbo].[BObjectsSensitiveInfo].bObject_id=[dbo].[BObjects].id  
WHERE object_name = '<vmname>'

Verify:

So we can now see that the VM in VMware has MoRef “vm-71326”. But Veeam database has “vm-992”. From here on you know what’s wrong and you need to open a Veeam support case to get the supported procedure.

If you don’t care about supported procedures you can update the database with VMware VM new MoRef ID and your VBR job should be running again. The SQL query would look like this:

UPDATE [dbo].[BOobjects]
SET [object_id] = 'new-id'
WHERE [object_id] = 'old-id'

Conclusion

It’s not that had to change the MoRef in the VBR database. But remember, if you care about having a supported installation. Then you need to create a Veeam support case and have them help you. Something could have changed in the VBR database schema since this post.

Change Veeam job repository

Posted on by Jesper Ramsgaard

Having to move jobs to another repository is something that can be time-consuming. If you have to do it with the existing job you will need to disable the job, move the data to the new location, point the job to the new location and then enable the job again.

I found the other approach to creating new jobs easier. You can do it in the GUI, but when having 200+ jobs it could take some time. Instead, I did a small script to list all jobs located on the old repo.

The script also looks a the retention point and writes into the old job that is can be deleted after x days. If you have very long retention then this way is not feasible and you will probably have to move data and point exing job to the new location.

Hope somebody else can use it 🙂

Add-PSSnapin -Name VeeamPSSnapIn
Connect-VBRServer -Server <VBRSERVER>

$Jobs = Get-VBRJob | where {$_.IsScheduleEnabled -eq $true} | where {$_.FindTargetRepository().name -eq "dc2sveeamrepo01-scaleout"} | where {$_.IsRunning -eq $false}

# Select first 15 jobs and list them afterwards.
$remainingJobs = $Jobs | select -first 40
$remainingJobs | select name

foreach ($job in $remainingJobs)
{
$oldjob = "$($job.Name)delete after $($job.BackupStorageOptions.RetainCycles) days"
$job.Info.CommonInfo.name = "$oldjob"
$Job.update()
Disable-VBRJob -Job $oldjob

$jobName = "$($job.Name.Split("_")[0])"

Copy-VBRJob -job $oldjob -name $jobName  -Repository "dc2sveeamrepo02-scaleout" 

Enable-VBRJob -job $jobName

sleep 5

start-vbrjob -Job $jobName -RunAsync 

write-host "$jobName : have been cloned and is now started...." -ForegroundColor Green
}

Veeam repository with local account

Posted on by Jesper Ramsgaard

For the matter of security, I consider it a good idea to isolate the Veeam repository server from Active Directory. So that a compromised domain admin account or other can not gain access to the repository.

But when wanting to do add the repository to the VBR its failing and saying “Access Denied”.

Alright, a bit of googling and found a short and precise article from another guy having solved this problem.

What was the solution?

Open regedit on the repository server and navigate to following

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System

Here you add a DWORD with the name of “LocalAccountTokenFilterPolicy” and value of “1”. This fixes the problem and without rebooting. 

### The PowerShell way
if((Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System')){New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System' -Name 'LocalAccountTokenFilterPolicy' -Value '1' -PropertyType DWORD}

Now you can add the repository server to the VBR. I always forgot where to find the info for the reg hack, so now it’s here so future Jesper can find it 🙂