Month: January 2022

Juniper upgrade process

Posted on by Jesper Ramsgaard

Junos is in my opinion an awesome OS for your network. I enjoy the CLI, where commands are alike across all of Juniper’s products. Also, the many features and the fact that it’s not cisco.

BUT it also has its drawbacks. Honestly, I have seen some weird bugs. And keeping track of all the PRs from Juniper is a full-time job. And last but not least, the software upgrades are kind of a pain. especially on Junos devices older than 18.x.

EX3400 – format/install

For this case, I had a new EX3400, but with older firmware, 15.1X53-D58.3. I needed to upgrade to the latest SR in the newest train but from the CLI of the device only jumping 3 firmware versions are supported.

15.1> 18.1 > 18.4 > 19.3 > 20.2 > 21.1

But you can also do a format/install where you interrupt the boot process and then load a new firmware image on the device from a TFTP server. This is all done outside of Junos. This way you can jump to whatever version you want. 

Jumping many version might make your config invalid, so beaware.

Juniper has a LOT of kb articles for this process and they all vary. So here is the process in my own writing

Process of format install

First, we need to get the right image from the juniper support side. It needs to the install image and the extension is .tgz

  • Download the image into your TFTP server.

In my case, the TFTP is a Linux box. If you prefer windows then TFTPd3264 is the way to go. Or MacOS then look here.

root@tftp:/srv/tftp# wget -O junos-install-media-net-ex-arm-32-21.4R1.12.tgz  'https://cdn.juniper.net/software/junos/21.4R1.12/junos-install-media-net-ex-arm-32-21.4R1.12.tgz?SM_USER=jv......5ce43fbdad2'
Resolving cdn.juniper.net (cdn.juniper.net)... 23.78.40.231
Connecting to cdn.juniper.net (cdn.juniper.net)|23.78.40.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 393745989 (376M) [application/octet-stream]
Saving to: ‘junos-install-media-net-ex-arm-32-21.4R1.12.tgz’

junos-install-media-net-ex-arm-32-21.4R1.12.tgz      100%[==================================================================>] 375.50M  3.48MB/s    in 2m 4s

2022-01-26 20:47:46 (3.03 MB/s) - ‘junos-install-media-net-ex-arm-32-21.4R1.12.tgz’ saved [393745989/393745989]

root@tftp:/srv/tftp# ls
junos-install-media-net-ex-arm-32-21.4R1.12.tgz
  • Now let’s reboot the switch and interrupt the “first” boot loader. just keep hitting ctrl+c after you powered rebooted when you see the “=>” you are in the right place. Here we set the IP address on the me0 interface and boot into the next boot loader.
Board: EX3400-24T
Base MAC: C00380FAAD2E
arm_clk=1000MHz, axi_clk=500MHz, apb_clk=125MHz, arm_periph_clk=500MHz
Net:   Registering eth
Broadcom BCM IPROC Ethernet driver 0.1
Using GMAC0 (0x18022000)
et0: ethHw_chipAttach: Chip ID: 0xdc14; phyaddr: 0x1
et0: gmac_serdes_init read sdctl(0xf4141c)
et0: gmac_serdes_init() serdes_status0: 0xf100ff00; serdes_status1: 0xf00
et0: gmac_serdes_init() PLL ready brought up exit
serdes_reset_core pbyaddr(0x1) id2(0xf)
bcmiproc_eth-0
Last Reset Reason: 0
Hit ^C to stop autoboot:  0
=>setenv ipaddr 10.1.100.253
=>setenv gatewayip 10.1.100.1
=>setenv netmask 255.255.255.0
=>setenv serverip 10.1.101.130
=>save
=>boot
Saving Environment to SPI Flash...
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB, mapped at 0001faa0
Erasing SPI flash...Writing to SPI flash...done
Erasing SPI flash...Writing to SPI flash...done
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
device 0 offset 0x3c0000, size 0x10000
SF: 65536 bytes @ 0x3c0000 Read: OK
  • Wait for a few seconds for the next bootloader to appear and press ctrl+c again. Now you will see a menu, this menu you choose 5 and 5 and you should see “loader>”
Hit ^C to stop autoboot:  0 
Options Menu

1.  Recover [J]unos volume
2.  Recovery mode - [C]LI

3.  Check [F]ile system
4.  Enable [V]erbose boot
5.  [B]oot prompt
6.  [M]ain menu
Choice: 
Type 'menu' to go back to the menu
Type 'boot-junos' to boot into Junos
Type 'reboot' to reboot

5 5
  • We now set use the install format with the TFTP location of the image we downloaded in the first step.
Type '?' for a list of commands, 'help' for more detailed help.
loader> install --format tftp://10.1.101.130/junos-install-media-net-ex-arm-32-21.4R1.12.tgz
/kernel text=0x105b888 data=0x640fc+0x1fbf04 syms=[0x4+0x914a0+0x4+0x9b821]
/ex3400.dtb size=0x1f76
/crypto.ko text=0x419e0 data=0xe58+0x2a0 syms=[0x4+0x4740+0x4+0x2ba5]
/iflib.ko text=0x11f10 data=0x910+0x58 syms=[0x4+0x2b10+0x4+0x2194]
/miibus.ko text=0x19f38 data=0x10c4+0x78 syms=[0x4+0x51f0+0x4+0x3491]
/if_gmac.ko text=0xbc3c data=0x688+0xc syms=[0x4+0x1cc0+0x4+0x15ad]
/contents.iso size=0x279b000
Using DTB from loaded file '/ex3400.dtb'.
Kernel entry at 0xc1000180...
Kernel args: (null)
---<<BOOT>>---
GDB: no debug ports present
K cache
Release APs
WARNING: WITNESS option enabled, expect reduced performance.
mwill now attempt to reach the remote host.
<====== LOADS OF OUTPUT TO CONSOLE ======>
<====== LOADS OF OUTPUT TO CONSOLE ======>
Downloading /junos-install-media-net-ex-arm-32-21.4R1.12.tgz from 10.1.101.130 ...
rmed on 1024 samples passed.t-up health tests perfo
  300.6MB  03:52random: unblocking device.
  393.7MB  05:04
Installing Junos OS release ...

After 15-20 minutes the switch will have the install finished and ready for you to log into and start loading your config.

FreeBSD/arm (Amnesiac) (ttyu0)
login:

Conclusion

This is a very helpful process and might come in handy when having new switches with old firmware that need to be applied. Skipping the smaller version jumps is a time saver.

This format install process can also be done with a USB key. This process is also quite simple but requires you to have physical access to the switch.

In my case, I have a console over ssh and can manage the switch out-of-band so TFTP is the easy way.

Veeam – retrive saved passwords from VBR

Posted on by Jesper Ramsgaard

Ever needed to retrieve a saved Veeam password? I did – Found the process for it on the Veeam forum.

  • Open SQL Studio as administrator and connect to the Veeam DB instance
  • Run query from below on the VeeamBackup database
SELECT TOP (1000) [id]
,[user_name]
,[password]
,[usn]
,[description]
,[visible]
,[change_time_utc]
FROM [VeeamBackup].[dbo].[Credentials]
Query the Veeam DB for all stored credentials to backup infrastructure components

Get the password hash from the results (match the description to the one you need). Then run PowerShell below with the hash you grabbed.

Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$encoded = 'AQAAANCM....RhQ'
[Veeam.Backup.Common.ProtectedStorage]::GetLocalString($encoded)
Password revealed and ready to use

Conclusion:

Is this a security problem? Depends, but it will give you a reminder of how important it is to keep your Veeam VBR server safe. Never domain join and have the firewall closed as much as possible. If a malicious person comes by your Veeam server they can grab the keys for the rest of your infrastructure, including your backup of cause. In most cases that would mean game over.

Faster and more scripted way:

$instance = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlInstanceName).SqlInstanceName
$server = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -name SqlServerName).SqlServerName
$result = Invoke-Sqlcmd -Query "SELECT TOP (1000) [user_name],[password],[description] FROM [VeeamBackup].[dbo].[Credentials]" -ServerInstance "$server\$instance"
Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$result | ForEach-Object { [Veeam.Backup.Common.ProtectedStorage]::GetLocalString($($_.password))}

Cloud Director 10.3 – Update certificates

Posted on by Jesper Ramsgaard

Since my last article on how to update Cloud Director SSL certificates, there has been a major change. No more binary java truststore – jaaay.

Cloud Director has changed over too, what I think, is a better and more normal way of storing the private and public keys, which is in PEM format. From release notes, the change actually happened in 10.2, but the certificate path changed again in 10.3. If you are in doubt of where the certificate path is then look inside global.properties

/opt/vmware/vcloud-director/etc/global.properties

VMware’s own documentation state that we can now just swap the .pem files, use the cell-management tool to import and restart the cell.

What we will do and what is needed

  • Get a new public signed certificate
    • Either in PEM format as .key and .pem(certificate including intermediate)
    • Or in PFX so it can be exported
  • Backup existing certificates
  • Replace existing certificates with your new certificate
  • Run VCD tool to import and define the private key encryption password
  • Restart cell(s)

Process

If you have a pfx you can use this article to extract the key and cert. If you already have the two files, .key end .pem then you can proceed.

We will follow VMware documentation and create a backup of the existing files. 

cp /opt/vmware/vcloud-director/etc/user.http.pem /opt/vmware/vcloud-director/etc/user.http.pem.original
cp /opt/vmware/vcloud-director/etc/user.http.key /opt/vmware/vcloud-director/etc/user.http.key.original
cp /opt/vmware/vcloud-director/etc/user.consoleproxy.pem /opt/vmware/vcloud-director/etc/user.consoleproxy.pem.original
cp /opt/vmware/vcloud-director/etc/user.consoleproxy.key /opt/vmware/vcloud-director/etc/user.consoleproxy.key.original

Now we can wither SCP in our key and certificate or edit and replace the content of the files on the server by copying and pasting in content from the files you have. Whatever you find to be the easiest.

Forgot your root password for the Cloud Director appliance, off cause not. But anyway, here is a link to reset it....

After the “user.http.pem/key” and “user.consoleproxy.pem/key” files have been updated with the new certificate data we can tell Cloud Dictor to update its config with the commands below. This is done to update the encryption password for the private key.

If you don’t care about security you can also update without –key-password, then off cause your private key will need to be in an unencrypted format in the .key files. 

/opt/vmware/vcloud-director/bin/cell-management-tool certificates -j --cert /opt/vmware/vcloud-director/etc/user.consoleproxy.pem --key /opt/vmware/vcloud-director/etc/user.consoleproxy.key --key-password PASSWD
/opt/vmware/vcloud-director/bin/cell-management-tool certificates -p --cert /opt/vmware/vcloud-director/etc/user.http.pem --key /opt/vmware/vcloud-director/etc/user.http.key --key-password PASSWD

If everything works out it will tell you the certificates have been updated and you need to restart VCD for it to take effect.

SSL configuration has been updated. You will need to restart the cell for changes to take effect.

Now safely shut down your cell(s) with the command below. this will ensure that VCD is the first shutdown when all tasks are done.

/opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s

Start again with the command below

systemctl start vmware-vcd

Conclusion

VMware has made it much easier to change a certificate in Cloud Director. The new way of storing certificates is a warm welcome change.

I did see a few different placements for the .key and .pem files depending on versions or if the cells have been created with raw Linux or an appliance, but you can always look in the conflig file placed in the same folder as the certificates.