The old Intel C2000 bug is still among us. Not all devices have been affected, yet. For me, this is the first. So here is my 2 cent on how to remediate the cluster with an RMA unit.
https://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64228.html
Process:
- Label all cables from the faulty unit, this is for you to not worry that any cables are mislocated when you are swapping the unit. I also disabled the links for the switch so that I with a calm can plug them in again without fearing that something will go wrong. Better safe than sorry.
- Make sure the firmware level on the new device is the same as the existing member. Firmware up/downgrade is needed.
- Backup the config of the HA member still running. Just in case something bad happens.
- Give an IP on the HA link interface, from this point it should find the existing member and start replication the config.
Firmware upgrade:
I have booted the unit and linked it only with a console cable. I can now see the firmware version and it needs an upgrade.
ciscoasa> en
Password:
ciscoasa#
ciscoasa# sh version
Cisco Adaptive Security Appliance Software Version 9.8(2)20
Firepower Extensible Operating System Version 2.2(2.63)
Device Manager Version 7.5(1)
Compiled on Fri 02-Feb-18 06:10 PST by builders
System image file is "disk0:/asa982-20-lfbff-k8.SPA"
Config file at boot was "startup-config"
ciscoasa up 30 secs
Hardware: ASA5516, 8192 MB RAM, CPU Atom C2000 series 2416 MHz, 1 CPU (8 cores )
Internal ATA Compact Flash, 8000MB
BIOS Flash M25P64 @ 0xfed01000, 16384KB
I have prepared a USB disk format in FAT and downloaded the matching firmware from cisco.com. When you plug in the USB key to the ASA you should now see a disk1 where you can copy from. If you want to see disk1 content issue command “show disk1”.
We now copy over the files and make the system boot the new firmware.
ciscoasa# copy disk1:/asa9-13-1-lfbff-k8.SPA disk0:/asa9-13-1-lfbff-k8.SPA
Source filename [asa9-13-1-lfbff-k8.SPA]?
Destination filename [asa9-13-1-lfbff-k8.SPA]?
Copy in progress...CCCCC
Verifying file disk0:/asa9-13-1-lfbff-k8.SPA...
Computed Hash SHA2: 80500c1790c76e90dde61488c3f977b8
69711278b6e550eeb8ea8830e19c4a23
8cf03fe64d1d9927d4a78e77b6090234
98485fbf9bc058eb3820b32e7a56f91f
Embedded Hash SHA2: 80500c1790c76e90dde61488c3f977b8
69711278b6e550eeb8ea8830e19c4a23
8cf03fe64d1d9927d4a78e77b6090234
98485fbf9bc058eb3820b32e7a56f91f
Digital signature successfully validated
Writing file disk0:/asa9-13-1-lfbff-k8.SPA...
107543456 bytes copied in 26.40 secs (4136286 bytes/sec)
ciscoasa# config t
ciscoasa(config)# boot system disk0:/asa9-13-1-lfbff-k8.SPA
ciscoasa(config)# wr mem
ciscoasa(config)# reloadAfter reload, the system is now up and I can confirm that it has booted on the new firmware.
ciscoasa> show version
Cisco Adaptive Security Appliance Software Version 9.13(1)
SSP Operating System Version 2.7(1.107)
Device Manager Version 7.5(1)
Compiled on Mon 23-Sep-19 09:28 PDT by builders
System image file is "disk0:/asa9-13-1-lfbff-k8.SPA"
Config file at boot was "startup-config"
ciscoasa up 29 secs
Hardware: ASA5516, 8192 MB RAM, CPU Atom C2000 series 2416 MHz, 1 CPU (8 cores)
Internal ATA Compact Flash, 8000MB
BIOS Flash M25P64 @ 0xfed01000, 16384KB
Joining the HA cluster
We now verified that the two ASA firewalls are on the correct firmware level. Now connect all the cables to the firewall, on the switch side all data links are administratively down, the HA link between the two ASA is a dedicated link. And those are the links we are now going to configure.
You can grab the lines from the existing member that are actively running. If you don’t have the failover key, you can also reset this on on the primary/existing member.
failover lan unit secondary
failover lan interface HA_FAILOVERLINK GigabitEthernet1/7
failover key ***
failover link HA_STATELINK GigabitEthernet1/8
failover interface ip HA_FAILOVERLINK 172.16.254.1 255.255.255.0 standby 172.16.254.2
failover interface ip HA_STATELINK 172.16.255.1 255.255.255.0 standby 172.16.255.2The new ASA is now ready to contact the primary member of the cluster and start the replication. In my case, the interfaces for HA were administratively down. So we are now going to enable the link and enable failover.
interface GigabitEthernet 1/7
no shut
interface GigabitEthernet 1/8
no shut
failoverIf something in the config is not ok, missing files or other is listed and you have to remediate this before you again can try to enable HA with the “failover” command. In the output beneath you can see what happens when the failover command is enabled with success.
Detected an Active mate
Beginning configuration replication from mate.
End configuration replication from mate.You can now check failover status and see if the standby member is in ready mode. if not try giving the standby a reload.
failover reload-standbyIf the standby member is now in a ready state you are now ready to do a live failover. Remember to enable the ports again on the switch side.
Conclusion
The process is not so bad as I thought. And there where no downtime involved. For me, I was missing ASDM and AnyConnect packages on the new standby node. I downloaded it from the existing primary node and then copied it to a USB disk. When the USB disk is plugged into the standby ASA I can then copy the files over the ASA flash.
copy /noconfirm disk1:/anyconnect-win-4.8.01090-webdeploy-k9.pkg disk0:/anyconnect-win-4.8.01090-webdeploy-k9.pkg
copy /noconfirm disk1:/anyconnect-macos-4.8.01090-webdeploy-k9.pkg disk0:/anyconnect-macos-4.8.01090-webdeploy-k9.pkg
copy /noconfirm disk1:/VPN_client_profile.xml disk0:/VPN_client_profile.xml
copy /noconfirm disk1:/anyconnect-linux64-4.8.01090-webdeploy-k9.pkg disk0:/anyconnect-linux64-4.8.01090-webdeploy-k9.pkg
copy /noconfirm disk1:/Management_client_profile.xml disk0:/Management_client_profile.xml
copy /noconfirm disk1:/asdm-7131.bin disk0:/asdm-7131.binFrom there on I could do a live failover and see the little “Active” light change on the physical ASA firewalls. With all traffic flowing uninterrupted. Mission accomplished.
