Cisco ASA cluster – upgrade

Cisco seems to have a good track record of there products, but I must say that there ASA firewalls have seen a lot of critical bugs in the last couple of years. Both in hardware and software…

The last critical bug I was not informed about, so didn’t catch it before the customer did. Always nice when a customer calls in with the problem of there primary ASA being down. It crashed in a way that meant that it did not come up again. It needed a physical reboot.

Before having the chance to have someone onsite locate the firewall and reboot it that secondary also died. And did not come up again! Customer needs to get online again, so there was no time to get a console cable and see what the heck was going on. So I told them to do a hard reboot on both firewalls. After the ASA booted they both became active again and could see each other. Great, customer online. But why and how.

Contact with Conscia Cisco support could confirm that the exact issue has been hitting multiple customers. Due to a bug, the firmware did a memory buffer overflow when being hit by a specific udp/500 attack. Great, now we know the problem and the fix is to upgrade ASA firmware.

It’s not something I do often, and I always forget to write down to procedure, so here goes.

Upgrade procedure

  1. Have a look at the cisco ASA upgrade guide, to see what version you and on and what is supported to go up to. I were on 9.8.2 and could go up to 9.13.x. So I did.
  2. Download and upload firmware to BOTH members of the cluster
  3. Change the boot image to the newly uploaded image
  4. Update the secondary, make a failover
  5. Update the primary and make a failover
  6. Done

Uploading the images to both nodes with TFTP.

I used the portable version of Tftpd64 by Jounin, simple and works out of the box. Copied the freshly downloaded images to both nodes.

### Primary
DS-ESB-ASA5516x# copy /noconfirm tftp:// disk0:/asa9-13-1-lfbff-k8.SPA
DS-ESB-ASA5516x# copy /noconfirm tftp:// disk0:/asdm-7131.bin

### Secondary
DS-ESB-ASA5516x# failover exec mate copy /noconfirm tftp://$
DS-ESB-ASA5516x# failover exec mate copy /noconfirm tftp://$

Change config to the new image

So now we will change over the config so that it will use the new boot images that we have uploaded. First, we remove the existing boot image, and afterwards, we set the new image together with the new ASDM image.

### Show current boot image
DS-ESB-ASA5516x# show running-config boot system
boot system disk0:/gf/asa982-20-lfbff-k8.SPA

### Remove existing boot image
DS-ESB-ASA5516x(config)# no boot system disk0:/gf/asa982-20-lfbff-k8.SPA

### Add new boot image that you just uploaded
DS-ESB-ASA5516x(config)# boot system disk0:/asa9-13-1-lfbff-k8.SPA

### Reload the standby node for the new firmware to take effect
DS-ESB-ASA5516x(config)# failover reload-standby

### Look at the output from show failover, check if the standby is up and verify the firmware version.

Failover and reload the second node

So now the secondary node is booted with the new firmware, time to failover to it so we can reload and have the new firmware running on the primary node. When doing the failover you might lose the SSH connection, just connect again. This time you will be connected to the second node, that is not the active node. Reload the primary, that is now standby and wait for it up come up. It will show in the console that its sending config to mate. Just like when we did it with the first reload of the standby, secondary node.

### Controlled failover to secoundary, standby node
DS-ESB-ASA5516x# no failover active
### reload the primary, standby node for firmware to take effect.
DS-ESB-ASA5516x# failover reload-standby

You are done

Now it’s only to test, and if you want, failback to the primary. But that up to you. I did not lose one ping through the upgrade process. So that cluster is indeed working as it should. While you are at it then why not also update the AnyConnect client and remember to clean up the flash so the old versions and file won’t fill it up. Enjoy your newly updated cluster.

Prevent Drive Failure at 32,768 Hours

This is one of the nasty bugs. Some SSD models will fail after they have been powered on for more than 32768 hours. Imagine running vSAN and you bought x amount of disks that where affected. They will all fail at the same time, so you are left alone with your backup(hopefully).

I seen this one time before, where Intel disks where the problems. Unfortunately the Intel SSDs where metadata disks in a Ceph storage cluster, and since they all failed at the same time, the cluster died!

This is of cause due to that nobody where informed of the bug. When buying hardware from HPE and other enterprise hardware vendors we cat a mail letting us know of the problem before it becomes a disaster.

Update procedure – VMware

We had to do a firmware update of the disks, we are running VMware and vSAN. And gladly HPE have allready released the patch. Also with guidance for VMware.

  1. download the patch, copy it to /tmp of the ESXi servers.
  2. Unzip and make the .vmexe file executable chmod +x CP****.vmexe
  3. Put one of the hosts into maintenance mode.
  4. Run the CP***.vmexe – ./CP****.vmexe. It will lists the disks that it found and you tell it the disk numbers for those you want to have firmware upgraded.
  5. After upgrade I did a reboot anyway.

Remember that reboot of vSAN nodes can take a long time, 10-30 min. On the console of the server it says: “vSAN initialising SSD XXX” Give it time, it will boot.

Fetching firmware version

When you use the HPE custom VMware image then we have all the HPE tools on the server, so that we can query hardware etc.

  1. cd /opt/smartstorageadmin/ssacli/bin
  2. Execute ./ssacli ctrl slot=0 pd all show detail

For more a command cheat sheet you could look at or the official documentation

This will give you all info on the disks behind the controller. The model number is the one that you can look up on HPEs site to see if its affected.

vRO – start a workflow with AQMP message

System to system interaction can be hard. API integrations are a way of doing it, but we can also use a message bus. Actually I think that using a message bus is a very awesome way of doing it, it’s a very loose couple between systems and we can queue multiple things and have the task or messages in RabbitMQ until system is ready to consume the messages.

This is a guide of using vRO to pick up the RabbitMQ message and start a workflow with the payload of the message.

Adding RabbitMQ to vRO:

  • Open vRO legacy client, login
  • Expand Library > AMQP > Configuration
  • Start the workflow Add a broker. This will pair vRO and RabbitMQ.
  • Follow the wizard. I’m using a virtual host, but input all depends on your RabbitMQ config.
Running the workflow “Add a broker” to pair vRO and RabbitMQ.

Subscribe vRO to a RabbitMQ queue:

From the same position in vRO, now run the workflow “Subscribe to queues“. This will make vRO aware of the queue and be able to use it as a trigger.

Subscribing to a RabbitMQ queue.
Give the queue subscription a name.
Choose the broker we added in the section before.
Add the name of the queue from RabbitMQ, must be identical from what the queue name is in RabbitMQ.

Now vRO is monitoring the queue, we are ready to proceed.

Creating a policy:

Now we need to create a policy that can tie the event of a message from RabbitMQ into a starting a workflow.

Navigate to the policy fane in vRO and create a new policy.
Inside the new policy, add a new policy element.
Choose the AMQP:Subscription.
From the popup you can now filter for the queue that you earlier made a subscription on.
Right-click the queue name and chose Add trigger event
Choose onMessage
Holding focus at the “onMessage” event we can choose a workflow that the policy will start on message.
You will need to search, this time its for the workflow that you want to start when there is a message in the queue.
Now we will need to map a variable to the payload of the message that we have received. You need to have an in variable declared as a string in the workflow that you have mapped to.
The in variable from your workflow will show and by double-clicking on the name of the variable you are able to choose the “event.key.body” to it. This will make the payload of the message become a variable for your workflow.


That was a lot of screen dumps, I hope it still makes sense.

Now you should make the policy start when vRO starts and for now also start the policy so that you can see it works. The case that is shown in the screen dumps will have a JSON as the message payload, it then sends it to the messageBody variable and then inside the script it will extract the values that it needs in for the workflow to run.

Example of how to parse the JSON to a JavaScript object and afterwards parse the needed parameters over to a new variable. In this case its parameters for a PowerShell script.

Killing a virtual machine the esxcli way

Rarely I run into a ghost VM. I can’t do anything with the VM from vCenter or local UI for the ESXi host. It looks like it’s powered off but in fact, it’s still running in a sort of ghost state. You can vMotion all VMs of and reboot that hosts that the ghost VM is on. Sometimes its standalone hosts and then killing the VMs world with esxcli is easier.

  1. Connect to the ESXi host with ssh
  2. Get a list of all running VM worlds
esxcli vm process list

3. Identify the world from the output and take note of the World ID. From here we will kill the world. Start with type soft and then escalate it if it doesn’t work.

esxcli vm process kill --type=[soft/hard/force] -–world-id=ID

VM should now be killed, the VMX files are unlocked and you can manage the VM with the GUI tools again. If it didn’t work then you are left with the option to reboot the host containing the ghost VM.

Clear Windows print queue

Even as an infrastructure guy in a datacenter you now and then have to deal with printers. The small beasts with there own life and horrible drivers! I always forget the path for where the spooler puts its temp files. So now I put it here for future me to find it again next time I have to clear spooler files and restart det print services.

GUI way is slow and by having the CMD edition you can always guess what should be done in the GUI.

  1. Open an elevated command prompt.
  2. Type net stop spooler then press “Enter“.
  3. Execute del %systemroot%\System32\spool\printers* /Q
  4. Type net start spooler then press “Enter“.
  5. The print queue on your Windows should now be cleared.

ESXCLI host upgrade procedure

Most of the time you would want to use VMware Update Manager when doing upgrade. Its part of vCenter and is necessary tool when having to maintain your environment. But for smaller deployments, with standalone hosts and no vCenter the following upgrade methods are desired and can help the upgrade time. Instead of having to upgrade with IPMI and an ISO.

Online mode:

This method is for getting the update online, no need to download ISO/offline bundles, etc. This will work for most of the upgrade use cases.

1: Connect to your ESXi host via the host client and enable SSH. Afterward ssh to the ESXi host and enable ESXi firewall rule to allow the host to access the internet.

esxcli network firewall ruleset set -e true -r httpClient

2: With the beneath command you will get a list of available ESXi packaged that are on the VMware repos. Enter this command to list all available profiles. We filter only those which are relevant to our case – upgrade to ESXi 6.7

esxcli software sources profile list -d | grep -i ESXi-6.7

3. Chose the desired profile and use the following command for choosing and upgrading the ESXi version. Before upgrade its a good idea to enter maintenance mode.

esxcli system maintenanceMode set --enable true
esxcli software profile update -p ESXi-6.7.0-20190402001-standard -d https://hostupdate.vm

4. After it’s done, you will need to restart the host, after its rebooted you will run on the new ESXi version.

Custom, with Offline bundle:

This method is for when you desire to install a custom update, or that your hosts down have access to the internet.

1: Download the offline bundle from the VMware webpage, in this upgrade I will use an HPE custom version. But if you run a generic version, that will also work.

2: After downloading the “” file, place it (upload it) to a datastore which is visible by your ESXi host. Best would be a local datastore if this host has some. If not, it can also be a shared datastore too.

3: Find the profile name from the depot offline bundle

 esxcli software sources profile list -d /vmfs/volumes/prd.r60lun01/ISO/VMware-ESXi-6.7.0-Up

Put your host into maintenance mode, enable SSH if you haven’t done yet.

3: Execute this command to upgrade your ESXi 6.x to 6.7

esxcli software profile update -p ESXi-6.7.0-13006603-standard -d /vmfs/volumes/your_datastore/

esxcli software profile update -p HPE-ESXi-6.7.0-Update2-Gen9plus-670.U2. -d /vmfs/volumes/prd.r60lun01/ISO/

After checking that your upgrade was successful, reboot your host. You should see a message saying that the upgrade completed successfully.


I have tried to get an error with:

Failed updating the bootloader: Execution of command /usr/lib/vmware/bootloader-installer/install-bootloader failed: non-zero code returned…. return code: 1”

Error when upgrading, due to “insufficient space”.

This problem is due to the SWAP is but on the installation of the ESXi, not a good thing. So let’s change it.

Go to the UI of the ESXi Hosts https://IP/ui, login and proceed to the following:

Manage > System > Swap > Edit Settings

Chose the dropdown and select a datastore. Apply and the swap space is not freed from the ESXi install device so that you can try to upgrade again.


After the upgrade, it’s a good idea to disable the ESXi firewall rule for “HTTP outside access”. Stop and disable SSH again, but it’s optional 🙂

esxcli network firewall ruleset set -e false -r httpClient

Now you should have an upgraded host.

Juniper – CheatSheet

I always forget how to do curtain stuff on juniper equipment, the more I do in the CLI the better I become. Many of below commands and settings will be obvious for most, but I am still learning, so bear with me. As time goes I will add more commands and tips and tricks in this post.

LLDP – Showing other LLDP or CDP enabled neighbours that are on the other end of the wire. I used the enable on specific interface, thinking strict is better. So that I know where its uses LLDP.

- To configure LLDP on all interfaces:
[edit protocols lldp]
user@srx# set interface all

- To configure LLDP on a specific interface:
[edit protocols lldp]
user@srx# set interface interface-name

- show lldp neighbors
user@srx> show lldp neighbors

Configuring access/trunk interface – my SRX delivered the native vlan on a accessport over to the switch, I wanted it to be a trunk port because I added more vlan to be routed in the SRX. Could not get it to work. Thanks to the “commit confirmed” feature is could easily try gain. Fix was to have the native vlan with as a member in the trunk configuration….

- Access port
user@srx> set interfaces ge-0/0/0 unit 0 family ethernet-switching port-mode access vlan members 10

- Trunk port with native vlan, remember to add the native vlan to member of trunk, else you cut your self off.
user@srx> set interfaces ge-0/0/1 native-vlan-id 3
user@srx> set interfaces ge-0/0/1 unit 0 family ethernet-switching interface-mode trunk
user@srx> set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members vlan-trust

Juniper have a great feature when having to change config, you do you configuration change and then you commit it all. The commit command can also be chained with confirmed. If there is something in the config that cut your off management or if has some other flaw you can tell it to rollback if you haven’t confirmed the change after a specified time of commit.

- Save changes with commit and tell it to rollback unless you confirm the change. The last number is the number of minuts before the srx does auto rollback.
user@srx# commit confirmed 2

- Confim the config after it all went well
user@srx# commit check

- An other nice paremeter to commit is comment. Here you can write what your config change does, and maybe even what change request number is an behalf of. 
user@srx# commit comment "This adds vlan trunk to uplink switch on port ge-0/0/1."

- Just another nice parameter. Prepare the config and make it active on other schedule or when the service windows is open. just nice.
user@srx# commit prepare

-- When its time you can active it.
user@srx# commit activate

- If you are interested in what's happening in the commit process, then you can monitor it.
user@srx# commit | display detail

DHCP – When you Junos device is doing DHCP its nice to know the IP it handed out to took for that matter

- Showing the DHCP leases that it handed out
user@srx> show dhcp server binding

- Showing the DHCP leases that it took with its own DHCP client.
user@srx> show dhcp client binding

Rollback and compare, another nice feature to help you see what have been happening on the device. You can compare older with newer or current configs.

- Give you a compare of the config that was 3 revisions back with current config. 
user@srx> show system rollback 3 compare 0

NTP – settings up NTP.

- Add NTP servers and Junos will change over to use NTP instead of local time setting. Setting 5 servers from
user@srx# set system ntp server
user@srx# set system ntp server
user@srx# set system ntp server
user@srx# set system ntp server

- Verify NTP servers
user@srx> show ntp associations

Configure DNS

- Add DNS server for lookups
user@srx# set system name-server

- Verify from configuration
user@srx> show configuration system name-server

Add user and insert ssh key for quick access. Feel free to add my public key 😉

set system login user jvradm class super-user authentication ssh-rsa "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC/VPq30qzyJHr8v6qh1vF1CVY8R9U09iCkqnIs9H6d9hBOeDu/e52rPj2BOQUfHwBmGRPVqZUyuOO20hDgT/BzP0QxISv9l2OpFariz8AmHu9m4kUwAdrBDvplw8fFeafppUwQF/aFsIF+t1PtFluz0Bp3N/sp3NQGWfkez7myctGc9X3eMc6oUAYrPPJeDZz1x5JoGdwdH/w6wjr3uK03kRx6TX1kNqxSypIQQ/8lYg1TG7yAuF5DhX4fJrPjpiLau1H6z0vChVpqY1q8oMntzHHtYtByFMrNtWFfAvG94BT27h/Lkmz5JM5d41TbL0YdZT8zCTrXzUG87wdEaRiB5ZeKy9LENgfxKO66scSU2gjiXwpyJTrHKZYz9g5EERH/41w+qMT90HAM3ArSIvk7pROoKhZy0IeOwfWbmMlQvKQFjS7OtKnFEeVUYRqnLvi3XeUiFbLxmW3ID8IqQy3iDNuESiVNRcp/PoN7lxL9cfGJdXBuJ3PBcaQZx/vQpePRqW9eBSmhhS1beIUlLV0UOFdRGTMMMjOlp7m7jaw5EnvztbInfPOdMPoUuSL9iGut7M0SVMgEzo0MiJDHNdLQYK0EKO8qrWMz76UHhpdnhOQNdi3/wtVVzxVUR/D9zBa1q2oL8ml7jKVubVbBd6Vm0lEEquDEN3I9Dan/Ev0j9w== jr@mbp"

NSX 6.3.6 to 6.4.5 – Controller problem encountered

NSX upgrades can be a delicate thing to upgrade, even though everything is in its finest shape.

After we successfully have upgrade the NSX managers we proceeded with upgrading of the NSX Controllers. We did pre-check and issued command “show control-cluster status” and it looked fine, upgrade to 6.4.6 went well and we could vMotion VMs around after the controller was booted. But post-checks was not ok, the “show control-cluster status” did not return as expected and we where not confident to proceed with the host upgrades.

After some trouble shooting we found that the /var/log partition on 2/3 of the controllers where full. Without any other evidence we concluded that this was the problem. After some google-fu we didn’t find any KB or blogs on how to purge logs.

But we found out that we could get into a engeering mode that would give us shell access. Long store short, we did the following:

1. to gain shell access on manager
1.1 password is IAmOnThePhoneWithTechSupport
2. Extracting root passwords for controllers with /home/secureall/secureall/sem/WEB-INF/classes/ controller-nn
3. Loged into each controller, and issued : debug os-shell and thereby gain root shell access.
4. Deleted /var/log/syslog.1 on each node.
5. Rolling restart of controllers and after they booted they all joined the cluster.


After this we got the status as we wanted. In the mean while we had create a case with VMware support and the supporter was on a remote session with us. We told him what we have done, we verified that the controlleres was health and they where.

Next step, VIB upgrade on the hosts.

Good commands to know:

Edit: This article from VMware have the exact problem we encountered. We also contacted VMware Support, but before they where able to assist us we had the problem solved. 🙂

Process of getting the root password for controllers.

Tomcat9 and java12 on FreeBSD

Quick post, had to make Tomcat9 and Java12 work together. The procedure is as follows:

1. pkg install tomcat9 (it will also install java8)
2. pkg install openjdk12

Now and edit /etc/rc.conf with a parameter to start tomcat on boot and set the tomcat java_home.

And not to part that took me a long time to figure out. In Java12, there is no longer a feature that tomcat is using in its startup parameters. But if you remove that from the init script you are able to start it up. The line is: Djava.endorsed.dirs=’/usr/local/apache-tomcat-8.0/endorsed’ \

After this, you are now able to boot tomcat9 with java12 🙂

Getting all domains from Office365 tenants

Mail spoffing etc. is a big problem, there are technologies that can help, but many domain owners have not yet implemented them. To help our customers we have started to monitor and see if the SPF, DKIM and DMARC policies have been implementened, and if not we can help 🙂

Our own spamfilter solution have a button that gives you an export over all the domains, nice and easy, but Office 365 CSP portal doesnt.

So there is a quick script to help with that. Next post will hopefully contain the checkscript for if the domain have implemented SPF, DKIM or DMARC.

NSX Edge PowerShell manipulation

This is from a VMware support experience. A customer could not change DNS server parameters of the NSX Edge IP Pool. But actually is was a problem due to a bug in VCD 9.5, where a Edge XML config was missing some tags and therefor not being able to validate the XML when VCD post the edited XML config back to NSX manager.

I have attached VMware support answer in the bottom of the post.

Script will get all edges from the NSX manager, then you find the correct one and fill into the next part of the script. Then you get the XML down to a file on your local machine, you then edit the file and put in the missing tags and lastly PUT the XML backup NSX manager. After this operation, it works from the GUI again.

From Support:
– The issue you are seeing is a known issue 9.5.
– Like I mentioned in the previous email, this is due to missing elements from the xml.
– From the xml in the logs, I could see there are 52 NAT rules on that edge.Correct me if I am wrong. The following 2 rules had the elements missing

I have attached the file with the list of all the NAT rules seen from the logs if you need to cross-verify.

– To fix the issue,please follow

If you have any further questions,let me know.

Have a good evening,

Best regards,


Make a clone of VMs to NAS – The PowerCLI way

Quick post, had a customer that yearly wants a clone of there VMs, copied to a NAS and then shipped to customers HQ. The owner of the company put this as a requirement. Fair enough. I have almost always done the clone of the VMs by GUI, in the start this was easy because they only had 5 servers, but they now have more. So this time i wanted to try and script it instead. It toke my some extra time, but in the end i think its worth it. My PowerShell skills is not great, still learning so bear with me.

FreeNAS installation failed – Operation not permitted

Ran into a annoying problem, have been having this problem multiple times of past, and never remember what the fix is. So now its on the blog for next time that i need it. Use the shell option in the freenas installer, the disks that i wanted to install onto was ada0 and ada1.

1. sysctl kern.geom.debugflags=0x10
2. dd if=/dev/zero of=/dev/ada0 bs=512 count=1 && dd if=/dev/zero of=/dev/ada1 bs=512 count=1

This will wipe out the sectors that keeps the partition schema and afterwards you can install FreeNAS without problems.

ESXi – physical memory population

Had en interesting problem where a ESXi host only showed it had 30GB of memory, but the motherboard was populated with 6*8GB modules. In earlier versions of ESXi 5.5< it was possible to use dmidecode to show how the physical hardware was populated. But since 6.0> that have been removed.

The new command to find those kind of information are now “smbiosDump”

You can also just run smbiosDump without any paramenters and you get a hole lot of information to crawl through.

Timedrift – domain controller and clients

Customers domain controllers where both virtual and due to CPU congestion it seems that time had been drifting.

So it was 5 minutes behind, and so was the clients. The fix is as follows.

1. Find the DC that have the PDC role.

2. Issue the follwing command to sync the time with some of the servers.

3. After the time on the PDC again is correct, then issue following on the other domain controllers, that are not PDC.

4. let the clients resync there time, either wait for it to happen or issue the following

Freebsd – Going from stable to release

I outdated FreeBSD 10.1-Stable server needed to be updated for it to install packages again. Problem was, it was deployed from stable, i normally never use stable because it not production ready, its a development branche. But this server was stable and here are the steps to get it to a release train.

1. Update the source tree on the server.

2. Follow the link below. Only change i did was to “-j 6” when making so that it used multiple cores.

After the merge-ui command you just choose “later” for all the promt with merge of config files.

And volia, The server is updated to FreeBSD 10.4-release. Then you can update with the binary process, freebsd-update upgrade -r 11.2-RELEASE.

In case you need to know more about the stable vs. release, here is a link i found en the FreeBSD forum.

P2V of a FreeBSD

Was looking for someone that had done a P2V of FreeBSD, but me google skills was not good enough. So here it goes.

1. Make a VM with disk a bit larger then the source phycial machine.
2. Boot the VM on a FreeBSD live cd
3. Give it a IP address

4. Make NC listen for any input on port xxxx ( )

5 on the physical maskine you run DD and pipe it to NC.

6. wait for it to finish….

Ceph – slow recovery speed

Onsite at customer they had a 36bays OSD node down in there 500TB cluster build with 4TB HDDs. When it came back online the Ceph cluster started to recover from it and rebalance the cluster.

Problem was, it was dead slow. 78Mb/s is not much when you have a 500TB Cluster. So what to do?

There a several settings within Ceph you can adjust. Here are the two settings that worked for me.

osd max backfills:
Description: The maximum number of backfills allowed to or from a single OSD.
Default value: 1

I set it to 8, and the recovery went to 350Mb/s. Set it to 16 and recovery was 700Mb/s, but clients where also affected. So 8 was a more moderat setting.

osd recovery max active

Description: The number of active recovery requests per OSD at one time. More requests will accelerate recovery, but the requests places an increased load on the cluster.
Default value: 3

Set it up a notch to 4.


Mysql replication

Had to resync master-slave replication setup. Here are my notes on how it’s done.

Binary bin-log files are kept for 7 days on DB1. If the replication is stopped for more than a week DB2 replication cannot start again due to the binary log files is no longer available. therefor a fresh dump is needed and DB2 replication can be started again from the master log position.



  • Single-transaction, makes it possible to do the dump without locking the database, very useful when having to dump from a production database. But while not locking the DB you may not create or alter table schema.  Mysql documentation link
  • master-data, is very useful because it records the master position when doing the dump and putting it into the output of the dump file. Therefore it is much easier to start the slave from the correct position. The number 2 is for only printing it to the output as a comment. Mysql documentation link
  • event and routines, if there are any stored procedures or like in the old server we take them with us. Mysql documentation link

mysqldump --single-transaction --quick --master-data=2 --events --routines <DATABASE> | gzip > /data/<DATABASE> _`date +%F`.sql.gz

When the dump is done we move the dump file over to the other server. Here we import it to the MySQL server if there already were an old database in place, drop it and create it again.
zcat <DATABASE>.sql.gz | mysql <database>

Also, have a look at the head of the dump file where we will find the master position data that we need to start the replication again.

gzip -cd <DATABASE>.sql.gz | head -n24

Now we have the position and need the user for replication. I did it on an older 5.5 database, in newer MySql servers it is done differently.


Or if the user is in place and you just need to reset the password:

When it is imported we need to setup the master to master(slave) replication again. Remember to have a user on DB1 that is allowing replication from the DB2 server and have the user and password ready CHANGE MASTER TO MASTER_HOST='<IP>',MASTER_USER='repl', MASTER_PASSWORD='happyS3ed99', MASTER_LOG_FILE='mysql-bin.000849', MASTER_LOG_POS=758329777 ;

It will now start to replicate from the master, now you can do a “mysql -e ‘show slave status\G'” and see if the slave IO is running as it should.


Rescuing a Juniper SRX550

Notthing more greate than getting a call from HQ 30 minutes after closing hours. Never the less i decided to take the call. Network problem onsite at customer…. After getting green light from women in charge, i got in the car an when on to the customer.

Connection with the USB cable to the SRX console port i got a weird boot sequence. Just like the following:

Either the Junos partition was corrupt or the disk inside of the unit was fried. Decided to try and install Junos again just see if that would help. Went to and downloaded the oldest Junos version available, junos-srxsme-12.3X48-D10.3-domestic.tgz. Found a USB drive and put the .tgz file on it and plugged it in the SRX. From the console i broke the bootloader while its was trying to find kernel and issued the following command.

I began to install Junos, but when it tried to create partitions on the card, it died with DMA errors. Great!

Since a SRX550 is not something you find everyday and spareparts a hard to get (support was also expired) i decided to take the srx apart. happily to find a CF card inside and luckily i found a kingstone CF card in my bag (I knew that would come in handy someday). Swapped the card and put it together again.

Power on and issued the install command again. This time with success.

The install of Junos take sometime, a long time 20 minutes. But then you also get a very nice login prompt. logged in with root and no password. Went into cli configuration mode and did a “delete” to wipe the factory config. then loaded the backup configuration with

Pasted the 55kb JSON config into the console and finished with a ctrl+d followed by a commit. commit success and all network was suddenly alive again.

just to make all the LEDs green on the SRX i did wrote the config to rescue config. This is in operational mode.

A happy consumer and hopefully a new Juniper SRX1500 firewall on its way to relive the SRX550 off its duties.