If there is free booze and Virtualization; I'm there!
April 16, 2014Posted by on
In my last post I covered how to create a scripted Recovery Plan that allows you to specify a boot order between Virtual Protection Groups (VPGs). If you missed this check it out here before continuing:
Just to clarify, the Recovery Plan script and GUI integration is not required to perform failover operations in Zerto. Failing over all, selected or individual VPGs can be done in just a few clicks without any scripting whatsoever.
While the Recovery Plan script is very powerful you might not want to rely on a user having to find and run the script in a disaster recovery scenario. Based on this assumption I will now cover how to integrate the script into the Zerto GUI in Part 1 and then lock it down using a Self-Service Portal (SSP) in Part 2. Using a SSP ensures that the defined disaster recovery user can only initiate this Recovery Plan and nothing else. It also makes the failover operation so simple for the entire environment that I’m 100% sure my mum could even do it and not get it wrong!
First we need to create a dummy VM that we can use to create a VPG which will run the Recovery Plan script when a move, failover or failover test is performed. This should be a VM with a tiny amount of resources allocated such as 1vCPU, 1GB RAM and a 1GB thin VMDK. No OS is required to be installed as we just the VM in the inventory. You should call it RPG1 (Recovery Protection Group) or whatever naming convention is preferred. For an extra level of protection if you place it in VM folder that no account apart from the Zerto Virtual Manager service account can delete it from then it prevents a VMware admin accidently breaking disaster recovery. The RPG1 VM needs to be powered on for an initial sync and placed in a non-root resource pool. The reason for using a resource pool will be explained in part 2. Here you can see my example RPG1 VM:
We now need to create a VPG called RPG1, select any SLA, a target resource pool, datastore and networks etc in the Recovery site. These settings don’t particularly matter as we don’t care about failing over the VM itself. The most important setting is specifying a start.bat file as the pre-failover script and the Zerto operation variable as per:
Ensure that the timeout value in seconds is greater than value of all of the time it will take to recover the VPGs specified in the recovery plan. More specifically the NextVPGFailoverDelays specified in the ZertoRecoveryPlan.csv. This ensures that Zerto does not timeout the script before finishing the Recovery Plan.
This start.bat should be placed on both the source and target site ZVM in the same folder with the same name. It should contain the following:
The start.bat will call the Recovery Plan PowerShell script then pass on the variable of the Zerto Operation type being a move, failover or test. The Recovery Plan script in this blog post is slightly modified from the standalone script covered in the first post. This is due to the failover operation being specified by the Zerto operation not the CSV and a few other minor modifications. I have also added in a transcript to make it easier to diagnose issues after initiation from the Zerto GUI due to a lack of visibility in the script operation. Following is a link to the required example script and files ZertoRecoveryPlan-v1-Integrated.zip:
We now need to modify the Recovery Plan PowerShell script with the variables in sections 1-3 and configure the CSV with the required values as covered in the first post. Placed on both ZVMs in the same folder tree as per the start.bat file. Here is the example PowerShell script:
Below is the example CSV which controls the boot order, scripts and time delays between each VPG. You can see the only change from the standalone version being the removal of the FailoverAction due it being decided by the action in the Zerto GUI:
You will now have VPG called RPG1 which will automatically initiate the Recovery Plan from the Zerto GUI when you move, failover or test failover the VPG. Pretty cool I hope you will agree.
The only problem with this being that it is now confusing for a non Zerto user to know which VPGs to initiate failover on, as you will either want to failover an individual VPG or all of them using the RPG, not both. To remove this complication in Part 2 of this blog post I will cover integrating the RPG1 VPG into a self service portal making it the only option available to the user. This removes all complication from a DR perspective yet keeps the flexibility of per VPG operations for the daily use of Zerto. Happy scripting,
March 26, 2014Posted by on
To protect VMs with Zerto they need to be placed into Virtual Protection Groups (VPGs) which are consistency groupings of VMs that are typically configured on a per application basis. A VM can only exist in 1 VPG at once, you can only failover the entire VPG and you can define the boot order of VMs inside each VPG.
A common request I receive is to specify a boot order between VPGs (a recovery plan) so that you can bring VPGs online in a specified order with time delays and pre/post failover scripts. A perfect use case is bringing Application 1 (I.E a Finance DB) online before Application 2 (I.E a CRM). You could work around this by placing all the VMs that form both applications in the same VPG, but this then removes the fidelity of failing over an individual application in the cases of logical failures. Another use case is running a site wide script that should only be initiated when failing over everything.
Based on this requirement I decided to create the Recovery Plan script. I could spend the next month breaking down each section of the script however I have decided to get some useful scripts out in the wild then explain key elements retrospectively. You can download it from:
The script uses a combination of PowerShell and a CSV to failover each VPG specified in the order listed to the most recent point in time. It utilises the Zerto Cmdlets, many IF statements and the Zerto API. I need to thank Zerto Cloud Solutions Engineer Tamir Solomon for his example code on building the API interaction and authentication URLs. Each section of the script is explained with comments so you can clearly see how it works. Ensure you have met the requirements listed in:
The PowerShell script requires the first 3 sections to be edited which includes the Zerto settings, a location for the CSV and location for the log file. Sorry to state the obvious but it is definitely recommend to configure the script to use your Recovery site Zerto Virtual Manager and store it there! You then need to configure the CSV with the VPGs in the order in which you want them to be recovered:
To explain the settings available in the CSV:
- VPGName – The name of the VPG to failover! – This is both case and space sensitive so triple check that the name of each VPG exactly matches as configured in Zerto. The script will check the VPGName exactly matches before attempting any operation and log the result if not found.
- Action – Whether you want to perform a Failover or Failover Test operation. I recommend starting with Failover Test first. – Parameters; FAILOVER, TEST
- CommitPolicy – When using a Failover operation this sets the default commit policy for operation allowing you to rollback out of the point in time selected. This script will always failover to the most recent point in time so it is therefore recommended to use a manual commit policy. – Parameters; none, commit, rollback
- CommitTime – The time before applying the Commit operation if commit or rollback is selected. – Parameters; 0+ – number of seconds
- ShutdownPolicy – The action to perform on source VMs (if contactable) before performing the failover action on the VPG. I recommend forceshutdown which covers VMs without VMware tools but will try a graceful shutdown first. – Parameters; None, shutdown, forceshutdown
- RunScriptsinTest – Whether you want to run any specified scripts for a TEST operation as well as for FAILOVER. – Parameters; TRUE, FALSE
- PreFailoverScript – A PowerShell script to execute before starting the operation. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell
- PostFailoverScriptDelay – Time delay before executing the PostFailoverScript. Zerto won’t wait for the VPG action to complete before running the post failover script so this is configurable. – Parameters; 0+ – number of seconds
- PostFailoverScript – A PowerShell script to execute after starting the operation and the delay specified. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell
Warning: both pre and post failover scripts will run irrespective of the outcome of the Zerto operation so ensure you have performed a failover test before executing a recovery plan.
Once you have configured the PowerShell default variables, entered your VPGs into the CSV (hopefully with the Action set to TEST to begin with) then you are ready to rock! Every action performed by the script is logged so you can easily troubleshoot any issues. The exact same script can be run for both failover and failback as it just initiates the failover or test, it doesn’t ask where from and to as this is already configured in Zerto.
The script is executed as a standalone tool which has both advantages and disadvantages. In my next post I will cover how to edit and integrate this script into a separate Zerto VPG so it can be initiated from the GUI. This will then give the added benefits of simplicity, integration and the ability to control vCenter permissions around its execution. My favoured term for this is RPG, which stands for Recovery Protection Group, I’m hoping if I say it enough it will catch on! When I have posted this I will create a new tab on the blog dedicated to this script.
Any feedback, questions or improvements feel free to comment. Happy scripting,
March 18, 2014Posted by on
Just so I have the majority of the basics covered; in order to run Zerto cmdlets inside a script you have to add the snapin. This is easily done with the following command:
My preferred method of doing this ensures that the script checks it is installed first. You should therefore put this at the start of any script calling Zerto or VMware cmdlets:
Once you have added the Zerto snapin you can list all of the cmdlets available by entering:
The cmdlets currently available are:
For a description on what each cmdlet does and the correct syntax use:
March 14, 2014Posted by on
When using Zerto PowerShell cmdlets it can be required to select a checkpoint (a point in time) in the journal of changes from which to perform an operation. The Zerto PowerShell cmdlets which require a checkpoint to be defined are:
A checkpoint is also required for the POST APIs “Performing a Failover of a Specified VPG” and “Test a VPG for Failing Over”. To select a checkpoint the following cmdlet is used:
Below is an example output from running this cmdlet:
To explain this output we have Identifier which is simply an internal ID number. Timestamp being the exact timestamp consistent across all the VMs in the Virtual Protection Group (VPG) that you can select to use. The Tag is free text which can be inserted by a manual or scheduled checkpoint. Any checkpoints without a tag are automatically generated by Zerto every few seconds showing crash consistent points in time. VSS indicates if the checkpoint is application consistent using the Zerto VSS agent.
We now need to select a checkpoint as a variable for use with other Zerto cmdlets. There are 3 main ways of selecting the required checkpoint:
Latest Point in Time
Latest VSS Point in Time
Point in Time by Checkpoint Tag
Now that we have selected a Checkpoint, using the $latesttimeobject variable, we can start to look at what we can do with it. I will cover my favourite use cases for this over the next few posts. One idea I’ve had is to use this to report to alert on any VPGs that don’t have a VSS (or tagged) checkpoint in their journal then send an email. This could be a handy notification to receive on a daily basis. If anybody is interested in this let me know and it will be built! Happy scripting,
March 11, 2014Posted by on
To run PowerShell scripts with Zerto I recommend the following requirements be fulfilled, on the host running the scripts, as a best practice to cover running anything and everything:
- PowerShell 3.0 installed (available here)
- Zerto PowerShell Cmdlets 3.1 onwards (available from the Zerto Self-Service Portal here)
- Zerto PowerShell Security configured (see below)
- vSphere PowerCLI 5.5 (available here)
- Remote signed scripts allowed to run by running PowerShell as admin then the below cmd:
- Ignore invalid certificate option set ignore to speed up connect-viserver operations by running PowerShell as admin then the below cmd:
“set-PowerCLIConfiguration -invalidCertificateAction “ignore” -confirm:$false”
In order to use Zerto PowerShell cmdlets you need to configure a username and password for authentication on each Zerto Virtual Manager (ZVM) server using the following file:
C:\Program Files (x86)\Zerto\Zerto Virtual Replication\users.txt
The default username and password is administrator and password. To configure additional users it is required to generate a SHA-1 hash of the password then enter a new line with the username, a space using the tab key, then the SHA-1 hash of the password. The website below is a simple tool to create the hash:
Below is an example of a configured users.txt file:
After making changes to the user.txt it is required to restart the ZVM service.
In order to integrate scripts into Zerto operations they need to be specified as either pre-recovery or post-recovery scripts on a per Virtual Protection Group (VPG) basis, which you can see below:
The script is specified in “Command to run” and will be run on the target site ZVM Windows server. If the same script is required for Failback then it should be placed with the same name and folder structure in the source site ZVM also.
Any script specified will run for all 3 Zerto operations of Move, Failover and Failover Test. This means that you should use a logical filter script in front of your desired script to handle when and when not to run the script. Otherwise you could perform Failover Test operation and a script that is only required for Failover is run in production. A great example is a script to automatically update companywide DNS servers after failover, which definitely shouldn’t be run when performing a failover test.
To create a logical filter it is required to pass parameters to a script and use a batch file to intercept and process them as per below:
By specifying the %ZertoOperation% it is possible control which scripts to run from the start.bat for Move, Failover or Failover Test. The %ZertoVPGName% allows the name of the VPG to be passed through to a PowerShell script so a separate script is not required for each VPG (if required). Below is an example of a start.bat file acting as a logical filter and passing the VPG name to the PowerShell scripts specified:
As you can see from the start.bat example, different PowerShell scripts can run depending on the operation. It is also possible to run different scripts depending on the operation being Failover or Failback by specifying a ZVM hostname check. I prefer to do this from within PowerShell, but to do this from within the batch file use:
“if %computername%==TargetZVMHostName c:\FailoverScriptToRun.ps1”
“if %computername%==SourceZVMHostName c:\FailbackScriptToRun.ps1”
For performing the same operation in PowerShell use:
Now that we have covered the basics you can start to think about what you’d possibly like to automate with Zerto scripting. In subsequent posts in this series I am going to give examples and explain the logic behind each.
Not all examples will be run as part of failover operations. Many scripts can be run independently for extending the functionality of Zerto, but they will still have the same basic requirements as above. Happy scripting,
March 11, 2014Posted by on
Hi and welcome to my blog. I will be blogging about common queries I often come across in my role as Solutions Engineer for Zerto, as well as cool Zerto features, but I will mainly cover the many different PowerShell scripts I write in my spare time to extend the functionality of Zerto. Some of the scripts I will be releasing as standalone tools. For these I will create a new page dedicated to each tool to go into more detail.
I will say this many times but I can’t stress enough that all scripts on this blog are not supported by Zerto in any shape or form. I am however happy to provide ad-hoc support if you have any questions or issues and I recommend using the comments section on the relevant post to ask.
I might also be tempted to throw in some reviews of the odd Virtualization event and subsequent party. As a Yorkshireman I definitely enjoy a few free drinks!
Don’t worry I didn’t get through all of these :-), but I certainly did try! If you have any spcecific PowerShell scripts you would like me to cover in a blog post feel free to contact me using the form below:
I’m looking forward to sharing some really good content! Thanks,