Virtually Sober

If there is free booze and Virtualization; I'm there!

Integrating a Scripted Recovery Plan into the Zerto GUI Part 1

In my last post I covered how to create a scripted Recovery Plan that allows you to specify a boot order between Virtual Protection Groups (VPGs). If you missed this check it out here before continuing:

https://virtuallysober.wordpress.com/2014/03/26/scripting-a-recovery-plan/

Just to clarify, the Recovery Plan script and GUI integration is not required to perform failover operations in Zerto. Failing over all, selected or individual VPGs can be done in just a few clicks without any scripting whatsoever.

While the Recovery Plan script is very powerful you might not want to rely on a user having to find and run the script in a disaster recovery scenario. Based on this assumption I will now cover how to integrate the script into the Zerto GUI in Part 1 and then lock it down using a Self-Service Portal (SSP) in Part 2. Using a SSP ensures that the defined disaster recovery user can only initiate this Recovery Plan and nothing else. It also makes the failover operation so simple for the entire environment that I’m 100% sure my mum could even do it and not get it wrong!

First we need to create a dummy VM that we can use to create a VPG which will run the Recovery Plan script when a move, failover or failover test is performed. This should be a VM with a tiny amount of resources allocated such as  1vCPU, 1GB RAM and a 1GB thin VMDK. No OS is required to be installed as we just the VM in the inventory. You should call it RPG1 (Recovery Protection Group) or whatever naming convention is preferred. For an extra level of protection if you place it in VM folder that no account apart from the Zerto Virtual Manager service account can delete it from then it prevents a VMware admin accidently breaking disaster recovery. The RPG1 VM needs to be powered on for an initial sync and placed in a non-root resource pool. The reason for using a resource pool will be explained in part 2. Here you can see my example RPG1 VM:

Dummy VM

We now need to create a VPG called RPG1, select any SLA, a target resource pool, datastore and networks etc in the Recovery site. These settings don’t particularly matter as we don’t care about failing over the VM itself. The most important setting is specifying a start.bat file as the pre-failover script and the Zerto operation variable as per:

RPGScreenshot2

Ensure that the timeout value in seconds is greater than value of all of the time it will take to recover the VPGs specified in the recovery plan. More specifically the NextVPGFailoverDelays specified in the ZertoRecoveryPlan.csv. This ensures that Zerto does not timeout the script before finishing the Recovery Plan.

This start.bat should be placed on both the source and target site ZVM in the same folder with the same name. It should contain the following:

:: The PowerShell script will run irrespective of operation. The BAT file is passing the Zerto Operation into the PowerShell script as a variable
Powershell -command "& {C:\rpg\ZertoRecoveryPlan-v1-Integrated.ps1 %ZertoOperation%}"

 

The start.bat will call the Recovery Plan PowerShell script then pass on the variable of the Zerto Operation type being a move, failover or test. The Recovery Plan script in this blog post is slightly modified from the standalone script covered in the first post. This is due to the failover operation being specified by the Zerto operation not the CSV and a few other minor modifications. I have also added in a transcript to make it easier to diagnose issues after initiation from the Zerto GUI due to a lack of visibility in the script operation. Following is a link to the required example script and files ZertoRecoveryPlan-v1-Integrated.zip:

http://1drv.ms/1fjoqcO

We now need to modify the Recovery Plan PowerShell script with the variables in sections 1-3 and configure the CSV with the required values as covered in the first post. Placed on both ZVMs in the same folder tree as per the start.bat file. Here is the example PowerShell script:

RPGScreenshot3

Below is the example CSV which controls the boot order, scripts and time delays between each VPG. You can see the only change from the standalone version being the removal of the FailoverAction due it being decided by the action in the Zerto GUI:

RPGScreenshot4

You will now have VPG called RPG1 which will automatically initiate the Recovery Plan from the Zerto GUI when you move, failover or test failover the VPG. Pretty cool I hope you will agree.

The only problem with this being that it is now confusing for a non Zerto user to know which VPGs to initiate failover on, as you will either want to failover an individual VPG or all of them using the RPG, not both. To remove this complication in Part 2 of this blog post I will cover integrating the RPG1 VPG into a self service portal making it the only option available to the user. This removes all complication from a DR perspective yet keeps the flexibility of per VPG operations for the daily use of Zerto. Happy scripting,

Joshua

Scripting a Recovery Plan

To protect VMs with Zerto they need to be placed into Virtual Protection Groups (VPGs) which are consistency groupings of VMs that are typically configured on a per application basis. A VM can only exist in 1 VPG at once, you can only failover the entire VPG and you can define the boot order of VMs inside each VPG.

A common request I receive is to specify a boot order between VPGs (a recovery plan) so that you can bring VPGs online in a specified order with time delays and pre/post failover scripts. A perfect use case is bringing Application 1 (I.E a Finance DB) online before Application 2 (I.E a CRM). You could work around this by placing all the VMs that form both applications in the same VPG, but this then removes the fidelity of failing over an individual application in the cases of logical failures. Another use case is running a site wide script that should only be initiated when failing over everything.

Based on this requirement I decided to create the Recovery Plan script. I could spend the next month breaking down each section of the script however I have decided to get some useful scripts out in the wild then explain key elements retrospectively. You can download it from:

http://1drv.ms/1fjoqcO

The script uses a combination of PowerShell and a CSV to failover each VPG specified in the order listed to the most recent point in time. It utilises the Zerto Cmdlets, many IF statements and the Zerto API. I need to thank Zerto Cloud Solutions Engineer Tamir Solomon for his example code on building the API interaction and authentication URLs. Each section of the script is explained with comments so you can clearly see how it works. Ensure you have met the requirements listed in:

http://virtuallysober.wordpress.com/2014/03/11/scripting-with-zerto-basics/

The PowerShell script requires the first 3 sections to be edited which includes the Zerto settings, a location for the CSV and location for the log file. Sorry to state the obvious but it is definitely recommend to configure the script to use your Recovery site Zerto Virtual Manager and store it there! You then need to configure the CSV with the VPGs in the order in which you want them to be recovered:

ZertoRecoveryPlanCSV

Zerto Recovery Plan CSV with all of the options configurable.

To explain the settings available in the CSV:

  • VPGName – The name of the VPG to failover! – This is both case and space sensitive so triple check that the name of each VPG exactly matches as configured in Zerto. The script will check the VPGName exactly matches before attempting any operation and log the result if not found.
  • Action – Whether you want to perform a Failover or Failover Test operation. I recommend starting with Failover Test first. – Parameters; FAILOVER, TEST
  • CommitPolicy – When using a Failover operation this sets the default commit policy for operation allowing you to rollback out of the point in time selected. This script will always failover to the most recent point in time so it is therefore recommended to use a manual commit policy. – Parameters; none, commit, rollback
  • CommitTime – The time before applying the Commit operation if commit or rollback is selected. – Parameters; 0+ – number of seconds
  • ShutdownPolicy – The action to perform on source VMs (if contactable) before performing the failover action on the VPG. I recommend forceshutdown which covers VMs without VMware tools but will try a graceful shutdown first.  – Parameters; None, shutdown, forceshutdown
  • RunScriptsinTest – Whether you want to run any specified scripts for a TEST operation as well as for FAILOVER.  – Parameters; TRUE, FALSE
  • PreFailoverScript – A PowerShell script to execute before starting the operation. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell
  • PostFailoverScriptDelay – Time delay before executing the PostFailoverScript. Zerto won’t wait for the VPG action to complete before running the post failover script so this is configurable.  – Parameters; 0+ – number of seconds
  • PostFailoverScript – A PowerShell script to execute after starting the operation and the delay specified. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell

Warning: both pre and post failover scripts will run irrespective of the outcome of the Zerto operation so ensure you have performed a failover test before executing a recovery plan.

Once you have configured the PowerShell default variables, entered your VPGs into the CSV (hopefully with the Action set to TEST to begin with) then you are ready to rock! Every action performed by the script is logged so you can easily troubleshoot any issues. The exact same script can be run for both failover and failback as it just initiates the failover or test, it doesn’t ask where from and to as this is already configured in Zerto.

The script is executed as a standalone tool which has both advantages and disadvantages. In my next post I will cover how to edit and integrate this script into a separate Zerto VPG so it can be initiated from the GUI. This will then give the added benefits of simplicity, integration and the ability to control vCenter permissions around its execution. My favoured term for this is RPG, which stands for Recovery Protection Group, I’m hoping if I say it enough it will catch on! When I have posted this I will create a new tab on the blog dedicated to this script.

Any feedback, questions or improvements feel free to comment. Happy scripting,

Joshua

Adding Zerto cmdlets to a script

Just so I have the majority of the basics covered; in order to run Zerto cmdlets inside a script you have to add the snapin. This is easily done with the following command:

add-pssnapin Zerto.PS.Commands

My preferred method of doing this ensures that the script checks it is installed first. You should therefore put this at the start of any script calling Zerto or VMware cmdlets:

function LoadSnapin{

  param($PSSnapinName)

  if (!(Get-PSSnapin | where {$_.Name -eq $PSSnapinName})){

    Add-pssnapin -name $PSSnapinName}}

LoadSnapin -PSSnapinName   "VMware.VimAutomation.Core"

LoadSnapin -PSSnapinName   "Zerto.PS.Commands"

Once you have added the Zerto snapin you can list all of the cmdlets available by entering:

Get-Command | Where-Object{$_.PSSnapin.Name -eq "Zerto.PS.Commands"}

The cmdlets currently available are:

image

For a description on what each cmdlet does and the correct syntax use:

get-help clone-vpg

Happy scripting,

Joshua

Selecting Zerto Checkpoints in PowerShell

When using Zerto PowerShell cmdlets it can be required to select a checkpoint (a point in time) in the journal of changes from which to perform an operation. The Zerto PowerShell cmdlets which require a checkpoint to be defined are:

Start-failovertest
Clone-vpg

A checkpoint is also required for the POST APIs “Performing a Failover of a Specified VPG” and “Test a VPG for Failing Over”. To select a checkpoint the following cmdlet is used:

get-checkpoints -virtualprotectiongroup WebApp1 -zvmip 192.168.0.116 -zvmport 9080 -username root -password Zerto123 -confirm:$false

Below is an example output from running this cmdlet:

image

To explain this output we have Identifier which is simply an internal ID number. Timestamp being the exact timestamp consistent across all the VMs in the Virtual Protection Group (VPG) that you can select to use. The Tag is free text which can be inserted by a manual or scheduled checkpoint. Any checkpoints without a tag are automatically generated by Zerto every few seconds showing crash consistent points in time. VSS indicates if the checkpoint is application consistent using the Zerto VSS agent.

We now need to select a checkpoint as a variable for use with other Zerto cmdlets. There are 3 main ways of selecting the required checkpoint:

Latest Point in Time

$cp_list = get-checkpoints -virtualprotectiongroup WebApp1 -zvmip 192.168.0.116 -zvmport 9080 -username root -password Zerto123 -confirm:$false

$last_cp = $cp_list[$cp_list.Count-1]

$latesttimeobject = $last_cp | select-object timestamp | select -expandproperty timestamp

$latesttimeobject

Latest VSS Point in Time

$cp_list = get-checkpoints -virtualprotectiongroup WebApp1 -zvmip 192.168.0.116 -zvmport 9080 -username root -password Zerto123 -confirm:$false

$lastvss_cp = $cp_list | where-object {$_.Vss -eq "True"}

$latesttimeobject = $lastvss_cp | select-object timestamp | select -expandproperty timestamp

$latesttimeobject

Point in Time by Checkpoint Tag

$cp_list = get-checkpoints -virtualprotectiongroup WebApp1 -zvmip 192.168.0.116 -zvmport 9080 -username root -password Zerto123 -confirm:$false

$lastcp = $cp_list | where-object {$_.Tag -eq "Tag Name Here"}

$latesttimeobject = $lastcp | select-object timestamp | select -expandproperty timestamp

$latesttimeobject

Now that we have selected a Checkpoint, using the $latesttimeobject variable, we can start to look at what we can do with it. I will cover my favourite use cases for this over the next few posts. One idea I’ve had is to use this to report to alert on any VPGs that don’t have a VSS (or tagged) checkpoint in their journal then send an email. This could be a handy notification to receive on a daily basis. If anybody is interested in this let me know and it will be built! Happy scripting,

Joshua

Scripting with Zerto Basics

To run PowerShell scripts with Zerto I recommend the following requirements be fulfilled, on the host running the scripts, as a best practice to cover running anything and everything:

  1. PowerShell 3.0 installed (available here)
  2. Zerto PowerShell Cmdlets 3.1 onwards (available from the Zerto Self-Service Portal here)
  3. Zerto PowerShell Security configured (see below)
  4. vSphere PowerCLI 5.5 (available here)
  5. Remote signed scripts allowed to run by running PowerShell as admin then the below cmd:
    “set-executionpolicy remotesigned”
  6. Ignore invalid certificate option set ignore to speed up connect-viserver operations by running PowerShell as admin then the below cmd:
    “set-PowerCLIConfiguration -invalidCertificateAction “ignore” -confirm:$false”

In order to use Zerto PowerShell cmdlets you need to configure a username and password for authentication on each Zerto Virtual Manager (ZVM) server using the following file:

C:\Program Files (x86)\Zerto\Zerto Virtual Replication\users.txt

The default username and password is administrator and password. To configure additional users it is required to generate a SHA-1 hash of the password then enter a new line with the username, a space using the tab key, then the SHA-1 hash of the password. The website below is a simple tool to create the hash:

http://www.sha1-online.com/

Below is an example of a configured users.txt file:

Users.txt Example

After making changes to the user.txt it is required to restart the ZVM service.

In order to integrate scripts into Zerto operations they need to be specified as either pre-recovery or post-recovery scripts on a per Virtual Protection Group (VPG) basis, which you can see below:

image

The script is specified in “Command to run” and will be run on the target site ZVM Windows server. If the same script is required for Failback then it should be placed with the same name and folder structure in the source site ZVM also.

Any script specified will run for all 3 Zerto operations of Move, Failover and Failover Test. This means that you should use a logical filter script in front of your desired script to handle when and when not to run the script. Otherwise you could perform Failover Test operation and a script that is only required for Failover is run in production. A great example is a script to automatically update companywide DNS servers after failover, which definitely shouldn’t be run when performing a failover test.

To create a logical filter it is required to pass parameters to a script and use a batch file to intercept and process them as per below:

image

By specifying the %ZertoOperation% it is possible control which scripts to run from the start.bat for Move, Failover or Failover Test. The %ZertoVPGName% allows the name of the VPG to be passed through to a PowerShell script so a separate script is not required for each VPG (if required). Below is an example of a start.bat file acting as a logical filter and passing the VPG name to the PowerShell scripts specified:

:: If this is a test failover the below powershell test failover script will run

IF "%ZertoOperation%"=="Test" Powershell -command "& {C:\zerto_scripts\Re-IP-Windows-Automation-FailoverTest.ps1 %ZertoVPGName%}"

:: If this is not a test failover then it is either a move or failover so the below powershell failover script will run

IF NOT "%ZertoOperation%"=="Test" (

Powershell -command "& {C:\zerto_scripts\Re-IP-Windows-Automation-Failover.ps1 %ZertoVPGName%}"

) ELSE (

::if there is some kind of error then neither script will run

echo Nothing to do

)

As you can see from the start.bat example, different PowerShell scripts can run depending on the operation. It is also possible to run different scripts depending on the operation being Failover or Failback by specifying a ZVM hostname check. I prefer to do this from within PowerShell, but to do this from within the batch file use:

“if %computername%==TargetZVMHostName c:\FailoverScriptToRun.ps1”
“if %computername%==SourceZVMHostName c:\FailbackScriptToRun.ps1”

For performing the same operation in PowerShell use:

# Taking the variable passed from the batch file to determine the VPG on which to initiate the IP changes on

$vpgnamefrombatchfile = $args

# Specifying the Windows hostnames of the 2 ZVMs on which this PowerShell Script can run

$sourcezvm = "site1zvm"

$targetzvm = "site2zvm"

# Setting the current hostname of the server on which this script is being run in a variable

$currentzvm = hostname

if ($currentzvm -eq $targetzvm)

{

## run this for failover

}

if ($currentzvm -eq $sourcezvm)

{

## run this for failback

}

Now that we have covered the basics you can start to think about what you’d possibly like to automate with Zerto scripting. In subsequent posts in this series I am going to give examples and explain the logic behind each.

Not all examples will be run as part of failover operations. Many scripts can be run independently for extending the functionality of Zerto, but they will still have the same basic requirements as above. Happy scripting,

Joshua

Introduction to my blog

Hi and welcome to my blog.  I will be blogging about common queries I often come across in my role as Solutions Engineer for Zerto, as well as cool Zerto features, but I will mainly cover the many different PowerShell scripts I write in my spare time to extend the functionality of Zerto. Some of the scripts I will be releasing as standalone tools. For these I will create a new page dedicated to each tool to go into more detail.

I will say this many times but I can’t stress enough that all scripts on this blog are not supported by Zerto in any shape or form. I am however happy to provide ad-hoc support if you have any questions or issues and I recommend using the comments section on the relevant post to ask.

I might also be tempted to throw in some reviews of the odd Virtualization event and subsequent party. As a Yorkshireman I definitely enjoy a few free drinks!

IMAG0492

Don’t worry I didn’t get through all of these :-), but I certainly did try! If you have any spcecific PowerShell scripts you would like me to cover in a blog post feel free to contact me using the form below:

I’m looking forward to sharing some really good content! Thanks,

Joshua

Follow

Get every new post delivered to your Inbox.

Join 407 other followers