How to perform a VMware recovery - x360Recover

Written By Tami Sutcliffe (Super Administrator)

Updated at August 16th, 2021

You can perform a Bare Metal Recovery when rescuing protected systems back into a VMware environment. Alternatively, you can perform a VMWare recovery.

VMware NFS Recovery

In a VMware environment, it is possible to utilize the NFS Export feature on an Appliance or Vault as a temporary data-store. This will simplify, and expedite recovery of protected systems when the underlying storage of the host server has been compromised, however, the physical host is still in operation. In order for this method to be employed, verify that there is sufficient free space remaining on the Appliance or Vault to hold the entire provisioned disk size for all protected systems you intend to recover. If insufficient disk space remains, refer to the VMware iSCSI Recovery section below.

To begin recovery of the affected protected systems:

  1. From the Web Interface, navigate to the Protected System Details page.
  2. Select the recovery point for the protected system that you want to restore.
  3. Click Export.
  4. Select VMDK as the format, and start the Export process.
  5. Monitor the progress of the operation from the Conversion tab in the Jobs menu pane.

    NOTE: For better performance, it is recommended not to select Export to USB. 

  6. From the Web Interface, navigate to the NFS Exports menu pane and select Enable NFS Exports. On a Vault, you will have to be logged in using a customer account in order to see this option.
  7. If not already present, click Add Allowed IP, and add the IP address of the VMware host server. In multi-server VMware environments, add all host addresses in your cluster to the allowed IP list if you desire to enable vMotion operations between hosts.
  8. From the VMware Client connect to each host and add the Axcient x360Recover device as an NFS storage location.
  9. Select Configuration, and then select Storage.
  10. Click Add Storage, choose Network File System, and then click Next
  11. For Server, enter the IP address of the Axcient x360Recover device.
  12. For Folder enter /export/admin for an Appliance or /export/<UserName> for a Vault, where <UserName> is the customer name of the protected systems you are trying to restore.

    Important: Do not select Mount NFS Read-Only.

  13. Enter a name for the new Datastore (for example, x360Recover ) and click Next.
  14. Click Finished. If successful, the new Datastore should appear in the list of storage locations.
  15.  Copy the VMDK export file (from the mounted iSCSI datastore) from the appliance to your local VMWare hosts datastore. Do this by going into the file browser, right clicking and copying the VMDK file, then pasting that into your local data store

Note:  If your VMware host storage is lost, skip step 15 and run the protected system directly from the BDR storage until such time as your host storage has been repaired.  Then complete step 15 to finish the recovery of the protected system.

When the Export jobs have finished for each protected system that you intend to recover, create a new guest server within the VMware Client.

  1. Choose the Axcient x360Recover data store as the server location, and configure the desired system resources and settings.
  2. When selecting disk drives, choose Use an Existing Hard Disk and browse to the exported disk volume(s) in the Axcient x360Recover datastore. 
  3. If necessary, edit the final guest server configuration, and add any additional disks to the system.
  4. Power on the guest server and complete any required configuration changes to the running system to return it to service. The recovered system may be run from Axcient x360Recover storage while waiting for repair or replacement of the failed host system storage array.
  5. When the host storage has been repaired, you can simply move the virtual disk and vmware guest files back onto the ESX host server.

VMware iSCSI Recovery

If disk space on the device is constrained, iSCSI export can be used to expose the protected system disks to VMware. This has the advantage of being faster, as we do not have to wait for the Export conversion process, but iSCSI is also more fragile.

Exported iSCSI target identifiers are not persistent through a reboot of the Axcient x360Recover device. The disks will have to be rediscovered and guest servers will have to be reconfigured if the Axcient x360Recover device is rebooted. 

IMPORTANT: Ensure you have selected Live Mode when starting iSCSI, or data loss will occur if the x360Recover device is rebooted. Data changes applied to the iSCSI disks is persistent and will not be lost when running iSCSI in Live mode, but the VMware configuration will have to be edited, as the iSCSI disk identification will change. It is recommended to use iSCSI Export with caution, and at present NFS Export is to be preferred.

VMware will only allow storing the virtual disk pointer files for Raw Disk Mapping (RDM) disks on native VMFS storage, so the host must have at least one small native VMFS datastore to allow creation of virtual machines using this method.  If the primary storage pool on the ESX host has been lost, you can install a temporary IDE/SATA disk in the system with which to create a temporary native VMFS storage pool.  This disk does not have to be very large, at it will only be used to hold the virtual machine definitions.

Configure each VMware host to allow iSCSI discovery from the Axcient x360Recover server:

  1. From Configuration, select Storage Adapters.
  2. If not already present, add the Software iSCSI Adapter to the host.

For ESX 5.1 and newer, configure iSCSI network communications as follows: 

  1. From Networking, select a virtual switch whose physical network adapters are connected to the same physical network as the Axcient x360Recover device. Typically this is vSwitch0.
  2. If there is not one already present, select properties and add a new VMKernel Adapter to the switch.
  3. Give it an available IP address in the same subnet as the Axcient x360Recover device. If the Axcient x360Recover appliance has multiple network adapters, you can use any configured subnet to isolate traffic onto separate controllers.
    • In complex networks, assign the VMKernel adapter to the correct VLAN.
  4. If more than one physical network adapter is associated with this vSwitch, edit the properties of the VMKernel adapter.
    • On the Nic Teaming page, move all but one network adapter to Unused, so that there is only a single network card active.
  5. From the Storage Adapters configuration page, edit the Software iSCSI adapter. Add the new VMkernel adapter on the Networking page.
  6. Add the IP address of the Appliance or Vault to the Dynamic Discovery tab.
  7. Save your settings.

For the protected system you want to recover, choose a recovery point, and click Start iSCSI to export the disks. 

From the VMware Client:

  1. select Storage Adapters from the Configuration page. 
  2. Select the Software iSCSI adapter and click Rescan All from the top right of the page.

Once complete, the newly exported iSCSI disks should appear in the lower storage management window. It is recommended to iSCSI export one system at a time and then change the default LUN names to a friendly name to simplify identification later. For example, change the random IQN names to <Server>_<Drive>.

Within the VMware client:

  1. Create a new Guest server stored on a native VMFS datastore. 
  2. When selecting disks, choose Raw Disk, and select the discovered iSCSI disks exported from the Axcient x360Recover device. The disk objects must be stored on a VMFS datastore.
  3. Boot the new Guest server on the VMware host and complete any reconfiguration necessary to return the system to service.

VMware: Relocating from iSCSI or NFS Back to Local Storage

When the host server storage has been repaired, perform the following steps to migrate the recovered systems from iSCSI or NFS back onto local storage.  

If you own VMware licensing and have a vCenter server, you may use Storage Migration from the VMware Client to easily relocate the guest system and storage back to the host.  Enterprise license owners can perform this operation while the guest is up and running, others will be required to power off the protected system first. 

  1. From the VMware Client, attach to vCenter and select the desired guest. 
  2. Right-click and select Migrate
  3. Chose Change Host and Storage Location and complete the remaining steps to pick a new host and storage location for the system. 
    • If recovering using iSCSI, in order to force the conversion of the Raw Disk (RDM) volumes into a VMDK, choose Advanced when choosing the storage location.
    • For each disk, explicitly select either Thick or Thin as the destination format.  The target datastore must be different than the current location of the virtual machine definition.

Those without vCenter licensing will have to perform a migration from the ESXi host shell. 

  1. If you have not already done so, enable and start the ESXi Shell and optionally the SSH server on the host. 
  2. From Security Profile in host Configuration, select Properties from the Services section. 
  3. Start ESXi Shell, and optionally SSH to enable access to the server command line interface.

    NOTE: It is recommended to perform the following steps from the actual console of the ESXi host rather than via a remote SSH shell in order to prevent the commands from failing if the remote connection is lost.

  4. Use vmkfstools to perform a disk conversion of the protected systems disks onto local storage.
  • Shut down the protected system and log in to the ESXi shell as root. 
  • If the target VMware volume is using NFS storage, copy the Guest virtual machine definition onto local storage as follows:

    cp -R <src> <dst>

    For example: cp -R /vmfs/volumes/x360Recover /Server01 /vmfs/volumes/Datastore1
  • If the Target VMWare is using VMSF, convert the existing disk file while relocating it to native storage as follows:

    vmkfstools -I <src path> <dst path> -d <thick|thin>

    For example: vmkfstools /vmfs/volumes/x360Recover/Server01_2016_02_01_11_00_00_pm_vmdk/ 57501948_e06c_4d4a_91e6_7ef6b7b689fe_C.vmdk /vmfs/volumes/Datastore1/Server01/C.vmdk -d thin

    The above command will work for both NFS Exported VMDK files and iSCSI exported Raw Disks as discussed in the topics above.

    NOTE: When typing names in the shell you may utilize tab completion to make entering long file paths easier.  Type the first few letters and press <Tab> to complete the name.  If the name is not unique, press <Tab> twice to list all matching possibilities.  Type a few more letters and press <Tab> again.

    There is no progress displayed during the conversion.  Wait for the shell to return to a new line with a cursor prompt.  You can monitor activity from the Performance tab within the VMware Client by observing disk read/write statistics.

When the conversion is complete, perform the following: 

  1. From the VMware Client, select the protected system and remove it. 

    Important: Do not select Delete from Disk.

  2. From Configuration select Storage and browse the Datastore location that you copied the machine definition files. 
  3. Open the folder containing the recovered server, right-click the .vmx file and import the Guest. 
  4. Edit the Guest configuration and remove all hard disks.

    Important: Do not select Delete from Disk.

  5. Save the virtual machine and then edit the Guest again.  
  6. Select Add Disk.
  7. Browse to the converted VMDK file within the local Datastore and select it. 
  8. Repeat to add any additional disks. 
  9. Save and boot the Guest.

IMPORTANT: After performing a full system recovery, regardless of the method used, it is imperative that a new Full Backup be run from the Appliance to synchronize the recovered system with the backup image.  Failing to perform a full backup may result in mismatched or missing data in the recovery image and can lead to corrupted backups and total loss of protection.