Best practices

x360Recover Essential User Guide

Written By Tami Sutcliffe (Super Administrator)

Updated at March 23rd, 2022


Below are some basic best practices that will improve your efficiency when working with x360Recover. 

  • If you want to increase the depth of your x360Recover knowledge, consider enrolling in our certification courses for x360Recover Direct-to-Cloud and x360Recover BDR (formally Replibit), which include interactive walkthroughs and quizzes  
  • And don't forget that you can always schedule a free, live, 15, 30 or 60-minute session with an Axcient expert, to review any specific question you might have: Free live training


Hardware and software

x360Recover recommends a minimum of 8GB of RAM and 4 CPU cores with support for Intel VT or AMD-V hardware virtualization. More RAM and CPU resources are recommended when protecting many systems, or if you plan to provide virtualization as a recovery option.  

Typically, appliances should be built using high-end workstation or low-end server equipment depending on the target environment. Vaults should be constructed on low- to mid-range server equipment for performance and reliability.

Dedicated IPMI or other out-of-band management is recommended to ensure robust remote management options when maintaining your fleet of devices.

External IP Addresses (if you are hosting your own cloud)

Each vault and management portal instance running in your data center is an independent machine.  Each of these devices will need their own dedicated Public IP address unless you are using a third-party VPN tunnel solution to route traffic from your customer appliances. Creating Public DNS records for each device is optional but recommended.

If you are using the Axcient cloud, these details will be taken care of for you.

Firewall Ports

  • LAN: Ports 9090-10000 need to be open between the protected system and the appliance.
  • WAN: Ports 443, 9080 and 9081need to be open between the appliance and the vault.
    Ports 22, 443, and 10000+ need to be open between all appliances and vaults and the Management Portal.
    • One port per managed device needs to be open above 10000. For example, if you have 50 appliances and vaults under management, you need to open ports 10000-10049.


Install on a vault or appliance

  • We recommend that Axcient x360Recover be installed on bare metal.
  • To install Axcient x360Recover as a virtual machine, refer to the Deploy x360Recover as a virtual machine (VM) article.
  • Axcient x360Recover offers the flexibility to select your own hardware or purchase pre-configured BDR appliances.
  • Axcient x360Recover BDR appliances can be specifically preconfigured for your customer location.
  • We recommend using a single disk, or a hardware RAID-1 disk volume for the operating system. Protection of the operating system disk is not critical, as there is a recovery procedure to reload and restore all device settings from the storage pool.
  • Unless you are using a SAN, we recommend deploying three or more physical disks and using ZFS Software RAID when creating the storage pool. ZFS RAID is more robust than hardware RAID and x360Recover has been optimized to provide disk failure and replacement of ZFS volumes.
  • When using a SAN, you may opt to create a RAID-0 storage pool consisting of iSCSI LUNs. (The disk redundancy will be provided by the SAN.)  For optimal performance, create a RAID-1 storage pool using multiple 10Gbe network adapters to provide access to the iSCSI disk volumes.  Ideally, each iSCSI volume should be provided by a separate SAN server.
  • DHCP should be enabled on the network when installing x360Recover to simplify the startup network configuration of the device for licensing and initial software upgrade. During configuration or after setup has completed, a permanent static IP address should be assigned to the device so that agents have consistent access to the device.

Deploy agents

Completely uninstall other backup software and reboot if possible

  • Some legacy backup software, such as Backup Exec or Acronis may not completely remove their proprietary VSS extensions when uninstalled from Add/Remove Programs. We recommend using their stand-alone uninstaller utility if available to ensure proper cleanup and removal of these agents.

Search for unknown large file creation

  • Perform a search for files greater than 50MB created within the last seven days. Verify that there are no ongoing large data dumps being performed on the server that can dramatically balloon the size of Axcient x360Recover incremental backup images. Some possible examples to avoid:  SQL database backups; Quickbooks company file backups; PST File exports or copies, Windows Server Backup images, etc.
  • If large file creation is discovered, alleviate the issue by disabling the source of the file creation, or retarget the file storage to a separate disk volume and then disable that volume from backup by the Axcient x360Recover agent.

[Additional details on best practices when working with x360Recover agents:]

Verify VSS settings

  • From an elevated command prompt, run ‘vssadmin list shadowstorage’. Examine the volume shadow copy size column and verify that it is NOT set to UNBOUNDED
  • The recommended default storage size is 10% of total space
  • If less than 15% of the disk volume is free, it is recommended to redirect shadowstorage to another volume with sufficient free space.
  • To modify shadowstorage configuration, run the following command
    ‘vssadmin resize shadowstorage /for=C: /on=D: /maxsize=3GB’
    Then replace the source, destination and maxsize values as appropriate.

Customers and locations (for x360Recover

  • When deploying Axcient x360Recover to customers with multiple locations, create only one customer account within the License Portal and assign multiple locations to the account.
  • Each customer location that has servers to be protected should have its own appliance. Axcient x360Recover does not recommend performing backup operations of remote servers over Intranet or VPN connections with less than 1Gigabit of bandwidth available.

Networks, switching, and routing

  • Appliances should be connected to the same physical LAN and VLAN as the protected systems they are guarding. Avoid transporting data through firewalls or routers to alleviate network congestion and performance bottlenecks.
  • Appliances and protected systems should be connected to 1Gigabit Ethernet switches or faster for best performance.

Backup time window

  • Microsoft’s best practice recommendation is to increase the minimum configured backup window by 15 minutes for every TB of disk size. (Total volume size, not space in use.) For example, a server with 2TB in total disk capacity should not be backed up any more frequently than every 30 minutes.

Backup scheduling

  • Axcient recommends performing backups on a 24/7 basis and not limiting backup windows strictly to expected business hours for the customer. Backups taken outside of working hours where no changes to data have been made will be extremely small and it is always better to have the option to restore from additional recovery points in the event of the unexpected.

Avoid scheduled defragmenting of disks

  • Performing disk defragment operations on systems protected by image-based backup solutions can generate unnecessarily large incremental backups due to excessive block changes. We do not recommend performing frequent or regularly scheduled defragmentation operations on systems protected by image-based backup solutions like x360Recover.
  • Prior to installing the Axcient x360Recover agent, we recommend performing a thorough disk defrag of both regular files and the Master File Table, if possible. This will help ensure that the initial base image for the protected system is in a healthy state.

Microsoft Exchange and SQL Server VSS agents

  • The Exchange VSS agent is disabled by default on Windows Server 2003 SBS. Ensure that this service is set to automatic startup and running in order to ensure consistent Exchange backups and log file cleanup.
  • The Exchange VSS agent is set to manual by default on Exchange 2008 and newer. Ensure that this service is set to automatic startup and running in order to ensure consistent Exchange backups and log file cleanup.
  • Verify that the Microsoft SQL VSS Writer service is set to automatic and is running in order to ensure consistent backups of Microsoft SQL databases.

OEM installations

  • Microsoft Windows systems distributed as OEM installations are not supported for virtualization. Many OEM distributions are deliberately altered so that they will not run as a virtual machine.
  • Virtualizing Microsoft OEM distributions is a violation of the Microsoft Licensing Terms and Conditions. Axcient x360Recover will not provide support for virtualizing OEM distributions of Microsoft operating systems.
  • Recovery of Microsoft OEM distributions to new hardware is a violation of the Microsoft Licensing Terms and Conditions. OEM distributions may only legally be recovered onto the original hardware on which it shipped.  Axcient x360Recover will not provide support for recovery of OEM distributions to replacement hardware.

Install agents

The x360Recover agent is designed and intended to work out of the box, without any special modifications in most cases.  

That said, there are still a few best practices that partners should consider:


Third party backup agents

In general, Axcient recommends running only a single backup agent on your protected system. 

However, we realize you may sometimes need to run concurrent backup solutions (for example, during a transition from one platform to another.)

The x360Recover agent has been tested to successfully coexist with a wide variety of third-party backup solutions, including both image-based and file-based solutions.  

  • If there are other non-Axcient backup agents installed on the machine we recommend that these other backups be configured to run at a different time than the x360Recover backups. This will prevent potential VSS snapshot collisions or high system utilization due to concurrent backup operations (which might impact user experience.)

Axcient x360Recover backups should be able to run successfully on the same machine configured with other backup vendors when necessary.


VSS snapshot disk space requirements

You'll need to verify that volumes have enough free disk space for Microsoft VSS snapshots. 

You'll also need enough free space available on the volume to satisfy data changes that might occur to the volume during the backup.  For the initial full base image backup, this might mean hours (or even days,  in extremely large systems.)  If the volume runs out of free space during a backup, Windows will automatically ‘free’ the VSS snapshot in use, causing the backup to fail.

  • We recommend that the shadow storage limit be set to at least 10% of the total size of the volume (or set to unlimited)
  • If a given volume has less than 15% free space, we recommend redirecting VSS shadow storage to another volume with more free space.

To do this, run the following command from an elevated command prompt: 

vssadmin resize shadowstorage /for=X: /on=Y: /maxsize=30GB

Notes:

  1. Change X: to the volume that needs to have its shadow storage area redirected
  2. Change Y: to the volume that should contain the shadow storage
  3. Change 30GB to the amount of space to reserve (This should be at least 10% of the total size of the volume being redirected)

Reduce application-level backups

Some applications have an option to generate file-level backups of their data and configurations.

This means scheduled application-level backup jobs might be writing large amounts of data to volumes that will also be backed up by x360Recover.

Examples:

  • Microsoft SQL server configured to take daily full database dumps, saving to a local volume
  • Windows backup configured to take full disk backups and save them to a local volume
  • Quickbooks generating large backup files on a regular basis.  

While x360Recover can backup these backups, doing so causes the total daily incremental data changes to be very large.  In fact, these large incremental data changes can possibly exceed what can be uploaded by the system’s internet bandwidth in one day. 

In general, x360Recover can capture the underlying application and database files successfully on its own. So, making such application-level backups is not necessary and can lead to excessive storage use on your backup appliance (along with prohibitively large incremental backups that are difficult to replicate off-site.)

Recommendations:

  • If you must make application-level backups, please store them to a local storage location that is excluded from x360Recover backups
  • QuickBooks may automatically generate application-level backups periodically when opening or closing the application. These daily backup files can be fairly large, but as long as the frequency of backups is not excessive, we recommend allowing Quickbooks to make such backups.
  • Some third-party business applications built on Microsoft SQL or other database platforms may produce automatic file-level backups on a schedule. Often, the product support for these types of applications requires that such backups be performed.  If the data change generated by such backups is excessive, we recommend configuring the backups to be stored on a disk volume excluded from x360Recover protection.

Check Windows system health

When deploying a new x360Recover agent, it is a good idea to evaluate the overall health of the system about to be protected.

Corrupt Windows system files can cause surprise issues with (a) Microsoft VSS snapshots (causing backups to fail) or (b) the P2V process required for instant backup virtualization.

To check for corruption in Windows system files (and get repair instructions), run:

sfc /scannow


Disk volume fragmentation

Disk volumes which are filled near to capacity (or have been filled near to capacity in the past) are likely to have a high level of file fragmentation.

Fragmentation occurs when there are no open disk spaces large enough to store a file in a single chunk. When this happens, the blocks must be written to disparate parts of the disk. 

  • Highly fragmented disk volumes can cause severe performance problems, especially when attempting to virtualize the system later during a disaster recovery.  If you suspect there is heavy fragmentation on the system, and the system is a physical machine with hard disks (and not flash storage), then defragment the volumes on the system prior to the first full backup.

The best time to defragment a volume is before the first full backup is taken of the system.

Note: Since x360Recover is an image-based backup solution, defragmenting a disk volume may cause an excessive amount of data change that will be included in the next backup.  Even though the files are not changed during a defrag operation, reorganizing the block data on the drive will be detected as change by the agent.  Remember: The best time to defragment a volume is before the first full backup is taken of the system.

Important note: Do not perform disk defragmentation on a schedule!  This will generate excessive and unnecessary block change and impact backup storage consumption for little reason.


Antivirus scanning issues

Axcient strives to conform to all recommended coding and security practices. 

Our x360Recover agent software is signed by an extended validation code signing certificate, and we follow Microsoft documented best practices and use cases. 

That said, the x360Recover agent is operating at a low system-hardware level to achieve successful image-based backups and occasionally, in rare instances, we have encountered issues with certain antivirus solutions. 

  • Generally, if problems occur, the issues involve heuristic-based scanners (like Sentinel One). Heuristic-based scanners rely on artfully analyzing applications behavior in real-time. This is in contrast with the more commonly used signature-based scanners whose detection uses specific chunks of code (or signatures) to identify malicious software. 

If you suspect that your antivirus solution is affecting your x360Recover agent, refer to this article Exclude an agent from anti-virus scans or contact Axcient Support for assistance.

Preload (Seeding)

What is preloading (seeding)?

A preload (seed) data set contains all of the initial backup files you wish to send to the Axcient Cloud. Preloading your backup data is an alternative method of sending the initial backup to Axcient (when transferring over the Internet is too slow.) 

For information on determining if a preload is your best option, review this knowledge base article, How long will my initial backup take?

What is WebSeed?

WebSeed is our web seeding platform. It provides tools to efficiently seed your backup data to your Axcient Data Center. [Web Seed Guide]

(If you do not yet have a WebSeed account, please create a new Support ticket in the x360 Portal.)

How to access WebSeed 

Depending on your data center, you will access your Webseed login using one of these links:





 SUPPORT    | 720-204-4500 | 800-352-0248 

  • To learn more about any of our Axcient products,  sign up for free one-on-one training.
  • Please contact your Partner Success Manager or Support if you have specific technical questions.
  • Subscribe to the Axcient Status page for a list of status updates and scheduled maintenance.

888