Print | Rate this content

CUSTOMER ADVISORY: DP 8.1x, configuration for known issues with Virtual Environment Integration backups

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04318203

Version: 1

CUSTOMER ADVISORY: DP 8.1x, configuration for known issues with Virtual Environment Integration backups
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2014-05-29

Last Updated: 2014-06-13


DESCRIPTION

Backup of Virtual Environment with DP 8.1x can fail intermittently with different errors or can run slow.

SCOPE

This advisory provides information about configuration changes for DP and for Virtual environment to have successful and faster backups.

Affected Versions:

Data Protector v8.1x

RESOLUTION

For successful and faster backups following configuration changes needs to be done, or some of them:
  1. Set omnirc variables.

    OB2_VEAGENT_RETRIEVE_PROPS_RETRY_COUNTER=10 to avoid problems with retrieving properties from vCenter. This problem is usually reflected as “Backup of object failed.” error in session output.

    OB2_VEAGENT_DISABLE_CBT_VALIDATION=1 to disable operation of CBT validation which creates additional snapshot. This will result in faster backups. However DP will not check CBT map and will not try to perform reset CBT if CBT map is not valid. If you see your CBT backups failing with error “Could not gather changed blocks” re-enable this validation so that DP can attempt to fix CBT map.

    Procedure:
    • Go to the location of omnirc file. This is by default “ProgramData\OmniBack\omnirc” on Windows. On Linux it is by default “opt/omni/.omnirc”.

    • Open omnirc file with text editor

    • Place variables with proposed values in the file and restart DP services.

  2. Use “Single” for snapshot behavior.

    In configuration of Virtual Machines setting for snapshot behavior needs to be set to “Single”. This will enable only incremental backups, or only differential backups, but not both in the same backup chain. This needs to be set to avoid errors “Could not backup disk scsiX:Y ...” and “Error removing snapshot” in some cases. This setting will not resolve all causes of these errors.

    Procedure:
    • Open DP GUI

    • Select “Backup” context

    • Open backup specification

    • Right click on some Virtual Machine (or some higher structure e.g. Resource Pool) and click “Configure Virtual Machines…” (If higher structure is selected, setting will apply to all Virtual Machines under that structure)

    • Deselect “Use common settings for selected VM”

    • Select “Single” in “Snapshot handling” section

  3. Increase “SmWaitForFirstBackupClient” and “SmWaitForFirstRestoreClient” timeout in global file.

    SmWaitForFirstBackupClient=120 (2 hours) and SmWaitForFirstRestoreClient=120 needs to be set in global file. This is needed to avoid failing of backups and restores due to timeout when a large amount of VM’s are selected. Error in session output is “Bar backup session was started but no client connected in 600 seconds. Aborting session!”.

    Procedure:
    • Go to the location of global file. This is by default “\ProgramData\OmniBack\Config\Server\Options\global” on Windows. On Linux it is by default “etc/opt/omni/server/options/global”.

    • Open global file with text editor

    • Place proposed value for timeout and restart DP services.

  4. Use CBT backups.

    To avoid “Error removing snapshot” failures or similar problems related to snapshots CBT needs to be enabled for backup.

    Procedure:
    • Open DP GUI

    • Select “Backup” context

    • Open backup specification

    • Right click on some Virtual Machine (or some higher structure e.g. Resource Pool) and click “Configure Virtual Machines…” (If higher structure is selected, setting will apply to all Virtual Machines under that structure)

    • Deselect “Use common settings for selected VM”

    • Go to “Advanced” tab

    • Select “Enable changed block tracking” in “Configure virtual machines” section (This will enable CBT on Virtual Machine/Machines)

    • Select “Use changed block tracking” in “Configure virtual machines” section (This will tell DP to use this functionality)

    • Select “Allow fallback to non-CBT backups” in “Configure virtual machines” section (In case CBT map is not stable, backup will fallback to non-CBT backup)

  5. Increase timeouts on vCenter and ESX hosts.

    In case of “Exception Occurred while creating VM snapshot” or “Backup of object failed.” errors increasing of timeout values in vCenter configuration file vpxd.cfg and ESX hosts configuration file vpxa.cfg can reduce occurrence of errors.

    Procedure for vCenter:
    • Log in to the vCenter Server with the appropriate permissions

    • Open the vpxd.cfg file in a text editor. The default location for the file is:
      C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
      For Windows 7 and Windows 2008, the default location for the file is:
      C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg

    • To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file between <config> and </config> tag :

      <task>
      <timeout>10800</timeout>
      </task>

      Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 30 hours.

    • To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file between <config> and </config> tag:

      <vmomi>
      <soapStubAdapter>
      <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
      </soapStubAdapter>
      </vmomi>

      Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 30 hours.

      KB document:

      http://kb.vmware.com/kb/1017253 Non-HPE site Extending the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager (1017253)

    • To increase heartbeat timeout value in seconds between ESX(I) hosts and vCenter Server, add the following values in the vpxd.cfg file between <config> and </config> tag:

      <heartbeat>
      <notRespondingTimeout>120</notRespondingTimeout>
      </heartbeat>

      Note: This parameter can be set to 60 or to 120 based on Vmware’s recommendation.

      KB document:

      http://kb.vmware.com/kb/1005757  Non-HPE site ESX/ESXi host keeps disconnecting and reconnecting when heartbeats are not received by vCenter Server (1005757)

    • Restart the vCenter Server service.

    Procedure for ESX hosts:
    • Log in to the ESXi/ESX host as root via the console or an SSH session.

    • Open the vpxa.cfg file in a text editor. The default location for the file is:
      /etc/opt/vmware/vpxa/vpxa.cfg

      Note: ESXi and ESX hosts come with open source tools which can be used for troubleshooting. The nano editor can be used in ESX , while the vi editor can be used in ESXi or ESX. For more information on these tools, see GNU nano and Vim.

    • To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file between <config> and </config> tag:

      <task>
      <timeout>10800</timeout>
      </task>

    • To increase the SOAP layer blocking call timeout, add the following values in the vpxa.cfg file between <config> and </config> tag:

      <vmomi>
      <soapStubAdapter>
      <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
      </soapStubAdapter>
      </vmomi>

      Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 30 hours.

    • Restart the vmware-vpxa service on the ESXi/ESX host.

  6. Increase operation timeouts on vCenter.

    In case of “Exception Occurred while creating VM snapshot” or “Backup of object failed.” errors increasing of operational timeout values on vCenter can reduce occurrence of errors.

    Procedure:
    • Connect to vCenter server with administrator privileges

    • Go to Home->Administration->vCenter Server Settings->Timeout Settings

    • Increase Normal Operations timeout to 600 in „Client Connection Timeout“

    • Increase Long Operations timeout to 2000 in „Client Connection Timeout“

  7. Check SAN settings.

    In case that backups are going over NBDSSL and not over SAN which is selected, settings for SAN need to be checked on backup host.

    Procedure for Windows backup host:
    • Run -> type regedit

    • Navigate to following path HKEY_LOCAL_MACHINE -> SOFTWARE -> Wow6432Node -> VMware, Inc. -> VMware Virtual Disk Development Kit

    • Open this folder

    • Inside should be placed variable: VerifySSLCertificates REG_DWORD (0)
      If this variable is not set, or set to different value, it needs to be set like this.

    Procedure for Linux backup host:
    • Go to /etc/opt/omni/client

    • Open vepa_vddk.config with text editor

      Inside the file the following variable should be set

      # whether to check SSL thumbprint on Linux
      vixDiskLib.linuxSSL.verifyCertificates=0

      If this variable is not set, or set to an different value, it need to be set as this.

    Details: http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vddk55_programming.pdf Non-HPE site

  8. If you are not using CBT backups:
    • Be careful when selecting Single/Mixed snapshot handling, because full backups will leave a snapshot open.
      • If a snapshot is left open for a long time the datastore space will be consumed and consolidating the snapshot can take hours.
      • If the system is quite busy it may suffer from performance problems.
    • See: VMware article: Estimating the time required to consolidate snapshots for VMware ESX and VMware ESXi (2053758)

    • Customer should plan long-lasting backups, if CBT is not used

  9. Upgrade to Data Protector 8.11 / 8.12 version from versions DP8.1 or below.
    • After upgrade customer should note that all planned first backups will run as full backup even when scheduled as incremental/differential, due to object presentations changes.

    • After upgrade from older version of DP in case of error "No Objects found for backup" recreate the barlist with the same selection as before and options and run the backup again.


Hardware Platforms Affected: Not Applicable
Components Affected: Not Applicable
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Third Party Products Affected: Not Applicable
Support Communication Cross Reference ID: IA04318203
©Copyright 2018 Hewlett Packard Enterprise Development LP
Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Legal Disclaimer: Products sold prior to the November 1, 2015 separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. may have older product names and model numbers that differ from current models.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!
Document title: CUSTOMER ADVISORY: DP 8.1x, configuration for known issues with Virtual Environment Integration backups
Document ID: emr_na-c04318203-3
How helpful was this document?
How can we improve this document?
Note: Only English language comments can be accepted at this time.
Please wait while we process your request.