How to handle the VOLUME_ALARM_DATA_CONTAINERS_NONLOCAL alarm when it is raised in an HPE Ezmeral Data Fabric cluster.

MapR Core

When the VOLUME_ALARM_DATA_CONTAINERS_NONLOCAL alarm is raised in an HPE Ezmeral Data Fabric cluster it indicates that a local volume and its containers are not fully populated on the local node.  The alarm is raised for each local volume on which the volumes and its containers are non-local.

Local volumes and their containers should reside on the same node. Some containers were created on another node, which may cause performance issues in YARN applications if the local YARN volumes are affected. When this happens the VOLUME_ALARM_DATA_CONTAINERS_NONLOCAL alarm will be raised and can be seen from the MCS and the output of the maprcli alarm list command.

To troubleshoot the cause of the alarm use the following steps:

1. Review the CLDB logs ($MAPR_HOME/logs/cldb.log) and find the alarm notification and the related volume. For example, in the below log message the volume "mapr.vm75-180.support.mapr.com.local.mapred" is the local volume with non-local containers:

   Copy
   cldb.log:2018-08-02 03:16:34,581 WARN Alarms [RScan]: composeEmailMessage: Alarm raised: VOLUME_ALARM_DATA_CONTAINERS_NONLOCAL:176615590:VOLUME_ALARM; 
   Cluster: my.cluster.com; Volume: mapr.vm75-180.support.mapr.com.local.mapred; Message: Containers of local volume are not on local node

2. Review the MFS logs on the corresponding node housing the local volume ($MAPR_HOME/logs/mfs.log-3) to identify any issues with container allocation for the local volume.
3. Recreate the local volume on that host if the volume contains only temporary data or contact the HPE Ezmeral Data Fabric Support team.

If the root cause of the alarm cannot be identified from the above logs and diagnostics or it appears the ALARM is due to a software defect and a fix is needed please open a support case with the HPE Ezmeral Data Fabric Support team via the Support Portal and provide all logs and diagnostic data collected as a result of the above steps.

https://docs.datafabric.hpe.com/62/ReferenceGuide/VolumeAlarms-VolumeWithNonLocalContainers.html