Advisory: (Revision) HPE Serviceguard for Linux - cmcld, cmproxyd, and qs Daemons May Fail To Run with Messages "Could Not/Failed To Set Realtime Priority"

Support Center

Document Subtype: Customer Advisory|Severity: High|Document ID: a00069245en_us|Last Updated: 2022-11-14|Document Version: 2

DESCRIPTION

Document Version	Release Date	Details
2	November 14, 2022	Updated the Resolution with the permanent fix, HPE Serviceguard for Linux version 12.50 (or later).
1	March 28, 2019	Original Document Release.

When CPU Accounting is enabled either explicitly or implicitly in systemd based Linux distributions, certain HPE Serviceguard for Linux processes cannot set their scheduler policy to SCHED_RR, which causes the Serviceguard cluster and/or Quorum Server to be unable to operate.

The following processes of HPE Serviceguard for Linux are affected:

cmcld, cmproxyd, and qs daemons

Additional information is available in the Red Hat Knowledge Base article, How to configure a RHEL 7 system to be able to run programs requiring Real-TimeScheduling, located at the following URL:

https://access.redhat.com/articles/3696121 Non-HPE site

This advisory will be updated when additional information specific to SUSE Linux Enterprise Server is available. The information below refers to the Red Hat Enterprise Linux Knowledgebase article.

When this occurs, the node may fail to join the cluster or the cluster may fail to start (i.e., cmruncl and cmrunnode will timeout and fail) or the Quorum Server may fail to start.

An error message similar to the following will be displayed in /var/log/messages or systemctl status output if running HPE Serviceguard for Linux (SGLX) 12.40 (or later) for all affected daemons:

Mar 12 14:13:19 node1 cmcld[1639]: Failed to set realtime priority.
Mar 12 14:13:19 node1 cmcld[1639]: This could be seen if CPU Accounting isenabled
Mar 12 14:13:19 node1 cmcld[1639]: on the system by any of the systemdservices.
Mar 12 14:13:19 node1 cmcld[1639]: For more information on how to resolve itplease refer to the
Mar 12 14:13:19 node1 cmcld[1639]: HPE Serviceguard for Linux documentation on
Mar 12 14:13:19 node1 cmcld[1639]: HPE Support Center athttps://support.hpe.com/hpesc/home
Mar 12 14:13:19 node1 cmcld[1639]: A Customer Advisory is published withadditional information on this problem.
Mar 12 14:13:19 node1 cmclconfd[1555]: The Serviceguard daemon, cmcld[1639], exited with a status of 1.

For versions of SGLX earlier than 12.40, the message will be similar to the following for the cmcld and cmproxyd daemons:

Mar 11 10:33:56 node1 cmcld[2939]: Could not set realtime priority: Operationnot permitted
Mar 11 10:33:56 node1 cmclconfd[2934]: The Serviceguard daemon, cmcld[2939], exited with a status of 1.

For Quorum Server, qs daemon in /var/log/messages will be displayed as follows:

Mar 24 19:12:18 node1 systemd: qs.service holdoff time over, schedulingrestart.
Mar 24 19:12:18 node1 systemd: start request repeated too quickly for qs.service
Mar 24 19:12:18 node1 systemd: Failed to start init script for ServiceguardQuorum Server.
Mar 24 19:12:18 node1 systemd: Unit qs.service entered failed state.
Mar 24 19:12:18 node1 systemd: qs.service failed.

When attempting to run the Quorum Server manually, a message similar to the following will be displayed:

# /usr/local/qs/bin/qs
Mar 24 19:14:45:0:Could not set realtime priority to 94: rtn=-1 Operation notpermitted
#

SCOPE

Any supported version of HPE Serviceguard for Linux running Linux operating systems that use systemd. As of the publication date of this advisory, Red Hat Enterprise Linux RHEL 7, SUSE Linux Enterprise Server 12, and SUSE Linux Enterprise Server 15 are affected.

RESOLUTION

To prevent this issue, upgrade to HPE Serviceguard for Linux version 12.50 (or later).

To download the HPE Serviceguard for Linux version 12.50 (or later), perform the following steps:

Click the following link: Hewlett Packard Enterprise Support Center
Enter a product name (e.g., "HPE Serviceguard for Linux") in the text search field and wait for a list of Suggested Products to display. From the Suggested Products list displayed, identify the desired product and select it.
The page should refresh to display the "DRIVERS AND SOFTWARE" tab and the components that support the selected product. From the "DRIVERS AND SOFTWARE" expandable filter menus on the top of the page:
Locate and select the appropriate HPE Serviceguard for Linux edition (Base, Advanced, or Enterprise) and version (12.50 or later).
- Note: To ensure that you have selected the latest version of the firmware/driver, click the Revision History tab to check if a new version of the firmware/driver is available.
- For more important information, review the Release Notes tab.
Click the Download button.

Typically, the daemons should run under the system slice directly and this slice will be enabled to allow real time priority. To determine if this issue is occurring, perform the following:

Below is an example of a system that is not affected:

# find /sys -name cpu.rt_runtime_us|wc -l
1
Note: The number = 1. This indicates that CPU Accounting is not enabled and this issue is most likely not occurring.

# grep cpuacct /proc/$$/cgroup
3:cpuacct,cpu:/
Note: The output ends in a trailing "/" character. This indicates that CPU Accounting is not enabled and that this issue is most likely not occurring.

Below is an example of a system that is affected:

# find /sys -name cpu.rt_runtime_us|wc -l
100
Note: The number is greater than 1. This is an indicator that this issue is occurring.

# grep cpuacct /proc/$$/cgroup
6:cpuacct,cpu:/user.slice
Note: There is no trailing "/" on the line. This is an indicator that this issue is occurring.

This typically occurs due to a service enabling CPU Accounting either explicitly or implicitly in its systemd startup file.

To determine which service is affected, run the following command as root on the system:

# find /etc/systemd/system.conf /etc/systemd/system /usr/lib/systemd
-type f | xargs grep -e CPUAccounting -e CPUWeight -e StartupCPUWeight
-e CPUShares -e StartupCPUShares -e CPUQuota |grep -v -e :# -e
"^Binary file"

Below is an example output that may be displayed:

# find /etc/systemd/system.conf /etc/systemd/system /usr/lib/systemd
-type f | xargs grep -e CPUAccounting -e CPUWeight -e StartupCPUWeight
-e CPUShares -e StartupCPUShares -e CPUQuota |grep -v -e :# -e
"^Binary file"

/etc/systemd/system/long_sleeper.service:CPUQuota=20%

As shown above, a service (long_sleeper) that is setting CPUQuota in its service unit file - This creates the scenario described in section 3.3 of the Red Hat Enterprise Linux Knowledgebase document (implicit case). Any of the keywords above may cause the issue to occur if they are enabled in a service unit file.

If turning off the service is an option, this is the recommended workaround. For questions regarding the service, refer to the vendor of the service.

# systemctl stop long_sleeper
# systemctl disable long_sleeper
# shutdown -r now

This may also occur due to CPUAccounting set to yes in the /etc/systemd/system.conf file, as shown below:

As shown above, the global system default setting DefaultCPUAccounting=yes enabled in /etc/systemd/system.conf - This creates the scenario described in section 3.2 of the Red Hat Enterprise Linux Knowledgebase document (explicit case).

If disabling CPU Accounting in the global config file is an option, this is the recommended workaround, as follows:

# edit the /etc/systemd/system.conf and set DefaultCPUAccounting=no
Save the file and reboot:
# shutdown -r now

If disabling the suspect service or disabling CPU Accounting globally is not an option, perform the following:

Create the /etc/sysctl.d/sglx_init.conf config file that contains the following line:
kernel.sched_rt_runtime_us = -1

Note: for more information about sysctl config files, refer to:
man sysctl.d
or

https://www.freedesktop.org/software/systemd/man/sysctl.d.html
Reboot the system:
# shutdown -r now

After the system is rebooted, cmcld, cmproxyd and qs daemons should run as expected.
Note: The setting kernel.sched_rt_runtime_us = -1 disable the kernel's built in protection against run-away real time processes that by default are limited to consuming 95% of the CPU. For more information, refer to the following kernel.org document: https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt Non-HPE site

RECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL: HPE Email Preference Center.

NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches and other support software downloads, refer to the Navigation Tips document.

SEARCH TIP: For hints on locating similar documents on HPE.com, refer to the Search Tips document.

Hardware Platforms Affected: HPE Serviceguard for Linux

Operating Systems Affected: Not Applicable

Software Affected: Not Applicable

Support Communication Cross Reference ID: SIK4311

Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.