Print | Rate this content

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 10

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2018-09-18

Last Updated: 2018-09-18


DESCRIPTION

Document Version
Release Date
Details
10
09/18/2018
Updated Resolution section with information on iLO 4 Firmware 2.61.
9
06/28/2018
Updated link in Resolution.
8
06/08/2018
Updated Resolution, including updated version of the iLO 4 firmware and Intelligent Provisioning considerations.
7
01/23/2018
Added online Intelligent Provisioning update to Resolution.
6
08/22/2017
Updated the Resolution with specific steps to be completed.
5
05/24/2017
Updated the Resolution with additional details on running the script.
4
12/05/2016
Updated the Resolution with additional details on running the script.
3
08/08/2016y
Updated the Resolution with details on running the script.
2
07/21/2016
Updated the Description with additional errors that are observed and updated Resolution to include Step 2 through Step 4.
1
03/31/2016
Original Document Release.

On an HPE Gen8-series or HPE ProLiant Gen9-series servers with HPE Integrated Lights-Out 4 (iLO 4), the NAND flash device may not initialize or mount properly, which may cause a variety of symptoms, which are listed below:

AHS Errors

  • AHS Logs display a blank date when performing the following:
    "Select a range of the Active Health log in days From: _______ To:_________"
    AHS file system mount may fail with (I/O Error) or (No Such Device).
  • The HP Active Health System (AHS) Logs are unable to be downloaded. AHS Data is not available due to a filesystem error.
  • The iLO Diagnostic tab will display the following error message: "Embedded media manager failed initialization" or "The AHS file system mount failed with (No such device) or "The AHS file system mount failed with (I/O error)" or "Controller firmware revision 2.09.00 Could not partition embedded media device."
  • Unable to download AHS log and "bb_dl_disabled" is displayed.

Embedded Media Errors

  • Controller Firmware Version 2.09.00 may fail to restart.
  • Unable to partition Embedded media device.
  • Embedded media manager may fail initialization.

Intelligent Provisioning Errors

  • Intelligent Provisioning will not execute when selecting F10.

OneView Errors

  • Remote Insight/Integrated Lights-Out Self-Test Error 8192.
  • Unable to register this HP OneView instance with iLO: There was a problem with posting a command to the iLO.
  • Unable to register this HP OneView instance with iLO: The iLO initialization was unable to complete.
  • Unable to determine if this server hardware is being managed by another management system. Received an error from iLO <ip address of iLO> with Error: Blob Store is not yet initialized. and Status: 126

SCOPE

Any HPE ProLiant Gen8-series or HPE ProLiant Gen9-series server running iLO 4.

RESOLUTION

The resolution for this issue may take several steps that need to be completed in the order specified below.

OVERVIEW

Step 1) Upgrade the iLO 4 firmware to version 2.61
Step 2) Perform a NAND format
Step 3) Check the iLO status If the iLO status is normal, then skip to Step 6 If the iLO status is still degraded, continue to Step 4
Step 4) Schedule downtime; AC power-cycle and repeat the NAND format
Step 5) Check the iLO status If the iLO status is normal, continue to
Step 6 If the iLO status is still degraded, then skip to Step 7
Step 6) Perform these final steps if the system board does not need to be replaced: Reboot the server; reinstall IP; and refresh the server in OneView (if server is managed by OneView)
Step 7) If steps 1-4 did not resolve the degraded iLO, replace the system board.

Note: The 2.61 iLO 4 firmware is a critical update. As such, HPE requires users to update to this version immediately. Install this update to take advantage of significant improvements to the write algorithm for the embedded 4 GB non-volatile flash memory (also known as the NAND). These improvements increase the NAND lifespan.

Additional considerations before performing a NAND format

A NAND format can be performed while the server is online in most cases.

Exception: For ESXi hosts booting from the Embedded SD Card - it is strongly recommended to perform the NAND format with the ESXi OS shutdown. This recommendation also applies when updating the iLO 4 firmware or resetting the iLO for ESXi hosts booting from the Embedded SD Card

A server AC power removal may be required (prior to the NAND format) in order for the NAND format to be successful. This can be accomplished for ML or DL servers by shutting down the server and disconnecting the power cables for a few seconds. For blade servers, an E-fuse can be accomplished by logging into the OA CLI and typing "reset server #" where "#" is the bay number of the blade.

An AuxPwrCycle feature was added in iLO 4 firmware version 2.55 so that the equivalent of an AC power removal can be performed remotely on a server.
Refer to Customer Notice "HPE Integrated Lights Out (iLO) 4 - RESTful Command to Allow an Auxiliary Power-Cycle Is Available in Firmware Version 2.55 (and Later)" located at the following URL:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00047494en_us

Required steps to perform after a successful NAND format

  • If the server is powered on when performing the NAND format, a server reboot is required after a successful NAND format. This reboot repopulates the RIS and RESTful data on the server during the next server POST with iLO 4 firmware version 2.53 or newer installed. If iLO 4 firmware version 2.50 or older is installed, connect to the iLO CLI (using putty) and use the following command "oemHP_clearRESTapistate" before rebooting the server to repopulate the RIS and RESTful data on the server.
  • Reinstall Intelligent Provisioning (IP) to ensure that it is working properly and reported as installed in the iLO 4 GUI "System Information - Firmware Information" page.
  • If using OneView: After the server is rebooted (to repopulate the RIS and RESTful data on the server), perform a server refresh in OneView; any existing errors in OneView will need to be marked as cleared after the refresh.

Detailed steps

Step 1. Upgrade the iLO 4 firmware to version 2.61. To download the firmware. The latest version of the iLO 4 firmware is available as follows:

Note: For ESXi hosts booting from the Embedded SD Card (Gen8/Gen9)- it is strongly recommended to perform this step with the ESXi OS shutdown.

  1. Click the following link:

    http://h20566.www2.hpe.com/portal/site/hpsc

  2. Enter a product name (e.g., "iLO 4") in the text field under Enter a Product Name or Number.
  3. Click Go.
  4. Select the appropriate product model from the Results list (if prompted).
  5. Click the "drivers, software & firmware" hyperlink under the Download Options tab.
  6. Select the system's specific operating system from the Operating Systems dropdown menu.
  7. Click the category Firmware - Lights-Out Management.
  8. Select the latest release of the HPE Integrated Lights-Out 4 Firmware version 2.61 (or later).
  9. Click Download.

Step 2. Perform NAND format using one of the methods detailed in the customer advisory: using one of the methods detailed in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning
Find the document at:

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=a00048622en_us


Note: iLO 4 firmware version 2.44 (or later) is required to format the NAND using the following steps.

Step 3. Check the iLO status using the options available in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning (find the document at the link in Step 2).

If the iLO status is normal based on the above criteria, then skip to Step 6 If the iLO status is still degraded, continue to Step 4

Step 4. If the iLO status is still degraded, perform the following steps:

a) Schedule a maintenance window
b) Shut down the server
c) Perform an E-fuse (server blade) or AC Power Pull (DL / ML series servers) d) Perform the NAND format again (refer to the instructions in Step 2 above)

Step 5. Check the iLO status (refer to the instructions in Step 3 above)
If the iLO status is normal, continue to Step 6
If the iLO status is still degraded, then skip to Step 7

Step 6. Perform these final steps after the NAND format is successful:
a) Reboot the server
b) Reinstall Intelligent Provisioning (see additional details below)
c) If the server is managed by HPE OneView, perform a server refresh to bring the server back under management.

Note: Any existing errors in OneView will need to be marked as cleared after the refresh.

Step 7. If steps 1-4 did not resolve the degraded iLO, contact HPE support to arrange a system board replacement. Follow these steps to complete the remediation:

a) Open an HPE support case to arrange for a replacement system board / arrange a maintenance window
b) During the maintenance window, shutdown the server
c) Unassign the OneView profile (if the server is under OneView management)
d) Replace the system board
e) Enter Server Model and Serial Number via RBSU
f) Update to iLO 4 firmware 2.61 (or later)
g) Check iLO status (refer to Step 3 above)
h) Reassign the OneView profile (if the server is under OneView management)

Note: Any existing errors in OneView will need to be marked as cleared after the OneView profile is applied.

If additional assistance is needed, contact HPE support and reference Advisory a00019495en_us as follows:

  • In North America, contact HPE Customer Support at 1-844-806-3425.

OR



OneView Considerations

OneView relies on the NAND to be accessible to perform many operations such as adding a new server or applying a profile. Because of this there are several things that need to be understood when dealing with this issue in an OneView environment.

  1. OneView interaction with the NAND - OneView uses a portion of the NAND called the iLO blob store. If the blob store is not accessible, adding a server to OneView or assigning a profile to the server cannot be completed.
  2. Impact of an inaccessible NAND when managed by OneView - If the NAND becomes inaccessible after the server was added in OneView, there are several things that can cause an outage:
    1. E-fuse - If an E-fuse reset is performed, the server will not be able to be brought back under management until the NAND issue is remediated and this will cause an unexpected outage.
    2. Server is removed and reinserted - This would essentially be the same as performing an E-fuse, so the same information in the E-fuse section applies here.
    3. Un-assign a server profile - If a server profile is unassigned, the server profile will not be able to be reassigned until the NAND issue is remediated.
  3. Migration from Virtual Connect to OneView - If an enclosure is migrated from Virtual Connect and a server has an inaccessible NAND, that server will not be able to be added properly until the NAND issue is remediated. If an in-service migration is attempted on a server with an inaccessible NAND, an unexpected outage will occur. Reference advisory

    c05384185

    for more detail.
  4. Formatting of the NAND - The NAND format wipes the iLO blob store that is used by OneView. It is important to issue a server refresh after a successful NAND format to recover the blob that OneView uses.

Reinstalling Intelligent Provisioning Considerations

There are online and offline options for reinstalling Intelligent Provisioning

  • Intelligent Provisioning can be reinstalled online if the server is running Windows (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers only) or Linux (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers). The available online Windows packages are older versions of Intelligent Provisioning.
  • Intelligent Provisioning must be reinstalled offline if the server is running VMware (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers).
  • The Intelligent Provisioning software download links are provided below. Intelligent Provisioning versions
  • HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x
  • " HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

Note: Intelligent Provisioning version 3.x is for HPE ProLiant Gen10-series servers only.

Offline method considerations:

  • Run the Intelligent Provisioning Restore Media to restore the Intelligent Provisioning data. Instructions to restore Intelligent Provisioning are as follows:
  • Instructions to create a bootable DVD with the IP image are provided on the HPE Intelligent Provisioning recovery media download site under the Installation Instructions tab.

Reference:
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_170e0ac3cb6e4439bf313f137a#tab3

Note: The DVD can be used multiple times.

Please note the Intelligent Provisioning Recovery Media DVD may be remotely mounted using HPE Integrated Lights-Out 4 (iLO 4) Virtual Media functionality, in order to reinstall Intelligent Provisioning. Additional information is available in the HPE iLO 4 User Guide at the following URL regarding how to mount an ISO image (federated or un-federated) and perform basic virtual media operations. Reference Pages 189 and 223-237:

HPE iLO 4 User Guide:
http://h20566.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c03334051-16

In addition, it is possible to write a script that utilizes HPE Integrated Lights-Out 4 (iLO 4) to reinstall Intelligent Provisioning on multiple servers. HPE Lights-Out management processors support an advanced scripting interface for group configuration and server actions. Scripts would need to be customized for the specific environment and task. Sample scripts are available for customers to reference at the following URL: HPE Lights-Out XML Scripting Sample for Windows: https://www.hp.com/swpublishing/MTX-459b8adc29c04317ad1d6a6752

Intelligent Provisioning software download links

HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x
HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

a.) Linux RPM located at the following URLs: For HPE Gen8-series servers: http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen8/hp-firmware-intelligentprovisioning-ip-1.70-1.1.x86_64.rpm

For HPE Gen9-series servers: http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen9/hp-firmware-intelligentprovisioning-ip-2.70-1.1.x86_64.rpm

b.) The Windows files are located at the following URLs:

For HPE ProLiant Gen8-series servers: Intelligent Provisioning for Microsoft Windows 64-bit Operating Systems 1.64.0.0 (2 Mar 2017) cp031302.exe https://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_01b900175afc4ae68be6c87b39
For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Windows x64 2.50.0.0(6 Jan 2017) cp031091.exe https://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_b031497eafd94b9ea2156cdef6

c) ISO images (e.g. for VMware) on Hewlett Packard Enterprise Support Center located at the following: http://h20566.www2.hpe.com/hpsc/swd/public/readIndex?sp4ts.oid=1008862660&lang=en&cc=us

For HPE ProLiant Gen8-series servers: Intelligent Provisioning Recovery Media, version 1.70(9 Oct 2017)
For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Gen9 Servers, version 2.70(b)(28 Feb 2018)




NOTE: Products sold prior to the November 1, 2015 separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. may have older product names and model numbers that differ from current models.




RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL: Proactive Updates Subscription Form.

NAVIGATION TIP : For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document .

SEARCH TIP : For hints on locating similar documents on HPE.com, refer to the Search Tips Document .

To search for additional advisories related to Linux, use the following search string:

+Advisory +ProLiant -"Software and Drivers" +Linux


Hardware Platforms Affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE ProLiant SL230s Gen8 Server, HPE ProLiant SL250s Gen8 Server, HPE ProLiant SL270s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant BL465c Gen8 Server Blade, HPE ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL320e Gen8 Server, HPE ProLiant DL360e Gen8 Server, HPE ProLiant DL385p Gen8 Server, HPE ProLiant ML310e Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE WS460c Gen8 Graphics Expansion Blade, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant ML310e Gen8 v2 Server, HPE ProLiant MicroServer Gen8, HPE ProLiant SL210t Gen8 Server, HPE ProLiant DL580 Gen8 Server, HPE ConvergedSystem 700x, HPE ProLiant XL220a Gen8 v2 Server, HPE ProLiant XL730f Gen9 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant ML350 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HPE ConvergedSystem 700x v1.1 VMware Kit, HPE ProLiant XL740f Gen9 Server, HPE ProLiant XL750f Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant SL4540 Gen8 1 Node Server, HPE ConvergedSystem 700 Virtualization 2.0 VMware Kit, HPE ConvergedSystem 700, HPE ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE WS460c Gen9 Graphics Expansion Blade, HPE ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Support Communication Cross Reference ID: SIK541
©Copyright 2018 Hewlett Packard Enterprise Development LP
Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!
Document title: Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors
Document ID: emr_na-c04996097-23
How helpful was this document?
How can we improve this document?
Note: Only English language comments can be accepted at this time.
Please wait while we process your request.