Riverbed Steelhead Appliance 5050 - Error "Flash disk error detected"

Support Center

Riverbed Steelhead Appliance 5050 - Error "Flash disk error detected" Issue A Riverbed Steelhead 5050 is reporting Flash Error : The Flash Error alarm is triggered under REPORTS, DIAGNOTICS, ALARM STATUS. System information: Hostname: XXXXXXXXXXXXXX Model: 5050 Serial Number: XXXXXXXXXXXXXX Version: rbt_sh 6.5.4 #147_15 2012-02-09 17:56:24 root@basel:svn://svn/mgmt/branches/canary_147_fix_branch Date: 2012/09/03 10:08:08 Uptime: 104d 22h 37m 18s Flash disk error detected. It is not writable and possibly not readable.Perform a "reload" to fix this issue. Solution There is a known issue reported and fixed by Riverbed: BUG # 63970 - FAILURE OF INTERNAL USB FLASH ON STEELHEAD 5050, 6050, AND 7050 APPLIANCES LEADS TO A VARIETY OF SERIOUS PROBLEMS Fixed in: RiOS 6.1.1, RiOS 5.5.9, RiOS 6.1.2, RiOS 6.0.4 Description: Fixed a problem where an unresponsive internal USB flash device could result in a failure to write changed configurations, a system reboot, or a ping but otherwise unresponsive system. When this occurred, the system would typically log messages like SCSI error : <1 0 0 0> return code = 0x70000. This behavior has been observed primarily on Steelhead 5050, 6050, and 7050 appliances that are also under heavy memory load. A common symptom is the message Internal Error (Code 1003) when an operator attempts to save a changed system configuration. The above version being reported rules out the bug mentioned as it is a newer release that already has the bug fix in it. The message above does say Perform a reload to fix this issue. So this would be the first thing to try. Steelhead xx50 models can have unexpected behavior due to issues with USB Flash Memory. The following list of symptoms can be experienced with an appliance affected by this issue: Inability to access the GUI/CLI. Inability to save the configuration using write mem or the GUI. Inability to see the configuration using show running. Inability to see the images configuration using show images. Inability to log in to the CLI and ending up in the CLI> prompt of the reduced shell. Inability to log in to the Steelhead appliance anymore. Inability to generate system dumps. Error messages about "holes" in the file system. Inability to complete the installation of new software. The issue is mostly seen on the 3U Steelhead 5050, 6050, and 7050 models, but it can also occur on the 1U Steelhead 1050 and 2050 models. CX1555, EX1160 and EX1260 are also affected but already have the required changes outlined below in all available RiOS versions for those appliances. Steelhead xx50, xx55 and xx60 models store the RiOS operating system and configuration files on an embedded USB flash memory disk (eUSB). Due to a bug in the eUSB controller firmware, certain functions on appliances may malfunction or fail and in early versions of RiOS may lock up completely. The xx20 series hardware is not affected. Riverbed has now changed how RiOS accesses the flash device which means the Steelhead will operate fully even in the event that the flash has become unavailable. For Steelhead models 5050, 6050, and 7050, please upgrade to at least: RiOS 5.5.9b RiOS 6.1.3a RiOS 6.5.0 or later For Steelhead models 1050 and 2050, please upgrade to at least: RiOS 6.5.5 RiOS 7.0.3 or later For Steelhead models CX1555, EX1160 and EX1260 the required changes are included in the initially supported RiOS versions (RiOS 6.5.4, RiOS 7.0.1 and EX 1.0.0). Here the user is on 6.5.4 so the user has the fix. That is, the SH will still work with this issue and not like the older versions. NOTE: If the eUSB flash disk is currently failed you will need to use the reload command to perform a full power cycle and reset the eUSB flash disk prior to upgrading. That is, as per the message that was sent. Technical description of software fix: The entire eUSB content is read into a disk-backed memory cache at boot time. Reads then come from cache instead of from the eUSB, and can no longer trigger the controller FW failure. Appliance configuration files are moved from eUSB to RAIDed disk. Configuration is also asynchronously synced to eUSB as a backup after every change. If one of these syncs triggers an eUSB controller FW problem, the syncs are simply suspended until after a power cycle resets the eUSB device. A user will still see a small number of the above SCSI log messages, which confirms that the eUSB became unavailable. At the time of image upgrade, a power cycle of the appliance will be required if the Flash Firmware is in a locked up state to clear the state. A Flash alarm is implemented to notify that eUSB controller FW lockup has occurred and will require a power cycle at the time of upgrade. However no user action is required to maintain full functionality at the time of alarm. Users may elect to disable this alarm for convenience. If upgrade is attempted without power cycle, upgrade will fail with an error message. Machine can then be power cycled and upgrade completed. For user convenience, when the eUSB controller is locked up, issuing the reload command will cause a power cycle. The following lines should appear in the log files if a user experiences this problem: kernel: SCSI error : X return code = 0x70000 kernel: end_request: I/O error, dev sdX, sector 2107009 kernel: Buffer I/O error on device sdXX, logical block 0 The values of the sector number and logical block can vary, the important parts are the value of the device sdX and the SCSI return code value. For different hardware models, the name (designated sdX above) of the eUSB flash Disk is different. Model eUSB Flash Disk 1050-L, 1050-M sdb 1050-H sdc 1050-LR, 1050-MR, 1050-HR sde 2050-L, 2050-M, 2050-H sde 5050-L, 5050-M sdi 5050-H sdm 6050 sdq So the recommendation is: Check back through the message log files to see if one can find this error. Ensure that the user is on a version of RiOS that fixes the above issue. Reboot of the SH 5050 to clear the problem as per the message. If the problem continues, then RMA the Steelhead. ©Copyright 2025 Hewlett Packard Enterprise Development LP Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.

Issue

A Riverbed Steelhead 5050 is reporting Flash Error : The Flash Error alarm is triggered under REPORTS, DIAGNOTICS, ALARM STATUS.

System information:
Hostname: XXXXXXXXXXXXXX
Model: 5050
Serial Number: XXXXXXXXXXXXXX
Version:  rbt_sh 6.5.4 #147_15 2012-02-09 17:56:24
          root@basel:svn://svn/mgmt/branches/canary_147_fix_branch
Date:     2012/09/03 10:08:08
Uptime:   104d 22h 37m 18s

Flash disk error detected. It is not writable and possibly not readable.Perform a "reload" to fix this issue.

Solution

There is a known issue reported and fixed by Riverbed:

BUG # 63970 - FAILURE OF INTERNAL USB FLASH ON STEELHEAD 5050, 6050, AND 7050 APPLIANCES LEADS TO A VARIETY OF SERIOUS PROBLEMS

Fixed in: RiOS 6.1.1, RiOS 5.5.9, RiOS 6.1.2, RiOS 6.0.4

Description:

Fixed a problem where an unresponsive internal USB flash device could result in a failure to write changed configurations, a system reboot, or a ping but otherwise unresponsive system. When this occurred, the system would typically log messages like SCSI error : <1 0 0 0> return code = 0x70000. This behavior has been observed primarily on Steelhead 5050, 6050, and 7050 appliances that are also under heavy memory load. A common symptom is the message Internal Error (Code 1003) when an operator attempts to save a changed system configuration.

The above version being reported rules out the bug mentioned as it is a newer release that already has the bug fix in it. The message above does say Perform a reload to fix this issue. So this would be the first thing to try.

Steelhead xx50 models can have unexpected behavior due to issues with USB Flash Memory.

The following list of symptoms can be experienced with an appliance affected by this issue:

Inability to access the GUI/CLI.
Inability to save the configuration using write mem or the GUI.
Inability to see the configuration using show running.
Inability to see the images configuration using show images.
Inability to log in to the CLI and ending up in the CLI> prompt of the reduced shell.
Inability to log in to the Steelhead appliance anymore.
Inability to generate system dumps.
Error messages about "holes" in the file system.
Inability to complete the installation of new software.

The issue is mostly seen on the 3U Steelhead 5050, 6050, and 7050 models, but it can also occur on the 1U Steelhead 1050 and 2050 models. CX1555, EX1160 and EX1260 are also affected but already have the required changes outlined below in all available RiOS versions for those appliances.

Steelhead xx50, xx55 and xx60 models store the RiOS operating system and configuration files on an embedded USB flash memory disk (eUSB). Due to a bug in the eUSB controller firmware, certain functions on appliances may malfunction or fail and in early versions of RiOS may lock up completely.

The xx20 series hardware is not affected.

Riverbed has now changed how RiOS accesses the flash device which means the Steelhead will operate fully even in the event that the flash has become unavailable.

For Steelhead models 5050, 6050, and 7050, please upgrade to at least:

RiOS 5.5.9b
RiOS 6.1.3a
RiOS 6.5.0 or later

For Steelhead models 1050 and 2050, please upgrade to at least:

RiOS 6.5.5
RiOS 7.0.3 or later

For Steelhead models CX1555, EX1160 and EX1260 the required changes are included in the initially supported RiOS versions (RiOS 6.5.4, RiOS 7.0.1 and EX 1.0.0).

Here the user is on 6.5.4 so the user has the fix. That is, the SH will still work with this issue and not like the older versions.

NOTE: If the eUSB flash disk is currently failed you will need to use the reload command to perform a full power cycle and reset the eUSB flash disk prior to upgrading. That is, as per the message that was sent.

Technical description of software fix:

The entire eUSB content is read into a disk-backed memory cache at boot time. Reads then come from cache instead of from the eUSB, and can no longer trigger the controller FW failure. Appliance configuration files are moved from eUSB to RAIDed disk. Configuration is also asynchronously synced to eUSB as a backup after every change. If one of these syncs triggers an eUSB controller FW problem, the syncs are simply suspended until after a power cycle resets the eUSB device. A user will still see a small number of the above SCSI log messages, which confirms that the eUSB became unavailable.

At the time of image upgrade, a power cycle of the appliance will be required if the Flash Firmware is in a locked up state to clear the state. A Flash alarm is implemented to notify that eUSB controller FW lockup has occurred and will require a power cycle at the time of upgrade. However no user action is required to maintain full functionality at the time of alarm. Users may elect to disable this alarm for convenience. If upgrade is attempted without power cycle, upgrade will fail with an error message. Machine can then be power cycled and upgrade completed. For user convenience, when the eUSB controller is locked up, issuing the reload command will cause a power cycle.

The following lines should appear in the log files if a user experiences this problem:

kernel: SCSI error : X return code = 0x70000
kernel: end_request: I/O error, dev sdX, sector 2107009
kernel: Buffer I/O error on device sdXX, logical block 0

The values of the sector number and logical block can vary, the important parts are the value of the device sdX and the SCSI return code value.

For different hardware models, the name (designated sdX above) of the eUSB flash Disk is different.

Model eUSB Flash Disk

1050-L, 1050-M sdb
1050-H sdc
1050-LR, 1050-MR, 1050-HR sde
2050-L, 2050-M, 2050-H sde
5050-L, 5050-M sdi
5050-H sdm
6050 sdq

So the recommendation is:

Check back through the message log files to see if one can find this error.
Ensure that the user is on a version of RiOS that fixes the above issue.
Reboot of the SH 5050 to clear the problem as per the message.
If the problem continues, then RMA the Steelhead.

©Copyright 2025 Hewlett Packard Enterprise Development LP Hewlett Packard Enterprise Development shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HPE nor its affiliates, subcontractors or suppliers will be liable for incidental, special or consequential damages including downtime cost; lost profits; damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett Packard Enterprise Development and the names of Hewlett Packard Enterprise Development products referenced herein are trademarks of Hewlett Packard Enterprise Development in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.