Node introspection fails with clean failed message

Symptom

During Overcloud installation, in the introspection operation, a node fails with the clean failed message.

Cause

This issue occurs intermittently without any specific reason.

Action
  1. Log in to Undercloud as a stack user with the appropriate credentials and run the following commands: 
    source stackrc
openstack baremetal node list
  2. Determine the UUID of the failed node from the output of the previous command.
  3. Retrieve the iLO IP address of the failed node using the following command: 
    openstack baremetal node show <failed node UUID>|grep ipmi_address
  4. Log in to the NPS toolkit VM and run the following commands: 
    export API_IP=<NPS Toolkit VM OAM/Customer Network IP address>
export NPSADMIN=<user name of admin mentioned in the nps_secret.yaml file>
export NPSPASSWD=<password of admin user mentioned in the nps_secret.yaml file>
export TOPOLOGY_NAME=<value of the field "topology_name" in the input.json file>
export VIM_TYPE=<VIM platform type like RHOCP/RHOCP/VMWARE>
nps show --data servers
  5. Determine the iLO IP addresses and the corresponding UUIDs of introspection failed servers from the result of the previous command.
  6. Delete the failed nodes using the following commands: 
    nps deploy -s vim_overcloud –a delete-nodes -i \
<serverUUID1,serverUUID2,serverUUID3>

    where serverUUID1 is the UUID of the NPS toolkit server. To view the UUID of the NPS toolkit server, run the following command: 

    nps show --data servers
  7. Register the nodes again using the following commands: 
    nps deploy -s vim_overcloud -a register-nodes
  8. Introspect the failed nodes again using the following command: 
    nps deploy -s vim_overcloud -a introspect-nodes