Shutting down VIM nodes, switches, and rack

Prerequisites

Ensure to create a backup of the systems (cloud, switches, and rack) before performing the shutdown procedure. For more information on creating backups, see Backing up and restoring VIM Undercloud.
Ensure to make a note of the iLO IP address of Undercloud Host, Overcloud controllers, and HCI or compute nodes before performing the shutdown procedure.

Procedure

Log into the Undercloud VM with username "stack".

Powering off the VMs in the Overcloud

To stop the VMs running in the Overcloud, run the following command:
```
source overcloudrc
nova list --all
nova stop <VM_UUID>
```
Where <VM_UUID> indicates the list of the UUIDs (separated by a blank space) of all the VMs.
To verify that no VMs are operational, run the following command:
```
nova list --all
```
Shut down the Overcloud.
- Shutting down Overcloud for HCI-based deployment
  1. To shut down overcloud for HCI-based deployment, do the following:
    1. Log in to the Undercloud VM with username "stack".
    2. To list all the HCI nodes running in the Undercloud, run the following command:
      source stackrc nova list
    3. Log in to the HCI node.
      ssh heat-admin@<hci_ip address>
    4. To stop nova services, execute the following steps:
      sudo docker ps | grep nova sudo docker stop nova_compute nova_migration_target nova_libvirt nova_virtlogd
    5. To verify that nova services are stopped run the following command:
      sudo docker ps -a | grep nova
      
      NOTE:
      The services must be in exited state.
    6. Repeat steps 'IV' and 'V' in all the remaining HCI nodes.
    Suspending the Ceph background operations
  2. Log in to the active overcloud controller with the username "heat-admin" using the VIP address.
    NOTE:
    The VIP address is mentioned in the field OS_AUTH_URL in the overcloudrc file available in the Undercloud VM.
    
    source stackrc ssh heat-admin@<VIP IP address>
  3. To suspend the Ceph background operations, run the following commands in the active Overcloud controller:
    sudo ceph osd set noout sudo ceph osd set norecover sudo ceph osd set norebalance sudo ceph osd set nobackfill sudo ceph osd set nodown sudo ceph osd set pause exit #Exit from the active controller
    Stopping the cluster service on the Overcloud controller
  4. To get the list of Overcloud controllers running in the undercloud, run the following command from the Undercloud VM:
    source stackrc nova list
  5. Log in to any one of the Overcloud controllers with the username "heat-admin".
    ssh heat-admin@<Overcloud_controller_IP>
  6. To verify that fencing is disabled, run the following command:
    sudo pcs property show stonith-enabled
    If value of the property "stonith-enabled" is set to false, then skip this step and proceed to Step 'g'.
    If value of the property "stonith-enabled" is set to true, it must be disabled using the following command:
    sudo pcs property set stonith-enabled=false
  7. To stop the cluster service on the Overcloud controller, run the following command:
    sudo pcs cluster stop --all
    
    NOTE:
    This command will stop the pcs cluster in all the Overcloud controllers.
  8. To verify the status of the cluster service on the controllers, run the following command:
    sudo pcs cluster status --all
    
    NOTE:
    Ensure that the cluster service is not running in any of the controllers.
    Powering off HCI or compute nodes
  9. To power off all the HCI nodes, do the following:
    1. Log in to Undercloud VM with the user name "stack".
    2. To list all the HCI nodes running in the Undercloud, run the following command:
      source stackrc nova list
    3. To power off each of the HCI node, do the following:
      
      Log in to each HCI node with username "heat-admin".
      ssh heat-admin@<HCI node IP address>
      
      Run the following command:
      sudo poweroff
      
      NOTE:
      Wait for the HCI nodes to power off.
      
      Perform shutdown on the remaining HCI nodes.
    4. Log in to the iLO console of each HCI node and ensure that the server is in powered off state.
      
      NOTE:
      All the HCI nodes must be in the Powered Off state.
    5. To verify that the status of all the HCI nodes in the Undercloud VM is displayed as "power off", run the following command:
      source stackrc openstack baremetal node list
      
      NOTE:
      All HCI nodes must be in powered off state before proceeding to shut down overcloud controllers.
    Powering off Overcloud controllers
  10. To power off all the Overcloud controllers, do the following:
    1. Log in to the Undercloud VM with username "stack" and list all the Overcloud controllers running in the Undercloud, using the following commands:
      source stackrc nova list
    2. To power off each of the overcloud controller nodes, do the following:
      
      Log in to each overcloud controller with username "heat-admin".
      ssh heat-admin@<VIM Overcloud controller IP address>
      
      Run the following command:
      sudo poweroff
      
      Repeat the power off step in the remaining overcloud controllers.
      
      NOTE:
      Wait for the Overcloud controller nodes to power off.
    3. Log in to the iLO console of each Overcloud controller and ensure that the server is in powered off state.
      
      NOTE:
      All the Overcloud controller nodes must be in the Powered Off state.
    4. To verify that the status of all the Overcloud controllers in the Undercloud VM is displayed as "power off", run the following command:
      source stackrc openstack baremetal node list
      
      NOTE:
      All overcloud controllers must be in powered off state before proceeding to Step 5''.
- Shutting down Overcloud for Ceph based deployment
  
  Disabling fencing in the Overcloud
  1. Log in to the Overcloud controller using VIP address and user "heat-admin". To disable fencing, execute the following steps:
    NOTE:
    The VIP address is mentioned in the field OS_AUTH_URL in the overcloudrc file available in the Undercloud VM.
    
    ssh heat-admin@<VIP IP address> sudo pcs property set stonith-enabled=false
  2. To verify that fencing property is disabled, execute the following command:
```
sudo pcs property show stonith-enabled
```
    The property "stonith-enabled" must be set to false.
    
    Powering off compute nodes
  3. To power off all the compute nodes, do the following:
    1. Log in to Undercloud VM with the user name "stack".
    2. To list all the compute nodes running in the Undercloud, run the following command:
      source stackrc nova list
    3. To power off each of the compute node, do the following:
      
      Log in to each compute with username "heat-admin".
      ssh heat-admin@<compute node IP address>
      
      Run the following command:
      sudo poweroff
      
      NOTE:
      Wait for the Compute nodes to power off.
    4. Log in to the iLO console of each compute and ensure that the server is in powered off state.
      
      NOTE:
      All the compute nodes must be in the Powered Off state.
    5. To verify that the status of all the computes in the Undercloud VM is displayed as "power off", run the following command:
      source stackrc openstack baremetal node list
      
      NOTE:
      All computes must be in powered off state before proceeding to the next step.
    Stopping the cluster service on the Overcloud controller
  4. To get the list of Overcloud controllers running in the undercloud, run the following command from the Undercloud VM:
    source stackrc nova list
  5. Log in to any one of the Overcloud controllers with the username "heat-admin".
    ssh heat-admin@<Overcloud_controller_IP>
  6. To stop the cluster service on the Overcloud controller, run the following command:
```
sudo pcs cluster stop --all
```
    NOTE:
    This command will stop the pcs cluster in all the Overcloud controllers.
  7. To verify the status of the cluster service on the controllers, run the following command:
```
sudo pcs cluster status --all
```
    NOTE:
    Ensure that the cluster service is not running in any of the controllers.
    
    Suspending the Ceph background operations
  8. Log in to the Overcloud controller with the username "heat-admin" using the VIP address.
```
source stackrc
ssh heat-admin@<VIP IP address>
```
    NOTE:
    The VIP address is mentioned in the field OS_AUTH_URL in the overcloudrc file available in the Undercloud VM.
  9. To suspend the Ceph background operations one after the other from the Overcloud controller, run the following commands:
    sudo ceph osd set noout sudo ceph osd set norecover sudo ceph osd set norebalance sudo ceph osd set nobackfill sudo ceph osd set nodown sudo ceph osd set pause exit #Exit from the active controller
    Powering off Ceph nodes
  10. To power off all the Ceph storage nodes, perform the following steps:
    1. To view the list of all the Ceph storage nodes running in the Undercloud, run the following command:
      source stackrc nova list
    2. To power off the Ceph nodes, log in with user name "heat-admin" and run the following commands:
      ssh heat-admin@<Ceph storage node IP address> sudo poweroff
    3. Repeat step ii for all the remaining Ceph storage nodes.
      
      NOTE:
      Wait for the ceph nodes to power off.
    4. Log in to the iLO console of each Ceph node and ensure that the server is in powered off state.
      
      NOTE:
      All the Ceph nodes must be in the Powered Off state.
    5. To verify the status of all the Ceph storage nodes, run the following command:
      source stackrc openstack baremetal node list
      
      NOTE:
      All Ceph nodes must be in powered off state before proceeding to the next step.
    Powering off Overcloud controllers
  11. To power off all the Overcloud controllers, do the following:
    1. Log in to the Undercloud VM with username "stack" and list all the Overcloud controllers running in the Undercloud using the following commands:
      source stackrc nova list
    2. To power off the Overcloud controllers, log in with user name "heat-admin" and run the following commands:
      ssh heat-admin@<VIM Overcloud controller IP address> sudo poweroff
    3. Repeat step ii for all the remaining Overcloud Controller nodes.
      
      NOTE:
      Wait for the controller nodes to power off.
    4. Log in to the iLO console of each overcloud controller node and ensure that the server is in powered off state.
      
      NOTE:
      All the overcloud controller nodes must be in the Powered Off state.
    5. To verify the status of all the Overcloud controller nodes, run the following command:
      source stackrc openstack baremetal node list
      
      NOTE:
      All overcloud controllers must be in powered off state before proceeding to the next step.

Powering off Undercloud VM

To power off the Undercloud VM, do the following:
1. Log in to Undercloud VM with username "stack".
2. To shut down the Undercloud VM, run the following command:
```
sudo shutdown
```
3. To verify if the Undercloud VM status is displayed as "shut off", log in to Undercloud host as "root" user.
  The password and IP details for the UnderCloud host can be obtained from the Input JSON file.
  
  OR
  1. Log in to the NPS VM.
  2. Export the environment variables.
    export API_IP=<NPS Toolkit VM OAM/Customer Network IP address> export NPSADMIN=<user name of admin mentioned in the nps_secret.yaml file> export NPSPASSWD=<password of admin user mentioned in the nps_secret.yaml file> export TOPOLOGY_NAME=<value of the field "topology_name" in the input.json file> export VIM_TYPE=RHOSP
  3. To get password, execute the following command:
    nps show --data platform --node operations_node
    Under the details column, look for the field named "root_pw" and "oam_ip".
    ssh root@<root_pw> virsh list --all
  4. Log in to the Undercloud host using the password and IP obtained from Step III.
    
    ssh root@<root_pw>
  5. To verify that Undercloud VM is in shutoff state, run the following command in the undercloud host:
    
    virsh list --all
    
    NOTE:
    It takes time for Undercloud VM to shut down. Wait until the status of Undercloud VM is displayed as "shut off".

Shutting down Undercloud host

To shut down the Undercloud host, do the following:
1. Take a backup of the Undercloud VM:
  1. To create a dump file of Undercloud VM (undercloud.xml), login to Undercloud host as a "root", and run the following command:
```
virsh dumpxml undercloud >/root/undercloud.xml
```
  2. Copy the undercloud.xmlfile and the Undercloud VM QCOW2 file (undercloud.qcow2), which is available in the /var/lib/libvirt/images/ directory to an external media.
2. To shut down the Undercloud host, run the following command:
```
sudo shutdown
```
3. Log in to the iLO console Undercloud host node and ensure that the server is in powered off state.
Manually power off the HPE StoreFabric SN2100M switches and the HPE FlexFabric 5900 switch.
Manually disconnect the rack power cable from the power supply.