NVMe Drive Firmware Upgrade Process

The software uses NVMe drives for storage because of their higher input or output (I/O) throughput than spinning media drives. These drives are referred to as data disks or data drives. When you run Ocient software, the Ocient System accesses these data drives using the User Mode I/O (UIO) driver during normal operation. This procedure guides a System Administrator through binding the drives back to the NVMe driver and upgrading the drive firmware. The procedure includes steps for upgrading the firmware of only NVMe drives. For other drive models, follow a similar process using the firmware upgrade process from the manufacturer.

Stop the rolehostd process prior to working on the NVMe drives. Otherwise, when the rolehostd process is running, the process becomes unstable because it has open file handles on the drives.

Execution of this process requires the sudo privilege on the node where the drive firmware is updated.

Prerequisites

Perform the firmware upgrade on one node of any specified type (Foundation, Loader, SQL) at a time. You can execute the firmware upgrade in parallel on different types of nodes.

Perform the upgrade during a maintenance window because issues with the upgrade might prevent the node from returning to service.

The prerequisites for this procedure are:

The following packages are installed on the node undergoing firmware upgrade:
- smartmontools
- nvme-cli
- intelmas
- sedutil
The target firmware version for the upgrade is known and documented.
If you upgrade the data drive firmware on a Loader Node, prior to beginning the procedure, take the node out of the loading path by updating the LAT configuration. Before you shut down the rolehostd process, drain the pages that are held by the node.

The following example illustrates the firmware update commands used for an Intel NVMe SSD. Please substitute the appropriate command for your drive manufacturer in each of the steps that use the intelmas command.

Firmware Upgrade Procedure

The following steps require sudo permissions on the node.

Prepare for the Firmware Upgrade

1. Log in to the node as an administrator user. 2. If you upgrade the drive firmware on a Foundation Node or Loader Node, save the status of the drives for later in the procedure. After you replace <IP_ADDRESS_OF_THE_NODE> with the IP address of your node, save the output of this command.

CURL

curl -s http://<IP_ADDRESS_OF_THE_NODE>:9090/v1/stats | jq -r '.[] | select (.name ==
 "localStorageService.device.status" ) | "(.device) (.value)" '

Alternatively, you can run a query on the SQL Node to display the status of the individual storage device and firmware version. After you replace <NODE_NAME> with the name of your node, connect to any SQL Node on the system and run this query.

SQL

SELECT
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.firmware_version
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';

Output

Text

serial_number     |pci_address |device_status |firmware_version
------------------+------------+--------------+-----------------
CVMD6356009M800GGN|0000:3b:00.0|ACTIVE        |VDV10184
CVMD636400BZ800GGN|0000:3c:00.0|ACTIVE        |VDV10184
CVMD636400JA800GGN|0000:5e:00.0|ACTIVE        |VDV10184
CVMD6365003C800GGN|0000:5f:00.0|ACTIVE        |VDV10184
CVMD6365003P800GGN|0000:86:00.0|ACTIVE        |VDV10184
CVMD63650092800GGN|0000:87:00.0|ACTIVE        |VDV10184
PHLJ0431019J8P0HGN|0000:88:00.0|ACTIVE        |VDV10184
PHLJ106500018P0HGN|0000:89:00.0|ACTIVE        |VDV10170

3. If you upgrade the drive firmware on a SQL Node, save the output of the df command.

Stop the `rolehostd` Process and Attach the Data Drives

1. Stop the rolehostd process to ensure that the drive replacement does not impact system stability.

Shell

sudo systemctl stop rolehostd

2. Prevent the rolehostd process from starting after reboot.

Shell

sudo systemctl disable rolehostd

3. Confirm that the output of this command indicates that the rolehostd process is not running and that the process auto start on reboot setting is disabled.

Shell

sudo systemctl status rolehostd

When you upgrade the drive firmware, you can impact data reliability while the rolehostd process is running on the node.

4. Attach the data drives to the NVMe driver.

Shell

sudo /opt/ocient/scripts/nvme-driver-util.sh bind-nvme

5. Ensure that all drives are not the operating system drive and are attached to the NVMe driver in the output of this command. Any drives that did not bind to the NVMe driver in the previous step do not upgrade, so you must investigate any failure to bind to the NVMe driver.

Shell

/opt/ocient/scripts/nvme-driver-util.sh

INCORRECT: Eight data drives are attached to the UIO driver. This is the incorrect system state for performing a firmware upgrade.

Text

./opt/ocient/scripts/nvme-driver-util.sh
NVMe device status

use option 'bind-uio' to bind uio_pci_generic driver to Ocient Payload/unpartitioned drives
use option 'bind-nvme' to bind nvme driver to drives

BDF             Numa Node       Driver name             Device name
0000:01:00.0    0               nvme                    nvme0n1
0000:3b:00.0    0               uio_pci_generic         -
0000:3c:00.0    0               uio_pci_generic         -
0000:5e:00.0    0               uio_pci_generic         -
0000:5f:00.0    0               uio_pci_generic         -
0000:86:00.0    1               uio_pci_generic         -
0000:87:00.0    1               uio_pci_generic         -
0000:88:00.0    1               uio_pci_generic         -
0000:89:00.0    1               uio_pci_generic         -

CORRECT: The data drives are attached to the NVMe driver. This is the correct system state for performing a firmware upgrade.

Text

./opt/ocient/scripts/nvme-driver-util.sh
NVMe device status

use option 'bind-uio' to bind uio_pci_generic driver to Ocient Payload/unpartitioned drives
use option 'bind-nvme' to bind nvme driver to drives

BDF             Numa Node       Driver name             Device name
0000:01:00.0    0               nvme                    nvme0n1
0000:3b:00.0    0               nvme                    nvme1n1
0000:3c:00.0    0               nvme                    nvme2n1
0000:5e:00.0    0               nvme                    nvme3n1
0000:5f:00.0    0               nvme                    nvme4n1
0000:86:00.0    1               nvme                    nvme5n1
0000:87:00.0    1               nvme                    nvme6n1
0000:88:00.0    1               nvme                    nvme7n1
0000:89:00.0    1               nvme                    nvme8n1

Perform Firmware Upgrade

1. Run this command and review the FirmwareUpdateAvailable field to identify the drives that need a firmware upgrade. Save the results of this step for comparison later in the process.

sudo intelmas show  -d Index, SerialNumber, DeviceStatus, Capacity, FirmwareUpdateAvailable -intelssd

Drive 8 needs the firmware update, but drive 7 does not need the update.

- 7 Intel SSD DC P4510 Series PHLJ0431019J8P0HGN -

Capacity : 8001.56 GB
DeviceStatus : Healthy
FirmwareUpdateAvailable : The selected drive contains current firmware as of this tool release.
Index : 7
SerialNumber : PHLJ0431019J8P0HGN

- 8 Intel SSD DC P4510 Series PHLJ106500018P0HGN -

Capacity : 8001.56 GB
DeviceStatus : Healthy
FirmwareUpdateAvailable : Firmware=VDV10170 Bootloader=VB1B0172
Index : 8
SerialNumber : PHLJ106500018P0HGN

2. Execute this command for every drive in the previous step that indicates an old firmware in the FirmwareUpdateAvailable field. Replace <DRIVE_INDEX> with the number shown in the Index field from the output of the previous command for all drives that require the update.

Shell

sudo intelmas load -intelssd <DRIVE_INDEX>

3. Ensure that the output shows Firmware updated successfully. This is the sample log output for the upgrade.

Text

- intelmas load -intelssd 8
WARNING! You have selected to update the drives firmware!
Proceed with the update? (Y|N): Y
Checking for firmware update...

- Intel SSD DC P4510 Series PHLJ106500018P0HGN -

Status : Firmware updated successfully. Please reboot the system.

4. After you upgrade all drives that require the firmware upgrade, reboot the node using the reboot command. After the node completes the reboot, log in to the node as an administrator. 5. Attach the data drives to the NVMe driver using this command.

Shell

sudo /opt/ocient/scripts/nvme-driver-util.sh bind-nvme

6. Compare the output of this command with the output of the same command captured earlier in this process. Ensure that the reported drive status and capacity are the same as before the upgrade and that the firmware shown is current.

Shell

sudo intelmas show -d Index, SerialNumber, DeviceStatus, Capacity, FirmwareUpdateAvailable -intelssd

7. Reattach the data drives to the UIO driver.

Shell

sudo /opt/ocient/scripts/nvme-driver-util.sh bind-uio

Enable the `rolehostd` Process and Perform Final Checks

1. Enable the rolehostd process to start after reboot, and then start the process.

Shell

sudo systemctl enable rolehostd && sudo systemctl start rolehostd

2. Ensure that the rolehostd process is enabled and started.

Shell

sudo systemctl status rolehostd

3. If you upgrade the firmware of the drive on a Foundation Node or Loader Node, ensure that the output of this command matches the output of the same command captured earlier in this process.

CURL

curl -s http://<IP_ADDRESS_OF_THE_NODE>:9090/v1/stats | jq -r '.[] | select (.name == "localStorageService.device.status" ) | "\(.device) \(.value)" '

Alternatively, you can run a query on the SQL Node to capture the individual device status and firmware version. Ensure that the drives and statuses in this query match the results from the query you executed earlier in this process. After you replace the node name <NODE_NAME> with the name of your node, connect to any SQL Node on the system and execute this query.

SQL

SELECT
  s.id as serial_number,
  s.pci_address,
  s.device_status,
  s.firmware_version
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';

Output

Text

serial_number     |pci_address |device_status |firmware_version
------------------+------------+--------------+-----------------
CVMD6356009M800GGN|0000:3b:00.0|ACTIVE        |VDV10184
CVMD636400BZ800GGN|0000:3c:00.0|ACTIVE        |VDV10184
CVMD636400JA800GGN|0000:5e:00.0|ACTIVE        |VDV10184
CVMD6365003C800GGN|0000:5f:00.0|ACTIVE        |VDV10184
CVMD6365003P800GGN|0000:86:00.0|ACTIVE        |VDV10184
CVMD63650092800GGN|0000:87:00.0|ACTIVE        |VDV10184
PHLJ0431019J8P0HGN|0000:88:00.0|ACTIVE        |VDV10184
PHLJ106500018P0HGN|0000:89:00.0|ACTIVE        |VDV10184

4. If you upgrade the firmware of the drive on a SQL Node, ensure that the output of the df command shows the same mount points as those captured earlier in this process. System Catalog

​Prerequisites

​Firmware Upgrade Procedure

​Prepare for the Firmware Upgrade

​Stop the rolehostd Process and Attach the Data Drives

​Perform Firmware Upgrade

​Enable the rolehostd Process and Perform Final Checks

​Related Links

Prerequisites

Firmware Upgrade Procedure

Prepare for the Firmware Upgrade

Stop the `rolehostd` Process and Attach the Data Drives

Perform Firmware Upgrade

Enable the `rolehostd` Process and Perform Final Checks

Related Links