Operating System Configuration

the {{ocient}} system configuration procedure assumes that these steps have been performed on the foundation nodes and loader nodes in the database cluster some of these steps require root user permissions u se sudo when performing these steps in your command line interface prepare the operating system configuration you must prepare the operating system with this configuration to run ocient software configure network interface controllers (nics) and firewall for use by ocient set the timezone to utc if opal drives are installed, unlock or revert them to factory state optional mount points /opt/ocient and /var/opt/ocient are mounted on separate storage devices confirm the ocient installation to check the installation, run this command on each node to show the software version /opt/ocient/current/rolehostd build info if the software is not installed or is the wrong version, see the ocient application installation docid\ dsxcbn7k6tarfy abjwu instructions apply required kernel settings in order to achieve high performance levels, you must set specific kernel settings you cannot set these settings at runtime therefore, the system must pass these parameters to the kernel at boot time automatically apply kernel settings you can automatically configure the kernel settings and apply them by running the ockernelparams script in the ocient software package this script detects the current system hardware and adjusts the grub bootloader kernel parameters then, the script updates the grub configuration with grub2 mkconfig or update grub (depending on the linux distribution) run this command and specify the appropriate node role /opt/ocient/scripts/ockernelparams node role {sql,loader,foundation,metadata} you can also manually configure kernel settings see operating system configuration docid\ rzs2qc1xvmkmwmiaq60mq for more information check current kernel settings run this command to show the current settings for the kernel that is running you need to reboot after any changes are made to kernel parameters in the bootloader configuration cat /proc/cmdlineboot image=(hd0,gpt2)/boot/vmlinuz 4 18 0 305 19 1 el8 4 x86 64 root=uuid=165df3b2 9221 43be 967d d91483c07219 ro crashkernel=auto rhgb quiet transparent hugepage=never isolcpus=0 23,26 49,52 75,78 101 hugepagesz=1g hugepages=1024 hugepagesz=2m hugepages=16384 the exact values can vary depending on the dram of the machine and the intended role for the node check that the number of free 1g huge pages matches the number specified at the command line you can check this number using this command cat /sys/kernel/mm/hugepages/hugepages 1048576kb/nr hugepages output 1024 the returned value should match your configuration manual kernel settings this section describes the optional kernel settings that you can manually configure to automatically configure these settings, see operating system configuration docid\ rzs2qc1xvmkmwmiaq60mq you must add these settings that follow the grub bootloader kernel parameters for the listed applicable node types setting transparent hugepages ocient requires that transparent hugepages be disabled because ocient software needs strict control over which huge pages it allocates applicable node types foundation nodes loader nodes sql nodes kernel setting transparent hugepage=never setting isolcpus to give ocient software exclusive use of the majority of the cores on the system, the kernel is isolated from the cores using the isolcpus parameter this isolation ensures that the system executes the kernel code only on a small number of cores where ocient is not running logic that is critical for performance the pattern is to isolate all but the last two physical cores of each cpu from the kernel for cpus with 26 or more cores this means that two cores (4 threads) per cpu are available to the kernel for cpus with fewer than 26 cores, all but the last physical core of each cpu is isolated applicable node types foundation nodes kernel setting dual 28 core cpus ( {{intel}} 6348) isolcpus=0 25,28 53,56 81,84 109 dual 32 core cpus (intel 6338) isolcpus=0 29,32 61,64 93,96 125 dual 96 core cpus ( {{amd}} 9654) isolcpus=0 93,96 189,192 285,288 381 setting huge pages ocient utilizes huge pages to streamline the management of large amounts of memory in the system the system uses 2mb and 1gb huge pages, depending on the node type these huge pages must be reserved at boot time because otherwise, the system memory quickly becomes too fragmented to allocate the number of huge pages needed at runtime these settings are the expected huge page settings metadata nodes do not use huge pages node type installed dram huge page kernel parameters foundation or sql 768 gb hugepagesz=1g hugepages=384 hugepagesz=2m hugepages=16384 foundation or sql 1024 gb hugepagesz=1g hugepages=680 hugepagesz=2m hugepages=16384 foundation or sql 1152 gb hugepagesz=1g hugepages=768 hugepagesz=2m hugepages=16384 foundation or sql 1536 gb hugepagesz=1g hugepages=1024 hugepagesz=2m hugepages=16384 foundation or sql 2048 gb hugepagesz=1g hugepages=1364 hugepagesz=2m hugepages=16384 foundation or sql 2304 gb hugepagesz=1g hugepages=1536 hugepagesz=2m hugepages=16384 loader 768 gb hugepagesz=1g hugepages=384 hugepagesz=2m hugepages=16384 loader 1024 gb hugepagesz=1g hugepages=512 hugepagesz=2m hugepages=16384 loader 1152 gb hugepagesz=1g hugepages=576 hugepagesz=2m hugepages=16384 loader 1536 gb hugepagesz=1g hugepages=768 hugepagesz=2m hugepages=16384 loader 2048 gb hugepagesz=1g hugepages=1024 hugepagesz=2m hugepages=16384 loader 2304 gb hugepagesz=1g hugepages=1152 hugepagesz=2m hugepages=16384 metadata (standalone administrator) n/a no huge pages use the nohugepages parameter when bootstrapping node nohugepages true prepare nvme drives these steps are for foundation nodes, loader nodes, and sql nodes only to prepare nvme drives on foundation nodes, loader nodes, and sql nodes, these nodes must have the state where they are connected to the uio pci generic or vfio pci driver and not contain a partition table to initialize the drives for use by ocient that have been previously partitioned, you must perform these steps 1 identify the os drives the os drives have partitions that are mounted to mount points such as / in this example, nvme0n1 is the os drive lsblk name maj\ min rm size ro type mountpoint nvme0n1 259 0 0 1 8t 0 disk ├─nvme0n1p1 259 1 0 512m 0 part /boot/efi └─nvme0n1p2 259 2 0 1 8t 0 part / 2 ensure all drives are bound to the nvme driver so the nvme drives show up as standard nvme devices list the nvme drives to identify the drive identifiers sudo /opt/ocient/scripts/nvme driver util sh bind nvme /opt/ocient/scripts/nvme driver util sh bdf numa node driver name device name 0000 01 00 0 0 nvme nvme0n1 0000 3b 00 0 0 nvme 0000 3c 00 0 0 nvme 0000 5e 00 0 0 nvme 0000 5f 00 0 0 nvme 0000 86 00 0 1 nvme 0000 87 00 0 1 nvme 0000 88 00 0 1 nvme 0000 89 00 0 1 nvme 3 for each nvme drive that you need to erase and prepare for ocient (not the os drive), run this command by replacing x and y to match the device name sudo nvme format /dev/nvmexny format ses=1 this command formats the specified drive and erases all data on the drive ensure that the os drive is not formatted if it is an nvme drive some drives do not support the nvme format command in that case, you can overwrite the partition table by zeroing out the beginning of the drive with this command this command does not securely erase the whole drive, but does allow it to be used by ocient sudo dd if=/dev/zero of=/dev/nvmexny bs=1024k count=1024 4 bind the drives to the uio or vfio driver by running this command the script automatically chooses the appropriate driver after detecting if the iommu setting is enabled any nvme drives with mounted partitions (i e the os drive) remain on the nvme driver show all expected nvme drives on the node attached to the uio or vfio driver sudo /opt/ocient/scripts/nvme driver util sh bind vfio /opt/ocient/scripts/nvme driver util sh bdf numa node driver name device name 0000 01 00 0 0 nvme nvme0n1 0000 3b 00 0 0 vfio pci 0000 3c 00 0 0 vfio pci 0000 5e 00 0 0 vfio pci 0000 5f 00 0 0 vfio pci 0000 86 00 0 1 vfio pci 0000 87 00 0 1 vfio pci 0000 88 00 0 1 vfio pci 0000 89 00 0 1 vfio pci clear files from any previous ocient setup to bootstrap a node that was previously part of an ocient system, you must clear a set of files from any previous setup of ocient from the /var/opt/ocient/ directory these commands remove these files sudo rm f /var/opt/ocient/rolehostd conf sudo rm f /var/opt/ocient/ocient key sudo rm f /var/opt/ocient/ocient crt sudo rm f /var/opt/ocient/ocient crt original sudo rm f /var/opt/ocient/ raft sudo rm f /var/opt/ocient/bootstrap completed do not remove files in the localkeystore directory if opal drives are present and encryption is not disabled if these files are removed, the drive encryption must be reset using an external utility related links ocient software upgrade docid 0h4hxxsrehalnzvveckdc install an ocient system docid\ clmx7aipvis6ctybuagzx