Amazon Web Services Ocient Installation
For details about AWS concepts, see these links:
Ocient supports deploying an Ocient System in AWS for pilot or testing purposes. There are no guarantees on data durability. Stopping Amazon instances will result in permanent data loss.
Steps for deploying an Ocient System in AWS:
- Prepare AWS resources.
- Prepare a CloudFormation template.
- Run the CloudFormation template with your customized parameters.
- Wait for nodes to be provisioned.
The standard Ocient implementation of the CloudFormation template deploys an Ocient cluster with the following number and type of nodes. Contact Ocient Support for a CloudFormation template. Note that a template can be created for any number of nodes.
Node Type | Instance Type |
|---|---|
Foundation Nodes (3) | i3en.metal |
Loader Nodes (1) | i3en.metal |
SQL Nodes (1) | r5d.metal |
This diagram shows an example of an Ocient cluster in AWS. The EC2 nodes (SQL, Loader, Foundation) are deployed within a single subnet of an Amazon . Each type of node is assigned a separate security group (sg1, sg2, sg3).
Prior to running the CloudFormation template, a set of AWS resources must be initialized. These resources are the input parameters to the template. You must choose the Region (e.g., N. Virginia - US East 1) where you will be deploying your Ocient system and an associated availability zone (e.g., us-east-1a).
Parameters
You pass the following parameters as input parameters to the CloudFormation template.
Name | Example | Description |
|---|---|---|
SystemName | test | Name of the Ocient system (prefix for all nodes) |
ImageId | ami-abc0123a1230abcef | Ocient AMI (contains Ocient software) |
AvailabilityZone | us-east-1a | Where the Ocient system will be deployed |
KeyName | myKeyPair | Name of an AWS KeyPair (used for SSH to nodes) |
SubnetId | subnet-0abc1230001230abc | Preset range of IP addresses for Ocient nodes |
SQLNodeSecurityGroupIds | sg-00011123000123abc | Defines access to the SQL Nodes |
FoundationNodeSecurityGroupIds | sg-012300abc12300123 | Defines access to Foundation Nodes |
LoaderNodeSecurityGroupIds | sg-0abcedf00123abcde | Defines access to the Loader Nodes |
LoaderNodeIamInstanceProfile | AmazonS3ReadOnlyAccessRole | Passes privileges to the Loader Nodes |
MetadataNodeSecurityGroupIds | sg-a123123bc12300123 | Defines access to the Metadata Nodes |
LogLevel | info | (optional) Sets the initial logging level for all nodes |
StorageSpaceWidth | 3 | The storage space width of the storage cluster |
StorageSpaceParityWidth | 1 | The storage space parity width of the storage cluster |
Many of the parameters are AWS resources. For each of those parameters, follow these steps to request the valid data to fill out the CloudFormation template. Save each parameter value to a temporary text file.
- Contact Ocient Support to request the latest AMI.
- The Ocient team shares the AMI with your AWS account.
- Navigate to EC2 in the AWS Management Console.
- Ensure that the appropriate region is set on the top navigation bar.
- Select AMIs under Images from the left navigation.
- Select Private Images from the drop down.
- The screen displays an image, such as ami-abc0123a1230abcef, with a name similar to Ocient 22.0.0.
- Save the image name.
Key Pair
In order to attach credentials for SSH access to the EC2 nodes, you must select a Key Pair. For details about key pairs, see Amazon EC2 key pairs and Linux instances.
- View Key Pairs in the console by entering key pairs in the search bar.
- Choose an existing key pair or create a new one.
- Save the name of the Key Pair.
Subnet
A single subnet is used to determine the range of IPs used for the EC2 nodes.
- View available subnets in the console by entering subnet in the search bar.
- Choose an existing subnet in the chosen region or create a new subnet.
- Save the Subnet ID.
The chosen subnet must match the chosen availability zone for the CloudFormation template to deploy correctly.
The CloudFormation template allows different security groups to be assigned for each node type.
The main requirements are to provide access to:
- Port 4050 on the SQL Nodes for JDBC client connectivity
- Port 8080 on the Loader Nodes for LAT client access
- Port 22 on all nodes for SSH access (recommended)
Create an individual security group with rules for these port requirements and apply them to the specified node types.
- View Security Groups in the AWS console by entering security groups in the search bar.
- Choose one or more existing security groups or create them as needed.
- Save the Security group ID or multiple identifiers if more than one security group is created.
An IAM Instance Role is used to pass an IAM Policy to the Loader Node. For more information about configure IAM roles, see Configuring IAM roles for Amazon EC2.
- View a list of Roles in the AWS console by entering roles in the search bar.
- Choose an existing role or create a new one as needed.
- Save the name of the role.
Minimum permissions for the policy should include Get/List permissions for Amazon S3 (where data is stored). For example, use the AWS standard policy AmazonS3ReadOnlyAccess.
You might optionally create a custom policy to limit access to a specific named resource (bucket).
- Contact Ocient Support to obtain the latest CloudFormation template (ocient-cf.yaml).
- Deploy the Ocient CloudFormation template using the Create Stack command.
- Run the Create Stack command in the AWS console using CloudFormation → Stacks → Create. A set of input screens guides you to specify the location of the template file.
- Choose a template file.
- Enter the name of the stack and enter the values for the parameters from the temporary text file.
Or, you can run the Create Stack command using the AWS command line interface. For example:
3. Accept the default configuration and click Create Stack. AWS creates the CloudFormation stack and provisions a set of Ocient nodes as EC2 instances.
4. Track the status of the CloudFormation stack in the AWS console using CloudFormation → Stacks → Events.
5. View the provisioned EC2 instances in the AWS console using EC2 → Instances.
After approximately 20 minutes, the instances are stable and accessible using SSH.
During this time, each instance will be restart at least once as kernel parameters are set up.
Initialization process can be observed in the /var/log/startup-script.log file on each instance.
Initialization is complete when the startup-script.log file on the sql01 node contains this message.
The Ocient system configuration is now complete with these items:
- Bootstrapping is completed.
- A storage space exists with all Foundation Nodes as members.
- The streamloader role has been applied to all Loader Nodes.
- The SQL and Administrator roles have been applied to the SQL Nodes.
- The Administrator role has been applied to all Metadata Nodes.
- The rolehostd service has been restarted for all role changes to take effect.
Load Data
Query Ocient