Amazon Web Services Ocient Installation

deploy a cloudformation template to install an {{ocient}} system in {{aws}} for details about aws concepts, see these links what is amazon ec2? https //docs aws amazon com/awsec2/latest/userguide/concepts html what is iam? https //docs aws amazon com/iam/latest/userguide/introduction html what is aws cloudformation? https //docs aws amazon com/awscloudformation/latest/userguide/welcome html a primer for amazon networking vpc, availability zones & subnets https //aws amazon com/blogs/apn/amazon vpc for on premises network engineers part one/ ocient supports deploying an ocient system in aws for pilot or testing purposes there are no guarantees on data durability stopping amazon {{ec2}} instances will result in permanent data loss steps for deploying an ocient system in aws prepare aws resources prepare a cloudformation template run the cloudformation template with your customized parameters wait for nodes to be provisioned target configuration the standard ocient implementation of the cloudformation template deploys an ocient cluster with the following number and type of nodes contact ocient support for a cloudformation template note that a template can be created for any number of nodes node type instance type foundation nodes (3) i3en metal loader nodes (1) i3en metal sql nodes (1) r5d metal this diagram shows an example of an ocient cluster in aws the ec2 nodes (sql, loader, foundation) are deployed within a single subnet of an amazon {{vpc}} each type of node is assigned a separate security group ( sg1 , sg2 , sg3 ) virtual private cloud for loading data from an s3 bucket using a jdbc client aws prerequisites prior to running the cloudformation template, a set of aws resources must be initialized these resources are the input parameters to the template you must choose the region (e g , n virginia us east 1 ) where you will be deploying your ocient system and an associated availability zone (e g , us east 1a ) parameters you pass the following parameters as input parameters to the cloudformation template name example description systemname test name of the ocient system (prefix for all nodes) imageid ami abc0123a1230abcef ocient ami (contains ocient software) availabilityzone us east 1a where the ocient system will be deployed keyname mykeypair name of an aws keypair (used for ssh to nodes) subnetid subnet 0abc1230001230abc preset range of ip addresses for ocient nodes sqlnodesecuritygroupids sg 00011123000123abc defines access to the sql nodes foundationnodesecuritygroupids sg 012300abc12300123 defines access to foundation nodes loadernodesecuritygroupids sg 0abcedf00123abcde defines access to the loader nodes loadernodeiaminstanceprofile amazons3readonlyaccessrole passes privileges to the loader nodes metadatanodesecuritygroupids sg a123123bc12300123 defines access to the metadata nodes loglevel info (optional) sets the initial logging level for all nodes storagespacewidth 3 the storage space width of the storage cluster storagespaceparitywidth 1 the storage space parity width of the storage cluster many of the parameters are aws resources for each of those parameters, follow these steps to request the valid data to fill out the cloudformation template save each parameter value to a temporary text file ocient amazon machine image (ami) amazon machine images (amis) contact ocient support to request the latest ami the ocient team shares the ami with your aws account navigate to ec2 in the aws management console ensure that the appropriate region is set on the top navigation bar select amis under images from the left navigation select private images from the drop down the screen displays an image, such as ami abc0123a1230abcef , with a name similar to ocient 22 0 0 save the image name key pair key pairs to attach credentials for ssh access to the ec2 nodes, you must select a key pair for details about key pairs, see amazon ec2 key pairs and linux instances https //docs aws amazon com/awsec2/latest/userguide/ec2 key pairs html view key pairs in the console by entering key pairs in the search bar choose an existing key pair or create a new one save the name of the key pair subnet subnets a single subnet is used to determine the range of ips used for the ec2 nodes view available subnets in the console by entering subnet in the search bar choose an existing subnet in the chosen region or create a new subnet save the subnet id the chosen subnet must match the chosen availability zone for the cloudformation template to deploy correctly security groups security groups the cloudformation template allows different security groups to be assigned for each node type the main requirements are to provide access to port 4050 on the sql nodes for jdbc client connectivity port 8080 on the loader nodes for lat client access port 22 on all nodes for ssh access (recommended) create an individual security group with rules for these port requirements and apply them to the specified node types view security groups in the aws console by entering security groups in the search bar choose one or more existing security groups or create them as needed save the security group id or multiple identifiers if more than one security group is created identity access management (iam) instance role iam roles an iam instance role is used to pass an iam policy to the loader node for details about configuring iam roles, see configuring iam roles for amazon ec2 https //docs aws amazon com/sdk for java/latest/developer guide/ec2 iam roles html view a list of roles in the aws console by entering roles in the search bar choose an existing role or create a new one as needed save the name of the role minimum permissions for the policy should include get/list permissions for amazon s3 (where data is stored) for example, use the aws standard policy amazons3readonlyaccess you might optionally create a custom policy to limit access to a specific named resource (bucket) deploy the ocient cloudformation template contact ocient support to obtain the latest cloudformation template ( ocient cf yaml ) deploy the ocient cloudformation template using the create stack command run the create stack command in the aws console using cloudformation → stacks → create a set of input screens guides you to specify the location of the template file choose a template file enter the name of the stack and enter the values for the parameters from the temporary text file create stack or, you can run the create stack command using the aws command line interface for example aws cloudformation create stack \\ \ stack name cloudformation test \\ \ template body file //ocient cf yaml \\ \ parameters \\"parameterkey=systemname,parametervalue=test" \\ "parameterkey=imageid,parametervalue=ami abc0123a1230abcef" \\ "parameterkey=availabilityzone,parametervalue=us east 1a" \\ "parameterkey=keyname,parametervalue=mykeypair" \\ "parameterkey=subnetid,parametervalue=subnet 0abc1230001230abc" \\ "parameterkey=sqlnodesecuritygroupids,parametervalue=sg 00011123000123abc" \\ "parameterkey=foundationnodesecuritygroupids,parametervalue=sg 012300abc12300123" \\ "parameterkey=loadernodesecuritygroupids,parametervalue=sg 0abcedf00123abcde" \\ "parameterkey=loadernodeiaminstanceprofile,parametervalue=amazons3roaccessrole" \\ "parameterkey=metadatanodesecuritygroupids,parametervalue=sg a123123bc12300123" \\ "parameterkey=loglevel,parametervalue=info" \\ "parameterkey=storagespacewidth,parametervalue=3" \\ "parameterkey=storagespaceparitywidth,parametervalue=1" 3\ accept the default configuration and click create stack aws creates the cloudformation stack and provisions a set of ocient nodes as ec2 instances 4\ track the status of the cloudformation stack in the aws console using cloudformation → stacks → events 5\ view the provisioned ec2 instances in the aws console using ec2 → instances after approximately 20 minutes, the instances are stable and accessible using ssh during this time, each instance restarts at least once as kernel parameters are set up the initialization process can be observed in the /var/log/startup script log file on each instance initialization is complete when the startup script log file on the sql01 node contains this message text initialization successfully completed the ocient system configuration is now complete with these items bootstrapping is completed a storage space exists with all foundation nodes as members the streamloader role has been applied to all loader nodes the sql and administrator roles have been applied to the sql nodes the administrator role has been applied to all metadata nodes the rolehostd service has been restarted for all role changes to take effect related links ingest data with legacy lat reference docid\ tt6tfoulap0mt aycm2ka query ocient docid 4ycq1d8tkfmlsacorynf6