Core Elements of an Ocient System

the system and nodes the {{ocient}} system is the collection of all of the nodes within a single environment this architecture diagram is an example of one ocient system across systems, the number of nodes can vary along with the total storage capacity within a single system, users can define varying numbers of databases, storage groups, users, etc an ocient system is made up of a collection of distributed nodes which each play a predefined role in the data warehouse the different roles are generally responsible for storing and analyzing data, administration, and moving data into ocient these nodes are connected to each other through a high speed 100 gbps network connection in addition to storing user data, the data warehouse also stores metadata about the system and data for its own internal operation and recovery this collection of nodes, their data, and the interconnection between them make up the ocient system the end user, analyzing data in ocient, connects to this system through a jdbc or a python based client from there, they interact with the database objects, such as tables and views the architecture itself is abstracted from users, allowing them to focus on sql queries and results while the number of nodes in an environment might vary, the inclusion of and purpose of the different roles do not the following sections briefly describe the components in the architecture ocient architecture diagram that shows the relationship between data sources, loading and transformation of the data, and data storage node types sql nodes a sql node is responsible for receiving incoming sql requests and parsing the statements this node also serves as the interface for system administrators and database administrators to configure and maintain an ocient system administrators can connect to a sql node using the ocient command line interface or their chosen client when a sql node receives a query, it creates a plan for it using one of two optimization methods after there is a plan for the query, the sql node distributes it to the foundation nodes when the foundation nodes finish their processing, they return data back to the sql node for further processing and packaging the result set to the client when the result set is finalized, the sql node returns the data to the client beyond the foundation nodes, further processing is done on the sql nodes to process intermediate result sets delivered by the foundation nodes depending on the query, the sql node is responsible for a different proportion of the overall work of the query common processing done here are aggregations and joins that happen across the intermediate results delivered by the foundation nodes administrators also use ddl or dcl statements to make changes to an ocient system when submitted to a sql node, the node executes the sql statement across the system or forwards the sql statement to the node assigned to execute it foundation nodes the foundation nodes store the user data in ocient and perform the bulk of query processing a key architectural concept of ocient is the {{casa}} , which collocates data and processing where possible the foundation node is the central element of this architectural principle when a query is deployed to the foundation node, it performs as much of the query as it can with the data on the node before having to join, aggregate, or compare it to other data it returns that result up to the sql node for further processing, packaging, and returning to the user foundation nodes contain the majority of the storage in an ocient system and are also typically the largest in number loader nodes loader nodes are responsible for extracting, transforming, indexing, and loading data ingested by ocient from batch file sources as well as streaming data sources such as {{kafka}} a user can specify the extraction source details and supply a transformation pipeline that manipulates the structured or semi structured source data before loading it into relational form into ocient tables loader nodes operate in a horizontal scale out fashion allowing the loading system to scale to fit various ingestion requirements transparently to the end user, the loader node also manages exactly once guarantees with data as it is loaded and converts pages into segments in the foundation nodes flows and networking between nodes data and communication flows across the nodes of an ocient system for a variety of different purposes there are two networks that connect nodes, a 100 gbps high speed network and a 10 gbps network these are used in different ways for data flows and administrative purposes a query in ocient when a user issues a query against the database, their sql statement moves from the jdbc client to the sql node the sql node parses and optimizes the query before handing the plan off to the foundation nodes the foundation nodes do the first level of processing against the data before handing the results back to the sql node the sql node further processes the data, constructs the results, and returns them to the client the connection between the sql nodes and foundation nodes is typically a 100 gbps network connection the speed of the connection between the client and the sql node is subject to the speed of that network (outside ocient) every foundation node is connected to every sql node administration flows when administrators configure or make changes to an ocient system, they do it using ddl and dcl using a sql client the sql client relays the statement to a sql node, which parses them into commands that the administrator role on the sql node executes this can impact the sql nodes, foundation nodes, or loader nodes, depending on the type of change or operation the administration flow connections with the ocient system are 10 gbps connections every node is connected to the sql nodes that perform the administrator role loading flow when the load job is defined and started, the loader nodes read data from the source files, stages the data, processes that data, and moves the data into the foundation nodes the loading and transformation process is responsible for managing this process and ensuring data is properly indexed and loaded the network connections between the loader and the foundation nodes are 100 gbps the loader nodes connect to every foundation node database, tables, and views similar to other databases, an {{ocienthyperscaledatawarehouse}} is a grouping of tables, views, users, and data within one ocient system as mentioned above, a single system can have multiple databases tables and views are defined within each database while ocient stores data on disk in columnar format, creating and working with tables in ocient is just like any other relational database ocient supports a wide variety of standard sql data types along with an expanding set of geospatial data types read more about data definition language (ddl) statement reference docid 2wcd8nmbi5m9fctufiqtm general sql syntax docid\ qcf0x9ao4a56x id39pkr functions overview docid\ yt6cjd9bnmg7zschejbce timekey and clustering key each table in ocient can contain a {{timekey}} column that is specified when defining the table the timekey column is used to partition the data within the foundation nodes time partitioning is an important performance mechanism used by the ocient hyperscale data warehouse because many, if not most, queries specify some time filter for the results, time partitioning allows ocient to quickly skip data from irrelevant times when defining a table, users can also specify a clustering key containing one or more of the columns to further subdivide records on disk this allows the database to quickly find records with the same key values read more in timekeys and clustering keys docid 21n4t9o37 dsjnmgsgj1o resilience to hardware failure in order to provide reliability in the case of a hardware outage, ocient uses erasure coding erasure coding is a mechanism to organize and compute parity blocks so that the system can rebuild missing data this means that ocient does not store redundant copies of the data for reliability as a result, an ocient system requires less overall storage than if it were storing one or more copies of the data for failover when a user defines their storage configuration, they specify the width of the group and the parity width these values can vary based on how many foundation nodes are in the system and how resilient users want to make the ocient system segments and segment groups segments in ocient are storage units that contain rows and are divided into a fixed number of coding blocks with a defined size segments are typically sized on the order of gigabytes you set the segment size when you define tables in ocient the size of the segment must be a multiple of the coding block size the coding block is the unit of parity calculation and the smallest unit of recovery as well multiple segments combine to form segment groups a segment group has a fixed number of segments, named the width, which is the number of segments in the group it also has a pre defined number of parity blocks per set of data blocks for resiliency each segment has a defined index in the group the segment group is physically stored in a storage cluster segment groups can also be nested in directories named segment directory groups when the user configures an ocient system, they need to define at least one storage space and storage cluster a storage cluster is a set of foundation nodes with an associated storage space also, these nodes coordinate together to store segment groups in a reliable manner the storage cluster also has a width in the number of nodes the width of the cluster and the segment groups stored on it must be the same at the storage space level, the user defines the width and parity width of the space segments and their segment groups are stored across these storage spaces and clusters to learn more about erasure coding, see compute adjacent input and output on large working sets docid 4 dsigjclxrf1rzztmwq4 ocient hyperscale data warehouse contains foundation nodes in segments grouped into segment groups that are grouped into storage clusters tables with metadata are stored in segments for details, see configure storage spaces docid\ vepqy7er3ewghfze1bvoa related links ocient architecture docid\ va ukzt3cfwu ghuqdn6s compute adjacent input and output on large working sets docid 4 dsigjclxrf1rzztmwq4 key concepts docid\ xpo4reeauxnlnh t3rss related videos at the whiteboard with ocient deployment options at the whiteboard with ocient data reliability