Manage Distributed Tasks

distributed tasks are processes that run in the background of the {{ocient}} system to keep the system running smoothly and efficiently these tasks provide persistent storage of task information in the global metadata storage cluster of the ocient system, allowing tasks to be tracked and retried until completion for details about storage and related concepts, see core elements of an ocient system docid\ adp4amtxi3djdrsq2khdb the distributed task infrastructure is completely generic the system instantiates tasks from factories using generic arguments and establishes communication using interfaces to abstract the business logic in whatever specific protocol is doing the actual work tasks must remain unchanged when duplicated and can be restarted on a new node by the infrastructure at any time tasks can also be structured recursively, with a parent task comprising several subtasks, which can also have subtasks, and so on distributed tasks provide persistence across outages to do this, the task infrastructure detects when the node running a task, or the task owner, has become unavailable and moves the task elsewhere the administrator owner manages node unavailability this owner is a node in the administrator cluster responsible for monitoring the status of a task and reassigning the task owner when needed this pairing of administrator and task owners provides load balancing across the administration cluster by removing the need for every task owner to communicate with the administrator owner task types the ocient system has different task types, such as data pipeline loading rebuild segments check for disk corruption by verifying segments rebalance data across storage clusters and nodes for data pipeline loading, see load data docid\ xq0tg7yph vn62uwufibu for details about rebuilding segments, checking the disk, and rebalancing data, see distributed tasks docid\ wyzmz0s4turygt1pbasjs task characteristics the characteristics of a task are a factory, serialized arguments, a location type, and a location identifier factory — a task class implementation that the ocient system instantiates generically and executes the actual work of the task arguments — the inputs to a task the ocient system stores generically as a string these inputs can contain anything the task needs, e g , flags modifying execution modes, identifiers on which to run, etc location type — the nature of the set of nodes where the task can run this location can be any node in the system, any node in a specific cluster, or a specific node location identifier — the specific location entity when the task location type is a cluster or node tasks have optional pre and post condition checks and must implement an asynchronous execute method you can also override cancel and status poll methods task execution execute tasks using a pair of nodes administrator and task owners the administrator owner must be an active node in the metadata storage cluster this node is responsible for starting and monitoring the task the task owner can be any node that meets the location constraints specified by the task creator and executes the actual task with the exception of subtasks, the ocient system assigns both owners upon task creation, chosen randomly to balance the load to see the disposition of tasks in the system, see the sys subtasks system catalog table for details, see sys subtasks administrator owner the administration owner manages and monitors all steps in the life cycle of their own tasks after the ocient system adds a task to the cluster state and assigns an administrator owner, the administrator node sends a message to the task owner to submit or poll the current task status after the task has started successfully, the administrator owner periodically polls the task owner for the task status, updating it in the cluster state as needed when a task completes, the administrator owner sets its state to a terminal status and no longer polls or interacts with the task the ocient system retains historical tasks in the cluster state until a configurable limit is reached to prevent the state from growing unboundedly task owner the task owner maintains an in memory map of running tasks by the identifier in the health protocol the ocient system pools tasks for execution status as needed when the ocient system receives a poll task request from an administrator owner, a task owner checks for the existence of the task, starting it if necessary and returning the current status the ocient system starts a task by instantiating a task object using the factory type and then executing its pre condition check, method, and post condition check if at any point the system encounters a failure, the system stops execution when a task completes, the task owner stores the results in memory until it receives an indication from the administrator owner that it is safe to remove the results handle outages administrator owner an important responsibility of the administrator owner is to detect and handle task owner outages if a poll fails for any reason, the administrator owner reassigns the owner in the cluster state and starts the task in the new location tasks must remain unchanged when duplicated and resume where they stopped when restarted if an administrator owner fails, the administrator leader detects the connectivity change and reassigns the administrator owner the new administrator owner begins polling the task owner when it receives the update, so the ownership change should not interfere with task execution task owner generally, the task owner does not need to handle outages however, in extreme situations, e g , network splits, an administrator owner might not be able to communicate with its task owner and orphan a task to prevent orphaned tasks from running in multiple locations in the system, the task owner times out and cancels a running task if it has not heard from the administrator owner in multiple poll cycles subtasks when you execute a task using a sql statement, the task performs the operation by executing one or multiple subtasks thus, when you query the system catalog tables for subtasks, you must look for numerous subtasks and their individual progress rather than just one task you can also execute tasks as a series of subtasks, that run sequentially, in parallel, or some combination of the two the series comprises a parent task with a number of child tasks each child task has a specified ordering the ocient system executes tasks with the same ordering in parallel and, the system executes groups of child tasks in sequence according to ascending order the ocient system assigns an administrator owner when the child task should begin running the system executes parent tasks directly on the administrator owner these tasks are only responsible for watching the status of their running child tasks and starting the next child tasks when applicable parent tasks complete after all their children tasks complete to see the disposition of subtasks in the system, you can query the sys subtasks system catalog table for details, see sys subtasks task cancellation a client can request to gracefully cancel a task by setting an is canceled flag on the task object the administrator owner evaluates this state on its next poll cycle and notifies the task owner that the task should be canceled the exact mechanism by which the running task is canceled varies by type, as some tasks might need to communicate with other protocols to stop execution canceling a parent task causes all of its subtasks to be canceled a subtask cannot be canceled individually if the task is successfully canceled, execution ends with a cancelled status and reverts back to the administrator owner and raft state using the normal success or failure paths for example, cancel the task named my task cancel task 'my task'; for the full syntax, see distributed tasks docid\ wyzmz0s4turygt1pbasjs related links core elements of an ocient system docid\ adp4amtxi3djdrsq2khdb distributed tasks docid\ wyzmz0s4turygt1pbasjs system catalog