Starting with Tetra Data Platform (TDP) version 3.6.0 and the TetraScience Software Development Kit (SDK) 2.0 release, the
protocol.ymlfile format replaces both the previous
script.jsfile formats when creating self-service Tetra Data pipelines (SSPs).
Protocols define the business logic of your pipeline by specifying the steps and the functions within task scripts that run those steps. You can configure your own custom protocols for use in an SSP by creating a
For instructions on how to create and deploy a custom protocol, see Create and Deploy a Protocol in the “Hello, World!” SSP Example.
protocolSchema: "v3" name: "Example Protocol" description: "This is just to show a basic protocol.yml file" config: idsFileCategory: label: "IDS File Category" description: "Optional File Category" type: "string" required: false default: "IDS" steps: - id: raw-to-ids task: namespace: common slug: akta-raw-to-ids version: v3.1.6 function: akta-raw-to-ids input: input_file_pointer: $( workflow.inputFile ) ids_file_category: $( config.idsFileCategory )
Instead of directly specifying everything in a protocol, you can use expressions to programmatically control the workflow at run time.
Expressions are specified by enclosing them in the following:
$( ... )
Within an expression, you have access to several contexts that contain data and a limited list of functions. These contexts can be combined however you want within a single expression.
input_file_pointer: $( workflow.inputFile )
The following contexts contain data and a limited list of functions that you can combine however you want within a single expression.
workflow context contains data relevant to the current workflow.
workflow Context Examples
|Current workflow |
|Pointer to the file that triggered the workflow|
|ISO 8601-formatted string of the date and time when the workflow started|
config context contains data configured by the pipeline user.
config Context Example
config: foo: label: Foo type: string bar: label: Hey Bear type: object
configcontext example, you could write
$( config.foo )or
$( config.bar ), but
$( config.baz )would result in an error.
Contains all the constant values defined in the protocol. If the constant contains an expression, it will be evaluated before access.
constants Context Example
constants: foo: "bar" baz: - 1 - 2
constantscontext example, you could write
$( constants.foo )or
$( constants.baz ), but
$( constants.qux )would result in an error.
Contains manifest fields. For example, a
protocol context could contain
$( protocol.name ) or
$( protocol.version ). If the value is not specified in the
protocol.yml file, it will return a
protocol Context Examples
For example protocol context values, see the Manifest Fields section.
Contains all of the step results. A step can be accessed in this context only once it has been run, or skipped. Steps can be referenced either by their
id or by their index within the steps list.
For example, if the first step has
id: stepOne, then its output could be accessed by any of the following:
steps Context Examples
|The output returned from the task script in the referenced step|
|Errors returned from the task script|
Note: This value is returned only if the referenced task script has its
|Status of the referenced step, listed as one of the following values:|
You can use the following
function: values in a
protocol.yml file to alter data within expressions.
Any functionality beyond the ones listed in the following table should be added to a task script, not a protocol.
|Returns the first value that isn’t ||c|
|Returns the negation of the value|
Type: String (must be
protocolSchema: "v3" property is required to indicate that this protocol is in the
The following manifest fields are optional, top-level metadata fields.
|String (must be ||Artifact type|
|String (must be ||Protocol namespace|
|String (must be in the following form: ||Protocol version number|
|List of strings||Used to look up the appropriate labels so that they can be added to the |
|List of key-value pairs|
|Used to annotate files and trigger pipelines without changing any file data|
For more information, see Labels.
config property is an object that maps a configuration ID to for a config object. Each config ID is unique so that it can be referenced in the
Config objects can’t contain expressions.
config: numberConfig: label: "Enter a Number" description: "This number will be used by the task scripts" type: number default: 9000 secretExample: label: "ELN Password" description: "The password to push data to some ELN" type: secret
config label that’s rendered in the TDP’s Pipeline Manager page.
The config type determines how the TDP UI renders the configuration element and how it is passed into the workflow.
config.<config_id>.type value must be one of the following:
Determines if the config is required or not. This requirement is reflected in the TDP UI and is also checked at workflow run time. If not set, this defaults to
A short description of the
config property. This description is what’s rendered on the Pipeline Manager page in the TDP.
config property. This default property is used if the
config property isn’t configured in the TDP UI. This property should contain a value whose type matches the config's
constants property is an object that maps a constant ID to any value. This property allows complex protocols to define common values once, instead of having them listed multiple times throughout the
Each constant ID is unique so that it can be referenced in the
steps property, or from other
constants. Constants can contain expressions. However, those expressions don't have access to the
steps context, because no steps have run when constants are evaluated.
Circular dependencies aren’t allowed and are detected by the TDP automatically.
constants: # a number REMOTE_PORT: 8080 # a string REMORE_URL: "http://example.com" # a list VALID_VALUES: - 80 - 443 - $( constants.REMOTE_PORT ) # an object CONFIG_OBJECT: url: $( constants.REMORE_URL ) ports: $( constants.VALID_VALUES )
steps property defines the tasks that the protocol runs and their inputs. The
steps property contains a list of step objects. Each step object runs a task script.
steps: - id: stepOne task: namespace: common slug: raw-to-ids version: v1.0.0 function: main input: input_file_pointer: $( workflow.inputFile ) - task: namespace: common slug: push-to-eln version: v2.1.0 function: "push-it" input: ids_file: $( steps.stepOne.output.ids_file ) options: timeoutInSec: $( config.pushToElnTimeout )
Defines a unique identifier for the step. The allows the step object to be referenced by its
id in later steps. If no
id is specified for a step, then later steps must refer to it by its index.
Type: Boolean or expression
If this property is set, then the step will run only if the value is
true, or if the expression evaluates to
true. If the
steps[*].if property isn’t set, then the step always runs.
You can use the
steps[*].ifproperty to conditionally run steps based on the output of a previous step or a configuration value.
Type: Boolean or expression
If this property is set and the task fails, then the protocol continues to run only if the value is
true, or the expression evaluates to
true. If the
steps[*].continueOnError property isn’t set, then the protocol always ends in failure if the task fails.
You can use the
steps[*].continueOnErrorproperty to handle errors generated by failing task scripts. For example, you can use this property to programmatically run a cleanup task after a failed task.
A short description of the step.
Type: Task object
Defines the task that the step runs.
Defines the step’s namespace. The value can be any one of the following:
Defines the step’s task slug, such as
Defines the step’s task version number, such as
Identifies the function that the task script runs. Task scripts can define multiple functions, so this property is required and must match one of the function slugs defined by the task script.
The input value that’s evaluated and passed directly into the task script function.
steps[*].inputproperty can contain expressions.
Specifies default options for the task script runner.
Type: Number or expression
Determines the default memory used to run this step’s task.
steps[*].options.memoryInMBproperty can be overridden on the Pipeline Manager page in the TDP UI.
Type: Number or expression
Determines how long before a task is considered failed, even if it hasn't completed.
Updated 27 days ago