This topic shows examples of the pipeline and workflow objects, and defines the parameters in each object.

Pipeline Object and Parameters

Sample pipeline objects appear below.

{
    "id": "8592187a-d3ed-4410-8ffb-ee47826bebf7",
    "name": "HG-Covid-Data",
    "description": "Reads in Covid USA states Data",
    "triggerType": "custom",
    "triggerCondition": {
        "groupOperator": "AND",
        "groupLevel": 1,
        "groups": [
            {
                "groupLevel": 2,
                "groupOperator": "AND",
                "groups": [
                    {
                        "key": "sourceId",
                        "operator": "is",
                        "value": "4095be8f-40c6-487d-99ce-bef42682c36b"
                    }
                ]
            },
            {
                "groupLevel": 2,
                "groupOperator": "AND",
                "groups": [
                    {
                        "key": "category",
                        "operator": "is",
                        "value": "raw"
                    }
                ]
            }
        ]
    },
    "protocolSlug": "hg-prot-covid-data",
    "protocolVersion": "v2.1.0",
    "createdAt": "2021-11-08T19:00:02.701Z",
    "updatedAt": "2022-02-03T17:54:46.962Z",
    "pipelineConfig": {
        "notificationsConfig": {
            "sendOnSuccessful": true,
            "sendOnFailed": true,
            "notificationEmailAddresses": [
                "[email protected]",
                "[email protected]"
            ]
        }
    },
    "masterScriptNamespace": "private-tetrascience",
    "masterScriptSlug": "hg-prot-covid-data",
    "masterScriptVersion": "v2.1.0",
    "status": null,
    "standby": 0,
    "retryBehavior": null
}

{
    "id": "28fd27a9-c931-49fe-9d21-661a7503cee5",
    "name": "SDC Measurement RAW to IDS to Solace",
    "description": null,
    "triggerType": "custom",
    "triggerCondition": {
        "groupOperator": "AND",
        "groupLevel": 1,
        "groups": [
            {
                "groupLevel": 2,
                "groupOperator": "AND",
                "groups": [
                    {
                        "key": "tags",
                        "operator": "has a tag that is",
                        "value": "igor_sdc"
                    }
                ]
            },
            {
                "groupLevel": 2,
                "groupOperator": "AND",
                "groups": [
                    {
                        "key": "category",
                        "operator": "is",
                        "value": "raw"
                    }
                ]
            }
        ]
    },
    "protocolSlug": "sdc-measurement-raw-to-ids-push-to-solace",
    "protocolVersion": "v1.0.0",
    "createdAt": "2021-04-22T20:20:24.968Z",
    "updatedAt": "2022-02-22T06:38:40.794Z",
    "pipelineConfig": {
        "solace-username": "solace-cloud-client",
        "solace-password": {
            "ssm": "/development/tetrascience/org-secrets/ea-solace",
            "secret": true
        },
        "solace-url": "https://mr1oqbbo5q4w2z.messaging.solace.cloud",
        "solace-port": "9443",
        "solace-topic": "SDC.measurement.results",
        "ignore-ssl": false,
        "notificationsConfig": {}
    },
    "masterScriptNamespace": "common",
    "masterScriptSlug": "sdc-measurement-raw-to-ids-push-to-solace",
    "masterScriptVersion": "v1.0.0",
    "status": "disabled",
    "standby": 0,
    "retryBehavior": null,
    "priority": null,
    "maxSlotLimit": null,
    "taskScriptTimeoutMins": null
}

Pipeline Object Parameters

Pipeline object parameters are defined in the following table.

Field	Description
id	Globally unique identifier for this specific pipeline.
name	User-defined and human-friendly name of the pipeline.
description	Description of the pipeline.
triggerType	Indicates the schema to be used for the `triggerCondition`.
triggerCondition	Defines trigger conditions of the pipeline, based on file source type, category, metadata, tags, and labels. If a pipeline is created/updated via the UI, the `triggerExpression` generates the `triggerCondition` that the pipeline API uses. If a pipeline is created/updated directly through the API, we require the `triggerCondition` and will generate the `triggerExpression` from the condition.
triggerCondition.groupLevel	Indicates which hierarchical level the operator and group belong to. Starts with 1 and goes up to 2.
triggerCondition.groupOperator	Comparator between groups at the same level. Possible values: `AND`, `OR`
triggerCondition.groups	List of trigger conditions that the `groupOperator` is applied to.
protocolSlug	Unique slug that identifies the protocol used in this pipeline.
protocolVersion	Version of the protocol used by this pipeline. Must always be in the following format: v.X.Y.Z For example: v2.1.3.
createdAt	Date/Time when this pipeline was created.
updatedAt	The last time (Date/Time) when this pipeline had a configuration parameter change.
pipelineConfig	Contains the following pipeline configuration parameters: - `notificationsConfig`: List of email addresses to notify on the success or the failure of the pipeline. - `sendOnSuccessful`: boolean (true/false). Indicates whether a notification should be sent out if the pipeline succeeds. - `sendOnFailed`: boolean (true/false). Indicates whether a notification should be sent out if the pipeline fails. - `notificationEmailAddresses`: Array of strings. Comma-separated list of email addresses to notify whether the user wants a notification to be sent on success and/or failure of the pipeline.
masterScriptNamespace	Namespace of the protocol used in this pipeline. Can be any of the following: `common`, `client`, `private`.
masterScriptSlug	Deprecated; duplicates `protocolSlug`.
masterScriptVersion	Deprecated; duplicates `protocolVersion`.
status	Indicates whether the pipeline is active. Possible values: - `disabled`: Indicates the pipeline is inactive. - `null`: Indicates that the pipeline is not active (disabled) - `standby`: Indicates how many instances of resources should be on hot standby to process files for this protocol. Must be between 0 and 5. Hot standby of an instance can add financial cost.
retryBehavior	Retry is per step (not workflow level). Each step has a starting memory that defaults to 512MB, but can be overridden in step `config.json`. Max memory will be 30GB. Possible values: - `Always retry 3 times (default)` (UI Value), `null` - (API Value) Retries step on any type of error. Available memory will be doubled with each retry. - `Retry 3 times (after OOM error only)` (UI Value), `oom_only` (API Value) Retry step ONLY on an out-of-memory error. Available memory will be doubled on each retry. - `No retry` (UI Value),`off` (API Value) Do not retry under any circumstance.

Workflow Object and Parameters

A sample workflow object appears below.

{
    "id": "4d78df0e-6990-4d89-bca3-73261a99ca66",
    "orgSlug": "tetrascience",
    "pipelineId": "8592187a-d3ed-4410-8ffb-ee47826bebf7",
    "masterScriptNamespace": "private-tetrascience",
    "masterScriptSlug": "hg-prot-covid-data",
    "masterScriptVersion": "v2.1.0",
    "protocolSlug": "hg-prot-covid-data",
    "protocolVersion": "v2.1.0",
    "protocol": {
        "protocolSchema": "v2",
        "name": "HG Protocol COVID-JSON to IDS",
        "description": "",
        "steps": [
            {
                "slug": "first-step-covid-json-ids",
                "description": "Generates COVID data row (IDS JSON) for each state.",
                "type": "generator",
                "script": {
                    "namespace": "private-tetrascience",
                    "slug": "hg-tscr-covid-data",
                    "version": "v2.0.1"
                },
                "functionSlug": "main"
            }
        ],
        "config": []
    },
    "pipelineConfig": {
        "notificationsConfig": {
            "sendOnSuccessful": true,
            "sendOnFailed": true,
            "notificationEmailAddresses": [
                "[email protected]",
                "[email protected]"
            ]
        },
        "pipelineName": "HG-Covid-Data"
    },
    "inputFile": {
        "meta": {
            "fileId": "18384a5f-9007-4028-85cd-e191a04a71d5",
            "source": {
                "box": {
                    "id": "56d9293c-2dc7-4133-b21b-c1468aefb41f",
                    "size": 2559900,
                    "filePath": "/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json",
                    "integrationSource": "4095be8f-40c6-487d-99ce-bef42682c36b"
                },
                "type": "box"
            },
            "traceId": "18384a5f-9007-4028-85cd-e191a04a71d5",
            "sourceId": "4095be8f-40c6-487d-99ce-bef42682c36b",
            "sourceName": "HG-Box-Covid-Data-States",
            "sourceType": "box",
            "integrationId": "56d9293c-2dc7-4133-b21b-c1468aefb41f",
            "integrationType": "box"
        },
        "type": "s3file",
        "bucket": "ts-platform-dev-datalake",
        "fileId": "18384a5f-9007-4028-85cd-e191a04a71d5",
        "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/RAW/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json",
        "version": "1kj7Fdl63JaVE_iBcSeS1EGKLX6UNdDR",
        "customTags": [],
        "customMetadata": {}
    },
    "tasks": [
        {
            "slug": "first-step-covid-json-ids",
            "input": {
                "input_file": {
                    "meta": {
                        "fileId": "18384a5f-9007-4028-85cd-e191a04a71d5",
                        "source": {
                            "box": {
                                "id": "56d9293c-2dc7-4133-b21b-c1468aefb41f",
                                "size": 2559900,
                                "filePath": "/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json",
                                "integrationSource": "4095be8f-40c6-487d-99ce-bef42682c36b"
                            },
                            "type": "box"
                        },
                        "traceId": "18384a5f-9007-4028-85cd-e191a04a71d5",
                        "sourceId": "4095be8f-40c6-487d-99ce-bef42682c36b",
                        "sourceName": "HG-Box-Covid-Data-States",
                        "sourceType": "box",
                        "integrationId": "56d9293c-2dc7-4133-b21b-c1468aefb41f",
                        "integrationType": "box"
                    },
                    "type": "s3file",
                    "bucket": "ts-platform-dev-datalake",
                    "fileId": "18384a5f-9007-4028-85cd-e191a04a71d5",
                    "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/RAW/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json",
                    "version": "1kj7Fdl63JaVE_iBcSeS1EGKLX6UNdDR",
                    "customTags": [],
                    "customMetadata": {}
                }
            },
            "retry": 0,
            "events": [
                {
                    "at": "2022-03-21T15:53:40.403+00:00",
                    "status": "pending"
                },
                {
                    "at": "2022-03-21T15:54:09.932+00:00",
                    "status": "in-progress"
                },
                {
                    "at": "2022-03-21T15:54:40.449+00:00",
                    "status": "completed"
                }
            ],
            "output": {
                "type": "success",
                "result": [
                    {
                        "type": "s3file",
                        "bucket": "ts-platform-dev-datalake",
                        "fileId": "59c769b3-f73b-41f9-8d76-9dc04fb5da9d",
                        "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-AK-2021-11-19.json",
                        "version": "2S3R7BGUAUGpRD8af7uxhp5lG56EAap_"
                    },
                    {
                        "type": "s3file",
                        "bucket": "ts-platform-dev-datalake",
                        "fileId": "7ab509dc-8f1d-48b2-8e73-1c682296b0f0",
                        "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-AL-2021-11-19.json",
                        "version": "PcB6tKOfnNds1ooPWQaVNieitqWSYiLd"
                    },
                    {
                        "type": "s3file",
                        "bucket": "ts-platform-dev-datalake",
                        "fileId": "99eb82e0-e89d-4d69-8f34-b9df3114c589",
                        "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-AR-2021-11-19.json",
                        "version": "5p4GQ6AhlIn_EPuSaMOivPRnxXYbO9Z1"
                    },
 
<...truncated for doc brevity>

                    {
                        "type": "s3file",
                        "bucket": "ts-platform-dev-datalake",
                        "fileId": "944f7d30-3193-493f-bf05-0a1681df6c4a",
                        "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-WY-2021-11-19.json",
                        "version": "2P5hobBY8GwPKS8jkDPeELqPdc0zhz2M"
                    }
                ]
            },
            "script": {
                "list": [
                    "requirements.txt",
                    "README-internal.md",
                    "config.json",
                    "README.md",
                    "Pipfile",
                    "main.py",
                    "Pipfile.lock",
                    "__test__/test_business_logic.py",
                    "__test__/test_config.py",
                    "__test__/test_integration.py",
                    "__test__/__pycache__/test_business_logic.cpython-37.pyc",
                    "__test__/__pycache__/test_config.cpython-37.pyc",
                    "__test__/__pycache__/test_integration.cpython-37.pyc",
                    "__test__/data/expected.json",
                    "__test__/data/input.json",
                    "__pycache__/main.cpython-37.pyc"
                ],
                "slug": "hg-tscr-covid-data",
                "docker": {
                    "image": "706717599419.dkr.ecr.us-east-2.amazonaws.com/ts-platform-development-task-private-tetrascience-hg-tscr-covid-data@sha256:c8efc76762d45bf74fe2ec9ae80f817224c715b8b2892b877862ce118b6980bb"
                },
                "version": "v2.0.1",
                "language": "python",
                "maxCount": 30,
                "createdAt": "2022-03-16T14:25:41.393Z",
                "functions": [
                    {
                        "slug": "main",
                        "function": "main.main"
                    }
                ],
                "hasSource": true,
                "namespace": "private-tetrascience",
                "timestamp": 1647440600360,
                "runnerType": "ecs",
                "buildLogSaved": true,
                "buildDurationMs": 78820
            },
            "status": "completed",
            "taskId": "3ec7ffec-ed45-440d-9375-0288f835b25e",
            "options": {},
            "createdAt": "2022-03-21T15:53:40.387+00:00",
            "containerId": "755298eb-da0a-47fb-a3a5-e02d02fab7d9",
            "functionSlug": "main",
            "lastUpdatedAt": "2022-03-21T15:54:40.455+00:00",
            "taskMemoryInMB": 512,
            "cloudWatchUrl": "https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logs-insights:queryDetail=~(end~'2022-03-21T15*3a59*3a40.455Z~start~'2022-03-21T15*3a48*3a40.387Z~timeType~'ABSOLUTE~tz~'Local~editorString~'filter*20*60taskId*60*3d*273ec7ffec-ed45-440d-9375-0288f835b25e*27*20and*20*60containerId*60*3d*27755298eb-da0a-47fb-a3a5-e02d02fab7d9*27*0a*7c*20fields*20*40timestamp*2c*20*40message*0a*7c*20sort*20*40timestamp*20asc~source~(~'*2fecs*2ftaskscripts*2fts-platform*2fcontainers))"
        }
    ],
    "supersededTasks": [],
    "status": "completed",
    "events": [
        {
            "at": "2022-03-21T15:53:40.280+00:00",
            "status": "in-progress"
        },
        {
            "at": "2022-03-21T15:54:40.465+00:00",
            "status": "completed"
        }
    ],
    "createdAt": "2022-03-21T15:53:40.010Z",
    "lastUpdatedAt": "2022-03-21T15:54:40.467Z",
    "restarted": false,
    "masterScriptLogs": [],
    "retryBehavior": null,
    "orchestratorId": null,
    "output": {
        "type": "success",
        "result": [
            {
                "type": "s3file",
                "bucket": "ts-platform-dev-datalake",
                "fileId": "59c769b3-f73b-41f9-8d76-9dc04fb5da9d",
                "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-AK-2021-11-19.json",
                "version": "2S3R7BGUAUGpRD8af7uxhp5lG56EAap_"
            },
            {
                "type": "s3file",
                "bucket": "ts-platform-dev-datalake",
                "fileId": "7ab509dc-8f1d-48b2-8e73-1c682296b0f0",
                "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-AL-2021-11-19.json",
                "version": "PcB6tKOfnNds1ooPWQaVNieitqWSYiLd"
            },
         
 ... <truncated>
 
            {
                "type": "s3file",
                "bucket": "ts-platform-dev-datalake",
                "fileId": "944f7d30-3193-493f-bf05-0a1681df6c4a",
                "fileKey": "tetrascience/4095be8f-40c6-487d-99ce-bef42682c36b/IDS/All Files/TetraScience/coviddata/statesdata/covid_states_tseries_2021_11_19.json/covid-state-tseries-WY-2021-11-19.json",
                "version": "2P5hobBY8GwPKS8jkDPeELqPdc0zhz2M"
            }
        ]
    }
}

Workflow Object Parameters

Workflow object parameters are defined in the following table.

Field	Description
id	Globally unique identifier for this specific workflow.
orgSlug	Indicates which organization this workflow was generated from.
pipelineId	Globally unique id of the pipeline that was triggered for this workflow.
masterScriptNamespace	Namespace of the protocol used in this workflow. Can be any of the following: `common` , `client` , `private` .
masterScriptSlug	This field is deprecated; duplicates the `protocolSlug` .
masterScriptVersion	This field is deprecated; duplicates `protocolVersion` .
protocolSlug	Slug of the protocol used in this workflow.
protocolVersion	Version of the protocol used by this workflow.
protocol	Protocol definition that includes protocol parameter definitions and taskscript information for each step.
protocol.protocolSchema	Version number of the schema.
protocol.name	Name of the protocol.
protocol.description	Description of the protocol.
protocol.steps	Slug, description, type, script, and functionSlug for the protocol.
protocol.config	Configuration details.
pipelineConfig	Pipeline configuration and pipeline parameter value settings that are passed to the workflow. These are: - `notificationsConfig`: List of email addresses to notify upon success or failure of the pipeline. - `sendOnSuccessful`: boolean (true/false). Indicates whether the notification should be sent if the pipeline succeeds. - `sendOnFailed`: boolean (true/false). Indicates whether the notification should be sent if the pipeline fails. - `notificationEmailAddresses`: An array of strings. Comma-separated list of email addresses to notify if the user wants notification on success and/or failure of the pipeline. - `pipelineName`: Name of the pipeline.
inputFile	Input parameter object that is passed into the taskscript as defined by protocol parameters. (Passed into the taskscript by script.js when calling `workflow.runTask` ).	meta: Provides file metadata information. This includes the fileId, source, and source type of the input file. It also includes the `traceId`, `sourceId`, `sourceName`, `sourceType`, `integrationId`, and `integrationType`, the type of file (e.g. s3file), the S3 bucket it is stored in, and the `fileId`, `fileKey`, `version`, `customTags`, and `customMetadata`.
`tasks[]`	An array that contains information on the last execution of each step in a protocol. If a step was run multiple times (for retries), the info in this array will contain info for ONLY the last run of each step. See `supersededTasks[]` for previous runs/retries. Each element in the list has the following fields: - `slug` - the taskscript slug for the given step - `input file` - includes the fileId, source, source ID, size, filepath, integration source and type. It also provides the traceId, sourceId, sourceName, sourceType, integrationId, and integrationType, the type of file (e.g. s3file), the bucket it is stored in, fileId, fileKey, version, customTags, and customMetadata. - `Retry` - indicates which retry run this represents. Since the items in the `task[]` are the last run of each step, this also indicates the total number of retries (after the initial run) that were run for the given task. Should be a number from zero to three. Zero indicates that the initial run of the script was successful. Three means that there was an initial run of the script and 3 additional retries (total of 4 runs of the script) - `events` - Presents a list of events that track status changes for individual taskscript executions (at the container level). Events provide the following information: - `at`: Timestamp of the event. - `status`: Indicates event type: (`pending` - step is waiting to be executed, `in progress` - step is currently executing, `complete` - step successfully completed. ) `output` - indicates the taskscript result info for the given run. On successful completion, this will contain whatever the task script returns on exit. For example, if the taskscript returns a list of files, then the output result will contain a list of files. If the taskscript returns number, then the `output.result` will be the returned number.
script	Detailed information about the list of files in the pipeline, the slug, docker image information, version, language, maxCount, when the script was created, function information, whether it hasSource, the namespace, timestamp, runnerType, and buildDuration in milliseconds. It also provides information about build log.
status	Last status event.
taskId	Globally unique id of the taskscript.
createdAt	Date/Time when the taskscript was started.
containerId	Id of the container that ran the task script.
functionSlug	Provides the taskscript code function name.
lastUpdatedAt	Date/Time this taskscript had an event update.
taskMemoryInMB	Starting memory for the taskscript container as defined by the taskscript configuration.
cloudWatchUrl	CloudWatch location of logs for this run of the taskscript.
supersededTasks[]	Contains run information for each retry for each step. Excludes the last run of each step. The last (latest) run of each steps is in the `tasks[]` field.
retry	Indicates the retries instance for this run. A number between zero and three.
output	The taskscript result info for the given run.
status	Final success/fail status of the script
createdAt	Date/Time when this workflow was started.
lastUpdatedAt	Date/Time when this workflow had an event update.
restarted	Indicates with the workflow was restarted.
masterScriptLogs	Indicates the location of the script logs
orchestratorId	Internal component processing id.
output	returned by script.js (in v3.1 this was called result). The standard output structure for `root.output, supersededTasks[].output, tasks[].output:` Successful case: `{ "type": ”success” "result": "anything that the taskscript or script.js returns, can be [], string, boolean, etc."} }` Error case: If a taskscript throws an exception, then the output will contain an error message. `{"type": ”error” "result": { "message": "error message details"} }`
events	List of events to track status for general workflow (not specific to a taskscript). `at` - Timestamp of the event `status` - Indicates event type: `in-progress` - indicates the workflow is currently executing (generated just before container startup, so includes container startup time) `completed` - workflow completed successfully.