This topic provides an example of the minimum files needed to create a self-service Tetra Data pipeline (SSP).

For more information about example SSP use cases, see Example Use Cases for SSPs. For best practices, see Data Engineering and Tooling and Automation in the TetraConnect Hub. For access, see Access the TetraConnect Hub.

Architecture

The following diagram shows an example “Hello, World!” SSP workflow:

The diagram shows the following workflow:

1. A task script named sspdemo-taskscript (v1.0.0) is created. The config.json file has one exposed function named print-hello-world (main.print_hello_world), which is the print_hello_world function found in the main.py file.
2. A protocol named hello-world (v1.0.0) is created. The protocol.yml file provides the protocol name, description, and outlines one step: hello-world-step. This step points to the sspdemo-taskscript task script and the exposed function, print-hello-world.

Prerequisites

Before you can create a self-service pipeline (SSP), you must do the following:

• Initialize the TetraScience Software Development Kit (SDK) environment.
• Configure Tetra Data Platform (TDP) dependencies and authentication.

For more information, see SSP Setup and Prerequisites.

Create an SSP

To create a basic SSP, do the following.

Create the SSP Folder Structure

Create the following basic SSP folder structure on your local machine in the directory where you want to keep your artifact code:

• A protocol folder that contains a protocol.yml file
• A task script folder that contains the following files:
  • config.json
  • main.py
  • requirements.txt

SSP Folder Structure Example

protocol
└── protocol.yml
task-script
├── config.json
└── main.py
└── requirements.txt

Create and Deploy a Task Script

Task scripts are the building blocks of protocols, so you must build and deploy your task scripts before you can deploy a protocol that uses them.

Task scripts require the following:

• A config.json file that contains configuration information that exposes and makes your Python functions accessible so that protocols can use them.
• A Python file that contains python functions (main.py in the following examples) that include the code that’s used in file processing.
• A requirements.txt file that either specifies any required third-party Python modules, or that is left empty if no modules are needed.

To create and deploy a task script that prints Hello World! to the TDP console, do the following.

📘

NOTE

For more information about creating custom task scripts, see Task Script Files. For information about testing custom task scripts locally, see Create and Test Custom Task Scripts.

Create a config.json File

Create a config.json file in your code editor by using the following code snippet:

{
    "language": "python",
    "runtime": "python3.11",
    "functions": [
        {
            "slug": "print-hello-world",
            "function": "main.print_hello_world"
        }
    ]
}

📘

NOTE

You can choose which Python version a task script uses by specifying the "runtime" parameter in the script's config.json file. Python versions 3.7, 3.8, 3.9, 3.10, and 3.11 are supported currently. If you don't include a "runtime" parameter, the script uses Python v3.7 by default.

Create a main.py File

Create a main.py file in your code editor by using the following code snippet:

from ts_sdk.task.__task_script_runner import Context

def print_hello_world(input: dict, context):
    print("Hello World!")
    return "Hello World!"

Create a Python Package (requirements.txt)

Within the task script folder that contains the config.json and main.py files, use Python Poetry to create a Python package and the necessary files to deploy them to the TDP.

The following Poetry export command example produces this package as a requirements.txt file.

Poetry Command Example

poetry init
[import packages with "poetry add"]
poetry export --without-hashes --format=requirements.txt > requirements.txt

📘

NOTE

If no packages are added, this poetry export command example produces text in requirements.txt that you must delete to create an empty requirements.txt file. A requirements.txt file is required to deploy a task script to the TDP.

Deploy the Task Script

To the deploy the task script, do the following:

1. Make sure that the local auth.json file you created while completing the SSP Setup and Prerequisites is formatted correctly.

Exampleauth.json File

{
    "api_url":"<TDP API endpoint base URL>",
    "auth_token":"<service token you generated>",
    "org":"<your organization slug name>",
    "ignore_ssl": <true to allow invalid SSL certificates>
}

2. Run the following command from your command line (for example, bash):

ts-cli publish --type  task-script --namespace private-{TDP ORG} --slug  sspdemo-taskscript --version v1.0.0 {task-script-folder} -c {auth-folder}/auth.json

📘

NOTE

Make sure to replace TDP ORG with your organization slug, {task-script-folder} with the local folder that contains your protocol code, and {auth-folder} with the local folder that contains your authentication information. To add the artifact to more than one organziation, see Add Artifacts to Multiple Organizations.

Create and Deploy a Protocol

Protocols define the business logic of your pipeline by specifying the steps and the functions within task scripts that execute those steps. For more information about how to create a protocol, see Protocol YAML Files.

In the following example, there’s one step: hello-world-step. This step uses the print-hello-world function that’s in the sspdemo-taskscript task script.

Create a protocol.yml File

Create a protocol.yml file in your code editor by using the following code snippet:

protocolSchema: "v3"
name: "Hello World - v3 protocol"
description: "Protocol that prints Hello World to TDP Console"

steps:
  - id: hello-world-step
    task:
      namespace: private-training-sspdemo
      slug: sspdemo-taskscript
      version: v1.0.0
      function: print-hello-world

Deploy the Protocol

To the deploy the protocol, do the following:

1. Make sure that the local auth.json file you created while completing the SSP Setup and Prerequisites is formatted correctly. For an example auth.json file, see Deploy the Task Script.

2. Run the following command from your command line (for example, bash):

📘

NOTE

Make sure to replace TDP ORG with your organization slug, {protocol-folder} with the local folder that contains your protocol code, and {auth-folder} with the local folder that contains your authentication information. To add the artifact to more than one organziation, see Add Artifacts to Multiple Organizations.

ts-cli publish --type protocol --namespace private-{TDP ORG} --slug hello-world --version v1.0.0 {protocol-folder} -c {auth-folder}/auth.json

📘

NOTE

To redeploy the same version of your code, you must include the -f flag in your deployment command. This flag forces the code to overwrite the file. The following are example protocol deployment command examples:

• ts-cli publish --type protocol --namespace private-xyz --slug hello-world --version v1.0.0 ./protocol -f -c auth.json
• ts-cli publish --type task-script --namespace private-xyz --slug hello-world --version v1.0.0 ./task-script -f -c auth.json

For more details about the available arguments, use the following command on the command line:
ts-cli publish --help

Create a Pipeline by Using the Deployed Protocol

To use your new protocol on the TDP, create a new pipeline that uses the protocol that you deployed. Then, upload a file that matches the pipeline’s trigger conditions.

Documentation Feedback

Do you have questions about our documentation or suggestions for how we can improve it? Start a discussion in TetraConnect Hub. For access, see Access the TetraConnect Hub.

📘

NOTE

Feedback isn't part of the official TetraScience product documentation. TetraScience doesn't warrant or make any guarantees about the feedback provided, including its accuracy, relevance, or reliability. All feedback is subject to the terms set forth in the TetraConnect Hub Community Guidelines.