This topic provides an example of the minimum files needed to create a self-service Tetra Data pipeline (SSP).

Architecture

The following diagram shows an example “Hello, World!” SSP workflow:

The diagram shows the following workflow:

1. A task script named sspdemo-taskscript (v1.0.0) is created. The config.json file has one exposed function named print-hello-world (main.print_hello_world), which is the print_hello_world function found in the main.py file.
2. A protocol named hello-world (v1.0.0) is created. The protocol.yml file provides the protocol name, description, and outlines one step: hello-world-step. This step points to the sspdemo-taskscript task script and the exposed function, print-hello-world.

SSP Folder Structure

Create the following basic SSP folder structure on your local machine in the directory where you want to keep your artifact code:

• A protocol folder that contains a protocol.yml file
• A task script folder that contains a config.json file and a Python file that contains Python functions ( main.py in the following example).

SSP Folder Structure Example

protocol
└── protocol.yml
task-script
├── config.json
└── main.py

Create and Deploy a Task Script

Task scripts are the building blocks of protocols, so you must build and deploy your task scripts before you can deploy a protocol that uses them.

Task scripts require the following:

• A config.json file that contains configuration information that exposes and makes your Python functions accessible so that protocols can use them.
• A Python file that contains python functions (main.py in the following examples) that include the code that’s used in file processing.
• A requirements.txt file that either specifies any required third-party Python modules, or that is left empty if no modules are needed.

To create and deploy a task script that prints Hello World! to the TDP console, do the following.

📘

NOTE

For more information about creating custom task scripts, see Task Script Files. For information about testing custom task scripts locally, see Create and Test Custom Task Scripts.

Create a config.json File

Create a config.json file in your code editor by using the following code snippet:

{
    "language": "python",
    "runtime": "python3.11",
    "functions": [
        {
            "slug": "print-hello-world",
            "function": "main.print_hello_world"
        }
    ]
}

📘

NOTE

You can choose which Python version a task script uses by specifying the "runtime" parameter in the script's config.json file. Python versions 3.7, 3.8, 3.9, 3.10, and 3.11 are supported currently. If you don't include a "runtime" parameter, the script uses Python v3.7 by default.

Create a main.py File

Create a main.py file in your code editor by using the following code snippet:

from ts_sdk.task.__task_script_runner import Context

def print_hello_world(input: dict, context):
    print("Hello World!")
    return "Hello World!"

Create a Python Package

Within the task script folder that contains the config.json and main.py files, use Python Poetry to create a Python package and the necessary files to deploy them to the TDP.

Poetry Command Example

poetry init
[import packages with "poetry add"]
poetry export --without-hashes --format=requirements.txt > requirements.txt

📘

NOTE

If no packages are added, this poetry export command example produces text in requirements.txt that you must delete to create an empty requirements.txt file. A requirements.txt file is required to deploy a task script to the TDP.

Deploy the Task Script

To the deploy the task script, run the following command from your command line (for example, bash):

ts-sdk put task-script private-{TDP ORG} sspdemo-taskscript v1.0.0 {task-script-folder} -c {auth-folder}/auth.json

📘

NOTE

Make sure to replace TDP ORG with your organization slug, {task-script-folder} with the local folder that contains your protocol code, and {auth-folder} with the local folder that contains your authentication information.

Create and Deploy a Protocol

Protocols define the business logic of your pipeline by specifying the steps and the functions within task scripts that execute those steps. For more information about how to create a protocol, see Protocol YAML Files.

In the following example, there’s one step: hello-world-step. This step uses the print-hello-world function that’s in the sspdemo-taskscript task script.

Create a protocol.yml File

Create a protocol.yml file in your code editor by using the following code snippet:

protocolSchema: "v3"
name: "Hello World - v3 protocol"
description: "Protocol that prints Hello World to TDP Console"

steps:
  - id: hello-world-step
    task:
      namespace: private-training-sspdemo
      slug: sspdemo-taskscript
      version: v1.0.0
      function: print-hello-world

Deploy the Protocol

To the deploy the protocol, run the following command from your command line (for example, bash):

ts-sdk put protocol private-{TDP ORG} hello-world v1.0.0 {protocol-folder} -c {auth-folder}/auth.json

📘

NOTE

Make sure to replace TDP ORG with your organization slug, {protocol-folder} with the local folder that contains your protocol code, and {auth-folder} with the local folder that contains your authentication information.

📘

NOTE

To redeploy the same version of your code, you must include the -f flag in your deployment command. This flag forces the code to overwrite the file. The following are example protocol deployment command examples:

• ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -f -c auth.json
• ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -f -c auth.json

For more details about the available arguments, use the following command on the command line:
ts-sdk put --help

Create a Pipeline by Using the Deployed Protocol

To use your new protocol on the TDP, create a new pipeline that uses the protocol that you deployed. Then, upload a file that matches the pipeline’s trigger conditions.