Deploy Self-Service Pipelines and Make Updates

📘

Prerequisites

Before you begin to deploy self-service pipelines:

• Read the Self-Service Pipeline Overview documentation.
• You should have created and tested the protocol and task scripts. For documentation details, click Create and Custom Task Scripts.

These instructions explain how to deploy:

• Artifacts to the Tetra Data Platform (TDP)
• A new version of your scripts

Deploy Artifacts to the TDP

To deploy the artifacts to the TDP:

📘

Deployment time

This may take a few minutes to complete.

1. Ensure that you run all of the following commands:

# note: you should be on the root of your task-script folder

# 1. Lock/freeze your dependencies
# If you do not need extra python packages, just create an empty requirements.txt
pipenv requirements > ./requirements.txt

# 2. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -c cfg.json

# 3. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -c cfg.json

📘

Redeploying the same version of your code

Sometimes you may need to redeploy the same version of your code. For example, you may need to fix a bug or you may want to build your code incrementally.
To redeploy the same version of your code, you must include the -f flag. This flag forces the code to overwrite the file. For example:

• ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -f -c cfg.json
• ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -f -c cfg.json

For more details about the available arguments, use: ts-sdk put --help

📘

Note

To reduce the size of the Docker image of the task script and ensure a faster initial startup, TetraScience offers the --exclude-folders argument that ignores these paths while building the image:

**/.git/**
**/.venv/**
**/example-input/**
**/example-output/**
**/_tests_/**
**/__tests__/**
**/__test__/**
**/example-files/**
**/raw_data/**

📘

Installation of third-party packages

Sometimes you may need to install third-party packages, for example, OpenJDK. To install a third-party package, you must include a special file ./init/before_install in your task script root folder.

task-script/
├── init/
     └── before_install
├── config.json
├── main.py
├── Pipfile
├── Pipfile.lock
└── requirements.txt

The content of this before_install file should include command line statements that use your specific package management system (here apt-get):

apt-get update && apt-get install -y openjdk-11-jdk

This code execute when the task script image builds.

Additionally, remember that all executable attributes of task script files are not preserved. To be able to execute them from your code, you must run chmod a+x ./path-to-your-exec-file in your before_install script.

📘

How do I find the namespace?

If you do not know what the namespace should be, review the namespace section in the self-service pipeline document.

2. Wait until the task script shows that the build has completed.
3. Verify that you can see the newly created artifacts in the View Artifacts page on the platform.
4. Create a new pipeline using the new protocol, triggered by the API upload and some custom metadata.
5. Upload a file that matches the trigger.
6. Verify that the pipeline runs.

Deploy a New Version

🚧

Pre-Deployment Notes

To build your task scripts, TetraScience runs an AWS Codebuild project with build.general1.small compute type (details).

• Memory: 3 GB
• vCPUs: 2
• Disk space: 64 GB

1. Make the necessary changes to your task script.
2. Increase the task script version to a newer version (for example, v1.1.0) in the protocol/protocol.json.
3. Deploy the scripts using the new version (see below in ts-sdk commands).

# 1. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <new version> <folder> -c cfg.json
# e.g. ts-sdk put task-script private-xyz hello-world v1.1.0 ./task-script -c cfg.json

# 2. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <new version> <folder> -c cfg.json
# e.g. ts-sdk put protocol private-xyz hello-world v1.1.0 ./protocol -c cfg.json

4. In the TDP, navigate to the Pipelines page and update your pipeline to include the new version. The pipeline version is the same as your protocol version.

5. Trigger the new pipeline and verify that the change was applied.

📘

Notes

• You can select to overwrite the existing artifact using the --force or -f flag for ts-sdk put ... command. However, please use this command cautiously.
  • If you need to bypass an invalid SSL certificate, you can use the --ignore-ssl or -i flag for ts-sdk put ... command.