Deploy Self-Service Pipelines and Make Updates

πŸ“˜

Prerequisites

Before you begin to deploy self-service pipelines:

These instructions explain how to deploy:

  • Artifacts to the Tetra Data Platform (TDP)
  • A new version of your scripts

Deploy Artifacts to the TDP

To deploy the artifacts to the TDP:

πŸ“˜

Deployment time

This may take a few minutes to complete.

  1. Ensure that you run all of the following commands:
# note: you should be on the root of your task-script folder

# 1. Lock/freeze your dependencies
# If you do not need extra python packages, just create an empty requirements.txt
pipenv lock -r > ./requirements.txt

# 2. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -c cfg.json

# 3. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -c cfg.json

πŸ“˜

Redeploying the same version of your code

Sometimes you may need to redeploy the same version of your code. For example, you may need to fix a bug or you may want to build your code incrementally.
To redeploy the same version of your code, you must include the -f flag. This flag forces the code to overwrite the file. For example:

  • ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -f -c cfg.json
  • ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -f -c cfg.json

For more details about the available arguments, use: ts-sdk put --help

πŸ“˜

Note

To reduce the size of the Docker image of the task script and ensure a faster initial startup, TetraScience ignores these paths while building the image:

**/.venv/**
**/example-input/**
**/example-output/**
**/_tests_/**
**/__tests__/**
**/__test__/**
**/example-files/**
**/raw_data/**

If this is not enough, you can extend the list by creating a .ignore file in the root of the task script folder.

Example:

folder1/**
folder2/subfolder2/*.raw
*.md

πŸ“˜

Installation of third-party packages

Sometimes you may need to install third-party packages, for example, OpenJDK. To install a third-party package, you must include this special file ./init/before_install at the next location relative to your task script root folder. The content of this file should be:

apt-get update && apt-get install -y openjdk-11-jdk

This code execute when the task script image builds.

Additionally, remember that all executable attributes of task script files are not preserved. To be able to execute them from your code, you must run tchmod a+x ./path-to-your-exec-file in your before_install script.

πŸ“˜

How do I find the namespace?

If you do not know what the namespace should be, review the namespace section in the self-service pipeline document.

  1. Wait until the task script shows that the build has completed.
  2. Verify that you can see the newly created artifacts in the View Artifacts page on the platform.
  3. Create a new pipeline using the new protocol, triggered by the API upload and some custom metadata.
  4. Upload a file that matches the trigger.
  5. Verify that the pipeline runs.

Deploy a New Version

🚧

Pre-Deployment Notes

To build your task scripts, TetraScience runs an AWS Codebuild project with build.general1.small compute type (details).

  • Memory: 3 GB
  • vCPUs: 2
  • Disk space: 64 GB
  1. Make the necessary changes to your task script.
  2. Increase the task script version to a newer version (for example, v1.1.0) in the protocol/protocol.json.
  3. Deploy the scripts.
# 1. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <new version> <folder> -c cfg.json
# e.g. ts-sdk put task-script private-xyz hello-world v1.1.0 ./task-script -c cfg.json

# 2. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <new version> <folder> -c cfg.json
# e.g. ts-sdk put protocol private-xyz hello-world v1.1.0 ./protocol -c cfg.json
  1. In the TDP, navigate to the Pipelines page and update your pipeline to include the new version. The pipeline version is the same as your protocol version.
11291129

Update pipeline version

  1. Trigger the new pipeline and verify that the change was applied.

πŸ“˜

Notes

  • You can select to overwrite the existing artifact using the --force or -f flag for ts-sdk put ... command. However, please use this command cautiously.
    • If you need to bypass an invalid SSL certificate, you can use the --ignore-ssl or -i flag for ts-sdk put ... command.