Deploy Self-Service Pipelines and Make Updates

📘

PREREQUISITES:

These instructions explain how deploy the artifacts to TDP. It also lists the steps for deploying a new version of your scripts.

Deploy Artifacts to TDP

To deploy the artifacts to TDP, complete the following steps. This will take a couple of minutes.

  1. Make sure you run all the commands below:
# note: you should be on the root of your task-script folder

# 1. Lock/freeze your dependencies
pipenv lock -r > ./requirements.txt

# 2. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -c cfg.json

# 3. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <version> <folder> -c <cfg.json path>
# e.g. ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -c cfg.json

📘

NOTE:

Sometimes you need to redeploy the same version of your code -- for example, you might need to fix a bug or you might build your code incrementally. If you need to redeploy the same version of your code, include the -f flag. This flag forces the file to be overwritten. Here are examples:

  • ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -f -c cfg.json
  • ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -f -c cfg.json

To get more information about available arguments use: ts-sdk put --help

📘

NOTE:

To reduce the size of the Docker image of the task script (faster initial startup), we ignore some paths while building the image:

**/.venv/**
**/example-input/**
**/example-output/**
**/_tests_/**
**/__tests__/**
**/__test__/**
**/example-files/**
**/raw_data/**

If this is not enough, you can extend the list by creating a .ignore file in the root of the task script folder.

Example:

folder1/**
folder2/subfolder2/*.raw
*.md

📘

NOTE:

Sometimes you may need to install 3rd party packages (OpenJDK for example). You have to use a special file at the next location ./init/before_install (relatively to your task script root folder). The content of this file should be:

apt-get update && apt-get install -y openjdk-11-jdk

This code will be executed during the task script image build.

Also, keep in mind that all executable attributes of task script files are not preserved so you will need to run chmod a+x ./path-to-your-exec-file in your before_install script in order to be able to execute them from your code.

📘

NOTE:

If you don't know what the namespace should be, check out namespace section in the self-service pipeline document.

  1. Wait until the task script shows that the build has completed.
  2. Verify that you are able to see the newly created artifacts in the “View Artifacts” page on the platform.
  3. Create a new pipeline using the new protocol, triggered by API upload and some custom metadata.
  4. Upload a file that matches the trigger.
  5. Verify that the pipeline runs.

Deploy a new version

  1. Make necessary changes to your task script.
  2. Bump the task script version to a newer version (e.g. v1.1.0) in protocol/protocol.json.
  3. Deploy the scripts.
# 1. Deploy task script
ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <new version> <folder> -c cfg.json
# e.g. ts-sdk put task-script private-xyz hello-world v1.1.0 ./task-script -c cfg.json

# 2. Deploy protocol
ts-sdk put protocol <namespace> <protocol script slug> <new version> <folder> -c cfg.json
# e.g. ts-sdk put protocol private-xyz hello-world v1.1.0 ./protocol -c cfg.json
  1. In the TDP, go to "Pipelines" page. Update your pipeline to pick up the new version. The pipeline version is the same as your protocol version.
Update Pipeline VersionUpdate Pipeline Version

Update Pipeline Version

  1. Trigger the new pipeline and verify the change is applied.

Notes:

  • You can also choose to overwrite the existing artifact (must be used cautiously) using --force or -f flag for ts-sdk put ... command.
  • When you need to overcome an invalid SSL certificate you can use --ignore-ssl or -i flag for ts-sdk put ... command.

🚧

PRE-DEPLOYMENT NOTES:

To build your task scripts, we run an AWS Codebuild project with build.general1.small compute type (details).

Memory: 3 GB, vCPUs: 2, Disk space: 64 GB.


Did this page help you?