Deploy Self-Service Pipelines and Make Updates
Before you begin to deploy self-service pipelines:
- Read the Self-Service Pipeline Overview documentation.
- You should have created and tested the protocol and task scripts. For documentation details, click Create and Test Protocol and Task Scripts.
These instructions explain how to deploy:
- Artifacts to the Tetra Data Platform (TDP)
- A new version of your scripts
Deploy Artifacts to the TDP
To deploy the artifacts to the TDP:
This may take a few minutes to complete.
- Ensure that you run all of the following commands:
# note: you should be on the root of your task-script folder # 1. Lock/freeze your dependencies # If you do not need extra python packages, just create an empty requirements.txt pipenv lock -r > ./requirements.txt # 2. Deploy task script ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <version> <folder> -c <cfg.json path> # e.g. ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -c cfg.json # 3. Deploy protocol ts-sdk put protocol <namespace> <protocol script slug> <version> <folder> -c <cfg.json path> # e.g. ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -c cfg.json
Redeploying the same version of your code
Sometimes you may need to redeploy the same version of your code. For example, you may need to fix a bug or you may want to build your code incrementally.
To redeploy the same version of your code, you must include the
-fflag. This flag forces the code to overwrite the file. For example:
ts-sdk put protocol private-xyz hello-world v1.0.0 ./protocol -f -c cfg.json
ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script -f -c cfg.json
For more details about the available arguments, use:
ts-sdk put --help
To reduce the size of the Docker image of the task script and ensure a faster initial startup, TetraScience offers the
--exclude-foldersargument that ignores these paths while building the image:
**/.git/** **/.venv/** **/example-input/** **/example-output/** **/_tests_/** **/__tests__/** **/__test__/** **/example-files/** **/raw_data/**
Installation of third-party packages
Sometimes you may need to install third-party packages, for example, OpenJDK. To install a third-party package, you must include a special file
./init/before_installin your task script root folder.
task-script/ ├── init/ └── before_install ├── config.json ├── main.py ├── Pipfile ├── Pipfile.lock └── requirements.txt
The content of this before_install file should include command line statements that use your specific package management system (here apt-get):
apt-get update && apt-get install -y openjdk-11-jdk
This code execute when the task script image builds.
Additionally, remember that all executable attributes of task script files are not preserved. To be able to execute them from your code, you must run
chmod a+x ./path-to-your-exec-filein your before_install script.
How do I find the namespace?
If you do not know what the
namespaceshould be, review the namespace section in the self-service pipeline document.
- Wait until the task script shows that the build has completed.
- Verify that you can see the newly created artifacts in the View Artifacts page on the platform.
- Create a new pipeline using the new protocol, triggered by the API upload and some custom metadata.
- Upload a file that matches the trigger.
- Verify that the pipeline runs.
Deploy a New Version
To build your task scripts, TetraScience runs an AWS Codebuild project with build.general1.small compute type (details).
- Memory: 3 GB
- vCPUs: 2
- Disk space: 64 GB
- Make the necessary changes to your task script.
- Increase the task script version to a newer version (for example, v1.1.0) in the
- Deploy the scripts using the new version (see below in ts-sdk commands).
# 1. Deploy task script ts-sdk put task-script <namespace> <steps.script.slug in protocol.json> <new version> <folder> -c cfg.json # e.g. ts-sdk put task-script private-xyz hello-world v1.1.0 ./task-script -c cfg.json # 2. Deploy protocol ts-sdk put protocol <namespace> <protocol script slug> <new version> <folder> -c cfg.json # e.g. ts-sdk put protocol private-xyz hello-world v1.1.0 ./protocol -c cfg.json
- In the TDP, navigate to the Pipelines page and update your pipeline to include the new version. The pipeline version is the same as your protocol version.
- Trigger the new pipeline and verify that the change was applied.
- You can select to overwrite the existing artifact using the
ts-sdk put ...command. However, please use this command cautiously.
- If you need to bypass an invalid SSL certificate, you can use the
ts-sdk put ...command.
Updated 9 days ago