API DocumentationProduct Documentation
Product Documentation

Setup and Deploy Instructions

Suggest Edits

These instructions explain how set up your environment, initialize the TetraScience Software Development Kit (TS SDK) and specify dependencies, create the scripts and test them, and deploy the artifacts to TDP. It also lists the steps for deploying a new version of your scripts.

Prerequisites

This topic should be read in tandem with the Self-Service Pipelines topic. If you haven't read it yet, you can still complete Step 1: Set Up the Environment. But performing the other steps will require some context.

Step 1: Set Up the Environment

To set up your environment, complete the following steps.

  1. Install Python 3.7. (Use pyenv if you have to work with multiple versions at the same machine.)
  2. Install pipenv: pip3 install pipenv.
  3. Install ts-sdk package: pip3 install ts-sdk.
  4. Check if the installation was successful by running ts-sdk --help.
  5. Log into the TDP as an admin user.
  6. Generate a service user with admin access on the “organization” page in your profile (or use an existing one).
  7. Generate a service token for the service user you have selected.
  8. Set environment variables in your shell:
# in your command prompt or `.bashrc` or `.bash_profile` file
export TS_API_URL=(TDP API endpoint base URL*)
export TS_AUTH_TOKEN=(service token you generated)
export TS_ORG=(your organization slug name)

# in your Windows commmand prompt
set TS_API_URL=(TDP API endpoint base URL*)
set TS_AUTH_TOKEN=(service token you generated)
set TS_ORG=(your organization slug name)

Step 2: Initialize the TS SDK and Specify Dependencies

To initialize the TS SDK and specify dependencies, do the following.

🚧

Auto-generated Scripts

ts-sdk init will create some template script files for you and they are fully functional (including test files). So you can deploy them to test self-service pipeline feature. However, they will be a little overwhelming if you haven't completed Self-Service Pipelines.

  1. Use ts-sdk init command to create a new folder, with a sample task script and master script based on a simple template.
ts-sdk init --org <your organization slug name> -m <master script slug> -t <task script slug> -f <folder name for the generated code>
# e.g. the following will generate folder `./decorate`,
# which contains a master script folder "ms" and task script folder "ts":
# $ ts-sdk init -m ms -t ts --org tetrascience-demo-1 -f ./decorate
  1. Install the dev dependencies.
# Go to the newly created tas-script folder
cd path/to/the/task-script-folder
pipenv install --dev

# (optional) install additional packages if your script uses any
pipenv install numpy pandas [...]

Step 3: Create Scripts and Test

To create the scripts and test, complete the following steps.

  1. Edit the following files in the newly created folder as needed - these files will be your scripts and tests:
  • task-script/main.py
  • task-script/__test__/test_business_logic.py
  • task-script/__test__/test_integration.py
  • task-script/__test__/data/input.txt
  • task-script/__test__/data/expected.json

As you edit, reference the Self-Service Pipelines sections Create a Master Script and Create a Task Script.

  1. In your terminal, run the following to test your code:
$ pipenv run python -m pytest

# or

$ pipenv shell
$ python -m pytest

Step 4: Deploy Artifacts to TDP

To deploy the artifacts to TDP, complete the following steps. This will take a couple of minutes.

  1. Make sure you run all the commands below:
# note: you should be on the root of your task-script folder

# 1. Lock/freeze your dependencies
pipenv lock -r > ./requirements.txt

# 2. Deploy master script
ts-sdk put master-script <namespace> <master script slug> <version> <folder>
# e.g. ts-sdk put master-script private-xyz hello-world v1.0.0 ./master-script

# 3. Deploy task script
ts-sdk put task-script <namespace> <task script slug> <version> <folder>
# e.g. ts-sdk put task-script private-xyz hello-world v1.0.0 ./task-script

📘

Namespace

If you don't know what should be the namespace, check out namespace section in the self-service pipeline document.

  1. Wait until the task script shows that the build has completed.
  2. Verify that you are able to see the newly created artifacts in the “View Master/Task Scripts” page on the platform.
  3. Create a new pipeline using the new protocol, triggered by API upload and some custom metadata.
  4. Upload a file that matches the trigger.
  5. Verify that the pipeline runs.

Step 5: (Optional Step) Deploy a new version

  1. Make necessary changes to your task script.
  2. Bump the task script version to a newer version (e.g. v1.1.0) in master-script/protocol.json.
  3. Deploy the scripts.
# 1. Deploy master script
ts-sdk put master-script <namespace> <master script slug> <new version> <folder>
# e.g. ts-sdk put master-script private-xyz hello-world v1.1.0 ./master-script

# 2. Deploy task script
ts-sdk put task-script <namespace> <task script slug> <new version> <folder>
# e.g. ts-sdk put task-script private-xyz hello-world v1.1.0 ./task-script
  1. In the TDP, go to "Pipelines" page. Update your pipeline to pick up the new version. The pipeline version is the same as your master script version.
1129

Update Pipeline Version

  1. Trigger the new pipeline and verify the change is applied.

Notes:

  • You can also choose to overwrite the existing artifact (must be used cautiously) using --force or -f flag for ts-sdk put ... command.
  • When you need to overcome an invalid SSL certificate you can use --ignore-ssl or -i flag for ts-sdk put ... command.

🚧

Pre-deployment Related Notes

To build your task scripts, we run an AWS Codebuild project with build.general1.small compute type (details).

Memory: 3 GB, vCPUs: 2, Disk space: 64 GB.

Updated about 1 year ago