To interact with Tetra Data Platform task scripts you will need to use the context object. The context object provides functions for you to read files, write files, update metadata, get pipeline configuration information, and more. This topic contains:

• An example of a task script entry-point function
• List of the context API properties
• List of context API functions
• More detailed information about context API functions

Task Script Entry-Point Function Example

If your task script entry-point function process_file is in file main.py, you should define process_file like this:

def process_file(input: dict, context: object):
    print("Starting task")
    
    input_data = context.read_file(input["inputFile"])
    # your business logic ...
    
    print("Task completed")

You can define input in the protocol's script.js. But you don't need to define context. It will be passed in as the second argument automatically.

Context API Properties

Context API provides properties that allow you to read and write files and more.

Context API Functions

Context API provides functions that allow you to retrieve data, write data, and modify/check data on TDP.

Retrieving Data from Data Lake

Writing Data to Data Lake

Modifying/Checking Data in Data Lake

Context API Functions - Detailed Information

context.read_file

Takes an input file reference, and returns a dictionary with keys. By using the form parameter, can control access to the file content, a file object, or the local path of a copy of the file.

Parameter(s)

Return(s)

Examples

body

f = context.read_file(input_file_pointer, form='body')
json.loads(f['body'])

file_ref = {
    "fileId": "8e58867d-f281-4251-81ec-baeb3fdbb2f5"
}

data = context.read_file(file_ref)["body"]

file_obj

f = context.read_file(input_file_pointer, form='file_obj')
with zipfile.ZipFile(f['file_obj']) as zf:
  with zf.open("somefile.csv", "r") as csvf:
    pd.read_csv(csvf)

download

f = context.read_file(input_file_pointer, form='download')
con = sqlite3.connect(f['download'])
df = pd.read_sql_query('SELECT * FROM foo', con)

context.write_file

Writes an output file to the data lake.

Parameters

The following are the parameters for this endpoint.

Returns

Examples

content: string, bytes object, byte stream or string

string

context.write_file(
    content='this is a string',
    file_name=file_name,
    file_category=file_category
)

bytes object

context.write_file(
    content=bytes('byte content', 'utf-8'),
    file_name=file_name,
    file_category=file_category
)
# or
context.write_file(
    content=b'byte content',
    file_name=file_name,
    file_category=file_category
)

byte stream / file object

📘

NOTE:

If a gzipped file is larger than 100MB, the file will be uploaded as multi-part upload.

For ts-sdk versions <= v1.2.31, the file size limit is 5GB. Starting with ts-sdk v1.2.32, the file size limit has been increased to 5TB.

with zf.open("somefile.csv", "rb") as f:
    context.write_file(
        content=f,
        file_name=file_name,
        file_category=file_category
    )

context.write_ids

Writes an output IDS file to the data lake.

🚧

WARNING:

Notice you can only define file_suffix, you cannot define the complete file name of the output IDS file. The file name is codified and follows a certain pattern, which includes ids_namespace and ids_slug. This means if you switch to a different IDS, the new IDS JSON produced will be a separate file not a new version of the old file.

Parameters

This function contains the following parameters.

Returns

context.get_file_name

Use this function to retrieve the filename of a file that is not downloaded locally.

Parameters

The following table shows the parameters for this function.

Returns

Example(s)

def function_in_main(input: dict, context):
    input_file = input["input_file_pointer"]  # passed by protocol script.js
	name = context.get_file_name(input_file)

context.get_ids

Use this function to get the Intermediate Data Schema that matches the namespace, slug, and version.

Parameters

The parameters for this function appear in the following table.

Returns

context.validate_ids

Checks the validity of IDS content provided in data. Throws an error if not valid.

Parameters

Returns

Example(s)

is_ids = validate_ids(ids_data_dict,
             		  namespace = "common",
             		  slug = "cell-counter",
             		  version = "v2.0.0")

context.get_logger

Returns the logger object, which currently has one method: log(data).

Parameters

There are no parameters for this function.

Returns

Example(s)

log an object

logger = context.get_logger()
logger.log({
    "message": "Starting the main parser",
    "level": "info"
})
# log output
# {"timestamp": "2020-04-22T20:45:02.540947", "message": "Starting the main parser", "level": "info"}
log a string, the default level is “info”


logger = context.get_logger()
logger.log("Writing IDS into the datalake")
# log output
# {"timestamp": "2020-04-22T20:45:02.971038", "message": "Writing IDS into the datalake", "level": "info"}

context.get_secret_config_value

📘

NOTE:

Instead of this function, we recommend as best practice to pass pipeline configs from your protocol's master-script script.js to your task script function and then using context.resolve_secret to resolve the secret. This makes your task script function less dependent on TS context function (more like a pure function). Please see the documentation for that function.

Retrieves the secret value from a configuration element for use in your script.

Parameters

Returns

context.get_presigned_url

Returns a time-limited HTTPS URL that can be used to access the file. If URL generation fails for any reason (except invalid value for ttl_sec parameter). None will be returned.

Parameters

Parameters for this function appear in the following table.

Returns

context.run_command

Invokes remote command/action on target (agent or connector) and returns its response.

📘

NOTE:

This function is still in the early stage and requires Tetra DataHub and Agent support. It's not an out-of-shelf function that you can use on any instrument. Please contact the TetraScience team for more information.

Parameters

Actions

TetraScience.Connector.gdc.HttpRequest
Send REST call to any network address as long as that address is reachable to GDC

• Requires TDP v3.0.0 and above
• Requires DataHub and Generic Data Connector (GDC)
• No agent needed

Returns

Raises

Throws an error when the following are missing: org_slug, target_id, action, payload, and ttl_sec.

Throws an error when the ttl_sec is less than 300 seconds or higher than 900 seconds.

Error messages include the following:

• command not created successfully
• TTL has expired
• command didn’t execute successfully

For SDK version 1.2.22 and above: If the command does not execute successfully, the exception and exception message contains the reason why the command failed. Previously, it was "command status ERROR". Now the error message is "error response from target system."

context.update_metadata_tags

Updates file's metadata and tags.

Parameters

Returns

Example Return object

{
  type: 's3file',
  bucket: 'bucket',
  fileKey: 'fileKey',
  fileId: 'fileId',
  // no version locally (from fake s3)
  version: 'versionId'
}

Raises

ts-sdk versions <= v1.2.31
Raises an error if the file size is over the limit of 5GB. Additionally, the function will raise an exception if no custom_meta or custom_tags are provided.

ts-sdk versions v1.2.32 onward
Raises an error if the file size is over the limit of 5TB. Additionally, the function will return the unmodified file instead of raising an exception.

context.resolve_secret

Returns the secret value. This function is used to convert the AWS Systems Manager Parameter Store (SSM) reference to the actual secret value.

Parameters

Example

If you defined a secret in protocol.json as:

{
...
  "config": [{
    "slug": "password",
    "name": "Password",
    "description": "This is a password",
    "type": "secret",
    "required": true
  }],
...
}

Then in the script.js, you will get password reference from the pipeline config:

...
const pipelineConfig = workflow.getContext('pipelineConfig');
const { password } = pipelineConfig;
await workflow.runTask('task-1', { password });
...

Then in your task script, you can get the password reference from input["password"]. The reference is a dictionary that contains SSM parameter store path to the secret value, like this:

{
  "secret": true,
  "ssm": "/development/tetrascience-demo/org-secrets/password",
  "version": 1
}

def main(input, context):
  password = context.resolve_secret(input.get('password'))
  ...

Returns

Returns the secret value. If you pass in a parameter that is not a secret, it will simply return the parameter.

context.add_labels

Allows you to add labels to a file.

If the file already has a label with the same key name, then a new label with the same key name will be created. The previous label with the same name will not be overwritten.

Parameters

Parameters appear in the table below.

Returns

context.get_labels

Use this function to get all of the labels associated with a file.

Parameters

Returns

context.delete_labels

Use this function to delete one or more labels from a file. Labels are discussed in detail in the Basic Concepts: Metadata, Tags, and Labels topic.

Parameters

Returns

context.add_attributes

This function has been added with TDP v3.0.0 and ts-sdk v1.2.20. This function allows you to add metadata, tags, or labels to an object.

Parameters

Returns

context.search_eql

This function has been added with TDP v3.1.0 and ts-sdk v1.2.24. It helps to search files via Elasticsearch query language. For more details, check search by Elasticsearch Query Language

Parameters

Returns