Context API

To interact with Tetra Data Platform task scripts you will need to use the context object. The context object provides functions for you to read files, write files, update metadata, get pipeline configuration information, and more. This topic contains:

  • An example of a task script entry-point function
  • List of the context API properties
  • List of context API functions
  • More detailed information about context API functions

Task Script Entry-Point Function Example

If your task script entry-point function process_file is in file main.py, you should define process_file like this:

def process_file(input: dict, context: object):
    print("Starting task")
    
    input_data = context.read_file(input["inputFile"])
    # your business logic ...
    
    print("Task completed")

You can define input in the protocol's script.js. But you don't need to define context. It will be passed in as the second argument automatically.

Context API Properties

Context API provides properties that allow you to read and write files and more.

ParameterTypeDescription
context.org_slugstringYour organization's slug.
context.pipeline_iduuidPipeline unique ID.
context.workflow_iduuidWorkflow unique ID.
context.master_script_namespacestringNamespace for the protocol script.
context.master_script_slugstringSlug for the protocol script (script.js).
context.master_script_versionstringVersion of the protocol script (e.g. v1.0.0).
context.protocol_slugstringProtocol slug.
context.protocol_versionstringProtocol version.
context.pipeline_configdictA dictionary from strings to strings, containing the configuration parameters set when configuring the pipeline that invoked this protocol.
context.input_filedictFile pointer that triggered the current workflow.
context.created_atstringWorkflow's creation time.

ISO timestamp string. Generated by moment().toISOString(). Indicates when the file is created.
context.task_iduuidID for the current task (step of a pipeline).
context.task_created_atstringTask's creation time.

ISO timestamp string. Generated by moment().toISOString(). Indicates whent the task is created.
context.platform_urlstringThe URL of the platform (i.e.: https://platform.tetrascience.com).
context.platform_versionstringThe version of the platform being used.
context.tmp_dirstringFolder for tmp dir. Usually it’s /tmp. If your task script wants to write a file and then call a shell command (like LibreOffice) to process the file, you should put it in context.tmp_dir . Other folders are not guaranteed to be writeable.

Context API Functions

Context API provides functions that allow you to retrieve data, write data, and modify/check data on TDP.

Retrieving Data from Data Lake

FunctionDescription
context.read_file()Takes an input file reference, and returns either the file content, a file object, or the local path of a copy of the file.
context.get_labels()Get all of the labels associated with a file.
context.get_file_name()Retrieve filename of a file that is not downloaded locally.
context.get_ids()Get the Intermediate Data Schema that matches the namespace, slug, and version.
context.search_eql()Searches files via Elasticsearch query language.
context.get_logger()Returns the logger object, so you can use log(data) to log an object.
context.get_presigned_url()Returns a time-limited HTTPS URL that can be used to access a file.
context.get_secret_config_value()Retrieves the secret's value.
context.resolve_secret()Returns the secret value. This function is used to convert the SSM reference to the actual secret value.

Writing Data to Data Lake

FunctionDescription
context.write_file()Writes an output file to the data lake.
context.write_ids()Writes an output IDS file to the data lake.

Modifying/Checking Data in Data Lake

FunctionDescription
context.add_labels()Allows you to add labels to a file.
context.delete_labels()Delete one or more labels from a file.
context.update_metadata_tags()Updates file's metadata and tags.
context.add_attributes()Add metadata, tags, or labels to an object.
context.validate_ids()Checks the validity of IDS content provided in data.
context.run_command()Invokes remote command/action on target (agent or connector) and returns its response.

Context API Functions - Detailed Information

context.read_file

Takes an input file reference, and returns a dictionary with keys. By using the form parameter, can control access to the file content, a file object, or the local path of a copy of the file.

Parameter(s)

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
formstring (optional)If form=body (the default), then result['body'] holds the contents of the file as a byte array. This approach cannot handle large files that don't fit in memory.

If form=file_obj, then result['file_obj'] is a file-like object that can be used to access the body in a streaming manner. This object can be passed to Python libraries such as Pandas.

If form=download, then result['download'] is the file name of a local file that has been downloaded from the specified data lake file. This is useful when the data needs to be processed by native code (e.g. SQLite) or an external utility program.

Return(s)

TypeDescription
dictbody, file_obj or download - exactly one of these keys is present, depending on the value of the form parameter.

resulta": {
is a byte array that holds the contents of the file. This approach cannot handle large files that don't fit in memory.

result-1": "body, is a file-like object that can be used to access the body in a streaming manner. This object can be passed to Python libraries such as Pandas.

result-2": "",
is a string that is the file name of a local file that has been downloaded from the specified data lake file. This is useful when the data needs to be processed by native code (e.g. SQLite) or an external utility program.

result-2": "Description"
is a dict that is the custom metadata of the document

results": 2,
"rows" is a list that is a list of custom tags of the document

Examples

body

f = context.read_file(input_file_pointer, form='body')
json.loads(f['body'])
file_ref = {
    "fileId": "8e58867d-f281-4251-81ec-baeb3fdbb2f5"
}

data = context.read_file(file_ref)["body"]

file_obj

f = context.read_file(input_file_pointer, form='file_obj')
with zipfile.ZipFile(f['file_obj']) as zf:
  with zf.open("somefile.csv", "r") as csvf:
    pd.read_csv(csvf)

download

f = context.read_file(input_file_pointer, form='download')
con = sqlite3.connect(f['download'])
df = pd.read_sql_query('SELECT * FROM foo', con)

context.write_file

Writes an output file to the data lake.

Parameters

The following are the parameters for this endpoint.

ParameterTypeDescription
contentstring, bytes object, byte stream, dictThe content to be written out.

The content can only be Dict type when file_category is set to ids, otherwise it will error. In this case, IDS validation will be automatically performed.
file_namestringThe name of the file to be written. The full S3 path is determined by the platform. As of ts-sdk v1.3.2, the name cannot navigate upward in the file path (for example, ../file_name).
file_categorystringFile category, can be IDS, TMP or PROCESSED
idsstring (required if file_category is "IDS", otherwise optional)If file_category is "IDS", then this parameter specifies the specific Intermediate Data Schema. The format is: namespace/slug:v1.2.3.
custom_metadatadict of string to string (optional)Metadata to be appended to a file.

Keys can only include letters, numbers, spaces and symbols +, - or _.

Values can only include letters, numbers, spaces and symbols +, -, _, /, . or ,.

Including metadata with a key that already existing within the file's current metadata keys will result in a ValueError.
custom_tagslist of strings (optional)Tags to be appended to a file.

Can only include letters, numbers, spaces, and the symbols +, -, ., /, or _).

Limited to 128 characters.

Including any tags that already existing within the file's current tags will result in a ValueError.
source_typestring (optional)Can be used to overwrite the source type S3 metadata value of the resulting document. Validation is performed if a value is passed against the regex /^",
"2+$/, if no value is passed, we fall back to the default logic (take source type value from the RAW file, or if not found on RAW file, set the value unknown).
labelslist of dicts (optional)List of dictionaries where each dictionary is in form { 'name': <LABEL_NAME>, 'value': <LABEL_VALUE> }
gzip_compress_levelint (optional)1 is fastest and 9 is slowest. 0 is no compression. The default is 5 (introduced in ts-sdk v2.0.0)

Returns

TypeDescription
dictReturns an object that indicates the file location.

Example:

{
"type": "s3file",
"bucket": "datalake",
"fileKey": "path/to/file",
"fileId": "uuidForFile",
"version": "versionId"
}

Note: version will not be available when you are running pipelines locally.

Examples

content: string, bytes object, byte stream or string

string

context.write_file(
    content='this is a string',
    file_name=file_name,
    file_category=file_category
)

bytes object

context.write_file(
    content=bytes('byte content', 'utf-8'),
    file_name=file_name,
    file_category=file_category
)
# or
context.write_file(
    content=b'byte content',
    file_name=file_name,
    file_category=file_category
)

byte stream / file object

📘

NOTE:

If a gzipped file is larger than 100MB, the file will be uploaded as multi-part upload.

For ts-sdk versions <= v1.2.31, the file size limit is 5GB. Starting with ts-sdk v1.2.32, the file size limit has been increased to 5TB.

with zf.open("somefile.csv", "rb") as f:
    context.write_file(
        content=f,
        file_name=file_name,
        file_category=file_category
    )

context.write_ids

Writes an output IDS file to the data lake.

🚧

WARNING:

Notice you can only define file_suffix, you cannot define the complete file name of the output IDS file. The file name is codified and follows a certain pattern, which includes ids_namespace and ids_slug. This means if you switch to a different IDS, the new IDS JSON produced will be a separate file not a new version of the old file.

Parameters

This function contains the following parameters.

ParameterTypeDescription
content_objdictDictionary that can be JSON-serialized.
file_suffixstringString that will be appended to the IDS file name generated from the provided information (IDS type and version). Useful for keeping names unique when a task script generates multiple IDS files of the same type. Document will have a systematic file name generated: ${idstype}${idsversion}${file_suffix}.
idsstringIf the file_category is "IDS", then this parameter specifies the specific Intermediate Data Schema. The format is: namespace/slug:v1.2.3.
custom_metadatadict of string to string (optional)Metadata to be appended to a file.

Keys can only include letters, numbers, spaces and symbols +, - or _.

Values can only include letters, numbers, spaces and symbols +, -, _, /, . or ,.
custom_tagslist of strings (optional)Tags to be appended to a file.

Can only include letters, numbers, spaces, and the symbols +, -, ., /, or _).

Limited to 128 characters.
source_typestring (optional)Used to overwrite the source type S3 metadata value of the resulting document. Validation is performed if a value is passed against the regex /^[-a-z0-9]+$/, if no value is passed, we fall back to the default logic (take source type value from the RAW file, or if not found on RAW file, set the value unknown).
file_categorystring (optional)Defaults to to IDS. Overwrites file category. Allowed values are IDS and TMP. If value is not provided or if other values is provided, it will be defaulted to IDS.
labelslist of dicts (optional)List of dictionaries where each dictionary is in form { 'name': <LABEL_NAME>, 'value': <LABEL_VALUE> }
gzip_compress_levelint (optional)1 is fastest and 9 is slowest. 0 is no compression. The default is 5 (introduced in ts-sdk v2.0.0)

Returns

TypeDescription
dictReturns an object that indicates the file location.

Example:

{
"type": "s3file",
"bucket": "datalake",
"fileKey": "path/to/file",
"fileId": "uuidForFile",
"version": "versionId"
}

Note: version will not be available when you are running pipelines locally.

context.get_file_name

Use this function to retrieve the filename of a file that is not downloaded locally.

Parameters

The following table shows the parameters for this function.

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.

Returns

TypeDescription
stringThe filename of the file. Note that you do not need to download the file locally to access the file name.

Example(s)

def function_in_main(input: dict, context):
    input_file = input["input_file_pointer"]  # passed by protocol script.js
	name = context.get_file_name(input_file)

context.get_ids

Use this function to get the Intermediate Data Schema that matches the namespace, slug, and version.

Parameters

The parameters for this function appear in the following table.

ParameterTypeDescription
namespacestringA namespace defines a realm; only those with the appropriate permissions can use the artifacts in that realm. There are three main categories of namespace:
• common
• client
• private
slugstringA slug is a unique identifier. The term is also used to denote a reference to a unique identifier (pointer).
versionstringVersion of the IDS.

Returns

typeDescription
dictReturns an IDS schema object.

context.validate_ids

Checks the validity of IDS content provided in data. Throws an error if not valid.

Parameters

ParameterTypeDescription
datadictThe JSON content of the IDS file.
namespacestringA namespace defines a realm; only those with the appropriate permissions can use the artifacts in that realm. There are three main categories of namespace:
• common
• client
• private
slugstringA slug is a unique identifier. The term is also used to denote a reference to a unique identifier (pointer).
versionstringVersion of the IDS.

Returns

TypeDescription
booleanReturns a boolean value indicating whether the IDS is valid (true). It throws an error if the IDS is not valid.

Example(s)

is_ids = validate_ids(ids_data_dict,
             		  namespace = "common",
             		  slug = "cell-counter",
             		  version = "v2.0.0")

context.get_logger

Returns the logger object, which currently has one method: log(data).

Parameters

There are no parameters for this function.

Returns

TypeDescription
objectThis returns the structured logger object, which currently has one method: log(input).

Example(s)

log an object

logger = context.get_logger()
logger.log({
    "message": "Starting the main parser",
    "level": "info"
})
# log output
# {"timestamp": "2020-04-22T20:45:02.540947", "message": "Starting the main parser", "level": "info"}
log a string, the default level is “info”


logger = context.get_logger()
logger.log("Writing IDS into the datalake")
# log output
# {"timestamp": "2020-04-22T20:45:02.971038", "message": "Writing IDS into the datalake", "level": "info"}

context.get_secret_config_value

📘

NOTE:

Instead of this function, we recommend as best practice to pass pipeline configs from your protocol's master-script script.js to your task script function and then using context.resolve_secret to resolve the secret. This makes your task script function less dependent on TS context function (more like a pure function). Please see the documentation for that function.

Retrieves the secret value from a configuration element for use in your script.

Parameters

ParameterTypeDescription
secret_namestringSecret slug
silent_on_errorbooleanOptional. Default: True. If set to True and the secret is missing, this function will return an empty string (otherwise an error will be thrown)

Returns

TypeDescription
strThe value of the secret.

context.get_presigned_url

Returns a time-limited HTTPS URL that can be used to access the file. If URL generation fails for any reason (except invalid value for ttl_sec parameter). None will be returned.

Parameters

Parameters for this function appear in the following table.

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
ttl_secnumber (optional)How long the URL will be valid before it expires. It is recommended that task scripts adjust this to be in line with the command TTL. Optional, defaults to 300. Must be between 0 and 900, otherwise an error is thrown.

Returns

TypeDescription
strA time-limited HTTPS URL that can be used to access the file.

context.run_command

Invokes remote command/action on target (agent or connector) and returns its response. This function requires Tetra Hub and Agent support. It's not an out-of-shelf function that you can use on any instrument. Please contact the TetraScience team for more information.

📘

NOTE

The context.run_command() function is on a deprecation path to be replaced with the context.run_cmd function. This change removes the need to supply an org_slug value in the function.

Parameters

ParameterTypeDescription
org_slugstringorganization slug
target_idstringUnique Identifier; identifies the connector or agent that will receive the command. It is recommended that task scripts receive this as a pipeline config parameter
actionstring, enumSee supported actions below.
payloaddictJSON object specific to the selected action
metadatadictA dict that describes command. Method will automatically add values for workflowId, pipelineId and taskId to metadata that will be sent to command service for execution.
ttl_secnumber
(optional, default: 300)
It is recommended that task scripts receive this as a pipeline config parameter. Range: 300 seconds (5min) to 900 seconds (15min)

Actions

TetraScience.Connector.gdc.HttpRequest
Send REST call to any network address as long as that address is reachable to GDC

  • Requires TDP v3.0.0 and above
  • Requires DataHub and Generic Data Connector (GDC)
  • No agent needed

Returns

TypeDescription
dictReturns a JSON response object returned by target (agent or connector) that ran the command.

Raises

Throws an error when the following are missing: org_slug, target_id, action, payload, and ttl_sec.

Throws an error when the ttl_sec is less than 300 seconds or higher than 900 seconds.

Error messages include the following:

  • command not created successfully
  • TTL has expired
  • command didn’t execute successfully

For SDK version 1.2.22 and above: If the command does not execute successfully, the exception and exception message contains the reason why the command failed. Previously, it was "command status ERROR". Now the error message is "error response from target system."

context.update_metadata_tags

Updates file's metadata and tags.

Parameters

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
custom_metadictUpdates the metadata. Passing "None" for this parameter removes the metadata entry.
custom_tagslistList of new tags where each tag must be a string.

Returns

TypeDescription
dictReturns an object that describes the file location.

Example Return object

{
  type: 's3file',
  bucket: 'bucket',
  fileKey: 'fileKey',
  fileId: 'fileId',
  // no version locally (from fake s3)
  version: 'versionId'
}

Raises

ts-sdk versions <= v1.2.31
Raises an error if the file size is over the limit of 5GB. Additionally, the function will raise an exception if no custom_meta or custom_tags are provided.

ts-sdk versions v1.2.32 onward
Raises an error if the file size is over the limit of 5TB. Additionally, the function will return the unmodified file instead of raising an exception.

context.resolve_secret

Returns the secret value. This function is used to convert the AWS Systems Manager Parameter Store (SSM) reference to the actual secret value.

Parameters

ParameterTypeDescription
secret_refdictIt contains SSM parameter store path to the secret value.

Example

If you defined a secret in protocol.json as:

{
...
  "config": [{
    "slug": "password",
    "name": "Password",
    "description": "This is a password",
    "type": "secret",
    "required": true
  }],
...
}

Then in the script.js, you will get password reference from the pipeline config:

...
const pipelineConfig = workflow.getContext('pipelineConfig');
const { password } = pipelineConfig;
await workflow.runTask('task-1', { password });
...

Then in your task script, you can get the password reference from input["password"]. The reference is a dictionary that contains SSM parameter store path to the secret value, like this:

{
  "secret": true,
  "ssm": "/development/tetrascience-demo/org-secrets/password",
  "version": 1
}
def main(input, context):
  password = context.resolve_secret(input.get('password'))
  ...

Returns

Returns the secret value. If you pass in a parameter that is not a secret, it will simply return the parameter.

context.add_labels

Allows you to add labels to a file.

If the file already has a label with the same key name, then a new label with the same key name will be created. The previous label with the same name will not be overwritten.

Parameters

Parameters appear in the table below.

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
labelslist of dictList of dictionaries where each dictionary is in form { 'name': <LABEL_NAME>, 'value': <LABEL_VALUE> }

Returns

TypeDescription
list of dictsReturns an list of added labels, where each label item shows that label's id and creation time.

e.g. ta": {
"h-0": "Type",
"h-1": "Description",
"0-0": "list of dicts",
"0-1": "Ret`

context.get_labels

Use this function to get all of the labels associated with a file.

Parameters

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.

Returns

TypeDescription
list of dictsReturns an list dicts related to the given file, where each dict contains information about a label.

e.g. ta": {
"h-0": "Type",
"h-1": "Description",
"0-1": "Returns an list dicts related t`

context.delete_labels

Use this function to delete one or more labels from a file. Labels are discussed in detail in the Basic Concepts: Metadata, Tags, and Labels topic.

Parameters

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
label_idsList(str)IDs of the labels you want to delete.

Returns

TypeDescription
objectAn empty object if the label was successfully deleted from the file.

context.add_attributes

This function has been added with TDP v3.0.0 and ts-sdk v1.2.20. This function allows you to add metadata, tags, or labels to an object.

Parameters

ParameterTypeDescription
filedictFile Pointer with the following keys:
- bucket
- fileKey
- version

Note: For ts-sdk v1.2.30 and higher, you can use a single fileId key. This key is internally resolved to the full file pointer. However, you must use the fileId for the latest file version. The fileId for a previous file version will not work.
custom_metadictOptional. List of metadata values and keys. Passing in a value of "None" will remove all metadata from the file.
custom_tagslistOptional. List of tags.
labelslistOptional. List of dictionaries where each dictionary is in form
{ 'name': <LABEL_NAME>, 'value': <LABEL_VALUE> }

Returns

TypeDescription
dictReturns a dict that describes the file location. A new file version will be generated if metadata/tags were provided.

context.search_eql

This function has been added with TDP v3.1.0 and ts-sdk v1.2.24. It helps to search files via Elasticsearch query language. For more details, check search by Elasticsearch Query Language

Parameters

ParameterTypeDescription
payloaddictElasticsearch Query Language (EQL). You can read more about the query format that Elasticsearch uses on their website
returns
(added in ts-sdk v1.2.30)
str- raw (default): raw response from ES
- filePointers: returns list of file pointers that can be used in context.read_file

Returns

varies

if returns=raw then dict
if returns=filePointers then list
Result of the ES query (raw or file pointers).