Context API

To interact with Tetra Data Platform task scripts you will need to use the context object. The context object provides functions for you to read files, write files, get pipeline configuration information, and more. This topic contains:

• An example of a task script entry-point function
• A description of the context API properties and functions
• A description of Local Context API functions for unit tests.

Task Script Entry-Point Function Example

If your task script entry-point function process_file is in file main.py, you should define process_file like this:

def process_file(input: dict, context: object):
    print("Starting task")
    
    input_data = context.read_file(input["inputFile"])
    # your business logic ...
    
    print("Task completed")

You can define input in the protocol' script.js. But, you don't need to define context. It will be passed in as the second argument automatically.

Context API Properties and Functions

Context API provides properties and functions that allow you to read and write files and more.

Properties

context.read_file

Takes an input file reference, and returns a dictionary with keys. By using the form parameter, can control access to the file content, a file object, or the local path of a copy of the file.

Parameter(s)

Examples

body

f = context.read_file(input_file_pointer, form='body')
json.loads(f['body'])

file_ref = {
    "fileId": "8e58867d-f281-4251-81ec-baeb3fdbb2f5"
}

data = context.read_file(file_ref)["body"]

file_obj

f = context.read_file(input_file_pointer, form='file_obj')
with zipfile.ZipFile(f['file_obj']) as zf:
  with zf.open("somefile.csv", "r") as csvf:
    pd.read_csv(csvf)

download

f = context.read_file(input_file_pointer, form='download')
con = sqlite3.connect(f['download'])
df = pd.read_sql_query('SELECT * FROM foo', con)

Return(s)

Returns an dictionary, as listed below:

• body, file_obj or download - exactly one of these keys is present, depending on the value of the form parameter.
  • result['body'] holds the contents of the file as a byte array. This approach cannot handle large files that don't fit in memory.
  • result['file_obj'] is a file-like object that can be used to access the body in a streaming manner. This object can be passed to Python libraries such as Pandas.
    *result['download'] is the file name of a local file that has been downloaded from the specified data lake file. This is useful when the data needs to be processed by native code (e.g. SQLite) or an external utility program.
• custom_metadata: custom metadata of the document
• custom_tags: list of custom tags of the document

context.write_file

Writes an output file to the data lake.

Parameters

The following are the parameters for this endpoint.

Examples

content: string, bytes object, byte stream or string

string

context.write_file(
    content='this is a string',
    file_name=file_name,
    file_category=file_category
)

bytes object

context.write_file(
    content=bytes('byte content', 'utf-8'),
    file_name=file_name,
    file_category=file_category
)
# or
context.write_file(
    content=b'byte content',
    file_name=file_name,
    file_category=file_category
)

byte stream / file object

📘

NOTE:

If a gzipped file is larger than 100MB, the file will be uploaded as multi-part upload.

For ts-sdk versions <= v1.2.31, the file size limit is 5GB. Starting with ts-sdk v1.2.32, the file size limit has been increased to 5TB.

with zf.open("somefile.csv", "rb") as f:
    context.write_file(
        content=f,
        file_name=file_name,
        file_category=file_category
    )

Returns

Returns an object that indicates the file location.

Here is an example:

{ 
  "type": "s3file", 
  "bucket": "datalake", 
  "fileKey": "path/to/file",
  "fileId": "uuidForFile",  
  "version": "versionId"
}.

📘

NOTE:

version will not be available when you are running pipelines locally.

context.write_ids

Writes an output IDS file to the data lake.

🚧

WARNING:

Notice you can only define file_suffix, you cannot define the complete file name of the output IDS file. The file name is codified and follows a certain pattern, which includes ids_namespace and ids_slug. This means if you switch to a different IDS, the new IDS JSON produced will be a separate file not a new version of the old file.

Parameters

This function contains the following parameters.

Returns

Returns an object that describes the file location, such as { type: 's3file', bucket: 'datalake', fileKey: 'path/to/file', fileId: 'uuidForFile' version: 'versionId'}. Note that version is not available when running pipelines locally.

context.get_file_name

Use this function to retrieve filename of a file that is not downloaded locally.

Parameters

The following table shows the parameters for this function.

Example(s)

N/A

Returns

A string; the filename of the file. Note that you do not need to download the file locally to access the file name.

context.get_ids

Use this function to get the Intermediate Data Schema that matches the namespace, slug, and version.

Parameters

The parameters for this function appear in the following table.

Returns

Returns an IDS schema object.

context.validate_ids

Checks the validity of IDS content provided in data. Throws an error if not valid.

Parameters

Returns

Returns a boolean value indicating whether the IDS is valid (true). It throws an error if the IDS is not valid.

context.get_logger

Returns the logger object, which currently has one method: log(data).

Parameters

There are no parameters for this function.

Example(s)

log an object

logger = context.get_logger()
logger.log({
    "message": "Starting the main parser",
    "level": "info"
})
# log output
# {"timestamp": "2020-04-22T20:45:02.540947", "message": "Starting the main parser", "level": "info"}
log a string, the default level is “info”


logger = context.get_logger()
logger.log("Writing IDS into the datalake")
# log output
# {"timestamp": "2020-04-22T20:45:02.971038", "message": "Writing IDS into the datalake", "level": "info"}

Returns

This returns the structured logger object, which currently has one method: log(input).

context.get_secret_config_value

📘

NOTE:

We recommend passing pipeline configs to your task script function in master-script script.js. Then in your task script function, use context.resolve_secret to resolve the secret value. This makes your task script function less dependent on TS context function (more like a pure function). Please see the documentation for that function.

Retrieves the secret's value.

Parameters

Returns

The value of the secret

context.get_presigned_url

Returns a time-limited HTTPS URL that can be used to access the file. If URL generation fails for any reason (except invalid value for ttl_sec parameter). None will be returned.

Parameters

Parameters for this function appear in the following table.

Returns

String that contains a time-limited HTTPS URL that can be used to access the file.

context.run_command

Invokes remote command/action on target (agent or connector) and returns its response.

📘

NOTE:

This function is still in the early stage and requires Tetra DataHub and Agent support. It's not an out-of-shelf function that you can use on any instrument. Please contact the TetraScience team for more information.

Parameters

Actions
TetraScience.Connector.gdc.HttpRequest
Send REST call to any network address as long as that address is reachable to GDC

• Requires TDP v3.0.0 and above
• Requires DataHub and Generic Data Connector (GDC)
• No agent needed

Returns

Returns a JSON response object returned by target (agent or connector) that ran the command.

It throws an error when the following are missing: org_slug, target_id, action, payload, and ttl_sec. Also throws an error when the ttl_sec is less than 300 seconds or higher than 900 seconds.

Error messages include the following:

• command not created successfully
• TTL has expired
• command didn’t execute successfully

For SDK version 1.2.22 and above: If the command does not execute successfully, the exception and exception message contains the reason why the command failed. Previously, it was "command status ERROR". Now the error message is "error response from target system."

context.update_metadata_tags

Updates file's metadata and tags.

Parameters

📘

Note

For ts-sdk versions <= v1.2.31, the file size limit is 5GB. Additionally, the function will raise an exception if no custom_meta or custom_tags are provided.

Starting with ts-sdk v1.2.32, the file size limit has been increased to 5TB. Additionally, the function will return the unmodified file instead of raising an exception.

Returns

Returns an object that describes the file location.

{
  type: 's3file',
  bucket: 'bucket',
  fileKey: 'fileKey',
  fileId: 'fileId',
  // no version locally (from fakes3)
  version: 'versionId'
}

context.resolve_secret

Returns the secret value. This function is used to convert the SSM reference to the actual secret value.

Parameters

Example

If you defined a secret in protocol.json as:

{
...
  config: [{
    "slug": "password",
    "name": "Password",
    "description": "This is a password",
    "type": "secret",
    "required": true
  }],
...
}

Then in the script.js, you will get password reference from the pipeline config:

...
const pipelineConfig = workflow.getContext('pipelineConfig');
const { password } = pipelineConfig;
await workflow.runTask('task-1', { password });
...

Then in your task script, you can get the password reference from input["password"]. The reference is a dictionary that contains SSM parameter store path to the secret value, like this:

{
  "secret": true,
  "ssm": "/development/tetrascience-demo/org-secrets/password",
  "version": 1
}

def main(input, context):
  password = context.resolve_secret(input.get('password'))
  ...

Returns

Returns the secret value. If you pass in a parameter that is not a secret, it will simply return the parameter.

context.add_labels

Allows you to add labels to a file.

If the file already has a label with the same key name, then a new label with the same key name will be created. The previous label with the same name will not be overwritten.

Parameter

Parameters appear in the table below.

Returns

Returns an array of added labels, e.g. [{"name": "Label A", "value": "Label B", "id": 22954, "createdAt": "2021-03-07T20:27:06.466Z"}]`

context.get_labels

Use this function to get all of the labels associated with a file.

Returns

Returns an array of labels for given file, e.g. [{"name": "Label A", "value": "Label B", "id": 22954, "createdAt": "2021-03-07T20:27:06.466Z"}]`

context.delete_labels

Use this function to delete one or more labels from a file. Labels are discussed in detail in the Basic Concepts: Metadata, Tags, and Labels topic.

Returns

An empty object if the label was successfully deleted from the file.

context.add_attributes

This function has been added with TDP v3.0.0 and ts-sdk v1.2.20. This function allows you to add metadata, tags, or labels to an object.

Returns

Returns a dict that describes the file location. A new file version will be generated if metadata/tags were provided.

context.search_eql

This function has been added with TDP v3.1.0 and ts-sdk v1.2.24. It helps to search files via Elasticsearch query language. For more details, check the next API call

Returns

Result of the ES query (raw or file pointers).