Tetra File-Log Agent Copy File Command
The following procedure shows how to download and drop a file to a designated folder programmatically by using the Tetra File-Log Agent and the Tetra Data Platform (TDP) Command Service.
Prerequisites
To complete the procedure, you must first do the following:
- Get a JWT Token and Org Slug from the Tetra Data Platform
- Create and configure a Tetra File-Log Agent that has its queue enabled
- Access to a tool that will allow you to run REST calls, such as curl, Postman, or some other method
NOTE:
FLA Drop File Command is available for File-Log Agent version 4.2.0 and above
Have a Tetra File-Log Agent Up and Running
Enable the Receive Commands option. For instructions, see Receive Commands/Enable Queue in the Tetra File-Log Agent Installation Guide.
Send Drop File Command Payload
Invoke the Command Service API to send a command. To send a successful command, the following items are required:
- Tetra Command API: POST https://api.tetrascience.com/v1/commands
- Headers, including x-or-slug and ts-auth-token
- Valid Request Body
Drop File Command Example
// POST https://api.tetrascience.com/v1/commands
// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
// Request body:
{
"targetId": "<agent-id>",
"action": "TetraScience.Agent.file-log.DropFile.CopyFile",
"metadata": {},
"expiresAt": "<command expiration datetimeoffset in ISO-8601 format, e.g. 2022-12-31T23:58:43.749Z>",
"payload": {
"source": "https://...",
"destination": "c:\\tmp\\test.txt"
}
}
Request Body Data Field Description
Field Name | Description | Sample |
---|---|---|
targetId | The Tetra File-Log Agent Id which receives the command (Required) | f777b552-e03b-4269-b186-5783beca430d |
createdAt | Command creation date (ISO-8601 format) (Optional). | 2021-12-13T03:06:22.564Z |
expiresAt | Command expire date (ISO-8601 format) (Required) | 2021-12-31T23:58:43.749Z |
action | Command Name (It is a constant variable) (Required) | TetraScience.Agent.file-log.DropFile.CopyFile |
payload | The body of command. It has predefined structure per the Command (action field) FLA CopyFile command validates the payload using the following JSON schema | |
payload.source | It can be any URL which will return the contents of a file when a GET HTTP request is performed. E.g. a pre-signed S3 object URL | https:// .... |
payload.destination | The target path including the file name to be created The path can be local path or the network path | C:\temp\SampleFile\G264_aT72_OD-T-220.xlsx |
payload.createPath | Specify if the agent should create the directory structure of the destination | true (If this property is not specified, default to false) |
payload.overwrite | Specify if the agent should overwrite the file if same named files exists in destination. | true (If this property is not specified, default to false) |
payload.rawMd5Checksum | MD5 Checksum of the file being downloaded | bb33d10c554e0b10f34f70f40faec4ee |
Drop File Command Payload JSON Schema
{
"$schema": "https://json-schema.org/2020-12/schema",
"$id": "https://tetrascience.com/command.schema.json/v1",
"title": "TetraScience.Agent.file-log.DropFile.CopyFile",
"description": "Payload Schema for Command TetraScience.Agent.file-log.DropFile.CopyFile",
"type": "object",
"properties": {
"source": { "type": "string", "format": "uri", "minLength": 1 },
"destination": { "type": "string", "minLength": 1 },
"createPath": { "type": "boolean" },
"overwrite": { "type": "boolean" },
"rawMd5Checksum": { "type": "string" }
},
"required": ["source" , "destination"]
}
Command Validation
The TDP validates the command JSON schema and the included payload JSON schema. If either of them aren't valid, the TDP returns the following response:
{
"statusCode": 400,
"error": "Bad Request",
"message": "Invalid request payload input"
}
If both schemas are valid, the TDP returns a response that contains the new generated command Id, such as the following:
{
"id": "<new_command_id>",
"orgSlug": "<org_slug>",
"targetId": "<agent_id>",
"action": "TetraScience.Agent.file-log.DropFile.CopyFile",
"expiresAt": "2022-12-31T23:58:43.749Z",
"metadata": {
"key1": "value1",
"key2": "value2"
},
"status": "PENDING",
"requestBody": {
...
}
}
When the Agent receives the Drop File Command, it will start to download the file and validate content of the JSON payload and the source file.
If any of the following rules aren't met, the Agent sends a FAILURE
status back to the TDP:
- The command is not expired.
- The file size won't exceed 100 MB
- The Agent group account has write permission of the destination folder
- The MD5 checksum in the command should match with the source file's
After the Agent successfully downloads and drops the file to the destination folder, the Agent then sends a SUCCESS
status back to TDP.
Best Practices
- Avoid setting the destination to a folder that a File-Log Agent is watching. Doing this can create a loop where the file downloaded by the Agent through the CopyFile command will be uploaded by the Agent to the TDP again.
- Calculate and send the MD5 checksum in the payload for integrity check
Including the CopyFile command in TDP task scripts
The ts-sdk
library provides two functions that greatly simplify creating and sending Copy File commands in TDP task scripts:
-
context.get_presigned_url - This is used to create a pre-signed S3 URL for a TDP File Pointer.
-
context.run_command - This is used to send the command to the agent, and receive the response.
With the pre-signed URL, you can construct the payload as described above and use the run_command
function to send it to the Agent.
Updated 10 months ago