Tetra File-Log Agent Copy File Command

The following procedure shows how to download and drop a file to a designated folder programmatically by using the Tetra File-Log Agent and the Tetra Data Platform (TDP) Command Service.

Prerequisites

To complete the procedure, you must first do the following:

  • Get a JWT Token and Org Slug from the Tetra Data Platform
  • Create and configure a Tetra File-Log Agent that has its queue enabled
  • Access to a tool that will allow you to run REST calls, such as curl, Postman, or some other method

๐Ÿ“˜

NOTE:

FLA Drop File Command is available for File-Log Agent version 4.2.0 and above

Have a Tetra File-Log Agent Up and Running

Enable the Receive Commands option. For instructions, see Receive Commands/Enable Queue in the Tetra File-Log Agent Installation Guide.

1157

Send Drop File Command Payload

Invoke the Command Service API to send a command. To send a successful command, the following items are required:

  1. Tetra Command API: POST https://api.tetrascience.com/v1/commands
  2. Headers, including x-or-slug and ts-auth-token
  3. Valid Request Body

Drop File Command Example

// POST https://api.tetrascience.com/v1/commands

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "targetId": "<agent-id>",
  "action": "TetraScience.Agent.file-log.DropFile.CopyFile",
  "metadata": {},
  "expiresAt": "<command expiration datetimeoffset in ISO-8601 format, e.g. 2022-12-31T23:58:43.749Z>",
  "payload": {
    "source": "https://...",
    "destination": "c:\\tmp\\test.txt"
  }
}

Request Body Data Field Description

Field NameDescriptionSample
targetIdThe Tetra File-Log Agent Id which receives the command (Required) f777b552-e03b-4269-b186-5783beca430d
createdAtCommand creation date (ISO-8601 format) (Optional).2021-12-13T03:06:22.564Z
expiresAtCommand expire date (ISO-8601 format) (Required)2021-12-31T23:58:43.749Z
actionCommand Name (It is a constant variable) (Required)TetraScience.Agent.file-log.DropFile.CopyFile
payloadThe body of command. It has predefined structure per the Command (action field)

FLA CopyFile command validates the payload using the following JSON schema
payload.sourceIt can be any URL which will return the contents of a file when a GET HTTP request is performed.

E.g. a pre-signed S3 object URL
https:// ....
payload.destinationThe target path including the file name to be created

The path can be local path or the network path
C:\temp\SampleFile\G264_aT72_OD-T-220.xlsx
payload.createPathSpecify if the agent should create the directory structure of the destinationtrue

(If this property is not specified, default to false)
payload.overwriteSpecify if the agent should overwrite the file if same named files exists in destination.true

(If this property is not specified, default to false)
payload.rawMd5ChecksumMD5 Checksum of the file being downloadedbb33d10c554e0b10f34f70f40faec4ee

Drop File Command Payload JSON Schema

{
    "$schema": "https://json-schema.org/2020-12/schema",
    "$id": "https://tetrascience.com/command.schema.json/v1",
    "title": "TetraScience.Agent.file-log.DropFile.CopyFile",
    "description": "Payload Schema for Command TetraScience.Agent.file-log.DropFile.CopyFile",
    "type": "object",
    "properties": { 
        "source": { "type": "string", "format": "uri", "minLength": 1 },
        "destination": { "type": "string", "minLength": 1 },
        "createPath": { "type": "boolean" },
        "overwrite": { "type": "boolean" },
        "rawMd5Checksum": { "type": "string" }
    },
    "required": ["source" , "destination"]
}

Command Validation

The TDP validates the command JSON schema and the included payload JSON schema. If either of them aren't valid, the TDP returns the following response:

{
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Invalid request payload input"
}

If both schemas are valid, the TDP returns a response that contains the new generated command Id, such as the following:

{
    "id": "<new_command_id>",
    "orgSlug": "<org_slug>",
    "targetId": "<agent_id>",
    "action": "TetraScience.Agent.file-log.DropFile.CopyFile",
    "expiresAt": "2022-12-31T23:58:43.749Z",
    "metadata": {
        "key1": "value1",
        "key2": "value2"
    },
    "status": "PENDING",
    "requestBody": {
      ...
    }
}

When the Agent receives the Drop File Command, it will start to download the file and validate content of the JSON payload and the source file.

If any of the following rules aren't met, the Agent sends a FAILURE status back to the TDP:

  1. The command is not expired.
  2. The file size won't exceed 100 MB
  3. The Agent group account has write permission of the destination folder
  4. The MD5 checksum in the command should match with the source file's

After the Agent successfully downloads and drops the file to the destination folder, the Agent then sends a SUCCESS status back to TDP.

Best Practices

  1. Avoid setting the destination to a folder that a File-Log Agent is watching. Doing this can create a loop where the file downloaded by the Agent through the CopyFile command will be uploaded by the Agent to the TDP again.
  2. Calculate and send the MD5 checksum in the payload for integrity check

Including the CopyFile command in TDP task scripts

The ts-sdk library provides two functions that greatly simplify creating and sending Copy File commands in TDP task scripts:

With the pre-signed URL, you can construct the payload as described above and use the run_command function to send it to the Agent.