Tetra File-Log Agent Copy File Command

This topic shows you how the Tetra File-Log Agent can be used to download and drop a file to a designated folder by utilizing Tetra Data Platform Command Service.

Prerequisites

To complete these instructions you'll need:

  • Get a JWT Token and Org Slug from the Tetra Data Platform
  • Set up a GDC or CDC Agent from Tetra Data Platform with queue enabled
  • Access to a tool that will allow you to run REST calls, such as curl, Postman, or some other method

๐Ÿ“˜

NOTE:

FLA Drop File Command is available for File-Log Agent version 4.2.0 and above

Have File-Log Agent up and running

Enable Receive Commands option

1157

Send Drop File Command Payload

You can invoke Tetra Command API to send a command. Below is the sample of the command,
To send a successful command, the following items are required:

  1. Tetra Command API: POST https://api.tetrascience.com/v1/commands
  2. Headers, including x-or-slug and ts-auth-token
  3. Valid Request Body
// POST https://api.tetrascience.com/v1/commands

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "targetId": "<agent-id>",
  "action": "TetraScience.Agent.file-log.DropFile.CopyFile",
  "metadata": {},
  "expiresAt": "<command expiration datetimeoffset in ISO-8601 format, e.g. 2022-12-31T23:58:43.749Z>",
  "payload": {
    "source": "https://...",
    "destination": "c:\\tmp\\test.txt"
  }
}

Request Body Data Field Description

Field NameDescriptionSample
targetIdThe Tetra File-Log Agent Id which receives the command (Required) f777b552-e03b-4269-b186-5783beca430d
createdAtCommand creation date (ISO-8601 format) (Optional).2021-12-13T03:06:22.564Z
expiresAtCommand expire date (ISO-8601 format) (Required)2021-12-31T23:58:43.749Z
actionCommand Name (It is a constant variable) (Required)TetraScience.Agent.file-log.DropFile.CopyFile
payloadThe body of command. It has predefined structure per the Command (action field)

FLA CopyFile command validates the payload using the following JSON schema
payload.sourceIt can be any URL which will return the contents of a file when a GET HTTP request is performed.

E.g. a pre-signed S3 object URL
https:// ....
payload.destinationThe target path including the file name to be created

The path can be local path or the network path
C:\temp\SampleFile\G264_aT72_OD-T-220.xlsx
payload.createPathSpecify if the agent should create the directory structure of the destinationtrue

(If this property is not specified, default to false)
payload.overwriteSpecify if the agent should overwrite the file if same named files exists in destination.true

(If this property is not specified, default to false)
payload.rawMd5ChecksumMD5 Checksum of the file being downloadedbb33d10c554e0b10f34f70f40faec4ee

Drop File Command Payload JSON Schema

{
    "$schema": "https://json-schema.org/2020-12/schema",
    "$id": "https://tetrascience.com/command.schema.json/v1",
    "title": "TetraScience.Agent.file-log.DropFile.CopyFile",
    "description": "Payload Schema for Command TetraScience.Agent.file-log.DropFile.CopyFile",
    "type": "object",
    "properties": { 
        "source": { "type": "string", "format": "uri", "minLength": 1 },
        "destination": { "type": "string", "minLength": 1 },
        "createPath": { "type": "boolean" },
        "overwrite": { "type": "boolean" },
        "rawMd5Checksum": { "type": "string" }
    },
    "required": ["source" , "destination"]
}

Command Validation

The Tetra Data Platform validates the command JSON schema and the included payload JSON schema
If either of them is invalid, Tetra Data Platform will return the response as below

{
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Invalid request payload input"
}

If both schema are valid, Tetra Data Platform returns response containing the new generated command Id

{
    "id": "<new_command_id>",
    "orgSlug": "<org_slug>",
    "targetId": "<agent_id>",
    "action": "TetraScience.Agent.file-log.DropFile.CopyFile",
    "expiresAt": "2022-12-31T23:58:43.749Z",
    "metadata": {
        "key1": "value1",
        "key2": "value2"
    },
    "status": "PENDING",
    "requestBody": {
      ...
    }
}

When the Agent receives the Dropfile Command, it will start to download the file and validate content of the JSON payload and the source file.

If any of the following rules doesn't meet, the Agent will send a FAILURE status back to Tetra Data Platform.

  1. The command is not expired.
  2. The file size won't exceed 100 MB
  3. The Agent group account has write permission of the destination folder
  4. The MD5 checksum in the command should match with the source file's

After the Agent successfully download and drop the file to the destination folder, the Agent will send SUCCESS status back to Tetra Data Platform.

Best Practices

  1. Avoid setting the destination to a folder that a File-Log Agent is watching as this will potentially create a loop, wherein the file downloaded by the agent through the CopyFile command will be uploaded by the agent to TDP again
  2. Calculate and send the MD5 checksum in the payload for integrity check

Including the CopyFile command in TDP task scripts

The ts-sdk library provides two functions which greatly simplify creating and sending CopyFile commands in TDP Task Scripts.

context.get_presigned_url - This is used to create a pre-signed S3 URL for a TDP File Pointer.

context.run_command - This is used to send the command to the agent, and receive the response.

With the pre-signed URL, you can construct the payload as described above and use the run_command function to send it to the agent.