API DocumentationProduct Documentation
Product Documentation

Tetra File-Log Agent Programmatic Agent Creation and Configuration

Suggest Edits

This topic shows how to create and configure a Tetra File-Log Agent by using the TetraScience API.

📘

NOTE

Online configuration is available for Tetra File-Log Agent versions 4.1 and higher. Agents that don’t have internet access also require a Tetra Hub.

Prerequisites

To complete these instructions you'll need the following:

  • A JSON Web Token (JWT) and org slug from the TDP
  • The Agent ID
  • Access to a tool that will allow you to run REST API calls, such as cURL or Postman

For instructions on how to get a JWT and org slug from the TDP, see Authentication in the TetraScience API documentation.

📘

NOTE

The following procedure works for both the TDP (No Connector) connection option and Tetra Hub v2 connections. To create and configure an Agent by using a Data Hub, you must replace integrationId with the Id of your Generic Data Connector (GDC), and integrationType with "datahub".

Create an Agent using the API

Now that you have your org slug, and JWT, you can use the Agents API to create an Agent by doing the following:

  1. To create an Agent in the platform, enter code that is similar to the following into a tool such as cURL or Postman (or your own code): 
// POST https://api.tetrascience.com/v1/agents

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "sourceType": null,
  "name": "File-Log Agent",
  "description": "your agent description",
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
  "type": "file-log",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "integrationType": "api",
  "labels": {
    "labelsToAdd": [
      {
        "name": "example-label-1",
        "value": "example label 1"
      }
    ]
  }
}

// Response body:
{
  "id": "<the-agent-id>",
  "name": "File-Log Agent",
  "description": "your agent description",
  "sourceType": null,
  "type": "file-log",
  "integrationType": "api",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "datahubId": null,
  "sourceId": "<internal-source-id>",
  "isEnabled": true,
  "queue": null,
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
    "config": null,
    "configStatus": null,
    "configStatusUpdatedAt": null,
    "configStatusError": null,
    "createdAt": "2022-02-12T00:50:04.159Z",
    "updatedAt": "2022-02-12T00:50:04.159Z"
	}
}
  1. Copy the Agent ID returned in response. The Agent ID is required for the next step.

Install the FLA on the Target Machine and Configure it to Receive Commands

To install the FLA on your target machine and configure it to receive commands, do the following:

  1. Install the File-Log Agent on the target machine.
  2. Open the File-Log Agent Management console and start to configure it. Use the Agent ID from the response above as the Agent Id, the API endpoint as the Connection Url. Then, enter your Org Slug and Authorization token (JWT).
  3. In the Agent Configuration section, for Receive Commands, choose Yes. Activating this setting allows the Agent to receive configuration commands from the platform.
1179

FLA Interface

  1. The Agent UI shows a Connected status once the Agent is configured correctly.
  2. When finished, start the Agent service.

Enabling the Command Queue on TDP

Before configuring the Agent programmatically, you will need to enable the command queue on the platform by doing the following:

  1. Use the following API endpoint to enable the command queue on the TDP for your Agent:
// PUT https://api.tetrascience.com/v1/agents/<your-agent-id>/command-queue

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>

// Request body:
{
  "enabled": true
}

// Response body:
{
  "enabled": true,
  "queue": "<your-queue-url>",
  "visibilityTimeout": 600
}
  1. Use the Agents API to check that your Agent is online (this may take a minute or two from when the Agent service is started). You should also see the Agent's queue information in the response. 
// GET https://api.tetrascience.com/v1/agents/<your-agent-id>

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
    "id": "<your-agent-id>",
    "name": "File-Log Agent",
    "description": "some description",
    "sourceType": null,
    "type": "file-log",
    "integrationType": "api",
    "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
    "datahubId": null,
    "sourceId": "<internal-source-id>",
    "isEnabled": true,
    "tags": [
        "example-tag-1"
    ],
    "metadata": {
        "meta1": "value1"
    },
    "config": null,
    "configStatus": null,
    "configStatusUpdatedAt": null,
    "configStatusError": null,
    "configChangedBy": null,
    "configChangedAt": null,
    "createdAt": "2022-05-18T17:36:33.447Z",
    "updatedAt": "2022-05-18T17:36:33.447Z",
    "labels": [
        {
            "id": 13409,
            "name": "example-label-1",
            "value": "example label 1"
        }
    ],
    "status": "Online",
    "version": "v4.1.0",
    "liveType": "file-log",
    "host": {
        "ip": null,
        "name": null
    },
    "queue": {
      "enabled": true,
      "queue": "<your-queue-url>",
      "visibilityTimeout": 600
    }
}

Updating the Agent's Configuration via the API

To update the Agent's configuration via the API, do the following:

🚧

NOTE:

This will remove any paths not included in the JSON. To preserve existing paths, you must include them each time you POST.

  1. POST a new config to the Agent using the API. The expected JSON format follows the following code sample. For the full schema of the payload, see Remotely configure a running File-Log Agent in the TetraScience API documentation.
// POST https://api.tetrascience.com/v1/agents/{agentId}/configuration

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
  "services_enabled": [
    "fileWatcher"
  ],
  "services_configuration": {
    "fileWatcher": {
      "paths": [
        {
          "path": "c:\\test-path-1",
          "interval": 30,
          "start_date": "2022-01-01",
          "patterns": [
            "*"
          ],
          "file_watch_mode": "file",
          "source_type": "unknown",
          "metadata": {
            "meta1": "value1"
          },
          "tags": "path-tag-1",
          "labels": [
            {
              "name": "path-label-1",
              "value": "label1-value"
            }
          ],
          "archive": {
            "path": "c:\\archive",
            "wait_time": 3,
            "wait_time_unit": "Days",
            "delete": {
              "wait_time": 30,
              "wait_time_unit": "Days"
            },
            "dry_run": false
          }
        },
        {
          "path": "c:\\test-path-2",
          "interval": 30,
          "start_date": "2022-01-01",
          "patterns": [
            "*"
          ],
          "file_watch_mode": "file",
          "source_type": "unknown",
          "metadata": {
            "meta2": "value2"
          },
          "tags": "path-tag-2",
          "labels": [
            {
              "name": "path-label-2",
              "value": "label2-value"
            }
          ],
          "archive": {
            "path": "c:\\archive",
            "wait_time": 24,
            "wait_time_unit": "Hours",
            "delete": null,
            "dry_run": false
          }
        }
      ],
      "use_path_configuration": true
    }
  }
}
  
// Response body:
{
  "commandId": "<some-command-id>"
}
  1. The updated configuration is shown in the Agent UI. There are also several ways to retrieve the Agent's configuration by using the API. This process is covered in the next section.

Retrieving Agent Information and Configuration

The GET Agent details API will include its latest applied configuration:

// GET https://api.tetrascience.com/v1/agents/<your-agent-id>

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "id": "d9ee8ce3-78f2-494a-82d1-b391c30eb879",
  "name": "File-Log Agent",
  "description": "some description",
  "sourceType": null,
  "type": "file-log",
  "integrationType": "api",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "datahubId": null,
  "sourceId": "a604d02a-ba51-469e-8706-b4efa7d2976a",
  "isEnabled": true,
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
  "config": {
  	"destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
          {
            "path": "c:\\test-path-1",
            "tags": "path-tag-1",
            "interval": 30,
            "metadata": {
              "meta1": "value1"
            },
            "patterns": [
              "*"
            ],
            "start_date": "2022-01-01",
            "source_type": "unknown",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-1",
                "value": "label1-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "wait_time": 3,
              "wait_time_unit": "Days",
              "delete": {
                "wait_time": 30,
                "wait_time_unit": "Days"
              },
              "dry_run": false
            }
          },
          {
            "path": "c:\\test-path-2",
            "tags": "path-tag-2",
            "interval": 30,
            "metadata": {
              "meta2": "value2"
            },
            "patterns": [
              "*"
            ],
            "start_date": "2022-01-01",
            "source_type": "unknown",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-2",
                "value": "label2-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "delete": null,
              "dry_run": false,
              "wait_time": 24,
              "wait_time_unit": "Hours"
            }
          }
        ],
        "interval": 30,
        "start_date": "2022-05-16",
        "use_path_configuration": true
      }
    }
  },
  "configStatus": "SUCCESS",
  "configStatusUpdatedAt": "2022-05-18T17:53:07.386Z",
  "configStatusError": null,
  "configChangedBy": "cloud",
  "configChangedAt": "2022-05-18T17:53:05.033Z",
  "createdAt": "2022-05-18T17:36:33.447Z",
  "updatedAt": "2022-05-18T17:53:04.961Z",
  "labels": [
    {
      "id": 13409,
      "name": "example-label-1",
      "value": "example label 1"
    }
  ],
  "status": "Online",
  "version": "v4.1.0",
  "liveType": "file-log",
  "host": {
    "ip": null,
    "name": null
  },
  "queue": {
    "enabled": true,
    "name": "https://sqs.us-east-2.amazonaws.com/706717599419/onprem-d9ee8ce3-78f2-494a-82d1-b391c30eb879.fifo",
    "visibilityTimeout": 600
  }
}

You can also get just the latest configuration applied to the Agent by using the following API:

// GET https://api.tetrascience.com/v1/agents/{id}/configuration

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "id": "4c483892-40b6-45b9-a352-cc7a46d1cb17",
  "config": {
  	"destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
          {
            "path": "c:\\online1",
            "tags": "path-tag-1",
            "interval": 30,
            "metadata": {
              "meta1": "value1"
            },
            "patterns": [
              "*.*"
            ],
            "start_date": "2022-01-01",
            "source_type": "something",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-1",
                "value": "label1-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "delete": {
                "wait_time": 30,
                "wait_time_unit": "Days"
              },
              "dry_run": false,
              "wait_time": 3,
              "wait_time_unit": "Days"
            }
          }
        ],
        "interval": 30,
        "start_date": "2022-07-07",
        "use_path_configuration": true
      }
    }
  },
  "by": "cloud",
  "at": "2022-07-08T19:04:34.993Z"
}

To see the latest cloud configuration and whether it's been applied by the Agent successfully or not, use the following API:

Note: In the following example API call, the configuration was not successfully applied because of a file_watch_mode that's not valid.

// GET https://api.tetrascience.com/v1/agents/{id}/configuration?type=cloud

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
{
  "id": "cf9d2576-b073-4744-9101-def5a37b32cf",
  "config": {
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
        {
          "path": "c:\\online1",
          "tags": "path-tag-1",
          "interval": 30,
          "metadata": {
            "meta1": "value1"
          },
          "patterns": [
            "*.*"
          ],
          "start_date": "2022-01-01",
          "source_type": "something",
          "file_watch_mode": "INVALID"
        }
      ],
      "use_path_configuration": true
     }
   }
 },
 "by": "cloud",
 "at": "2022-07-08T19:15:43.246Z",
 "commandStatus": "FAILURE",
 "commandId": "cf9d2576-b073-4744-9101-def5a37b32cf"
}

To see the Agent's configuration history, use the following API:

Note: Responses are listed from most to least recent. In the following example, the earlier configuration command failed because paths weren't valid, and the later one succeeded.

// GET https://api.tetrascience.com/v1/agents/{id}/configuration/list

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "page": 1,
  "size": 10,
  "hasNext": false,
  "hits": [
    {
      "id": "<command-id>",
      "config": {
        "services_enabled": [
          "fileWatcher"
        ],
        "services_configuration": {
          "fileWatcher": {
            "paths": [
              {
                "path": "c:\\test-path-1",
                "tags": "path-tag-1",
                "interval": 30,
                "metadata": {
                  "meta1": "value1"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              },
              {
                "path": "c:\\test-path-2",
                "tags": "path-tag-2",
                "interval": 30,
                "metadata": {
                  "meta2": "value2"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              }
            ],
            "interval": 30,
            "start_date": "2022-05-16",
            "use_path_configuration": true
          }
        }
      },
      "by": "cloud",
      "at": "2022-05-18T17:53:05.033Z",
      "commandId": "8819af94-b9ac-4ea9-9540-97328ffae96d",
      "commandStatus": "SUCCESS"
    },
    {
      "id": "f8c11ec3-fbfb-45e5-a358-b23de23714df",
      "config": {
        "services_enabled": [
          "fileWatcher"
        ],
        "services_configuration": {
          "fileWatcher": {
            "paths": [
              {
                "path": "c:\\<your-test-path-1>",
                "tags": "path-tag-1",
                "interval": 30,
                "metadata": {
                  "meta1": "value1"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              },
              {
                "path": "c:\\<your-test-path-2>",
                "tags": "path-tag-2",
                "interval": 30,
                "metadata": {
                  "meta2": "value2"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              }
            ],
            "use_path_configuration": true
          }
        }
      },
      "by": "cloud",
      "at": "2022-05-18T17:52:27.487Z",
      "commandId": "f8c11ec3-fbfb-45e5-a358-b23de23714df",
      "commandStatus": "FAILURE"
    }
  ]
}

Commands Pending, Failure, and Expiration

When a configuration command is waiting to be received and applied by an Agent, the Agents UI shows a CONFIGURATION PENDING status.

If a configuration command fails to apply (for example, because it has a configuration that isn't valid), the Agents UI shows a CONFIGURATION FAILED: status.

If the Agent doesn't receive a command within four minutes, the Agent returns the following error message: Configuration has failed due to no response for 4m.

To troubleshoot the error, first make sure that the Receive Command setting is turned on in the Agent UI. For instructions, see the Install the FLA on the Target Machine and Configure it to Receive Commands section on this page.

If the Receive Command setting is set to Yes and you still receive the error, submit a support ticket to TetraScience. Before submitting a support ticket, make sure that you collect the local logs for the session at /logs/nxlog/ and/or /logs/archive/backup.

When the configuration is applied to the Agent successfully, the Agent UI shows a CONFIGURATION APPLIED: status.

Cloud Configuration superseded by local

Because the Agent can be configured locally as well as by these commands, it may receive a cloud command that's already been superseded by a later local configuration. When that happens, the Agent will ignore the superseded configuration, and it will be considered successful.

Configuration via TDP UI

The APIs listed above are the preferred way to configure your Agents, because they support repeatability and automation. You can also configure through the TDP UI by doing the following:

  1. Sign in to the TDP.
  2. In the left navigation pane, select the hamburger menu icon.
  3. Choose Data Sources. Then, choose Agents. The Agents page appears.
  4. On the Agents page, select the name of the agent that you want to configure. A menu appears on the right.
  5. Choose Configure. The Configure page opens.
  6. On the Configure page, set the Path Configurations and File Watcher settings. For more information, see the Tetra File-Log Agent Installation Guide.
  7. Choose Apply.
    Add a comment. Markdown allowed.

Updated about 2 months ago