API DocumentationProduct Documentation

Tetra File-Log Agent Programmatic Agent Creation and Configuration

Suggest Edits

This topic shows you how the Tetra File-Log Agent can be created and configured using APIs to enable automation.

Prerequisites

To complete these instructions you'll need the following:

  • A JSON Web Token (JWT) and org slug from the TDP
  • The Agent ID
  • Access to a tool that will allow you to run REST API calls, such as cURL or Postman

For instructions on how to get a JWT and org slug from the TDP, see Authentication in the TetraScience API documentation.

📘

NOTE

Online configuration is available for File-Log Agent 4.1 or later. A Datahub with the L7 proxy is required for Agents that don’t have internet access.

📘

NOTE

These instructions explain how to connect the Agent without a Connector, but you can create and configure Agents in a Datahub GDC in the same way. Just replace integrationId with the Id of your GDC, and integrationType with "datahub".

Create an Agent using the API

Now that you have your org slug, and JWT, you can use the Agents API to create an Agent by doing the following:

  1. To create an Agent in the platform, enter code that is similar to the following into a tool such as cURL or Postman (or your own code): 
// POST https://api.tetrascience.com/v1/agents

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "sourceType": null,
  "name": "File-Log Agent",
  "description": "your agent description",
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
  "type": "file-log",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "integrationType": "api",
  "labels": {
    "labelsToAdd": [
      {
        "name": "example-label-1",
        "value": "example label 1"
      }
    ]
  }
}

// Response body:
{
  "id": "<the-agent-id>",
  "name": "File-Log Agent",
  "description": "your agent description",
  "sourceType": null,
  "type": "file-log",
  "integrationType": "api",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "datahubId": null,
  "sourceId": "<internal-source-id>",
  "isEnabled": true,
  "queue": null,
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
    "config": null,
    "configStatus": null,
    "configStatusUpdatedAt": null,
    "configStatusError": null,
    "createdAt": "2022-02-12T00:50:04.159Z",
    "updatedAt": "2022-02-12T00:50:04.159Z"
	}
}
  1. Copy the Agent ID returned in response. The Agent ID is required for the next step.

Install the FLA on the Target Machine and Configure it to Receive Commands

To install the FLA on your target machine and configure it to receive commands, do the following:

  1. Install the File-Log Agent on the target machine.
  2. Open the File-Log Agent Management console and start to configure it. Use the Agent ID from the response above as the Agent Id, the API endpoint as the Connection Url. Then, enter your Org Slug and Authorization token (JWT).
  3. In the Agent Configuration section, for Receive Commands, choose Yes. Activating this setting allows the Agent to receive configuration commands from the platform.
1179

FLA Interface

  1. The Agent UI shows a Connected status once the Agent is configured correctly.
  2. When finished, start the Agent service.

Enabling the Command Queue on TDP

Before configuring the Agent programmatically, you will need to enable the command queue on the platform by doing the following:

  1. Use the following API endpoint to enable the command queue on the TDP for your Agent:
// PUT https://api.tetrascience.com/v1/agents/<your-agent-id>/command-queue

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>

// Request body:
{
  "enabled": true
}

// Response body:
{
  "enabled": true,
  "queue": "<your-queue-url>",
  "visibilityTimeout": 600
}
  1. Use the Agents API to check that your Agent is online (this may take a minute or two from when the Agent service is started). You should also see the Agent's queue information in the response. 
// GET https://api.tetrascience.com/v1/agents/<your-agent-id>

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
    "id": "<your-agent-id>",
    "name": "File-Log Agent",
    "description": "some description",
    "sourceType": null,
    "type": "file-log",
    "integrationType": "api",
    "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
    "datahubId": null,
    "sourceId": "<internal-source-id>",
    "isEnabled": true,
    "tags": [
        "example-tag-1"
    ],
    "metadata": {
        "meta1": "value1"
    },
    "config": null,
    "configStatus": null,
    "configStatusUpdatedAt": null,
    "configStatusError": null,
    "configChangedBy": null,
    "configChangedAt": null,
    "createdAt": "2022-05-18T17:36:33.447Z",
    "updatedAt": "2022-05-18T17:36:33.447Z",
    "labels": [
        {
            "id": 13409,
            "name": "example-label-1",
            "value": "example label 1"
        }
    ],
    "status": "Online",
    "version": "v4.1.0",
    "liveType": "file-log",
    "host": {
        "ip": null,
        "name": null
    },
    "queue": {
      "enabled": true,
      "queue": "<your-queue-url>",
      "visibilityTimeout": 600
    }
}

Updating the Agent's Configuration via the API

To update the Agent's configuration via the API, do the following:

🚧

NOTE:

This will remove any paths not included in the JSON. To preserve existing paths, you must include them each time you POST.

  1. POST a new config to the Agent using the API. The expected JSON format follows the following code sample. For the full schema of the payload, see Remotely configure a running File-Log Agent in the TetraScience API documentation.
// POST https://api.tetrascience.com/v1/agents/{agentId}/configuration

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Request body:
{
  "destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
  "services_enabled": [
    "fileWatcher"
  ],
  "services_configuration": {
    "fileWatcher": {
      "paths": [
        {
          "path": "c:\\test-path-1",
          "interval": 30,
          "start_date": "2022-01-01",
          "patterns": [
            "*"
          ],
          "file_watch_mode": "file",
          "source_type": "unknown",
          "metadata": {
            "meta1": "value1"
          },
          "tags": "path-tag-1",
          "labels": [
            {
              "name": "path-label-1",
              "value": "label1-value"
            }
          ],
          "archive": {
            "path": "c:\\archive",
            "wait_time": 3,
            "wait_time_unit": "Days",
            "delete": {
              "wait_time": 30,
              "wait_time_unit": "Days"
            },
            "dry_run": false
          }
        },
        {
          "path": "c:\\test-path-2",
          "interval": 30,
          "start_date": "2022-01-01",
          "patterns": [
            "*"
          ],
          "file_watch_mode": "file",
          "source_type": "unknown",
          "metadata": {
            "meta2": "value2"
          },
          "tags": "path-tag-2",
          "labels": [
            {
              "name": "path-label-2",
              "value": "label2-value"
            }
          ],
          "archive": {
            "path": "c:\\archive",
            "wait_time": 24,
            "wait_time_unit": "Hours",
            "delete": null,
            "dry_run": false
          }
        }
      ],
      "use_path_configuration": true
    }
  }
}
  
// Response body:
{
  "commandId": "<some-command-id>"
}
  1. The updated configuration is shown in the Agent UI. There are also several ways to retrieve the Agent's configuration by using the API. This process is covered in the next section.

Retrieving Agent Information and Configuration

The GET Agent details API will include its latest applied configuration:

// GET https://api.tetrascience.com/v1/agents/<your-agent-id>

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "id": "d9ee8ce3-78f2-494a-82d1-b391c30eb879",
  "name": "File-Log Agent",
  "description": "some description",
  "sourceType": null,
  "type": "file-log",
  "integrationType": "api",
  "integrationId": "6f166302-df8a-4044-ab4b-7ddd3eefb50b",
  "datahubId": null,
  "sourceId": "a604d02a-ba51-469e-8706-b4efa7d2976a",
  "isEnabled": true,
  "tags": [
    "example-tag-1"
  ],
  "metadata": {
    "meta1": "value1"
  },
  "config": {
  	"destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
          {
            "path": "c:\\test-path-1",
            "tags": "path-tag-1",
            "interval": 30,
            "metadata": {
              "meta1": "value1"
            },
            "patterns": [
              "*"
            ],
            "start_date": "2022-01-01",
            "source_type": "unknown",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-1",
                "value": "label1-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "wait_time": 3,
              "wait_time_unit": "Days",
              "delete": {
                "wait_time": 30,
                "wait_time_unit": "Days"
              },
              "dry_run": false
            }
          },
          {
            "path": "c:\\test-path-2",
            "tags": "path-tag-2",
            "interval": 30,
            "metadata": {
              "meta2": "value2"
            },
            "patterns": [
              "*"
            ],
            "start_date": "2022-01-01",
            "source_type": "unknown",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-2",
                "value": "label2-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "delete": null,
              "dry_run": false,
              "wait_time": 24,
              "wait_time_unit": "Hours"
            }
          }
        ],
        "interval": 30,
        "start_date": "2022-05-16",
        "use_path_configuration": true
      }
    }
  },
  "configStatus": "SUCCESS",
  "configStatusUpdatedAt": "2022-05-18T17:53:07.386Z",
  "configStatusError": null,
  "configChangedBy": "cloud",
  "configChangedAt": "2022-05-18T17:53:05.033Z",
  "createdAt": "2022-05-18T17:36:33.447Z",
  "updatedAt": "2022-05-18T17:53:04.961Z",
  "labels": [
    {
      "id": 13409,
      "name": "example-label-1",
      "value": "example label 1"
    }
  ],
  "status": "Online",
  "version": "v4.1.0",
  "liveType": "file-log",
  "host": {
    "ip": null,
    "name": null
  },
  "queue": {
    "enabled": true,
    "name": "https://sqs.us-east-2.amazonaws.com/706717599419/onprem-d9ee8ce3-78f2-494a-82d1-b391c30eb879.fifo",
    "visibilityTimeout": 600
  }
}

You can also get just the latest configuration applied to the Agent by using the following API:

// GET https://api.tetrascience.com/v1/agents/{id}/configuration

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "id": "4c483892-40b6-45b9-a352-cc7a46d1cb17",
  "config": {
  	"destination_id": "afd7e5f0-f9ef-499e-9c02-4566750fb260",
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
          {
            "path": "c:\\online1",
            "tags": "path-tag-1",
            "interval": 30,
            "metadata": {
              "meta1": "value1"
            },
            "patterns": [
              "*.*"
            ],
            "start_date": "2022-01-01",
            "source_type": "something",
            "file_watch_mode": "file",
            "labels": [
              {
                "name": "path-label-1",
                "value": "label1-value"
              }
            ],
            "archive": {
              "path": "c:\\archive",
              "delete": {
                "wait_time": 30,
                "wait_time_unit": "Days"
              },
              "dry_run": false,
              "wait_time": 3,
              "wait_time_unit": "Days"
            }
          }
        ],
        "interval": 30,
        "start_date": "2022-07-07",
        "use_path_configuration": true
      }
    }
  },
  "by": "cloud",
  "at": "2022-07-08T19:04:34.993Z"
}

To see the latest cloud configuration and whether it's been applied by the Agent successfully or not, use the following API:

Note: In the following example API call, the configuration was not successfully applied because of a file_watch_mode that's not valid.

// GET https://api.tetrascience.com/v1/agents/{id}/configuration?type=cloud

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
{
  "id": "cf9d2576-b073-4744-9101-def5a37b32cf",
  "config": {
    "services_enabled": [
      "fileWatcher"
    ],
    "services_configuration": {
      "fileWatcher": {
        "paths": [
        {
          "path": "c:\\online1",
          "tags": "path-tag-1",
          "interval": 30,
          "metadata": {
            "meta1": "value1"
          },
          "patterns": [
            "*.*"
          ],
          "start_date": "2022-01-01",
          "source_type": "something",
          "file_watch_mode": "INVALID"
        }
      ],
      "use_path_configuration": true
     }
   }
 },
 "by": "cloud",
 "at": "2022-07-08T19:15:43.246Z",
 "commandStatus": "FAILURE",
 "commandId": "cf9d2576-b073-4744-9101-def5a37b32cf"
}

To see the Agent's configuration history, use the following API:

Note: Responses are listed from most to least recent. In the following example, the earlier configuration command failed because paths weren't valid, and the later one succeeded.

// GET https://api.tetrascience.com/v1/agents/{id}/configuration/list

// Headers:
x-org-slug: <your-org-slug>
ts-auth-token: <your-auth-token>
  
// Response body:
{
  "page": 1,
  "size": 10,
  "hasNext": false,
  "hits": [
    {
      "id": "<command-id>",
      "config": {
        "services_enabled": [
          "fileWatcher"
        ],
        "services_configuration": {
          "fileWatcher": {
            "paths": [
              {
                "path": "c:\\test-path-1",
                "tags": "path-tag-1",
                "interval": 30,
                "metadata": {
                  "meta1": "value1"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              },
              {
                "path": "c:\\test-path-2",
                "tags": "path-tag-2",
                "interval": 30,
                "metadata": {
                  "meta2": "value2"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              }
            ],
            "interval": 30,
            "start_date": "2022-05-16",
            "use_path_configuration": true
          }
        }
      },
      "by": "cloud",
      "at": "2022-05-18T17:53:05.033Z",
      "commandId": "8819af94-b9ac-4ea9-9540-97328ffae96d",
      "commandStatus": "SUCCESS"
    },
    {
      "id": "f8c11ec3-fbfb-45e5-a358-b23de23714df",
      "config": {
        "services_enabled": [
          "fileWatcher"
        ],
        "services_configuration": {
          "fileWatcher": {
            "paths": [
              {
                "path": "c:\\<your-test-path-1>",
                "tags": "path-tag-1",
                "interval": 30,
                "metadata": {
                  "meta1": "value1"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              },
              {
                "path": "c:\\<your-test-path-2>",
                "tags": "path-tag-2",
                "interval": 30,
                "metadata": {
                  "meta2": "value2"
                },
                "patterns": [
                  "*"
                ],
                "start_date": "2022-01-01",
                "source_type": "unknown",
                "file_watch_mode": "file"
              }
            ],
            "use_path_configuration": true
          }
        }
      },
      "by": "cloud",
      "at": "2022-05-18T17:52:27.487Z",
      "commandId": "f8c11ec3-fbfb-45e5-a358-b23de23714df",
      "commandStatus": "FAILURE"
    }
  ]
}

Commands Pending, Failure, and Expiration

When a configuration command is waiting to be received and applied by an Agent, the Agents UI shows a CONFIGURATION PENDING status.

If a configuration command fails to apply (for example, because it has a configuration that isn't valid), the Agents UI shows a CONFIGURATION FAILED: status.

If the Agent doesn't receive a command within four minutes, the Agent returns the following error message: Configuration has failed due to no response for 4m.

To troubleshoot the error, first make sure that the Receive Command setting is turned on in the Agent UI. For instructions, see the Install the FLA on the Target Machine and Configure it to Receive Commands section on this page.

If the Receive Command setting is set to Yes and you still receive the error, submit a support ticket to TetraScience. Before submitting a support ticket, make sure that you collect the local logs for the session at /logs/nxlog/ and/or /logs/archive/backup.

When the configuration is applied to the Agent successfully, the Agent UI shows a CONFIGURATION APPLIED: status.

Cloud Configuration superseded by local

Because the Agent can be configured locally as well as by these commands, it may receive a cloud command that's already been superseded by a later local configuration. When that happens, the Agent will ignore the superseded configuration, and it will be considered successful.

Configuration via TDP UI

The APIs listed above are the preferred way to configure your Agents, because they support repeatability and automation. You can also configure through the TDP UI by doing the following:

  1. Sign in to the TDP.
  2. In the left navigation pane, select the hamburger menu icon.
  3. Choose Data Sources. Then, choose Agents. The Agents page appears.
  4. On the Agents page, select the name of the agent that you want to configure. A menu appears on the right.
  5. Choose Configure. The Configure page opens.
  6. On the Configure page, set the Path Configurations and File Watcher settings. For more information, see the Tetra File-Log Agent Installation Guide.
  7. Choose Apply.
    Add a comment. Markdown allowed.

Updated 7 months ago