Tetra File-Log Agent Installation Guide (Version 4.1.x - 4.2.x)

This installation guide is for Tetra File-Log Agent version 4.1.x. and 4.2.x.
This page describes how to:

• Install the Tetra File-Log Agent
• Configure the Tetra File-Log Agent
• Configure the FileWatcher Tetra File-Log Agent
• Start and Stop the Tetra File-Log Agent
• Monitor Tetra File-Log Agent Progress and Metrics

📘

NOTE

As of the Tetra File-Log Agent version 4.1.x, configuration changes, as well as when the agent is stopped and started are now recorded in the User Audit Trail.

📘

NOTE:

The File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NFTS, do not make any directory you want to watch case-sensitive.

Install the Tetra File-Log Agent

🚧

Verify File-Log Agent Requirements

Before you install the File-Log Agent, verify that you meet the requirements.

TetraScience provides a Windows Installation Package (.msi) file. To access the server and run the installation package, you must log in as an Administrator.

To install the Tetra File-Log Agent:

1. Double-click the installation package file to start the installation wizard.

2. Click Next > and follow the prompts. The default Tetra File-Log Agent installation path is: C:\Tetrascience\Tetrascience.Agent.File-Log.v4.1.0 (if you are installing version 4.1.x) or C:\Tetrascience\Tetrascience.Agent.File-Log.v4.2.0 (if you are installing version 4.2.x).
3. When you complete the Wizard, click Close to exit.
4. After the Tetra File-Log Agent is successfully installed on the server, you can access its Windows-based Management Console under the TetraScience folder from the Windows Programs.

Configure the Tetra File-Log Agent

After you have successfully installed the Tetra File-Log Agent, use the Windows-based Management Console to continue the Agent configuration.

The Management Console is divided into four sections:

• Windows Scheduled Task
• File-Log Group User
• Agent Configuration
• Service Settings

Section 1: Set a Windows Scheduled Task

You can increase the Agent's reliability by ensuring that the windows service is always running without having to manually check.

1. Select Yes to enable the Windows Scheduled Task to check Agent Service Status option on a daily basis. The Windows Task that you create runs under the LocalSystem account.
2. Specify the time in the Run Time field when you want the scheduled Windows task to check on the status of the Tetra File-Log Agent windows service.

How it works

• If the Agent windows service is stopped, then the scheduled Windows task automatically restarts the Agent windows service.
• If the Agent windows service is running, nothing will happen.
• If you manually stop the Tetra File-Log Agent, then the scheduled Windows task is removed.

Section 2: File-Log Group User

The File-Log Group User is the service account that runs the Tetra File-Log Agent.

• If you leave the User Name field blank, then the Tetra File-Log Agent runs using the Windows predefined LocalSystem account.

• If the Tetra File-Log Agent monitors the network's shared folders, then we suggest that you provide a dedicated service account that:
  • Has at least read and List folder content permission of the network shared folders (including the subfolders and files contained in those folders).
  • Is part of the local user group of the host server.
  • Has no account expiration date. If an account expires, then the Tetra File-Log Agent cannot extract data and data acquisition is interrupted.
  • Requires the log on as service permission.

When you enter the File-Log Group User Name and Password, the Agent validates the account immediately. If the user name and password are correct, then the Tetra File-Log Agent displays a Valid status next to the field. If the user account does not have the proper permissions, then the word Invalid displays.

Section 3: Agent Configuration

Use this section to configure the Agent connection to the Tetra Data Platform. TetraScience provides several options to connect the Tetra Data Platform. Click this link to select the best option for your company's needs.

Area A: S3 Direct Upload

To have the Tetra File-Log Agent directly upload files to the AWS S3 bucket bypassing GDC or API Agent, set the S3 Direct Upload option to Yes. When you set this option to Yes, you can upload a maximum file size of 5 TB. No is the default setting.

To learn more about connectors see the following topics:

• Selecting the Connector Type to Use You can add an L7 Proxy Data Connector if the Agent host machine doesn't have internet access, but it is not necessary.
• GDC Connector.
• "No Connector" Agent.

📘

SQLite Database File Backups

To enable the Tetra File-Log Agent to automatically perform regular backups of the SQLite database file, set the S3 Direct Upload option to Yes. When you enable this option, the SQLite database file (which stores agent configuration data) is uploaded to the backup bucket in the TDP. If a Tetra File-Log Agent failure occurs, you can restore the database file from the backup bucket and continue processing.

If you do not use the S3 Direct Upload option, then we recommend that you create periodic backups of the Tetra Agent database, which is typically stored in the installation folder under C:\TetraScience<agent>\Bin\Database. In the event of a failure (for example, if the host server drive is lost), this would enable the data extraction to continue from where it left off without having to re-upload all of the data.

Area B: Receive Commands/Enable Queue

The Tetra File-Log Agent can receive commands from the Tetra Data Platform when setting the Receive Commands option to Yes. (If you do this, you'll need to enable the queue in TDP via the UI or via an API endpoint command.) When you set this option to Yes, the Tetra File-Log Agent keeps polling the queue messages sent from the Tetra Data Platform. The messages provides instructions to the Tetra File-Log Agent. In version 4.0.0 and above, the TDP can push the folder path configuration. In version 4.2.0 and above, the TDP can push a file to File-Log Agent. No is the default setting.

📘

TDP Commands and Queue Security

Although queues enable TDP to send commands to the Tetra Agent, the command execution request process is secured.

• TDP does not send actual commands, but rather messages that request commands for execution.
• The Tetra Agent has its own queue to receive messages from TDP and is not shared by any other agent.
• When queues are enabled, the Tetra Agent proactively reads the queue for messages containing command execution requests and determines whether to initiate the command execution. You can control whether to enable local command execution by toggling the Receive Commands option for the Tetra Agent.
• The Tetra Agent can only execute certain a limited set of pre-built commands. As of Tetra File-Log Agent version 4.0, the only available command for execution is to update its configuration based on cloud configuration. For details about cloud configuration, click [here.] (https://developers.tetrascience.com/docs/centralized-configuration-of-tetra-data-platform-components)

Area C: Advanced Settings

1. Click Advanced Settings to open the Agent Configuration Advanced Settings pane.

You can specify the time intervals for these fields (default value for each field is 30 seconds):

• Data Connection status check every - Indicates how often the software checks the status of the connection (heartbeat) between the Tetra Data Platform and the Tetra File-Log Agent. If the Tetra Data Platform does not receive a message indicating that the Tetra File-Log Agent is "alive" for more than 5 minutes, it assumes that the Agent is offline.
• File Upload Job runs every - Indicates how often to upload files to the TDP. The difference between this field and the File Change Interval field (located in the Service Settings section) is that the File Change Interval field indicates the minimum amount of time that the file must remain unchanged before uploading. For example, if you set the File Upload Job runs every field to five minutes, and you set the File Change Interval field to one minute, then the file must have remained unchanged for at least one minute before uploading the file to the TDP. Reducing the time interval can improve the Agent throughput, however, it will increase the CPU usage as tradeoff.
• Agent log files upload every - Indicates how often to upload log files to the TDP.

Area D: Agent ID and Connection URL

Before you set up a Tetra File-LogAgent from GDC or API Agent, you must enter:

• (Required) An Agent ID. The Agent ID (a UUID) is used to connect the Tetra Data Platform. You can retrieve Agent ID from the Tetra Data Platform when you set set up GDC or "No Connector".
• A full connection URL from GDC or "No Connector" (as shown in these URL examples):
  • GDC URL is http://10.100.1.1:8888/generic-connector/v1/agent
  • API URL is https://api.tetrascience.com/v1/data-acquisition/agent

📘

Verify the Agent ID and Connection URL

• Agent ID is required. The Connection URL is required if you are uploading RAW files to the Tetra Data Platform; optional if you are not.
• Before you use the Agent ID and URL with the Tetra Data Platform, we strongly suggest that you verify both with your TetraScience Customer Success Team.

If you use the "No Connector" agent, then you must:

1. When using the JWT token, the Org Slug field is required. Enter the Org Slug to attach it to the header.
2. Enter the JWT token in the Authorization field to attach it to the header. To learn how to get the JWT token, see No Connector Agents.
3. Click Add/Edit to open a dialog and enter the JWT token.
4. Click Save to close the dialog and encrypt and save the JWT token. The Agent validates the connection immediately and displays the updated connection status next to the Connection URL field.

Configure Tetra File-Log Agent FileWatcher Service

📘

NOTE:

The File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NFTS, do not make any directory you want to watch case-sensitive.

The Tetra File-Log Agent has a FileWatcher Service that provides the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags.

❗️

Functionality Change Introduced in Software Version 3.4

When the Tetra File-Log Agent sends data to the TDP, the data is stored in the AWS S3. The Tetra File-Log Agent sends data from the Windows file storage (which is not case sensitive) and uploads it to the S3 storage, which is case sensitive. To avoid unintentional duplication of data based on case, the Tetra File-Log Agent normalizes all file path information to lower case. This ensures a behavior similar to one that users would expect from their Windows file storage.

Configure the FileWatcher Service

Use the Tetra File-Log Agent FileWatcher Service to provide the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags. When you set the Receive Commands option to Yes, in the Agent Configuration section, the Tetra File-Log Agent will be able to receive the path configuration sent from Tetra Data Platform.

You can also add paths in the Tetra File-Log Agent. To do this, complete the following steps.

1. Click the New Path button.
2. Enter the following in the table.

• Change Interval (seconds)
• Start Date
• Path (required)
• Source Type - (required)
• Patterns (required)
• Metadata
• Tags
• File Watch Mode (required)

3. You can specify which Path folder to monitor and the glob pattern used to select the files or folders.
   You can add, edit, or delete a path. To create a new path, click New Path, or double-click the existing path in the table open the Watcher Path Editor pane:

📘

NOTE

Glob patterns are a set of wildcard characters. For more information on glob patterns, including examples, see this topic.

Additional File Path Specification

📘

Path Specification

• The Tetra File-Log Agent supports absolute paths for local folders and UNC paths for Network Folders. The Agent doesn't support mapped drives or relative paths.
  • The folder path doesn't support umlaut or diaresis characters, e.g Ö, or ö
  • You can specify up to 500 paths in FileWatcher.

4. After you complete configuring the path, you can save the changes and start the Tetra File-Log Agent. After the Tetra File-Log Agent starts, the path configuration changed in the local is uploaded to the S3 bucket. A copy of that path configuration will be available on the Tetra Data Platform. You can then modify the path and push a new path configuration to the Tetra File-Log Agent.

📘

Metadata and Tags

To learn more about metadata and tags, including which characters you can use, see this topic.

5. You can click Re-upload to regenerate and re-upload the result files for a file path. From the dialog, set the Reupload Start Date and Reupload End Date. If you leave the Reupload End Date blank, then the Agent uses the current date as the end date. Click OK to save the date range.

As of v3.5.0, the Agent verifies the folder path and the user permission when iterating the folder paths during scanning. The Agent displays an icon in front of every folder path to indicate whether the folder path is valid. For example, these two folders are invalid:

• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder3\
• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder4\

The Agent keeps checking the folder paths at each scan interval, and updates the status if there are any changes.

Start and Stop the Agent

The top section of the Management Console indicates the Agent's processing using these statuses:

• Agent Service Not Installed
• Running
• Stopped

• To save the Tetra File-Log Agent configuration changes, click Save. Configuration settings are saved in the database.

• To start the Tetra File-Log Agent with the new configuration settings, click Start.

• For every path, the Agent uploads the newest files first. Files from other paths won’t be blocked by a single path.

  • When the Tetra File-Log Agent starts, the status indicator shows Running for an Agent that is successfully up and running. When the Tetra File-Log Agent starts, the Stop button is enabled. To stop the Tetra File-Log Agent, click Stop.

Monitor Tetra File-Log Agent Progress and Metrics

To review the Tetra File-Log Agent's run time progress and host server's system metrics, select Summary to open the Summary dashboard:

Processing Summary

You can review how many File-Log result files were detected, generated, and uploaded. If a failure occurs, the numbers display here.

The Processing Summary dashboard displays this information:

• Scanned files/folders - Number of files or folders scanned by the agent
• Generated files - Number of files generated
• Failed to generate - Number of files that failed to generate
• To upload - A pending state. When the Tetra File-Log Agent stops, the To upload state is reset to zero. Any file that was not completely uploaded is removed from the Tetra File-Log Agent's database and resets the counter. When the Tetra File-Log Agent restarts, it calculates how may files it needs to upload and starts from the beginning instead of resuming from where the file upload left off.
• Uploaded - Number of files that have been uploaded to the agent successfully
• Failed to upload - Number of files that failed to upload to the agent

Additional File Properties Uploaded

File Properties, such as file creation date, last modified date and file size are sent with files when uploading to TDP. These become part of the S3 object. These properties provide file context. See the table below for details.

Error Messages

This section displays any detailed error messages, such as when a file has failed to upload or has been copied to the temp folder in the Agent host machine.

System Messages

This section displays the host server system metrics and messages every minute and provides these metrics:

• Data Connection (connection status between the Agent and the Tetra Data Platform)
• Available Disk Space (in GB)
• Disk Usage (as a % of the total)
• CPU Usage (as a % of the total)
• Memory Usage (in MB)

For help with any issues, review the Troubleshooting Issues and Solutions section.

To learn more about the Tetra File-Log Agent, please review the FAQ section or please contact the TetraScience Customer Success team.