Tetra File-Log Agent Installation Guide (Version 4.0.x)

This installation guide is for Tetra File-Log Agent version 4.0.x.
This page describes how to:

  • Install the Tetra File-Log Agent
  • Configure the Tetra File-Log Agent
  • Configure the Tetra File-Log Agent Services:
    • File Watcher
    • Log Watcher
  • Start and Stop the Tetra File-Log Agent
  • Monitor Tetra File-Log Agent Progress and Metrics

📘

NOTE:

The File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NFTS, do not make any directory you want to watch case-sensitive.

Install the Tetra File-Log Agent

🚧

Verify File-Log Agent Requirements

Before you install the File-Log Agent, verify that you meet the requirements.

TetraScience provides a Windows Installation Package (.msi) file. To access the server and run the installation package, you must log in as an Administrator.

To install the Tetra File-Log Agent:

  1. Double-click the installation package file to start the installation wizard.
462

Tetra File-Log Agent Installation Wizard

  1. Click Next > and follow the prompts. The default Tetra File-Log Agent installation path is: C:\Tetrascience\Tetrascience.Agent.File-Log.v4.0.1.
  2. When you complete the Wizard, click Close to exit.
  3. After the Tetra File-Log Agent is successfully installed on the server, you can access its Windows-based Management Console under the TetraScience folder from the Windows Programs.

Configure the Tetra File-Log Agent

After you have successfully installed the Tetra File-Log Agent, use the Windows-based Management Console to continue the Agent configuration.

2412

Management Console

The Management Console is divided into four sections:

  • Windows Scheduled Task
  • File-Log Group User
  • Agent Configuration
  • Service Settings

Section 1: Set a Windows Scheduled Task

You can increase the Agent's reliability by ensuring that the windows service is always running without having to manually check.

  1. Select Yes to enable the Windows Scheduled Task to check Agent Service Status option on a daily basis. The Windows Task that you create runs under the LocalSystem account.
  2. Specify the time in the Run Time field when you want the scheduled Windows task to check on the status of the Tetra File-Log Agent windows service.

How it works

  • If the Agent windows service is stopped, then the scheduled Windows task automatically restarts the Agent windows service.
  • If the Agent windows service is running, nothing will happen.
  • If you manually stop the Tetra File-Log Agent, then the scheduled Windows task is removed.
630

Windows Scheduled Task

Section 2: File-Log Group User

The File-Log Group User is the service account that runs the Tetra File-Log Agent.

  • If you leave the User Name field blank, then the Tetra File-Log Agent runs using the Windows predefined LocalSystem account.
  • If the Tetra File-Log Agent monitors the network's shared folders, then we suggest that you provide a dedicated service account that:
    • Has at least read and List folder content permission of the network shared folders (including the subfolders and files contained in those folders).
    • Is part of the local user group of the host server.
    • Has no account expiration date. If an account expires, then the Tetra File-Log Agent cannot extract data and data acquisition is interrupted.
    • Requires the log on as service permission.

When the you enter the File-Log Group User Name and Password, the Agent validates the account immediately. If the user name and password are correct, then the Tetra File-Log Agent displays a Valid status next to the field. If the user account does not have the proper permissions, then error messages display.

820

Account validation

Section 3: Agent Configuration

Use this section to configure the Agent connection to the Tetra Data Platform. TetraScience provides several options to connect the Tetra Data Platform. Click this link to select the best option for your company's needs.

Area A: S3 Direct Upload

To have the Tetra File-Log Agent directly upload files to the AWS S3 bucket bypassing GDC or API, set the S3 Direct Upload option to Yes. When you set this option to Yes, you can upload a maximum file size of 2 TB. No is the default setting.

  • If you use the S3 Direct Upload option with GDC, you must add an L7 Proxy Data Connector in the same Data Hub where you set up the GDC. Additionally, the port of the L7 Proxy Data Connector is open. To learn more about GDC, see this link.
  • To learn more about No Connector Agents, see this link.

📘

SQLite Database File Backups

To enable the Tetra File-Log Agent to automatically perform regular backups of the SQLite database file, set the S3 Direct Upload option to Yes. When you enable this option, the SQLite database file (which stores agent configuration data) is uploaded to the backup bucket in the Data Lake. If a Tetra File-Log Agent failure occurs, you can restore the database file from the backup bucket and continue processing.

If you do not use not use the S3 Direct Upload option, then we recommend that you create periodic backups of the Tetra Agent database, which is typically stored in the installation folder under C:\TetraScience<agent>\Bin\Database. In the event of a failure (for example, if the host server drive is lost), this would enable the data extraction to continue from where it left off without having to re-upload all of the data.

Area B: Receive Commands

To enable the Tetra File-Log Agent to receive commands from the Tetra Data Platform, set the Receive Commands option to Yes. When you set this option to Yes, the Tetra File-Log Agent keeps polling the queue messages sent from the Tetra Data Platform. The messages provides instructions to the Tetra File-Log Agent. In version 4.0.0, the TDP can push the folder path configuration. No is the default setting.

📘

TDP Commands and Queue Security

Although queues enable TDP to send commands to the Tetra Agent, the command execution request process is secured.

  • TDP does not send actual commands, but rather messages that request commands for execution.
  • The Tetra Agent has its own queue to receive messages from TDP and is not shared by any other agent.
  • When queues are enabled, the Tetra Agent proactively reads the queue for messages containing command execution requests and determines whether to initiate the command execution. You can control whether to enable local command execution by toggling the Receive Commands option for the Tetra Agent.
  • The Tetra Agent can only execute certain a limit set of pre-built commands. As of Tetra File-Log Agent version 4.0, the only available command for execution is to update its configuration based on cloud configuration. For more information, see Cloud Configuration of Tetra Agents.

Area C: Advanced Settings

  1. Click Advanced Settings to open the Agent Configuration Advanced Settings pane.
485

Agent Configuration Advanced Settings

You can specify the time intervals for these fields (default value for each field is 30 seconds):

  • Data Connection status check every - Indicates how often the software checks the status of the connection (heartbeat) between the Tetra Data Platform and the Tetra File-Log Agent. If the Tetra Data Platform does not receive a message indicating that the Tetra File-Log Agent is "alive" for more than 5 minutes, it assumes that the Agent is offline.
  • File Upload Job runs every - Indicates how often to upload files to the Data Lake. The difference between this field and the File Change Interval field (located in the Service Settings section) is that the File Change Interval field indicates the minimum amount of time that the file must remain unchanged before uploading. For example, if you set the File Upload Job runs every field to five minutes, and you set the File Change Interval field to one minute, then the file must have remained unchanged for at least one minute before uploading the file to the Data Lake.
  • Agent log files upload every - Indicates how often to upload log files to the Data Lake.

📘

Software Version 3.4 Update

To improve readability, the After a watched file is modified, wait for it to stop changing field was renamed to File Upload Job runs every, and the Upload Log File field was renamed to Agent log files uploads every.

Area D: Agent ID and Connection URL

Before you set up a Tetra File-LogAgent from GDC or without a connector, you must enter:

  • (Required) An Agent ID. The Agent ID (a UUID) is used to connect the Tetra Data Platform. You can retrieve Agent ID from the Tetra Data Platform when you set set up GDC or Agent.
  • A full connection URL from GDC or agent (as shown in these URL examples):
    • GDC URL is http://10.100.1.1:8888/generic-connector/v1/agent
    • No Connector API URL is https://api.tetrascience.com/v1/data-acquisition/agent

📘

Verify the Agent ID and Connection URL

  • Agent ID is required. The Connection URL is required if you are uploading RAW files to the Tetra Data Platform; optional if you are not.
  • Before you use the Agent ID and URL with the Tetra Data Platform, we strongly suggest that you verify both with your TetraScience Customer Success Team.

If you use an Agent without a Connector, then you must:

  1. When using the JWT token, the Org Slug field is required. Enter the Org Slug to attach it to the header.
  2. Enter the JWT token in the Authorization field to attach it to the header. To learn how to get the JWT token, see No Connector Agents.
  3. Click Add/Edit to open a dialog and enter the JWT token.
  4. Click Save to close the dialog and encrypt and save the JWT token. The Agent validates the connection immediately and displays the updated connection status next to the Connection URL field.

Configure Tetra File-Log Agent Services

📘

NOTE:

The File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NFTS, do not make any directory you want to watch case-sensitive.

The Tetra File-Log Agent provides two services:

  • FileWatcher Service - Use to provide the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags.
  • LogWatcher Service - Use to provide the path to monitor the log or folder, log source type, glob pattern, file metadata, and tags.

Select Yes to enable one service, or both of these services.

❗️

Functionality Change Introduced in Software Version 3.4

When the Tetra File-Log Agent sends data to the TDP, the data is stored in the AWS S3. The Tetra File-Log Agent sends data from the Windows file storage (which is not case sensitive) and uploads it to the S3 storage, which is case sensitive. To avoid unintentional duplication of data based on case, the Tetra File-Log Agent normalizes all file path information to lower case. This ensures a behavior similar to one that users would expect from their Windows file storage.

Configure the FileWatcher Service

Use the Tetra File-Log Agent FileWatcher Service to provide the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags.
To enable and configure the FileWatcher Service:

  1. When you select Yes for the Enable FileWatcher Service field, the FileWatcher Service information displays in the Service Settings section at the bottom of the Management Console pane:
1806

File Watcher Service

📘

File Watcher Paths

You can specify up to 500 paths in File Watcher.

  1. Set the File Change Interval (Seconds) field to indicate the time span for the Tetra File-Log Agent to determine if the file should be uploaded by comparing the file's Last Write Time.
  2. Set the Start Date to have the Tetra File-Log Agent ignore files/folders older than the set date.
  3. Select Yes for Use Path Configuration to have the Tetra File-Log Agent apply the File Change Interval and Start Date settings for all of the file paths, or individual file paths (Feature is available for version 3.3.1 and later).
  4. Click Local Config or Cloud Config to indicate the source of the path configuration sent from the Tetra Data Platform.

When you set the Receive Commands option to Yes, the Tetra File-Log Agent can receive the path configuration sent from Tetra Data Platform. After the path configuration is imported successfully, the path configuration source should switch from Local Config back to Cloud Config. To toggle the path configuration source from Cloud Config back to Local Config, click Local Config. A modal dialog displays for your confirmation.

  1. After you complete configuring the path, you can save the changes and start the Tetra File-Log Agent. After the Tetra File-Log Agent starts, the path configuration changed in the local is uploaded to the S3 bucket. A copy of that path will be available on the Tetra Data Platform. You can then modify the path and push a new path configuration to the Tetra File-Log Agent.

  2. The table specifies the Tetra File-Log Agent path containing these fields:

  • Path (required)
  • Source Type - Uploaded from the TDP
  • Patterns (required)
  • Metadata - Uploaded from the TDP
  • Tags - Uploaded from the TDP
  • File Watch Mode (required)

You can specify which Path folder to monitor and the glob pattern used to select the files or folders.
You can add, edit, or delete a path. To create a new path, click New Path, or double-click the existing path in the table open the Watcher Path Editor pane:

829

Watcher Path Editor

If you selected Yes for the Use Path Configuration option is turned, then you can specify the File Change Interval and Start Date for that file path.

📘

Metadata and Tags

To learn more about metadata and tags, including which characters you can use, see this topic.

  1. You can click Re-upload to regenerate and re-upload the result files for a file path. From the dialog, set the Reupload Start Date and Reupload End Date. If you leave the Reupload End Date blank, then the Agent uses the current date as the end date. Click OK to save the date range.
870

Reupload result files within specified date range

As of v3.5.0, the Agent verifies the folder path and the user permission when iterating the folder paths. It regenerates and re-uploads any result files that have their Result Creation Date within the specified time range. The Agent also displays an icon in front of every folder path to indicate whether the folder path is valid. For example, these two folders are invalid:

  • \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder3\
  • \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder4\
953

Invalid folder paths

The Agent keeps checking the folder paths at each interval, and updates the status if there are any changes.

Configure the LogWatcher Service

Use the Tetra File-Log Agent LogWatcher Service to provide the path to monitor the log or folder, log source type, glob pattern, file metadata, and tags.
To enable and configure the LogWatcher Service:

  1. When you select Yes for the Enable LogWatcher Service field, the LogWatcher Service information displays in the Service Settings section at the bottom of the Management Console pane:
883

Log Watcher Service

  1. Set the Scan Interval to File Change (Seconds) field to indicate how often the LogWatcher Service gathers the log's metadata, determines any changes, and then generates new content if the logs have been updated.
  2. Select the LogWatcher Output Folder to where you want to temporarily store the newly generated content before uploading. You can use the default (C:\Temp) location, or provide a different folder. If the folder does not exist, then the Agent creates the folder.
  3. Set the Start Date to have the Tetra File-Log Agent ignore logs/folders older than the set date.
  4. The table specifies the Tetra File-Log Agent path containing these fields:
  • Path (required)
  • Source Type - Uploaded from the TDP.
  • Patterns (required)
  • Metadata - Uploaded from the TDP
  • Tags - Uploaded from the TDP
  • Max Batch Size

You can specify which Path folder to monitor and the glob pattern used to select the files or folders.
You can add, edit, or delete a path. To create a new path, click New Path, or double-click the existing path in the table open the Watcher Path Editor pane:

619

Watcher Path Editor

If you leave the Source Type field empty, then the Agent will use unknown as the default source.

As of v3.0.0, if you update the Source Type, Metadata, or Tags of a folder path that has already processed, then the Agent will reupload those files to the TetraScience Data Platform by attaching the updated Source Type, Metadata, or Tags values. For details, see FAQ - 7. How the Agent behaves if Source Type, Metadata or Tag is changed?

Start and Stop the Agent

The top section of the Management Console indicates the Agent's processing using these statuses:

  • Agent Service Not Installed
  • Running
  • Stopped
922

Agent Processing Status

  • To save the Tetra File-Log Agent configuration changes, click Save.
  • To start the Tetra File-Log Agent with the new configuration settings, click Start.
    Before the Tetra File-Log Agent starts, the Agent validates if the File-Log Group User account has the correct read privilege of the folder paths that were specified. If not, then a popup displays and indicates which user account path is invalid and has access issues:
476

Invalid user account

Click OK to close popup and fix the user account issue. After you fix the errors, you can start the Tetra File-Log Agent again by clicking Start.

  • When the Tetra File-Log Agent starts, the status indicator shows Running for an Agent that is successfully up and running. When the Tetra File-Log Agent starts, the Stop button is enabled. To stop the Tetra File-Log Agent, click Stop.
617

Agent is running successfully

Monitor Tetra File-Log Agent Progress and Metrics

To review the Tetra File-Log Agent's run time progress and host server's system metrics, select Summary to open the Summary dashboard:

602

File-Log Agent Summary Dashboard

Processing Summary

You can select to review processing summaries for each of the Tetra File-Log Agent's services:

  • File Watcher
  • Log Watcher

For each service, you can review how many File-Log result files were detected, generated, and uploaded. If any failure occurs, the numbers display here.

The Processing Summary dashboard displays this information:

  • Scanned files/folders - Number of files or folders scanned by the agent
  • Generated files - Number of files generated
  • Failed to generate - Number of files that failed to generate
  • To upload - A pending state. When the Tetra File-Log Agent stops, the To upload state is reset to zero. Any file that was not completely uploaded is removed from the Tetra File-Log Agent's database and resets the counter. When the Tetra File-Log Agent restarts, it calculates how may files it needs to upload and starts from the beginning instead of resuming from where the file upload left off.
  • Uploaded - Number of files that have been uploaded to the agent successfully
  • Failed to upload - Number of files that failed to upload to the agent

Error Messages

This section displays any detailed error messages.

System Messages

This section displays the host server system metrics and messages every minute and provides these metrics:

  • Data Connection (connection status between the Agent and the Tetra Data Platform)
  • Available Disk Space (in GB)
  • Disk Usage (as a % of the total)
  • CPU Usage (as a % of the total)
  • Memory Usage (in MB)

For help with any issues, review the Troubleshooting Issues and Solutions section.

To learn more about the Tetra File-Log Agent, please review the FAQ section or please contact the TetraScience Customer Success team.