This guide shows how IT Administrators can install and configure Tetra File-Log Agent versions 4.4.x.

To set up the Tetra File-Log Agent, do the following:

1. Verify the prerequisites:
   • Make sure that your host server meets the Agent's [hardware and software requirements
2. Install and configure the Agent:
   • Determine the upload method and data connection type
   • Create a new Agent in the TDP
   • Install the Agent
   • Configure the Agent
3. Verify the installation and monitor the Agent:
   • Start the Agent
   • Monitor progress, metrics, and errors

For instructions, see the procedures in each of the following sections.

📘

NOTE:

The File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NTFS, do not make any directory you want to watch case-sensitive.

Prerequisites

Hardware and Software Requirements

To install and run Tetra File-Log Agent v4.4.x, the following hardware and software is required:

• One of the following Microsoft Windows versions:
  • Windows 7 Service Pack 1 (SP1) or later, with TLS 1.2 enabled
  • Windows Server 2008 R2 SP1 or later, with TLS 1.2 enabled
  • Windows Server 2022
• .NET Framework 4.8 (Download from Microsoft)
• 8 GB RAM minimum (16 GB is recommended)
• 4 core CPU minimum (Intel Xeon 2.5.0 GHz or equivalent)
• Verify that the available storage space in host server's Group User temporary folder is more than the maximum file size that the Agent will upload. (The Agent copies the source file to the Group User temporary folder before uploading it to the TDP.)

Networking Requirements

Make sure that you verify the following networking requirements are also met:

• The Windows server hosting the Agent must have network access (SMB over port 445, TCP, and UDP) to any computers whose shared folders need to be monitored as well as the Group User Account that has the necessary access
• (For Tetra Hub connections only) If you select the Enable S3 Direct Upload or Receive Commands option when you configure the Tetra File-Log Agent, then you must add the following endpoints to your organization's allow list before you can use a Tetra Hub:
  • For required Data Hub endpoints, see Tetra Data Hub Allow List Endpoints.
  • For required Hub endpoints, see Tetra Hub Allow List Endpoints.

📘

NOTE

When using the TetraScience API, this access must be direct. When using a Tetra Data Hub with a Generic Data Connector (GDC), this access can be provided by installing an L7 Proxy Connector in the same Data Hub as the GDC. In this case, the Agent also requires access to the port selected when configuring the L7 Proxy (for example, https://192.168.1.1:3129).

Install and Configure a Tetra File-Log Agent

To install and configure a Tetra File-Log Agent v4.4.x, do the following.

Step 1: Determine the Upload Method and Data Connection Type

The Tetra File-Log Agent generates RAW files that are then uploaded to the Tetra Data Platform (TDP) through a variety of potential upload methods and connection types. Before you install the Tetra File-Log Agent, you must set up one of these data connections when you create the Agent in the TDP.

To determine which upload method and data connection type is required for your use case, see Agent Deployment Options.

Step 2: Create a New Tetra File-Log Agent in the TDP

To create a new Tetra File-Log Agent, follow the instructions in Create a New Agent. For Agent Type, make sure that you select File-Log Agent.

Step 3: Install the Tetra File-Log Agent

TetraScience provides a Windows Installation Package (.msi) file to install the Tetra File-Log Agent on the host server.

To get the installation package and install the Agent, do the following:

1. Get the link to the .msi installation package by contacting your TetraScience Customer Success Manager (CSM) or account executive.
2. After you have the link to the installation package, download the Agent to a host Windows machine that meets the Agent's hardware, software, and networking requirements.

🚧

IMPORTANT

To access the server and run the installation package, you must sign in as an Administrator.

3. Start the installation wizard by double-clicking the Tetra Agent installation file. The installation wizard guides you through the prompts. The following is the default Tetra File-Log Agent installation path: C:\Tetrascience\Tetrascience.Agent.File-Log.v4.4.0

4. After the Tetra File-Log Agent is installed, open the TetraScience folder within Windows Programs. Then choose Tetra File-Log Agent to open the Agent Management Console.

📘

NOTE

To restrict access to the local Tetra Agent Management Console, you can edit the Agent’s installation folder’s permissions to grant least privilege access. For more information, see Restrict Access to Agent Installation.

Step 4: Configure the Tetra File-Log Agent

To configure the Tetra File-Log Agent after it's installed on the host server, do the following:

1. In the Agent Management Console, select the hamburger menu icon in the left navigation menu. Then, choose Configuration. The Agent configuration page appears.

2. Configure the Agent by filling out the fields in the following sections on the Agent Configuration page:
   • Section 1: Agent Status
   • Section 2: Windows Task Scheduler
   • Section 3: File-Log Group User (Windows User service account)
   • Section 4: Agent Configuration
   • Section 5: Service Settings

Configure the Agent Status

The Agent Status section indicates the current state of the Agent's processing status, which can be any of the following:

• Agent Service Not Installed
• Stopped
• Running

To change the Agent's status, do the following:

• To save the Agent configuration, choose Save.
• To start the Agent after configuring new settings, choose Start. When the Agent starts, the status indicator shows Running.
• To stop a running Agent, choose Stop. You can select the Stop button only if the Agent is running.

Automate Status Checks by Using Windows Task Scheduler

To avoid needing to check the Agent’s service status manually, it’s recommended that you create a Windows task to perform daily status checks.

To create a Windows task, do the following:

1. Under Windows Scheduled Task to check Agent Service Status, choose Yes.
2. For Run Time, enter the time that you want the status check to run each day. If the Agent is stopped, then the scheduled Windows task restarts the Agent service automatically. If the Agent is running, it continues to run, and the Windows task will run at the time you selected.

📘

NOTE

If you manually stop the Agent, the task is removed from Windows Task Scheduler. When you run the Agent without specifying an Agent Group User account, the task runs under the LocalSystem account.

Configure A Windows User Service Account for the Agent

The File-Log Group User is the Windows User service account that runs the Tetra File-Log Agent. To configure a user account, do the following:

1. On the local host server, add a user account for the Agent to use. For instructions, see Add a user account in the Microsoft documentation.

   📘

   NOTE

   If the Tetra File-Log Agent monitors the network's shared folders, then we suggest that you provide a dedicated service account that has the following:

   • At least read and List folder content permission of the network shared folders (including the subfolders and files contained in those folders).
   • Is part of the local user group of the host server.
   • Has no account expiration date. If an account expires, then the Tetra File-Log Agent can't extract data and data acquisition is interrupted.
   • Requires the log on as service permission.

2. For User Name, enter the Windows User's user name.
3. For Password, enter the Window User's password. The Agent validates the account immediately. If the user name and password are correct, then the Tetra File-Log Agent displays a Valid status next to the field. If the user account does not have the proper permissions, then the word Invalid displays.

📘

NOTE

If you leave the User Name field blank, then the Tetra File-Log Agent runs using the Windows predefined LocalSystem account. When the File-Log Group User is left blank the Tetra File-Log Agent can only access local folders. If you want to access remote folders you MUST provide a File-Log Group User.

Set Up the Agent Configuration

📘

NOTE

Make sure that you’ve set up a data connection before completing the following procedure. You will need the Agent ID and Connection URL that you created during the data connection setup.

To set up the Agent configuration, enter the following information in the Agent Configuration section:

🚧

IMPORTANT

• Agent ID is required.
• The connection URL is required only if you're uploading RAW files to the TDP.
• Before you use your Agent ID and connection URL, it’s strongly recommended that you verify both with your TetraScience Customer Success Team.

S3 Direct Upload

Files that are larger than 500 MB must use the S3 Direct Upload option. Activating this setting tells the Tetra File-Log Agent to directly upload files to the Amazon S3 bucket.

• Set the S3 Direct Upload option to Yes.
• When you set this option to Yes, you can upload a maximum file size of 5 TB.
• Yes is the default setting for new 4.4 installs.
• For agents getting upgraded, the existing setting for this will be preserved.

For more information, see Agent Deployment Options.

📘

SQLite Database File Backups

To enable the Tetra File-Log Agent to automatically perform regular backups of the SQLite database file, set the S3 Direct Upload option to Yes. When you enable this option, the SQLite database file (which stores agent configuration data) is uploaded to the backup bucket in the TDP. If a Tetra File-Log Agent failure occurs, you can restore the database file from the backup bucket and continue processing.

If you do not use the S3 Direct Upload option, then we recommend that you create periodic backups of the Tetra Agent database, which is typically stored in the installation folder under C:\\TetraScience\<agent>\\Bin\\Database. In the event of a failure (for example, if the host server drive is lost), this would enable the data extraction to continue from where it left off without having to re-upload all of the data.

Receive Commands/Enable Queue

The Tetra File-Log Agent can receive commands from the TDP when setting the Receive Commands option to Yes. (If you do this, you'll need to enable the queue in TDP through the UI or through an API endpoint command.) When you set this option to Yes, the Tetra File-Log Agent keeps polling the queue messages sent from the Tetra Data Platform. The messages provide instructions to the Tetra File-Log Agent. In version 4.0.0 and above, the TDP can push the folder path configuration. In version 4.2.0 and above, the TDP can push a file to File-Log Agent. No is the default setting.

📘

TDP Commands and Queue Security

Although queues enable TDP to send commands to the Tetra Agent, the command execution request process is secured.

• TDP does not send actual commands, but rather messages that request commands for execution.
• The Tetra Agent has its own queue to receive messages from TDP and is not shared by any other agent.
• When queues are enabled, the Tetra Agent proactively reads the queue for messages containing command execution requests and determines whether to initiate the command execution. You can control whether to enable local command execution by toggling the Receive Commands option for the Tetra Agent.
• The Tetra Agent can only execute a limited set of pre-built commands. As of Tetra File-Log Agent version 4.0, the only available command for execution is to update its configuration based on cloud configuration. For details about cloud configuration, click here.

Agent ID, Destination ID, and Connection URL

Before you set up a Tetra File-Log Agent, you must enter the following information in the Agent Configuration section:

• (Required) An Agent ID. The Agent ID (a UUID) is used to connect the Tetra Data Platform. You can retrieve Agent ID from the Tetra Data Platform when you set set up either a Tetra Hub or TDP (No Connector) option.
• Destination ID - When the Destination Id is entered, the Agent uploads the files to the platform using the Destination Id instead of the Agent Id. Agents sharing the same Destination Id will upload files to the same Data Lake location defined using the Destination Id instead of Agent Id, which allows files from multiple Agents to be uploaded to the same Amazon Simple Storage Service (Amazon S3) location.
• For the Connection Url field, enter the complete URL of the Tetra Hub, Generic Data Connector (GDC) attached to a Data Hub, or No Connector, based on your connection type.

📘

Verify the Agent ID and Connection URL

• Agent ID is required. The Connection URL is required if you are uploading RAW files to the Tetra Data Platform; optional if you are not.
• Before you use the Agent ID and URL with the Tetra Data Platform, we strongly suggest that you verify both with your TetraScience Customer Success Team.

📘

Destination ID

Destination ID is editable in the Agent Management Console from Tetra File-Log Agent version 4.3.1 and higher.

If you use the "No Connector" agent, then you must do the following:

1. Enter an Org Slug.
2. Enter the JWT token in the Authorization field to attach it to the header. To learn how to get the JWT token, see Configure the "TDP (No Connector)" Option.
3. Click Add/Edit to open a dialog and enter the JWT token.
4. Click Save to close the dialog and encrypt and save the JWT token. The Agent validates the connection immediately and displays the updated connection status next to the Connection URL field.

Advanced Settings

To configure Advanced Settings for the agent, do the following:

1. Choose Advanced Settings to open the Agent Configuration Advanced Settings dialog.

You can specify the time intervals for these fields (default value for each field is 30 seconds):

• Data Connection status check every: Indicates how often the software checks the status of the connection (heartbeat) between the Tetra Data Platform and the Tetra File-Log Agent. If the Tetra Data Platform does not receive a message indicating that the Tetra File-Log Agent is "alive" for more than 5 minutes, it assumes that the Agent is offline.
• Agent log files upload every: Indicates how often to upload log files to the TDP.
• Retry limit for Upload/Archive/Delete: Determines how many times the agent will retry upload, archive, and delete actions before erroring out.
• Compression type: Indicates how files will be compressed when uploaded to the TDP.
• TDP API URL Override: Indicates the TDP API endpoint for your deployment (for example, platform.tetrascience.com)
• Proxy: Indicates what type of proxy the agent is using (System or Custom), or if it's not using a proxy (None).
  - System (default setting): Uses the system proxy specified in Windows for the service user account. If basic authentication is required, a username and password may be specified for the proxy.
  - Custom: Makes all of the agent's proxy options configurable, regardless of any other settings. This proxy takes precedence over any existing system proxy or L7 proxy.
  - None: Explicitly configures the agent to not use any proxies for web requests.

  📘

  NOTE

  If a system proxy requires credentials, make sure that you enter them in the Proxy Username and Proxy Password fields.

Configure the Tetra File-Log Agent FileWatcher Service

📘

NOTE:

The Tetra File-Log Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NTFS, do not make any directory you want to watch case sensitive.

The Tetra File-Log Agent has a FileWatcher Service that provides the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags.

❗️

Functionality Change Introduced in Software Version 3.4

When the Tetra File-Log Agent sends data to the TDP, the data is stored in Amazon Simple Storage Service (Amazon S3). The Tetra File-Log Agent sends data from the Windows file storage (which is not case sensitive) and uploads it to the Amazon S3 storage, which is case sensitive. To avoid unintentional duplication of data based on case, the Tetra File-Log Agent normalizes all file path information to lower case. This ensures a behavior similar to one that users would expect from their Windows file storage.

Use the Tetra File-Log Agent FileWatcher Service to provide the path to monitor the file or folder, file source type, glob pattern, file metadata, and tags. When you set the Receive Commands option to Yes, in the Agent Configuration section, the Tetra File-Log Agent will be able to receive the path configuration sent from Tetra Data Platform.

You can also add paths in the Tetra File-Log Agent. To do this, complete the following steps.

1. Click the New Path button.
2. Configure the following options:

• Change Interval (seconds)—Indicates the minimum amount of time that a file in the path must remain unchanged before it is uploaded, and determines the minimum time between the scans of each path. The Agent scans two paths in parallel.

Change Interval File Scanning Example Scenario

The following table shows how file scanning works when three paths are configured with different Change Interval settings.

• Start Date(required)
• End Date—An optional end date for the time range of files that the agent will upload. If unspecified, all files with last modified dates that are on or after the Start Date are uploaded. If an End Date is specified, only files with last modified dates that are on or after the Start Date and on or before the End Date are uploaded.
• Path(required)—You can specify the Path of the folder to monitor and the glob pattern used to select the files or folders.
  You can add, edit, or delete a path. To create a new path, click New Path, or double-click the existing path in the table open the Watcher Path Editor pane:

📘

NOTE

• Glob patterns are a set of wildcard characters. For more information on glob patterns, including examples, see Common Glob Patterns.
• Archive/Delete settings in File mode are configurable in the TDP user interface for TDP v3.5.0 and higher, and in the TetraScience API for Tetra File-Log Agent v4.3.0 and higher.
• Archive/Delete settings in File mode are configurable in the Tetra File-Log Agent management console for agent v4.3.1 and higher.

• Source Type (required)
• Patterns (required)
• Labels
• Metadata
• Tags
• File Watch Mode (required)—File or Folder
• Fetch OS File/Folder Owner—Only available from agent version 4.3.1 and above. When this is Yes, agent will fetch the OS file (or folder, depending the File Watch Mode) creator and add as a metadata to the file during upload
• Archive Settings—includes Archive/Delete settings that provide the option to have the agent automatically archive and/or delete source files after they're uploaded to the TDP.

Archive Settings

🚧

IMPORTANT

If you configure Data Access Rules for the organization that the Tetra File-Log Agent is in, you must add the agent's service user to an access group that has at least Read permissions for any files the agent uploads. Otherwise, archive and delete actions fail and return a 403 error.

If File Watch Mode is set to File, the Archive to section displays the following options:

• Archive checkbox - if checked, archive settings are enabled
• Archive to - the root folder for inside where the archived files will be stored

  📘

  NOTE

  Archive to paths must be specified as absolute paths. Relative paths are not supported.

• Archive time and unit - time from successful upload after which the file will be eligible to be archived from the source folder

Delete Settings

• Delete checkbox - if checked, delete settings are enabled
• Delete time and unit - time from successful archive after which the file will be eligible to be deleted from the archive folder

Additional File Path Specification

📘

Path Specification

1. Tetra File-Log Agent supports both local folders and UNC paths for Network Folders, but not mapped drive.
2. You can specify up to 500 paths in FileWatcher

4. After you complete configuring the path, you can save the changes and start the Tetra File-Log Agent. After the Tetra File-Log Agent starts, the path configuration changed in the local is uploaded to the S3 bucket. A copy of that path configuration will be available on the Tetra Data Platform. You can then modify the path and push a new path configuration to the Tetra File-Log Agent.

📘

Metadata and Tags

To learn more about metadata and tags, including which characters you can use, see this topic.

5. You can click Re-upload to regenerate and re-upload the result files for a file path. From the dialog, set the Reupload Start Date and Reupload End Date. If you leave the Reupload End Date blank, then the Agent uses the current date as the end date. Click OK to save the date range.

As of v3.5.0, the Agent verifies the folder path when iterating the folder paths during scanning. The Agent displays an icon in front of every folder path to indicate whether the folder path is valid. For example, these two folders are invalid:

• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder3\
• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder4\

The Agent keeps checking the folder paths at each scan interval, and updates the status if there are any changes.

Verify the installation and monitor the Agent

Step 1: Start the Agent

In the Agent Management Console, select the top Start button. Then, verify that the Agent Status displays as Running.

Step 2: Monitor Progress, Metrics, and Errors

To review the Tetra File-Log Agent's run time progress and host server's system metrics, select Summary to open the Summary dashboard:

Processing Summary

You can review how many File-Log result files were detected, generated, and uploaded. If a failure occurs, the numbers display here.

The Processing Summary dashboard displays this information:

• Scanned files/folders - Total number of files or folders detected by the agent across all paths.
• Generated files - Number of file versions* generated.
• Failed to generate - Number of files that failed to generate.
• Pending upload - A pending state representing files which have been scanned and saved to the database, awaiting upload. When the Tetra File-Log Agent stops, the To upload state is maintained. Any file that was not completely uploaded is retained in the Tetra File-Log Agent's database. When the Tetra File-Log Agent restarts, it may resume uploading files which were previously scanned.
• Uploaded - Number of file or folder versions* that have been uploaded by the agent successfully.
• Failed to upload - Number of files that failed to upload to the agent
• Pending archive - If there are paths in file mode with archive setting enabled, and files uploaded are eligible for archive, this will show the total number of files waiting to be archived.
• Archived - If there are paths in file mode with archive setting enabled, this will show the total number of files that have been archived from such paths.
• Failed to archive - If there are paths in file mode with archive setting enabled, and any file from those paths encounters errors in the archival process, this will show the total number of such files that could not be archived.
• Pending delete - If there are paths in file mode with archive/delete setting enabled, and archived files from such paths become eligible for delete, this will show the total number of files waiting to be deleted from archive folder.
• Deleted - If there are paths in file mode with archive/delete setting enabled, and files have been deleted based on the setting, this will show the total number of files deleted from the archive folder.
• Failed to delete - If there are paths in file mode with archive/delete setting enabled, and any file from the archived folder encounters errors during the delete process, this will show the total number of such files that could not be deleted from archive folder.

* - If a single file/folder is scanned and uploaded, then later modified and uploaded again, this will result in the generated and upload counts having a count higher than the Scanned count.

Additional File Properties Uploaded

File Properties, such as file creation date, last modified date and file size are sent with files when uploading to TDP. These become part of the S3 object. These properties provide file context. See the table below for details.

Error Messages

This section displays any detailed error messages, such as when a file has failed to upload or has been copied to the temp folder in the Agent host machine.

System Messages

This section displays the host server system metrics and messages every minute and provides these metrics:

• Data Connection (connection status between the Agent and the Tetra Data Platform)
• Available Disk Space (in GB)
• Disk Usage (as a % of the total)
• CPU Usage (as a % of the total)
• Memory Usage (in MB)

For help with any issues, review the Troubleshooting Issues and Solutions section.

To learn more about the Tetra File-Log Agent, please review the FAQ section or please contact the TetraScience Customer Success team.