This guide shows how to configure and use Tetra File-Log Agent versions 4.5.x after an Agent is installed.

To install the Agent, see the Tetra File-Log Agent Installation Guide (v4.5.x).

Agent Management Console Configuration

You can use the Agent Management Console to do the following:

• Configure Agent settings
• Configure Tetra Data Platform (TDP) connection settings
• Review file processing

To get started, do the following.

Open the Agent Management Console

To open the Agent Management Console on the host server, do the following:

1. On the Agent's host server, open Windows Programs.
2. In the Windows Programs menu, choose TetraScience File-Log Agent. The Agent Management Console appears.

Configure the Agent

To configure the Tetra File-Log Agent after it's installed, do the following:

1. In the Agent Management Console, select the menu icon in the left navigation menu. Then, choose Configuration. The Agent Configuration page appears.

📘

NOTE

The Configuration page provides settings to save, start, and stop the Agent along with its Tetra Data Platform (TDP) connectivity settings.

2. Configure the Agent by filling out the fields in the following sections on the Configuration page:

• Agent Status
• Windows Task Scheduler
• File-Log Group User
• TDP Connection Settings
• FileWatcher Service Settings

Change the Agent's Status

The Agent Status section indicates the current state of the Agent's processing status, which can be either of the following:

• Stopped
• Running

To save, start, or stop the Agent, do the following:

• To save the Agent configuration, choose Save.
• To start the Agent after configuring new settings, choose Start. When the Agent starts, the status indicator shows Running.
• To stop a running Agent, choose Stop. When the Agent stops, the status indicator shows Stopped. (The Stop button can be selected only if the Agent is running.)

Automate Status Checks by Using Windows Task Scheduler

To avoid needing to check the Agent’s service status manually, it’s recommended that you create a Windows task to perform daily status checks.

To create a Windows task, do the following:

1. Under Windows Scheduled Task to check Agent Service Status, choose Yes.
2. For Run Time, enter the time that you want the status check to run each day. If the Agent is stopped, then the scheduled Windows task restarts the Agent service automatically. If the Agent is running, it continues to run, and the Windows task will run at the time you selected.

📘

NOTE

If you manually stop the Agent, the task is removed from Windows Task Scheduler. When you run the Agent without specifying an Agent Group User account, the task runs under the LocalSystem account.

Configure A Windows User Service Account for the Agent

The File-Log Group User is the Windows User service account that runs the Tetra File-Log Agent. To configure a user account, do the following:

1. On the local host server, add a user account for the Agent to use. For instructions, see Add a user account in the Microsoft documentation.

   📘

   NOTE

   If the Tetra File-Log Agent monitors the network's shared folders, then we suggest that you provide a dedicated service account that has the following:

   • At least read and List folder content permission of the network shared folders (including the subfolders and files contained in those folders).
   • Is part of the local user group of the host server.
   • Has no account expiration date. If an account expires, then the Tetra File-Log Agent can't extract data and data acquisition is interrupted.
   • Requires the log on as service permission.

2. For User Name, enter the Windows User's user name.
3. For Password, enter the Window User's password. The Agent validates the account immediately. If the user name and password are correct, then the Tetra File-Log Agent displays a Valid status next to the field. If the user account does not have the proper permissions, then the word Invalid displays.

📘

NOTE

If you leave the User Name field blank, then the Tetra File-Log Agent runs using the Windows predefined LocalSystem account. When the File-Log Group User is left blank the Tetra File-Log Agent can only access local folders. If you want to access remote folders you MUST provide a File-Log Group User.

Configure TDP Connection Settings

The Agent Configuration section on the Configuration tab provides settings to do any of the following:

• Set the S3 Direct Upload option (activated by default).
• Set the Receive Commands option to allow the Agent to receive commands from the TDP (activated by default).
• Establish a data connection between the Agent and the TDP.
• Specify the Destination Id so that files from multiple Agents can be uploaded to the same Amazon S3 bucket. It usually helps to split the load into multiple agents for scaling horizontally.

Set the S3 Direct Upload Option

To have the Agent upload files directly to your Amazon Simple Storage Service (Amazon S3) bucket, set the S3 Direct Upload option to Yes. For new installs, Yes is the default setting. When upgrading the Agent, the setting from the previous version of the Agent is retained.

If you choose this option, keep in mind the following:

• You can upload a maximum file size of 5 TB.
• If you use the S3 Direct Upload option with a Tetra Data Hub that uses a Generic Data Connector (GDC), you must add an L7 Proxy Connector in the same Hub where you set up the GDC. The port of the L7 Proxy Connector must also be open. For more information, see GDC Connections.

For more information, see Agent Deployment Options.

📘

SQLite Database File Backups

To have the Agent automatically perform regular backups of the SQLite database file, set the S3 Direct Upload option to Yes. When you enable this option, the SQLite database file (which stores agent configuration data) is uploaded to the backup bucket in the Tetra Data Lake. If an Agent failure occurs, you can restore the database file from the backup bucket and continue processing.

If you don't use the S3 Direct Upload option, it's recommended that you create periodic backups of the Agent database, which is typically stored in the installation folder under C:\\TetraScience\<agent>\\db. In the event of a failure (for example, if the host server drive is lost), this would enable the data extraction to continue from where it left off, without having to re-upload all of the data.

Allow the Agent to Receive Commands from the TDP

To allow the Agent to securely receive and run commands from the TDP, set the Receive Commands option to Yes. The default setting is No. If you do this, you'll need to also enable the Agent's queue in the TDP through the UI or through an API endpoint command.

To be able to use these APIs, the Receive Commands setting must be set to Yes.

📘

TDP Commands and Queue Security

Although queues enable TDP to send commands to the Tetra Agent, the command execution request process is secured.

• TDP does not send actual commands, but rather messages that request commands for execution.
• The Tetra Agent has its own queue to receive messages from TDP and is not shared by any other agent.
• When queues are enabled, the Tetra Agent proactively reads the queue for messages containing command execution requests and determines whether to initiate the command execution. You can control whether to enable local command execution by toggling the Receive Commands option for the Tetra Agent.
• The Tetra Agent can only execute a limited set of pre-built commands. As of Tetra File-Log Agent version 4.0, the only available command for execution is to update its configuration based on cloud configuration. For details about cloud configuration, click here.

Establish a Data Connection Between the Agent and the TDP

To set up a data connection between the Agent and the TDP, enter the following information in the Agent Configuration section on the Configuration tab:

🚧

IMPORTANT

When establishing a connection between the Agent and the TDP, keep in mind the following:

• Agent ID is required.
• The connection URL is required only if you're uploading RAW files to the TDP.
• Before you use your Agent ID and connection URL, it’s strongly recommended that you verify both with your customer account leader.

1. For Agent Id, enter the Agent’s ID. The Agent ID is a Universal Unique Identifier (UUID) value that’s used to connect the Agent to the TDP. To find your Agent ID, do the following:

• Sign in to the TDP.
• In the left navigation menu, choose Data Sources. Then, choose Agents. The Agents page appears.
• In the AGENT column, find the name of the Agent that you’re configuring. The Agent ID is listed below the Agent name.

2. For Connection Url, enter the complete connection URL of the Connector or integration type that you’re using:

• TDP (No Connector) URL: <https://api.tetrascience-dev.com/v1/data-acquisition/agent/>
• Hub: http://10.100.1.2:8443/generic-connector/v1/agent
• Generic Data Connector (GDC) URL: <http://10.100.1.1:8888/generic-connector/v1/agent>

3. (For “TDP (No Connector)” setups only) If you’re using an Agent without a Connector, make sure that you do the following:

• For Org Slug, enter your organizational slug. To get your organizational slug, see Viewing Organization Details. The organizational slug is required when using a JSON Web Token (JWT). Adding the slug to this field attaches it to the Agent’s API request headers.
• For Authorization, select Add/Edit. Then, enter your JWT in the dialog that appears. To get a JWT, see Generate a JWT for a Service User. Then, choose Save to encrypt and save the JWT. The Agent validates the connection immediately and displays the updated connection status next to the Connection Url field.

🚧

IMPORTANT

If the Upload Raw Files to TDP setting in the Agent Run Time section (this is described later in this page) is set to No, the Agent will not connect to the TDP to upload files. The status here will still show Connected.

Configure Advanced Settings

You can configure the following options by selecting Advanced Settings on the Configuration page:

• Data Connection status check every: Indicates how often the software checks the status of the connection (heartbeat) between the Tetra Data Platform and the Tetra File-Log Agent. If the Tetra Data Platform does not receive a message indicating that the Tetra File-Log Agent is "alive" for more than 5 minutes, it assumes that the Agent is offline.
• Agent log files upload every: Indicates how often to upload log files to the TDP.
• Retry limit for Upload/Archive/Delete: Determines how many times the agent will retry upload, archive, and delete actions before erroring out.
• Compression type: Indicates how files will be compressed when uploaded to the TDP.
• TDP API URL Override: Indicates the TDP API endpoint for your deployment (for example, platform.tetrascience.com)
• Proxy: Indicates what type of proxy the agent is using (System or Custom), or if it's not using a proxy (None).
  - System (default setting): Uses the system proxy specified in Windows for the service user account. If basic authentication is required, a username and password may be specified for the proxy.
  - Custom: Makes all of the agent's proxy options configurable, regardless of any other settings. This proxy takes precedence over any existing system proxy or L7 proxy.
  - None: Explicitly configures the agent to not use any proxies for web requests.

  📘

  NOTE

  If a system proxy requires credentials, make sure that you enter them in the Proxy Username and Proxy Password fields.

Configure FileWatcher Service Settings

You can use the Agent's FileWatcher Service to provide the Agent the path to monitor target files or folders. To configure paths for the Agent, you can use either Agents page in the TDP user interface (When you set the Receive Commands option to Yes), or the Agent Management Console.

📘

NOTE

Keep in mind the following when configuring the FileWatcher Service:

• The Agent can only retrieve files stored on the NTFS file system. Other file systems are not supported. Also, when using NTFS, do not make any directory you want to watch case sensitive.

• The Agent sends data from the Windows file storage (which is not case sensitive) and uploads it to the Amazon S3 storage, which is case sensitive. To avoid unintentional duplication of data based on case, the Tetra File-Log Agent normalizes all file path information to lower case. This ensures a behavior similar to one that users would expect from their Windows file storage.

• The Agent supports both local folders and UNC paths for Network Folders, but not mapped drives.

• You can specify up to 500 paths.

To configure paths remotely in the TDP user interface, see Configure the Tetra File-Log Agent in the Cloud.

To configure paths in the local Tetra File-Log Agent Management Console, do the following:

1. In the local Tetra File-Log Agent Management Console, click the New Path button.
2. Configure the following options:

• File Change Interval (seconds)—Indicates the minimum time a file in the configured path must remain unchanged before it can be uploaded. The default is 30 seconds.
• Scan Interval (seconds)—Indicates the frequency between when paths are rescanned for new or changed files. The default is 30 seconds.
• Start Date(required)—Indicates the earliest date the Agent will look at for a file or folder's last modified date to determine files for upload.¸
• End Date—An optional end date for the time range of files that the agent will upload. If unspecified, all files with last modified dates that are on or after the Start Date are uploaded. If an End Date is specified, only files with last modified dates that are on or after the Start Date and on or before the End Date are uploaded.
• Path(required)—You can specify the Path of the folder to monitor and the glob pattern used to select the files or folders.
  You can add, edit, or delete a path. To create a new path, click New Path, or double-click the existing path in the table open the Watcher Path Editor pane:

📘

NOTE

• Glob patterns are a set of wildcard characters. For more information on glob patterns, including examples, see Common Glob Patterns.
• Archive/Delete settings in File mode are configurable in the TDP user interface for TDP v3.5.0 and higher, and in the TetraScience API for Tetra File-Log Agent v4.3.0 and higher.
• Archive/Delete settings in File mode are configurable in the Tetra File-Log Agent management console for agent v4.3.1 and higher.

• Source Type (required)
• Patterns (*required)
• Enable Extended Glob Patterns—When this is Yes, Agent can support extended patterns supporting ranges and repetition in the pattern to allow more customization over which files should be ingested into the TDP. The default is No.
• Labels
• Metadata
• Tags
• File Watch Mode (required)—File or Folder
• Fetch OS File/Folder Owner—Only available from agent version 4.3.1 and above. When this is Yes, agent will fetch the OS file (or folder, depending the File Watch Mode) creator and add as a metadata to the file during upload
• Symlink Behavior (required for Agent v4.4.1 and higher)—Determines how the Agent handles file symbolic links. Choose one of the following options:
  • Legacy: When the Agent scans a file symbolic link, it will record the metadata of the symbolic link itself. Then, when uploading the file, the Agent follows the link and uploads the contents of the target file.
  • Ignore: The Agent skips file symbolic links entirely during scan and event handling. The Agent doesn't record any metadata for these files and doesn't upload them. Directory symbolic links are still followed.
• For File Watch Mode: You can configure the following settings:
  • Stream Uploads: When set to Yes, the Agent uploads files directly to Amazon S3 without requiring the file to be stored locally in the Group User System temporary folder. Minimum available local disk space will still be required if Stream Uploads is enabled. The default setting is No. It's recommended that you enable this for new configured paths and paths without files pending upload.
  • Archive Settings—includes Archive/Delete settings that provide the option to have the agent automatically archive and/or delete source files after they're uploaded to the TDP.

After you complete configuring the path, you can save the changes and start the Tetra File-Log Agent. After the Tetra File-Log Agent starts, the path configuration changed in the local Agent Management Console is uploaded to the Amazon S3 bucket. A copy of that path configuration is then available on the TDP. You can then modify the path either through the TDP UI or the local Agent Management Console.

📘

Attributes

To learn more about applying attributes to files, including which characters you can use, see Attributes.

Re-Upload Files

You can regenerate and re-upload the result files for a file path by clicking Re-upload. From the dialog, set the Reupload Start Date and Reupload End Date. If you leave the Reupload End Date blank, then the Agent uses the current date as the end date. Click OK to save the date range.

The Agent verifies the folder path when iterating the folder paths during scanning. The Agent displays an icon in front of every folder path to indicate whether the folder path is valid. For example, these two folders aren't valid:

• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder3\
• \\tsempowercli2.tsempower.local\Shared-Folder\Performance_Test_01\folder4\

The Agent keeps checking the folder paths at each scan interval, and updates the status if there are any changes.

Archive Settings

🚧

IMPORTANT

If you configure Data Access Rules for the organization that the Tetra File-Log Agent is in, you must add the agent's service user to an access group that has at least Read permissions for any files the agent uploads. Otherwise, archive and delete actions fail and return a 403 error.

If File Watch Mode is set to File, the Archive to section displays the following options:

• Archive checkbox - if checked, archive settings are enabled
• Archive to - the root folder for inside where the archived files will be stored

  📘

  NOTE

  Archive to paths must be specified as absolute paths. Relative paths are not supported.

• Archive time and unit - time from successful upload after which the file will be eligible to be archived from the source folder

Delete Settings

• Delete checkbox - if checked, delete settings are enabled
• Delete time and unit - time from successful archive after which the file will be eligible to be deleted from the archive folder

Run Commands

You can use the Tetra File-Log Agent and TDP Command Service to programmatically do the following:

• Processes file change events for both file and folder modes from any external deployed system by using the File Changed command.
• Download and drop a file to a designated folder programmatically by using the Copy File command.

Documentation Feedback

Do you have questions about our documentation or suggestions for how we can improve it? Start a discussion in TetraConnect Hub. For access, see Access the TetraConnect Hub.

📘

NOTE

Feedback isn't part of the official TetraScience product documentation. TetraScience doesn't warrant or make any guarantees about the feedback provided, including its accuracy, relevance, or reliability. All feedback is subject to the terms set forth in the TetraConnect Hub Community Guidelines.