Tetra File-Log Agent Frequently Asked Questions

1. What is File-Log Group User and How it is used by the Agent?

One of the core applications in File-Log Agent is run as Windows Services. This application is running in the background to detect the changes in files and folders. When configuring the File-Log Agent, the user needs to specify what is the account (File-Log Group User) to run that Windows Services application.

The File-Log Group User account could be a local Windows User Account, a Service Account, or a Domain User Account.

Please check the guideline for selecting the account from Microsoft.

Also, the File-Log Group User account should have Log on as service right and the account has read permission as the minimum for the path under monitoring.

If the File-Log Group User account is left empty, the Agent will use the local system account as default.

2. What is the required permission for the folders or the files

The TetraScience File-Log Agent requires the File-Log Group User has at least read permission on the files and folders, including the subfolders and files contained in those folders.

If the File-Log Group User doesn't have read permission on the paths defined in the Management Console. The Agent won't start until the permission issue is fixed.

If the File-Log Group User doesn't have read permission on the subfolders or the files contained in the paths defined in the Management Console. The Agent will skip those folders and files. The error message is written to the Agent log if the path is local. If the path is UNC, the Agent won't be able to generate any error message due to limited permission.

3. What should the user do if there is a connection issue when starting the Agent?

When the user starts the Agent, the Agent will verify the connection to the TetraScience platform. If she sees the message as below, it means the Agent is not able to connect to the TetraScience platform.

The user can take the following steps to investigate and correct that issue.

  • Verify the configuration on TetraScience Platform
  • Verify the Generic Data Connector or User-defined Integration is accessible from the computer hosting the File-Log Agent
  • Verify the Agent ID and URL (including the protocol, IP and Port) are correct.
  • Verify the protocol used for Data Connection.
    If using Http, the Agent can take the connection URL as Http://IP/port or just IP/Port
    If using Https, the user has to provide Https explicitly, otherwise, the Agent will assume the protocol is Http.

4. Why the files I expect to see are not uploaded?

There could have multiple reasons that could cause that. Please verify the following:

  • Understand how the Agent detects the file or folder changes. Please check the detecting logic from fileWatcherService and logWatcherService respectively.

  • Check the defined patterns. Please check the common usage of Glob pattern implemented in File-Log Agent.

  • Make sure Start Date is selected correctly

5. How to specify the Data Connection Protocol?

The user can select Http or Https to connect the TetraScience Platform when using the Generic Data Connector. It is part of the configuration setting in the Data Connector

The user can set the port for HTTP or HTTPs, as long as those ports are available.

When connecting the TetraScience Platform through User-Defined Integration, only Https is supported.

6. How to upgrade from previous version of File-Log Agent?

❗️

Stop the existing Agent Windows service

Make sure you stop the previous agent before upgrade/reinstall.

The user can choose Upgrade when installing the File-Log Agent. The Agent will retain the configuration setting and Sqlite database from the previous version.

If the user selects New Installation during installation. The File-Log Agent will be installed from scratch. The configuration setting and Sqlite database which stores the file processing history will be refreshed.

7. How the Agent behaves if Source Type, Metadata or Tag is changed?

❗️

Functionality changes

The Agent has changed that behavior after v3.4.0.
The behavior below is applied to the following versions

  • v3.0.0
  • v3.1.0
  • v3.2.0
  • v3.2.1
  • v3.3.1

From v3.4.0, the behavior that Agent will re-upload existing files when any of Source Type, Metadata or Tags change has been reverted. Instead, there have two changes applied

  1. When the Metadata, Source Type, or Tag is updated, the change will be applied to only the new files or the updated files after the update.

  2. The Agent provides a Re-upload button for users to determine what are files should be re-upload explicitly.

The detail is described in v3.x.x installation guide

When the user updates any of the Source Type, Metadata, or Tag, the Agent will reprocess the files.
Taking an example that the Agent is configured using FileWatcher Service. Six files are being processed and uploaded to TetraScience Data Platform.

The user can stop the Agent and add a new Metadata entry. The Agent notifies the user that the files will be reprocessed.

After the files are reprocessed, the Summary page updates accordingly.

Please beware that there have no duplicated files that exist in the Tetra Data Platform. The files have a new version. Taking the example above, there still have six files that exist in TetraScience Data Platform. Each file has two versions.

8. What should we do if we want to re-upload the files to TetraScience Data Platform

We can add new Metadata or tag to trigger the Agent to re-process/re-upload the files. More detail can be found in Question 7 (How the Agent behaves if Source Type, Metadata or Tag is changed).

9. How to use File Path Start Date and File Change Interval field in File Watcher service?

When a file path is under monitoring by File Watcher Service, it needs to determine the following:

  • What are the files in the file path that should be monitored

Besides the glob pattern defined in the file path, another criterion is Start Date (file age). The File-Log Agent monitors the files with their last write time is greater than the date specified in Management Console. It impacts the initial file upload only. The Agent can exclude historical files that the user don't want to upload.

  • When the files should be uploaded

The Agent scans the file paths using its predefined time interval. When the Agent detected the file change, the Agent doesn't upload it immediately which tries to avoid uploading any file in an intermediate transition stage. Instead, the Agent has a File Change Interval feature. The Agent compares the File Last Write Time from two continuous scans. If the values are the same, the Agent thinks the file change is done which is eligible for upload. When the file is uploaded, it won't be reloaded unless it is updated.

This setting can be configurated either on the global level which applied to all of the file paths under monitoring or on the individual file path level. There has an option button on the Windows Management Console for the user to choose.

10. How to enable long path for Windows?

In editions of Windows before Windows 10 version 1607, the maximum length for a path is MAX_PATH, which is defined as 260 characters. In the later versions of Windows, changing a registry or using the Group Policy is required to remove the limit.

The following editions of Windows OS support long path:

  • Windows 10 (version 1607 and later)
  • Windows Server 2016
  • Windows Server 2019

Windows 8.1 and Windows Server 2012 (which is based on Windows 8.1) does not support paths longer than 260 characters.

Please read the following posts regarding how to remove the 260 character limit

11. How to report errors to TetraScience?

In case anything unexpected occurred, please report to TetraScience and providing the following two items:

  • Complete Log folder (It is located in the File-Log Agent installation folder. It is suggested to compress the entire folder.)
  • Sqlite Database file, applicationdb.sqlite (it is located in the Database folder under Bin)

12. Which AWS Endpoints Does the File-Log Agent Use?

The Tetra File-Log Agent uses these AWS endpoints:

AWS Endpoint

Description

When Required

s3.[region].amazonaws.com

Uploads files

When the Enable S3 Direct Upload option is selected

sqs.[region].amazonaws.com

Fetches the command message and then returns the command processing status

When the Enable Queue option is selected

logs.[region].amazonaws.com

Posts Agent Heart Beats and Agent logs

When the Enable S3 Direct Upload option is selected

monitoring.[region].amazonaws.com

Sends Metrics Data (such as CPU, Memory, and Disk usage)

When the Enable S3 Direct Upload option is selected


Did this page help you?