This topic contains Frequently Asked Questions (FAQs) for the Tetra File-Log Agent (FLA).
One of the core applications in File-Log Agent runs as a Windows Service. This application runs in the background to detect the changes in files and folders. When configuring the File-Log Agent, the user needs to specify which account (File-Log Group User) runs that Windows Services application.
The File-Log Group User account could be a local Windows User Account, a Service Account, or a Domain User Account.
Please check the Microsoft guideline for selecting the account.
Also, the File-Log Group User account should have Log on as service right and the account has read permission as the minimum for the path under monitoring.
If the File-Log Group User account is left empty, the Agent will use the local system account as default.
The Tetra File-Log Agent requires that the File-Log Group User has at least read permission on the files and folders, including the subfolders and files contained in those folders.
If the File-Log Group User doesn't have read permission on the paths defined in the Management Console, the Agent won't start until the permission issue is fixed.
If the File-Log Group User doesn't have read permission on the subfolders or the files contained in the paths defined in the Management Console, the Agent will skip those folders and files. The error message is written to the Agent log if the path is local. If the path is UNC, the Agent won't be able to generate any error message due to limited permission.
When the user starts the Agent, the Agent will verify the connection to the TetraScience platform. If the user sees the following message, it means the Agent is not able to connect to the TDP.
The user can take the following steps to investigate and correct that issue.
- Verify the configuration on the TDP.
- Verify the Generic Data Connector or User-defined Integration is accessible from the computer hosting the File-Log Agent.
- Verify the Agent ID and URL (including the protocol, IP and Port) are correct.
- Verify the protocol used for Data Connection.
If using HTTP, the Agent can take the connection URL as HTTP://IP/port or just IP/Port
If using HTTPS, the user has to provide HTTPS explicitly, otherwise, the Agent will assume the protocol is HTTP.
There could be multiple causes for this.
- Understand how the Agent detects the file or folder changes. Please check the detecting logic from fileWatcherService and logWatcherService respectively. (The LogWatcher Service is only available for versions before the File-Log Agent 4.1.0 release.)
- Check the defined patterns. Please check the common usage of Glob pattern implemented in File-Log Agent.
- Make sure Start Date is correct.
- Make sure Group User Temp folder has enough free space to store the largest file being uploaded by the File-Log Agent.
The Group User Temp folder is used to move the file into by the File-Log Agent before being uploaded to TDP.
Theoretically the Temp folder should have enough free space to store the largest file being uploaded. For this purpose if the Temp folder location/path needs to be changed, please refer to these articles:
- How to change TEMP folder and allocate free space on your system disk
- Change location of TEMP files folder to another drive
The user can select HTTP or HTTPS to connect to TDP when using the Generic Data Connector. It is part of the configuration setting in the Data Connector.
The user can set the port for HTTP or HTTPS, as long as those ports are available.
When connecting to TDP through User-Defined Integration, only HTTPS is supported.
Stop the existing Agent Windows service
Make sure you stop the previous agent before upgrade/reinstall.
The user can choose Upgrade when installing the File-Log Agent. The Agent will retain the configuration setting and Sqlite database from the previous version.
If the user selects New Installation during installation. The File-Log Agent will be installed from scratch. The configuration setting and Sqlite database which stores the file processing history will be refreshed.
The Agent has changed this behavior after v3.4.0.
The behavior below is applied to the following versions
From v3.4.0, the behavior that Agent will re-upload existing files when any of Source Type, Metadata or Tags change has been reverted. Instead, there are two changes applied
When the Metadata, Source Type, or Tag is updated, the change will be applied to only the new files or the updated files after the update.
The Agent provides a Re-upload button for users to determine what files should be re-upload explicitly.
The detail is described in v3.x.x installation guide
When the user updates any of the Source Type, Metadata, or Tag, the Agent will reprocess the files.
Taking an example that the Agent is configured using FileWatcher Service. Six files are being processed and uploaded to TetraScience Data Platform.
The user can stop the Agent and add a new Metadata entry. The Agent notifies the user that the files will be reprocessed.
After the files are reprocessed, the Summary page updates accordingly.
Please beware that there have no duplicated files that exist in the Tetra Data Platform. The files have a new version. Taking the example above, there still have six files that exist in TetraScience Data Platform. Each file has two versions.
Add new Metadata or tag to trigger the Agent to re-process/re-upload the files.
When a file path is under monitoring by File Watcher Service, it needs to determine the following:
- What are the files in the file path that should be monitored
Besides the glob pattern defined in the file path, another criterion is Start Date (file age). The File-Log Agent monitors the files with their last write time is greater than the date specified in Management Console. It impacts the initial file upload only. The Agent can exclude historical files that the user doesn't want to upload.
- When the files should be uploaded
The Agent scans the file paths using its predefined time interval. When the Agent detected the file change, the Agent doesn't upload it immediately which tries to avoid uploading any file in an intermediate transition stage. Instead, the Agent has a
File Change Interval feature. The Agent compares the File Last Write Time from two continuous scans. If the values are the same, the Agent thinks the file change is done which is eligible for upload. When the file is uploaded, it won't be reloaded unless it is updated.
This setting can be configurated either on the global level which is applied to all of the file paths under monitoring or on the individual file path level. There has an option button on the Windows Management Console for the user to choose.
In editions of Windows before Windows 10 version 1607, the maximum length for a path is MAX_PATH, which is defined as 260 characters. In the later versions of Windows, changing a registry or using the Group Policy is required to remove the limit.
The following editions of Windows OS support long path:
- Windows 10 (version 1607 and later)
- Windows Server 2016
- Windows Server 2019
Windows 8.1 and Windows Server 2012 (which is based on Windows 8.1), does not support paths longer than 260 characters.
Please read the following posts regarding how to remove the 260-character limit
- How to Make Windows 10 Accept File Paths Over 260 Characters
- Enable NTFS long paths in Windows Server 2016 and 2019 by Group Policy
If something unexpected occurs, contact us. Provide the following two items.
- Complete Log folder, which is located in the File-Log Agent installation folder. We suggest that you compress the entire folder.
- Sqlite Database file, applicationdb.sqlite, which is located in the Database folder under Bin.
The Tetra File-Log Agent uses these AWS endpoints:
|AWS Endpoint||Description||When Required|
|s3.[region].amazonaws.com||Uploads files||When the Enable S3 Direct Upload option is selected|
|sqs.[region].amazonaws.com||Fetches the command message and then returns the command processing status||When the Enable Queue option is selected|
|logs.[region].amazonaws.com||Posts Agent Heart Beats and Agent logs||When the Enable S3 Direct Upload option is selected|
|monitoring.[region].amazonaws.com||Sends Metrics Data (such as CPU, Memory, and Disk usage)||When the Enable S3 Direct Upload option is selected|
Alphabetic characters A-Z and a-z are supported in the paths and file names watched. If the path or filename has any other alphabets, accents, dieresis etc. the FLA may not behave correctly and it is suggested not to use these alphabetic characters in paths/files which FLA is watching.
Updated 2 months ago