One of the core applications in File-Log Agent is run as Windows Services. This application is running in the background to detect the changes in files and folders. When configuring the File-Log Agent, the user needs to specify what is the account (File-Log Group User) to run that Windows Services application.
The File-Log Group User account could be a local Windows User Account, a Service Account, or a Domain User Account.
Please check the guideline for selecting the account from Microsoft.
Also, the File-Log Group User account should have Log on as service right and the account has read permission as the minimum for the path under monitoring.
If the File-Log Group User account is left empty, the Agent will use the local system account as default.
The TetraScience File-Log Agent requires the File-Log Group User has at least read permission on the files and folders, including the subfolders and files contained in those folders.
If the File-Log Group User doesn't have read permission on the paths defined in the Management Console. The Agent won't start until the permission issue is fixed.
If the File-Log Group User doesn't have read permission on the subfolders or the files contained in the paths defined in the Management Console. The Agent will skip those folders and files. The error message is written to the Agent log if the path is local. If the path is UNC, the Agent won't be able to generate any error message due to limited permission.
When the user starts the Agent, the Agent will verify the connection to the TetraScience platform. If she sees the message as below, it means the Agent is not able to connect to the TetraScience platform.
The user can take the following steps to investigate and correct that issue.
- Verify the configuration on TetraScience Platform
- Verify the Generic Data Connector or User-defined Integration is accessible from the computer hosting the File-Log Agent
- Verify the Agent ID and URL (including the protocol, IP and Port) are correct.
- Verify the protocol used for Data Connection.
If using Http, the Agent can take the connection URL as Http://IP/port or just IP/Port
If using Https, the user has to provide Https explicitly, otherwise, the Agent will assume the protocol is Http.
There could have multiple reasons that could cause that. Please verify the following:
Check the defined patterns. Please check the common usage of Glob pattern implemented in File-Log Agent.
Make sure Start Date is selected correctly
The user can select Http or Https to connect the TetraScience Platform when using the Generic Data Connector. It is part of the configuration setting in the Data Connector
The user can set the port for HTTP or HTTPs, as long as those ports are available.
When connecting the TetraScience Platform through User-Defined Integration, only Https is supported.
Stop the existing Agent Windows service
Make sure you stop the previous agent before upgrade/reinstall.
The user can choose Upgrade when installing the File-Log Agent. The Agent will retain the configuration setting and Sqlite database from the previous version.
If the user selects New Installation during installation. The File-Log Agent will be installed from scratch. The configuration setting and Sqlite database which stores the file processing history will be refreshed.
The Agent has changed that behavior after v3.4.0.
The behavior below is applied to the following versions
From v3.4.0, the behavior that Agent will re-upload existing files when any of Source Type, Metadata or Tags change has been reverted. Instead, there have two changes applied
When the Metadata, Source Type, or Tag is updated, the change will be applied to only the new files or the updated files after the update.
The Agent provides a Re-upload button for users to determine what are files should be re-upload explicitly.
The detail is described in v3.x.x installation guide
When the user updates any of the Source Type, Metadata, or Tag, the Agent will reprocess the files.
Taking an example that the Agent is configured using FileWatcher Service. Six files are being processed and uploaded to TetraScience Data Platform.
The user can stop the Agent and add a new Metadata entry. The Agent notifies the user that the files will be reprocessed.
After the files are reprocessed, the Summary page updates accordingly.
Please beware that there have no duplicated files that exist in the Tetra Data Platform. The files have a new version. Taking the example above, there still have six files that exist in TetraScience Data Platform. Each file has two versions.
We can add new Metadata or tag to trigger the Agent to re-process/re-upload the files. More detail can be found in Question 7 (How the Agent behaves if Source Type, Metadata or Tag is changed).
When a file path is under monitoring by File Watcher Service, it needs to determine the following:
- What are the files in the file path that should be monitored
Besides the glob pattern defined in the file path, another criterion is Start Date (file age). The File-Log Agent monitors the files with their last write time is greater than the date specified in Management Console. It impacts the initial file upload only. The Agent can exclude historical files that the user don't want to upload.
- When the files should be uploaded
The Agent scans the file paths using its predefined time interval. When the Agent detected the file change, the Agent doesn't upload it immediately which tries to avoid uploading any file in an intermediate transition stage. Instead, the Agent has a
File Change Interval feature. The Agent compares the File Last Write Time from two continuous scans. If the values are the same, the Agent thinks the file change is done which is eligible for upload. When the file is uploaded, it won't be reloaded unless it is updated.
This setting can be configurated either on the global level which applied to all of the file paths under monitoring or on the individual file path level. There has an option button on the Windows Management Console for the user to choose.
In editions of Windows before Windows 10 version 1607, the maximum length for a path is MAX_PATH, which is defined as 260 characters. In the later versions of Windows, changing a registry or using the Group Policy is required to remove the limit.
The following editions of Windows OS support long path:
- Windows 10 (version 1607 and later)
- Windows Server 2016
- Windows Server 2019
Windows 8.1 and Windows Server 2012 (which is based on Windows 8.1) does not support paths longer than 260 characters.
Please read the following posts regarding how to remove the 260 character limit
- How to Make Windows 10 Accept File Paths Over 260 Characters
- Enable NTFS long paths in Windows Server 2016 and 2019 by Group Policy
In case anything unexpected occurred, please report to TetraScience and providing the following two items:
- Complete Log folder (It is located in the File-Log Agent installation folder. It is suggested to compress the entire folder.)
- Sqlite Database file, applicationdb.sqlite (it is located in the Database folder under Bin)
The Tetra File-Log Agent uses these AWS endpoints:
|AWS Endpoint||Description||When Required|
|s3.[region].amazonaws.com||Uploads files||When the Enable S3 Direct Upload option is selected|
|sqs.[region].amazonaws.com||Fetches the command message and then returns the command processing status||When the Enable Queue option is selected|
|logs.[region].amazonaws.com||Posts Agent Heart Beats and Agent logs||When the Enable S3 Direct Upload option is selected|
|monitoring.[region].amazonaws.com||Sends Metrics Data (such as CPU, Memory, and Disk usage)||When the Enable S3 Direct Upload option is selected|
Updated 12 months ago