API DocumentationProduct Documentation
Product Documentation

Tetra File-Log Agent Release Notes (v4.3.x and Below)

Suggest Edits

The following are the Tetra File-Log Agent release notes for versions 4.x and earlier.

📘

NOTE

To upgrade the Tetra File-Log Agent, see Common Tetra Agent Upgrade and Installation Information..

For installation instructions, see Tetra File-Log Agent Installation Guide (Version 4.3.x).

v4.3.5

Release date: 14 June 2024

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.3.5. This release resolves an issue with how files are handled that are pending upload but don't match any scanned path.

🚧

IMPORTANT

Data integrity is our top priority. For this reason, the Tetra File-Log Agent Archive and Delete feature is temporarily deactivated until we can be sure it meets the necessary data integrity standards in all supported scenarios. For v4.3.5, the Archive and Delete feature is behind a feature flag that is disabled by default. We are working to ensure that all files—regardless of which version of the Agent and which connection method they were uploaded with—are only archived when they're available on the TDP.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect Intended Use for GxP validation purposes.

  • There is no new functionality in this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system.

  • There are no new enhancements in this release.

Bug Fixes

The following bugs are now fixed:

  • Files that are pending upload but don't match any scanned path no longer block other file uploads in that path's queue. Previously, if a file was scanned but not uploaded, and then the Agent's configuration was changed in a way that the file no longer matched the path's criteria (for example, pattern or start date), and the file was then modified so that it triggered another upload, the file would get stuck in pending upload.

Deprecated Features

There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.5 release:

  • For a path configured in File mode, if a glob pattern does not start with “\*\*/” and the path has system directories that the agent can't access, that path's scan process terminates. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1, and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
  • In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. However, this action doesn't actually remove the path from the agent. A fix for this issue is in development and testing and is planned for a future Tetra File-Log Agent release. To delete a path from the agent in the Agent Management Console, customers must use the Delete option in the Action column of the grid for each path, and then choose Save.
  • Modifying only agent connection information does not cause the agent to upload configuration to the TDP.

Upgrade Considerations

There are no additional upgrade considerations for this release.

v4.3.4

Release date: 1 March 2024

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.3.4. This release resolves several known issues related to file system event archiving.

🚧

IMPORTANT

Data integrity is our top priority. For this reason, the File-Log Agent Archive and Delete feature is temporarily deactivated until we can be sure it meets the necessary data integrity standards in all supported scenarios. For v4.3.4, the Archive and Delete feature is behind a feature flag that is disabled by default. We are working to ensure that all files—regardless of which version of the agent and which connection method they were uploaded with—are only archived when they're available on the TDP.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect Intended Use for GxP validation purposes.

  • There is no new functionality in this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system.

  • There are no new enhancements in this release.

Bug Fixes

The following bugs are now fixed:

  • Files detected through file system events now consistently receive the correct archive configuration, even if they occur in a batch with files from other paths that also have archive configuration and are within ~10 seconds of each other. Any files previously affected will be checked before archival and skipped if a matching path isn't located, or if the archive configuration is incorrect.
  • File system event handling was improved to better handle events that might have been skipped previously and relied on scanning.
  • The handling of temporary files created during upload was improved to pre-allocate space (which reduces the chance of an error), and then automatically clean up if an error does occur.

Deprecated Features

There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.4 release:

  • For a path configured in File mode, if a glob pattern does not start with “\*\*/” and the path has system directories that the agent can't access, that path's scan process terminates. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1, and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
  • In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. However, this action doesn't actually remove the path from the agent. A fix for this issue is in development and testing and is planned for a future Tetra File-Log Agent release. To delete a path from the agent in the Agent Management Console, customers must use the Delete option in the Action column of the grid for each path, and then choose Save.

Upgrade Considerations

There are no additional upgrade considerations for this release.

v4.3.3

Release date: 19 December 2023 (Last updated: 25 January 2024)

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.3.3.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect Intended Use for GxP validation purposes.

  • There is no new functionality in this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system.

  • The TetraScience.Agent.file-log.DropFile.CopyFile command now supports specifying the rawMd5Checksum payload property in either hexadecimal or Base64 encoding. This allows customers to use the rawMd5Checksum metadata value from a file uploaded by the File-Log Agent v4.3.0 or later in the command directly, without decoding and encoding in hexadecimal.
  • The compression method the Agent uses in Folder mode can now be specified in the Agent Management Console, under Advanced Settings in the Agent Configuration section. The default method is Zip, but the legacy method (7zip) is also available.

Bug Fixes

The following bugs are now fixed:

  • Files scanned by Tetra File-Log Agent versions 4.0.5 and earlier are now uploaded by the Agent when the files are modified. To implement this bug fix, customers must run the database upgrade script that the Agent Management Console provides after the latest Agent version is installed. For more information, see Upgrade Considerations.
  • Instead of polling requests sometimes hanging indefinitely, there's now an explicit 30-second timeout in addition to the existing 20-second "long" poll timeout. (Issue #2764)
  • To better monitor and diagnose an indefinite file hang that was identified in Tetra File-Log Agent v4.3.x, additional logging was added to command polling jobs. (Issue #2764)
  • Job scheduling triggers that fire while jobs are already executing are now dropped instead of queued. This fix resolves adverse effects for some jobs, such as command polling.
  • Tetra File-Log Agent commands now support Unicode characters in labels and values, except for new line Unicode characters.
  • When using Folder mode, the original (source) file metadata of all files included in a zip file is now preserved. (Issue #2974 and #3010)

Deprecated Features

  • There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.3 release:

  • For a path configured in File mode, if a glob pattern does not start with “**/” and the path has system directories which the agent cannot access, the scan process for that path will terminate. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1, and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
  • In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. This doesn't actually remove the path from the agent. The issue will be addressed in a future Tetra File-Log Agent version. To delete a path from the agent in the management console, use the Delete option in the Action column of the grid for each path, and then choose Save.
  • Multiple changed files detected by the Agent via file system events may receive the incorrect archive configuration if they occur in a batch with files from other paths which have archive configuration (within ~10 seconds of each other). Although the probability that this condition occurs is small, it could cause files to be incorrectly archived. Please ensure the “Archive Files after Upload to TDP” option in Archive Settings is disabled on your FLA configuration. As an added safety measure, you can ensure the permissions for the service user used by the Agent is set to Read-only access.

Upgrade Considerations

After customers complete the upgrade to Tetra File-Log Agent v4.3.3 and open the management console for the first time, the console prompts them to run a database upgrade script. Running this script is required to have files scanned by Tetra File-Log Agent versions 4.0.5 and earlier uploaded by the Agent when the files are modified.

📘

NOTE

The upgrade process will consume about 2.5x the current database size while running. The database size returns to the original size after the upgrade process completes.

v4.3.2

Release date: 17 July 2023 (Last updated: 25 January 2024)

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.3.2. This release focuses on bug fixes and enhancements, which include making scans more resilient and a new setting in the management console for managing archive configurations.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect “intended use” for GxP validation purposes.

  • There is no new functionality for this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system. The following are the enhancements included in the Tetra File-Log Agent v4.3.2 release:

  • A new Archive files with no checksum setting is now available for when the file watch mode for a path is set to File. The setting allows customers to still archive files when the files' MD5 checksums weren’t stored in the Tetra Data Platform (TDP) during upload. The new setting is visible as a toggle in the Tetra File-Log Agent management console when archive settings are activated for a path. When using this new setting, keep in mind the following:
    • If the value is set to No and a file’s MD5 checksum wasn’t stored during upload, the agent doesn’t archive the file. The agent then logs a warning message with the following event type: agents.filelog.fileArchiveSkipped.v1
    • If the value is set to Yes and a file’s MD5 checksum wasn’t stored during upload, the agent compares the file’s last modified date, file size, and the state recorded by the agent during upload. If these properties match, the agent then archives the file. If the properties don't match, the agent doesn’t archive the file, and then logs a warning message with the following event type: agents.filelog.fileArchiveFailed.v1
    • When a new path is added to an agent, or an agent is upgraded to include paths that contain archive settings, the setting’s default value is set to No. The Archive files with no checksum setting is also available through the TetraScience API by using the following field name: archive_without_checksum. In the API, the field accepts a boolean value only: true for the Yes value in the management console, and false for No.

📘

NOTE

Files uploaded to the Tetra Data Platform (TDP) by Tetra File-Log Agent v4.2.3 or earlier won’t have their MD5 checksums stored. The ability to calculate and store each file’s MD5 checksum wasn’t added to the Tetra File-Log Agent until v4.3.0.

Bug Fixes

The following bugs are now fixed:

  • A race condition related to paths with multilayer nested-directory structures is now resolved. The issue had previously resulted in scans sometimes showing a Complete status, even though some files in the path hadn’t been scanned. The agent now consistently completes the scan of all of a path’s files, except the ones that couldn't be accessed (for example, files with no error won’t be skipped). The files that return errors log a warning message and raise an event.

Deprecated Features

The following features have been deprecated for this release or are now on a deprecation path:

  • There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.2 release:

  • For a path configured in File mode, if a glob pattern does not start with “**/” and the path has system directories which the agent cannot access, the scan process for that path will terminate. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1, and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
  • In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. This doesn't actually remove the path from the agent. The issue will be addressed in a future Tetra File-Log Agent version. To delete a path from the agent in the management console, use the Delete option in the Action column of the grid for each path, and then choose Save.
  • In Folder mode, the file metadata of individual files within the uploaded zip file (created at, last modified, and last accessed) are set to the time that the agent creates the zip file, rather than retaining the source files' values. A fix for this issue is in development and testing and is planned for Tetra File-Log Agent v4.3.3.
  • Multiple changed files detected by the Agent via file system events may receive the incorrect archive configuration if they occur in a batch with files from other paths which have archive configuration (within ~10 seconds of each other). Although the probability that this condition occurs is small, it could cause files to be incorrectly archived. Please ensure the “Archive Files after Upload to TDP” option in Archive Settings is disabled on your FLA configuration. As an added safety measure, you can ensure the permissions for the service user used by the Agent is set to Read-only access.

Upgrade Considerations

After customers complete the upgrade to Tetra File-Log Agent v4.3.2 and open the management console for the first time, the console prompts them to run a database upgrade script. Customers must run this script to set the new Archive files with no checksum setting’s default value to No for any path that already has archive settings configured.

v4.3.1

Release date: 22 June 2023 (Last updated: 25 January 2024)

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.3.1. This release focuses on bug fixes and enhancements, which include improving agent scan performance, making scans more resilient, and new configuration options in the management console.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect “intended use” for GxP validation purposes.

  • There is no new functionality for this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system. The following are the enhancements included in the Tetra File-Log Agent v4.3.1 release:

  • The following settings can now be managed from the Tetra File-Log Agent management console:
    • Destination Id can be added, modified, or removed for an agent
    • Path-level labels can be added, modified, or removed for a path
    • If a path is configured in File mode, the archive and delete settings can be added, modified or removed for that path
  • Destination Id is configurable in the Tetra Data Platform (TDP) version 3.5.x, when the Tetra File-Log Agent version is 4.3.x or higher.
  • There is a new setting at path level Fetch OS File/Folder Owner. If this value is set to Yes, the OS file (or folder if path is configured as Folder) creator will be fetched by the agent during upload of a file from the path and then added as Amazon Simple Storage Service (Amazon S3) metadata. If this value is set to No, the OS file (or folder if path is configured as Folder) creator will not be fetched by the agent for any files of that path. When a new path is added, the default value for this setting is No. For paths upgraded from previous versions of the agent, the value is set to Yes. This setting can be configured both in the management console and TDP v3.5.x or higher (only if the corresponding agent is 4.3.1 or above). The setting can be configured in the TetraScience API as well. For more information, see the TetraScience API Documentation.
  • If a path is configured in File mode, containing System Volume Information directory (or other system directories), the agent can encounter “access denied” or similar errors, because it may not have permissions to access the system folders. The agent will raise an agents.filelog.scanError.v1 event, which includes the error details.

Bug Fixes

The following customer-reported bugs are now fixed:

  • Retrieving OS file creator was introduced in Tetra File-Log Agent v4.3.0. The scan of files was impacted, reducing its performance. This feature is configurable in 4.3.1 (see Enhancements for more details) and if set to Yes for a path, the OS file creator (or folder creator in Folder mode) is only fetched once during the upload process. The scan process now does not fetch the information, and its performance has been improved so that it’s now comparable to Tetra File-Log Agent versions prior to v4.3.0.
  • If a path is configured in File mode with a pattern starting with “**/” and containing System Volume Information directory (or other system directories), in previous versions, the scan would terminate as the agent does not have permission to access these system folders. In Tetra File-Log Agent v4.3.1, the scan will log an error, raise an event and continue to scan other files in the path. (Issue #2236)

Deprecated Features

The following features have been deprecated for this release or are now on a deprecation path:

  • There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.1 release:

  • For a path configured in File mode, if a glob pattern does not start with “**/” and the path has system directories which the agent cannot access, the scan process for that path will terminate. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1 and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
  • In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. This does not actually remove the path from the agent. The issue will be addressed in a future Tetra File-Log Agent version. To delete a path from the agent in the management console, use the Delete option in the Action column of the grid for each path, and then choose Save.
  • In Folder mode, the file metadata of individual files within the uploaded zip file (created at, last modified, and last accessed) are set to the time that the agent creates the zip file, rather than retaining the source files' values. A fix for this issue is in development and testing and is planned for Tetra File-Log Agent v4.3.3.
  • Multiple changed files detected by the Agent via file system events may receive the incorrect archive configuration if they occur in a batch with files from other paths which have archive configuration (within ~10 seconds of each other). Although the probability that this condition occurs is small, it could cause files to be incorrectly archived. Please ensure the “Archive Files after Upload to TDP” option in Archive Settings is disabled on your FLA configuration. As an added safety measure, you can ensure the permissions for the service user used by the Agent is set to Read-only access.

Upgrade Considerations

For any path upgraded from the previous Tetra File-Log Agent version, the Fetch OS File/Folder Owner setting will be set to Yes. If this metadata is not required for files uploaded from a particular path, consider setting this toggle off from the Tetra File-Log Agent management console, TDP UI, or TetraScience API.

v4.3.0

RELEASE DATE: 4/3/2023 (Last updated: 25 January 2024)

What's New?

TetraScience has released its next version of the Tetra File-Log Agent (FLA) version 4.3.0. This release focuses on new functionalities including the ability to archive/delete from local storage, define path level labels, and horizontally scale using destination ID, as well as enhancements for data integrity using MD5 checksums and support for non-ASCII characters in paths, to name a few. Several bug fixes and improvements are also included in this release.

For more details, review these sections:

  • New Functionality
  • Enhancements
  • Bug fixes
  • Deprecated features
  • Known and Possible Issues
  • Upgrade Considerations
  • Miscellaneous

New Functionality

New Functionality includes features not previously available in the File-Log agent. New functionality introduced in this version of the software includes:

  • When using File Watch Mode for paths, you have the option to configure file archive and delete settings. These settings can be applied through the TDP programmatic API (POST v1/agents/{agentId}/configuration). If enabled, files uploaded will be archived to the archive folder (and removed from the source folder), and optionally (if the delete setting is also enabled) deleted from the archive location after the configured time. These settings can be configured from the programmatic API, and the archive/delete settings for each path can be viewed in read-only mode in the agent management console. The archive settings include:
    • Archive path
    • Archive time (specified in minutes, hours, or days)
    • Delete time (specified in minutes, hours, or days)
      To know more about the considerations for using the archive/delete feature, see this section in FAQs
  • The summary tab in the agent management console displays the count of files that have been archived/deleted after the agent has been started (if archive/delete settings are configured for paths, these metrics will show the corresponding file counts). The following cumulative count of files from all paths the agent is watching, are displayed:
    • Pending Archive
    • Archived
    • Failed to archive
    • Pending delete
    • Deleted
    • Failed to delete
  • Destination ID can be configured through the TDP programmatic API (POST v1/agents/{agentId}/configuration) for one or more agents. Agents sharing the same Destination ID will upload files to the same data lake location defined using the Destination ID instead of Agent ID, allowing files from multiple agents to be uploaded to the same S3 location. This will allow paths to be split across multiple agents and scale horizontally.
  • Events for different agent operations (e.g. start/stop/ database backup), path scans and file scan/upload/archive/delete are emitted by the agent. Every event will have an Event Type and timestamp along with other pertinent information for that operation. The event types are explained below.
Event TypeDescription
agents.common.agentStarted.v1Occurs when the agent is started.
agents.common.agentStopped.v1Occurs when the agent is stopped.
agents.common.heartbeat.v1Occurs on a regular interval to indicate the agent is running.
agents.common.sqliteBackupStarted.v1Occurs when the SQLite database backup begins.
agents.common.sqliteBackupCompleted.v1Occurs when the SQLite database backup upload completes.
agents.common.sqliteBackupFailed.v1Occurs when an error occurs during SQLite database backup or upload.
agents.filelog.scanStarted.v1Occurs when a scan of a given path has started. This particular scan operation has a unique identifier associated with it and included in the event as scanId.
agents.filelog.scanCompleted.v1Occurs when a scan operation (identified by scanId) has completed. Contains the number of files scanned.
agents.filelog.pathValidationFailed.v1Occurs when a file's path validation fails (is inaccessible) before scan operation starts.
agents.filelog.fileScanned.v1Occurs when a file has been scanned and identified for upload.
agents.filelog.fileUploadStarted.v1Occurs when the upload of a file begins.
agents.filelog.fileUploadCompleted.v1Occurs when the upload of a file has completed successfully.
agents.filelog.fileUploadFailed.v1Occurs when an error occurs before or during the upload of a file.
agents.filelog.fileArchiveStarted.v1Occurs when archival of a file begins.
agents.filelog.fileArchiveCompleted.v1Occurs when a file is successfully archived.
agents.filelog.fileArchiveFailed.v1Occurs when an error occurs during file archival.
agents.filelog.archiveFileDeleteCompleted.v1Occurs when a file is successfully deleted from the archive location.
agents.filelog.archiveFileDeleteFailed.v1Occurs when an error occurs during archive file deletion.
  • For improved data integrity of files uploaded, when the file watch mode for the path is File, each file’s MD5 checksum is calculated and stored as S3 metadata on the platform (the MD5 checksum is base 64 encoded). If archive setting is enabled for the path, the checksum is validated during the archive process. If the checksum validation fails, the file will not be archived and a File Archive Failed event (agents.filelog.fileArchiveFailed.v1) will be raised. If the checksum validation fails, that would mean probably the file contents have changed. This would be detected by scan and a new version of the file will be uploaded. The archive will be attempted again after the defined archival window has elapsed since the last successful upload of the file.
  • For improved data integrity of folders uploaded, when the file watch mode for the path is Folder, once the folder is zipped during upload, the MD5 checksum of the zip is calculated and stored as S3 metadata on the platform (the MD5 checksum is base 64 encoded). Note that if the same folder/files are zipped multiple times, the checksum of the zip may be different each time due to the way zip programs access the files during the zipping process.
  • Path level label name/value pairs can be defined for paths using the TDP programmatic API (POST v1/agents/{agentId}/configuration). The settings can be viewed, in read-only mode, in the path configuration in the agent management console UI. If the same name/value combination is specified multiple times in the API as path level labels, the name/value combination is added to the path only once. The path level labels are applied to any file/folder uploaded from the monitored path when the agent has “Direct S3 upload” enabled.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system. Enhancements for this release appear below:

  • Agent ID, host, IP and Destination ID (if configured) are uploaded to TDP with agent heartbeat.
  • Agent audit and configuration API will contain all the new settings including agent level Destination ID, path level Archive/Delete and path level Labels (if specified).
  • OS file path (when the file watch mode for the path is File) and OS folder path (when the file watch mode for the path is Folder) are stored as S3 metadata. These metadata will be stored when S3 Direct Upload setting is enabled in the File-Log Agent Management Console. These will also be available in ElasticSearch doc and FileInfo API in TDP 3.5. The file path is stored in the original casing.
  • OS create user for file (when the file watch mode for the path is File) or folder (when the file watch mode for the path is Folder) are stored as S3 metadata. This metadata will be stored when S3 Direct Upload setting is enabled in the File-Log Agent Management Console. This will also be available in ElasticSearch doc and FileInfo API in TDP 3.5.
  • For new installation of 4.3, the “S3 direct upload” setting will be enabled by default. For agents upgraded from earlier versions to 4.3, the existing setting for this property will be preserved.
  • Upload algorithm has been optimized so that the upload of a large file does not block the upload of other files. In earlier agent versions, the upload job would pull a batch of files and wait for the upload of those to be completed before starting on the next batch of files to upload. In previous agent versions, the agent would process the upload in a batch of 500 files with the file upload job running on a schedule defined by “File upload job runs every” advanced setting. The upload job is now continuously running and loading files in the upload queue without batching (i.e. it will not wait for the largest file in a batch to finish uploading before starting to upload any new files). The algorithm now implements an improved round robin approach to select files from multiple paths so that a large number of files from one path do not block the upload of from other paths. The agent queries for the next file ready to upload in each path and pulls it into the upload queue so that the files from every path ready to be uploaded have even a chance to get uploaded. The “File upload job runs every” advanced setting is removed now and not used in the upload process.

Bug Fixes

The bug fixes for this release appear below:

  • Agent can now handle non-ASCII characters in paths e.g. accents, dieresis etc. This includes Unicode characters except those noted here (Issue# 1814).
  • In previous versions, the agent did not stop cleanly when a path with large number files was getting scanned. This has been fixed in 4.3 and the scan is canceled as soon as the agent is stopped so that the File-Log service/winapp is correctly stopped (Issue# 2335).
  • If there are multiple files in error (e.g. due to a network connectivity issue, etc.) during upload process, this may consume all the upload threads which will continue trying to upload these in previous agent versions, which may lead to scenario of the upload getting blocked altogether. In 4.3, the agent will try to upload and if the files are in error, File Upload Failed event (agents.filelog.fileUploadFailed.v1) will be raised for these files and they will be added to the end of the upload queue (the queue which processes files ready for upload). The agent will move on to the next files which are ready to be uploaded and process them. The events API (will be available in TDP 3.5) may be used to query the File Upload Failed to understand if any file is in a state of error and also the API can be used to aggregate such events to identify if certain files are always in an error state.

Deprecated Features

With the addition of events to the Agent, we’re going to deprecate some messages which were previously logged and thus may be relied upon by clients for monitoring purposes. In a future release, the following messages will stop being written to the logs by the Agent:

LevelJobMessageEvent Replacement
warnScanFolder Path Validation Erroragents.filelog.pathValidationFailed.v1
infoScanScanning path {Path}agents.filelog.scanStarted.v1
infoScanTotal detected files in ‘{Path}'agents.filelog.scanCompleted.v1
errorUploadRead file folder attribute error: {message}agents.filelog.fileUploadFailed.v1
warnUploadFile is too bigagents.filelog.fileUploadFailed.v1
warnUploadNot enough disk spaceagents.filelog.fileUploadFailed.v1
errorUpload(various error messages)agents.filelog.fileUploadFailed.v1
infoUploadjob_completeagents.filelog.fileUploadCompleted.v1
warnUploadjob_erroragents.filelog.fileUploadFailed.v1
infosqlite_uploadStart to backup Sqlite databaseagents.common.sqliteBackupStarted.v1

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.3.0 release:

  • If upgrading from File-Log agent 4.0.5 or earlier to 4.3, there may be a default path that will be present in the agent path configuration screen. After the agent upgrade, if the default path is present (that is not needed to be watched), then remove the default path before starting the agent.
  • In Folder mode, the file metadata of individual files within the uploaded zip file (created at, last modified, and last accessed) are set to the time that the agent creates the zip file, rather than retaining the source files' values. A fix for this issue is in development and testing and is planned for Tetra File-Log Agent v4.3.3.
  • Multiple changed files detected by the Agent via file system events may receive the incorrect archive configuration if they occur in a batch with files from other paths which have archive configuration (within ~10 seconds of each other). Although the probability that this condition occurs is small, it could cause files to be incorrectly archived. Please ensure the “Archive Files after Upload to TDP” option in Archive Settings is disabled on your FLA configuration. As an added safety measure, you can ensure the permissions for the service user used by the Agent is set to Read-only access.

Upgrade Considerations

If an agent is upgraded, the agent may require certain database objects to be updated. The user will be prompted for the upgrade of the database (a separate “Database updates required” prompt will be displayed) when the agent management console or service is started after the agent is upgraded to 4.3. The user may choose to update or exit. Since it is undesirable for the agent to run with an outdated database, the only options are to perform the updates, or not perform the updates and close the application. If the user chooses to “Close and Exit”, they will be prompted again next time the application is started. If the user clicks “Update”, a progress dialog will appear with the number of updates that need to be applied, and a progress bar for the current update. Depending upon the size of the database, the update may take several minutes to complete.

Miscellaneous

The File upload job runs every setting has been removed from the Advanced Settings section in the Configuration tab of the management console, as it is no longer used by the agent in 4.3.

v4.2.3

RELEASE DATE: 11/16/2022

Enhancements

There are no enhancements for this release.

Bug Fixes

The bug fixes for this release appear below.

  • AG-2579: FLA Uploads stop after several thousand using GDC/L7 Proxy

This resolves a potential hanging issue when using the GDC with L7 proxy in some workloads and environments. Versions 4.1.x through 4.2.2 are using the asynchronous AWS SDK methods and a customer has reported a scenario of the upload hanging when using GDC and L7 proxy on a single instance. Based on internal testing, we’ve found the synchronous AWS SDK methods to be more reliable for our use case and this release reverts to using those.

  • AG-2612: Memory leak when scanning path using recursive glob pattern ("**/")

The memory leak is due to task cancellation tokens were not disposed properly in the path scanning job.

📘

Note

This is not a critical patch and may only be necessary when experiencing this specific scenario.

Minor Functionality Changes

There are no minor functionality changes for this release.

v4.2.2

RELEASE DATE: 9/23/2022

🚧

Critical Patch

v4.2.2 is a critical patch. TetraScience strongly recommends that you upgrade to v4.2.2 if you have already upgraded to v4.2.x.

📘

NOTE

The release note for v4.1.4, which has the same bug fix as v4.2.2, has also been released.

Enhancements

There are no enhancements for this release.

Bug Fixes

The bug fixes for this release appear below.

  • AG-2367/AG-2396 Improves error handling and logging when re-initializing file system event listeners.

v4.1.4

RELEASE DATE: 9/23/2022

🚧

Critical Patch

v4.1.4 is a critical patch. TetraScience strongly recommends that you upgrade to v4.1.4 if you have already upgraded to v4.1.x.

Enhancements

There are no enhancements for this release.

Bug Fixes

The bug fixes for this release appear below.

  • AG-2367/AG-2396 Improves error handling and logging when re-initializing file system event listeners.

v4.2.1

RELEASE DATE: 9/1/2022

🚧

Critical Patch

v4.2.1 is a critical patch. TetraScience strongly recommends that you upgrade to v4.2.1 if you are using a previous version of the agent.

Enhancements

There are no enhancements for this version of the software.

Bug Fixes

AG-2338 - Paths configured with File Watch Mode “file” using a recursive glob pattern.
Any pattern starting with [**/], such as the following: 

**/*.txt, **/*, or  **/*.*

It can get "hung" indefinitely scanning (but not actually doing anything) if some unhandled errors occur. These errors are correctly handled and logging improved to better diagnose and troubleshoot issues of this nature.

Deprecated Features

There are no deprecated features for this version of the software.

Minor Functionality Changes

There are no minor functionality changes for this version of the software.

v4.1.3

RELEASE DATE: 9/1/2022

🚧

Critical Patch

v4.1.3 is a critical patch. TetraScience strongly recommends that you upgrade to v4.1.3 if you have already upgraded to v4.1.x.

Enhancements

There are no enhancements for this version of the software.

Bug Fixes

AG-2338 - Paths configured with File Watch Mode “file” using a recursive glob pattern.
Any pattern starting with [**/], such as the following: 

**/*.txt, **/*, or  **/*.*

It can get "hung" indefinitely scanning (but not actually doing anything) if some unhandled errors occur. These errors are correctly handled and logging improved to better diagnose and troubleshoot issues of this nature.

Deprecated Features

There are no deprecated features for this version of the software.

Minor Functionality Changes

There are no minor functionality changes for this version of the software.

v4.2.0

RELEASE DATE: 8/22/2022

Enhancements

Enhancements are new features and functionality that have been added to the software. Please check the section below to review the enhancements made for this version of this software:

AG-2104 - A file can be sent from TDP to the agent via a command service and the agent can receive the file and place it in a directory it can access. Refer to FLA Drop File Command to learn how to use this command service.

AG-1975/AG-2019/AG-2084 - The path watching/scanning algorithms and database performance have been optimized to improve the agent's scan performance. If you upgrade the Agent from the previous version, it might experience some delay when starting the Agent.

Bug Fixes

There are no bug fixes for this version of the software.

Deprecated Features

There are no deprecated features for this version of the software.

Minor Functionality Changes

There are no minor functionality changes for this version of the software.

v4.1.2

RELEASE DATE: 6/30/2022

Enhancements

There are no enhancements for this version of the software.

Bug Fixes

The following bug fixes were made to this version of the agent.

  • AG-2087 - When the agent is started, the scrollbar for the list of paths is no longer disabled. The scrollbar for the paths is enabled whether the agent is started or stopped.

Deprecated Features

Removed the LogWatcher functionality.

v4.1.1

RELEASE DATE: 6/8/2022

🚧

Critical Patch

v4.1.1 is a critical patch. TetraScience strongly recommends that you upgrade to v4.1.1 if you have already upgraded to v4.1.0.

Note also that the enhancements, bug fixes, and other changes found in v4.1.0 are in v4.1.1 as well.

Enhancements

No new features and functionality have been added to the software for this release.

Bug Fixes

Here are the bug fixes made for this version of the agent software:

  • AG-2017 - Fixes a bug that was introduced in the Tetra File-Log Agent v4.1.0, which caused the Agent to use the S3 Direct Upload option, even if it was not selected. Problem was resolved and the files are now uploaded using the option that the user selects for file upload.

Deprecated Features

There are no deprecated features for this version of the software.

Minor Functionality Changes

There are no minor functionality changes for this version of the software.

v4.1.0

RELEASE DATE: 31 May 2022

📘

NOTE

Version 4.1.0 of the Tetra File-Log Agent is no longer available. Please upgrade to version 4.1.1 instead. All enhancements, bug fixes, and other changes are in version 4.1.1 of the software.

Enhancements

Enhancements are new features and functionality that have been added to the software.

  • AG-1194/AG-1196/AG-1925: You can programmatically modify agent configuration using the API. (You can also modify the agent configuration via the user interface in the TDP.) New documentation has been produced that shows you how to use API endpoints to programmatically create and configure an agent. See the link to the documentation FLA Programmatic Agent Creation and Configuration here.
  • AG-1158: There is now a more equal priority for the ingestion of files from all paths. For every path, the Agent uploads the newest files first. Files from other paths won’t be blocked by a single path.
  • AG-1595: Global settings for the File Change Interval, Start Date, and Use Path Configurations in File-Log Agent Management Console have been removed. Also:
    • The File Change Interval and Start Date for each path are now displayed and must be filled in.
    • When the configuration is pushed from TDP to the agent or locally saved in the File-Log Agent Management Console, the Use Path Configurations is now always true.
    • If you are migrating file paths from a pre-FLA v4.1.0 version of the software:
      • If Use Path Configurations is false, the global File Change Interval and Start Date are copied to all path level information.
      • If Use Path Configurations is true, then there is no change.
  • AG-978/AG-1004/AG-1005: Configuration changes, as well as when the agent is stopped and started are now recorded in the Agent User Audit Trail.
  • AG-1177: When the FLA path configuration is changed locally or in the cloud, a copy of the path configuration is saved in the manual.json file, which is uploaded to TDP.
  • AG-947: File Properties, such as file creation date, last modified date, and file size are sent with files when uploading to TDP. They become part of the metadata of the S3 object.
  • AG-1584: The default folder path setting in the FileWatcher Service has been removed.
  • Several enhancements were made to improve throughput, performance, validation, logging, consistency, and usability. These include, (but are not limited to):
    • AG-1198: Improving how the FLA configuration is updated.
    • AG-1262: Adding more information in the logs, such as the agent host machine's IP, agent's host machine name, and connection type (CDC or GDC).
    • AG-1440: Allowing log files larger than 1 MB to be uploaded to Cloudwatch.
    • AG-1959: Changing “Enable Queue” label to "Receive Commands".
    • Improving caching, streamlining workflows, and improving performance.

Bug Fixes

The following bug fixes were made to this version of the agent.

  • AG-1845: Changes to files monitored by folder mode paths are now detected by the Event monitoring.

Deprecated Features

  • Removed the LogWatcher functionality.

Minor Functionality Changes

  • Logs include more information. See the enhancements section of these release notes.
  • AG-1949: Command response timestamps are now in UTC standard format.
  • AG-1560: The File-Log Agent now uploads files up to 5 TB when you use the S3 Direct Upload option. Large files that are uploaded no longer automatically timeout after an hour.

v.4.0.5

RELEASE DATE: 22 April 2022

Enhancements

Enhancements are new features and functionality that have been added to the software. Click the arrows next to each section to review the enhancements made for this version of this software:

  • AG-1750/AG-1795 We have improved FLA logs to help with troubleshooting. Logs now capture more information. Logs now include:
    • “changeDetectedBy” - how changes were detected, by an Event or by Scan - which is a task that the File-Log Agent software performs.
    • “fileWatchMode” - whether a file change was detected in file or folder mode.
    • “orgSlug” - the organizational identifier for your organization in TDP.
    • “message” - indicates when the folder path scanning starts and provides more details about what has been changed, such as the last write time, the length of the file or folder, and the file count. Additionally, permission issues are logged.

Bug Fixes

Click the arrows next to each section to review the bug fixes made for this version of the agent software:

  • AG-1796 - If FLA does not have the permissions needed to process a file that is part of a folder, it now shows an error that is displayed on the Management Console’s Summary page. This will allow you to be aware that it was not processed so that you can address the issue.

Deprecated Features

The LogWatcher feature is on the deprecation path.

Minor Functionality Changes

Logs include more information. See the enhancements section of these release notes.

v.4.0.4

RELEASE DATE: 16 March 2022

🚧

Critical Patch

Version 4.0.4 is a critical patch. TetraScience strongly recommends that you upgrade to v4.0.4.

Enhancements (New Features and Functionality)

There are no new features and functionality for this release.

Bug Fixes

The following bug fixes are available for the Tetra File-Log Agent:

  • AG-1432 The Start Date field in the File Log Watcher now works properly. The Start date field is documented here.
  • AG-1433 The File Watcher now correctly processes files that are stored in different paths but that have the same file name.

Other Notes

This release includes the enhancements and fixes made to Patch 4.0.2 and 4.0.3.

v.4.0.3

RELEASE DATE: 17 January 2022

📘

Note

Patch 4.0.3 is no longer available. Please upgrade to v4.0.4 instead.

v4.0.2

RELEASE DATE: 21 December 2021

Enhancements (New Features and Functionality)

This version of the software has a number of performance improvements due to transitioning the Tetra File-Log Agent from a 32-bit to a 64-bit architecture, and a reduced footprint.

Bug Fixes

The following bug fix(es) were made to the Tetra File-Log Agent.

  • AG-1209 - OOM while uploading system metrics
    This issue, which involved high memory usage, was resolved by performance improvements.

v4.0.1

RELEASE DATE: 3 August 2021

Bug Fixes

The following bug fix(es) were made to the agent.

  • [AG-781] - FLA Agent uses S3 direct upload as default
    This bug caused the S3 Direct Load option to always be enabled, even if you tried to manually disable it in the File-Log Agent's Management Console. This issue was fixed and you can now disable the S3 Direct Load option.

v4.0.0

RELEASE DATE: 21 July 2021

Enhancements (New Features and Functionality)

The following enhancements were made in this version of the software.

  • You can enable or disable the queue service. The queue service allows you to determine whether to receive a command from the TDP.
  • If you choose to enable the service, the Agent can input the path configuration pushed from TDP in the Cloud Config.
  • The end-to-end performance has improved 2x -3x. (It is measured from the time when the file is detected to the time file is uploaded to TDP).

Bug Fixes

The following bug fixes were made to the agent.

  • [AG-548] The Agent is able to upload empty file using the S3 direct upload option
  • [AG-531] The Agent installer can delete the installation folder from the previous version

Minor Functionality Change

  • The Agent is able to upload Agent Log with a file size greater than 1 MB using the S3 direct upload option.

v3.5.0

RELEASE DATE: 17 May 2021

This topic provides information on enhancements, bug fixes, deprecated features, as well as minor functionality changes for version 3.5.0 of the Tetra File-Log Agent.

Enhancements (New Features and Functionality)

There was an enhancement in this version of the software.

  • The Tetra File-Log Agent metadata and tag logic matches the latest TDP updates. For more information about TDP metadata updates, see the TDP 3.0.0 Release Notes.
  • A new option has been added that can check the status of the Tetra File-Log Agent daily and restarts it if it has stopped. For more information about how to enable this option see this section of the Tetra File-Log Agent installation guide.

Bug Fixes

The following bug fix(es) were made to the agent.
AG-292 - Tetra File-Log Agent Windows Management UI Agent Log Upload Interval Is Bound to a Wrong Field.

The Agent Log files Uploads Every field is now bound to the correct field (Interval In Seconds).

Deprecated Features

The following feature(s) are currently deprecated.

  • N/A

Minor Functionality Changes

The following minor functionality changes are in this version of the software.

  • The Tetra File-Log Agent now starts even if the File Path is invalid. The status has been added near the file path to indicate whether the path is valid. The status is refreshed periodically (with each scan interval).

  • When the agent shuts down or is backing up, "pending" records that have been scanned and not yet uploaded are deleted from the queue. (Please note that the records themselves are not deleted.) When the agent restarts again, those files will be rescanned and uploaded to the TDP. This solves an edge-case issue (AG-281).

  • Source Type in the File Watcher section (Folder Path Configuration) is now mandatory. Previously if this was left blank, the value was "unknown". Note that for the Log Watcher section, Source Type is still optional.

Additional Notes
There are no other notes for this version of the software.

v3.4.0

New Features

  • Users can re-upload the files with the file Last Write Date in a specific date-time range.
  • The agent can now notify the user of the potential errors when starting or stopping the agent Windows Service application.

Bug Fixes

  • The Agent is able to delete files with a read-only attribute from the local temp folder after the files are uploaded successfully to Tetra Data Platform.

Functionality Changes

  • The agent won't trigger file reupload when the source type, metadata, or tag change on the file path.

  • The agent converts the file path as the lower case letters when uploading the files to Tetra Data Platform (TDP).

  • If you have previously installed a v3.x.x File Log Agent, after you upgrade to v3.4.0, note the following.

    • New file paths will all be converted to lowercase.
    • For already-scanned and uploaded files by v3.x.x File Log Agent, if there is a change to the last modified date or the file size, it will be uploaded with the original casing (the same casing the first time this file was uploaded).

v3.3.1

New Features

  • Users can set up the file Start Date and File change interval on the individual file path level.

Enhancements

  • Long file path (longer than 260 characters) for Windows 10, Windows Server 2016, and Windows Server 2019 can now be enabled. (Please check here to learn more
  • The agent now notifies the user when s/he tries to close the Agent Windows Management Console UI without saving the changes.

Bug Fixes

  • The search result of the file path should be case-insensitive when the Agent checks whether that file path is already under monitoring.

v3.2.1

Enhancements

  • Agent doesn't require the user account configuring the Agent to have access permission to the folder paths. It delegates the folder path validation to the Windows Service part of the Agent.
  • The Windows Management Console logs the detailed errors for diagnosis if the connection to the TDP fails.

v3.2.0

Enhancements

  • Enhanced the internal algorithm to improve the performance of file change (new or updated) detection by 2-4 times.

Other Changes

  • Adjusts the order of file upload to prioritize the latest updated files.
  • Logs file access errors (e.g. file is moved or deleted) to Database. The Agent won't try to reprocess errored files in the later time interval.

v3.1.0

New Features and Functionality Changes

  • The Agent can now run using a Service Account without Local Logon permission.

v3.0.0

New Features and Functionality Changes

  • Upgraded to .NET Framework 4.8.
  • Provides the option to upload the files directly to AWS S3 bucket. With this option turned on, the user is able to upload up to a 500 GB file.
  • Files can be reuploaded when the associated Source Type, Metadata, or Tags are changed.
  • Added Summary page to display the file processing status and the Agent hosting server performance metrics.
  • Server performance metrics are now uploaded to AWS CloudWatch Metrics.
  • Sqlite database is uploaded to AWS S3 bucket for disaster recovery. (Please note this feature requires turning on the S3 direct upload option.)
  • No shortcut is added to the user desktop, instead, a link is added to the Windows Program menu.

Enhancements

  • Preserves the case of file and folder name.
  • Validates user account and permissions to the folder paths before the Agent starts.

Other Changes

  • Now uses a data grid to manage Tags.

v2.3.2

New Features and Functionality Changes

  • .NET Framework 4.7.2 is required.

Enhancements

  • In LogWatcherService, the output files are removed after they are successfully uploaded to the Platform.
  • Retains the Agent log files for 72 hours.
  • Allows users to check the configuration setting for both LogWatcherService and FileWatcherService when the Agent is running.
  • Popup Error notification message if File-Log Group User is invalid.
  • Popup Notification messages if File-Log Group User doesn't have read permission of the paths. defined in the Filewatcherservice or LogWatcherService.
  • The authentication errors are written to Agent Log files.

Bug Fixes

  • Fixed Group User account is not used for running the File-Log Agent Service issue.
  • Fixed System.Net.Http dll (4.2.0.0) reference issue.
  • Fixed validation message when the pattern is missing from folder mode.
  • Fixed NullReferenceException if JWT Token is not provided issue.

Installation Changes

  • Applied Full Control permission to the installation folder for Machine\Users to avoid database permission issues.
  • In migration, if the config.json from the previous version is incomplete, the Agent migrates the sections available in the config.json file.
  • In migration, if the config.json from the previous version is invalid or missing, the Agent sets up configuration using default values.

Configuration Changes

  • By default, FileWatcher is enabled and LogWatcher is disabled.
  • Default agent ID and connector URL is blank.
  • Default path is blank.
  • All job time intervals are set as 10 seconds as the minimum.

v2.3.1

  • Added a Windows-based Management Console UI to replace Config.json for Agent configuration.
  • In the LogWatcher Service, grouped related output files into a subfolder in the output folder, to prevent potential file name collision.

Updated 6 months ago