The following are the Tetra File-Log Agent release notes for versions 4.4.x.

📘

NOTE

To upgrade the Tetra File-Log Agent, see Common Tetra Agent Upgrade and Installation Information..

For installation instructions, see Tetra File-Log Agent Installation Guide (Version 4.4.x).

v4.4.0

Release date: 4 September 2024

What's New

TetraScience has released its next version of the Tetra File-Log Agent, version 4.4.0. This release re-enables the Archive and Delete features with new checks in place to ensure data integrity. The Archive and Delete features allow customers to programmatically archive and delete source files uploaded by the agent after a configured time period. This release also includes enhanced proxy support, a new End Date setting for path configurations, and several bug fixes.

New Functionality

New Functionality includes features not previously available in the Tetra File-Log Agent. These features may or may not affect Intended Use for GxP validation purposes.

• There is no new functionality in this release.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system.

Archive and Delete Workflow Enhancements

🚧

IMPORTANT

Before you activate the Tetra File-Log Agent's Archive and Delete features, keep in mind the following:

• Tetra Data Platform (TDP) v4.0.3 or higher is required for the Tetra File-Log Agent v4.4.0's Archive and Delete features to work.
• If you configure Data Access Rules for the organization that the Tetra File-Log Agent is in, you must add the agent's service user to an access group that has at least Read permissions for any files the agent uploads. Otherwise, archive and delete actions fail and return a 403 error.

The Archive and Delete features have been enhanced to ensure files are archived and present in the TDP before any source files are archived or deleted.

New Archive Workflow

1. A file is uploaded from a path that includes an Archive configuration. The archive date and time is calculated based on the upload time and the configuration on the path.
2. When the archive time is reached, the agent performs simple last-modified and length checks of the file to ensure it has not been changed since it was uploaded.
3. The agent then calls the TDP to retrieve a verified checksum from the platform for the file version.
   • These checksums are either verified and provided by Amazon Simple Storage Service (Amazon S3) or by the TDP when not available from Amazon S3.
   • These checksums are now available regardless of upload method (S3 Direct Upload or using a Data Connector) and the version of the agent that uploaded the file.
4. If no verified checksums are available for the file on the TDP, the file isn't archived.
5. If a verified checksum is present, the agent will calculate the checksum by using the same algorithm as the TDP-verified checksum.
6. If the checksums match, the file is archived to the configured archive path. If the checksums do not match, the archival fails and the file remains in the source location.

📘

NOTE

If archival fails for any reason, it is retried after a delay of two minutes (up to a maximum of 24 hours). Each file can be attempted up to 40 times by default (configurable through Advanced Settings). This behavior results in 10 attempts in the first 34 hours, then one attempt a day for the next 30 days.

New Delete Workflow

1. A file is archived following the new Archive workflow. The delete date and time is calculated based on the archive time and the configuration on the path.
2. When the delete time is reached, the agent calls the TDP to retrieve a verified checksum from the platform for the file version.
3. If no verified checksums are available for the file on the TDP, the file is not deleted.
4. If a verified checksum is present, the agent will calculate the checksum by using the same algorithm as the TDP-verified checksum.
5. If the checksums match, the file is deleted from the archive path. If the checksums do not match, the deletion fails and the file remains in the archive location.

📘

NOTE

If deletion fails for any reason, it is retried after a delay of two minutes (up to a maximum of 24 hours). Each file can be attempted up to 40 times by default (configurable through Advanced Settings). This behavior results in 10 attempts in the first 34 hours, then one attempt a day for the next 30 days.

Important Considerations

• Archive and Delete features are available when the File Watch Mode is set to File only.
• The Archive and Delete features are automatically re-enabled when upgrading to Tetra File-Log Agent v4.4.0. Any paths which have Archive configuration will begin archiving files by using the new workflow unless the Archive configuration is removed from the paths before starting the Agent service.
• Installations that rely on a Tetra Data Hub, will need to configure the TDP API URL Override in the agent's Advanced Settings, Org Slug, and Authorization to retrieve verified checksums from the TDP for the archival process.
• As part of this improvement, the Archive without checksum setting is no longer an option when configuring file paths for the Tetra File-Log Agent. This option had allowed files to be archived that didn’t have a checksum in the Agent database, because they had been uploaded by an earlier Agent version originally. Now, files will always have verified checksum values calculated, regardless of how they were first uploaded.

Proxy Enhancements

Advanced Settings now include the following options for configuring how the Agent interacts with a proxy:

• System (default setting): Uses the system proxy specified in Windows for the service user. If basic authentication is required, a username and password may be specified for the proxy.
• Custom: Makes all of the Agent's proxy options configurable, regardless of any other settings. This proxy takes precedence over any existing system proxy or L7 proxy.
• None: Explicitly configures the Agent service to not use any proxies for web requests.

Other Enhancements

• File uploads with S3 Direct Upload enabled now use a SHA256 checksum to ensure data integrity on uploads (instead of the MD5 checksum used previously).
• File upload, archive, and delete operations now have limited, delayed retries—also known as a retry with exponential backoff pattern. Previously, the agent would retry failed operations indefinitely, which could overload network bandwidth and cause contention. Now, these operations are retried by increasing wait times for a specified number of retry attempts. The agent now enqueues the file for retry after a delay starting at 1 minute, doubling the delay every retry up to a maximum of 24 hours. Each operation is attempted up to 40 times by default (configurable through Advanced Settings). This behavior results in 10 attempts in the first 34 hours, then one attempt each day for the next 30 days.
• File uploads performed through a Generic Data Connector (GDC) or TDP (No Connector) connection now also perform checksum validation for the leg of the upload from the Agent to the connection endpoint on TDP version 4.1.0 and higher.
• Customers can now specify path End Date. Similar to the Start Date, an End Date defines the end of the time range for identifying files to upload. The End Date is inclusive. Any files with a Last Modified date that is on or before the end of the End Date are uploaded.
• Proactive permission checks have been removed before scanning. This will alleviate false positives typically seen with Network Attached Storage solutions that could cause path validation to fail.
• Paths configured via the TDP UI or API will have a trailing backslash appended to be consistent with paths configured via the Agent Management Console.
• During upgrades, the Agent now verifies and removes the previous Agent version's Windows Service application.
• Customers can now download the Agent's local configuration file in JSON format from the Tetra Data Platform (TDP) by selecting the Agent's record on the Agent page, and then selecting the Download Configuration button under the Info tab.
• The Agent Management Console now has an About tab with specific version details. The new tab includes a Copy button that you can use when communicating with the TetraScience Support Team to provide the specific version and machine configuration (for example, Operating System, OS version, and locale information).
• The Agent Management Console now displays a warning if an instance is already running on the machine.
• The Agent Management Console now displays more detailed connection errors when unable to connect.
• The agents.filelog.scanCompleted.v1 event now includes a durationInSeconds property with a floating point number representing the duration of the scan in seconds. This is in addition to the existing duration field which is a string representing a timespan.
• The agents.filelog.scanCompleted.v1 event now includes the duration and fileCount properties, even if they are zero. The duration property is also more accurate now, which helps avoid zero values.
• Dates displayed in the Agent Management Console are now consistent with the TDP format (yyyy-mm-dd).

Bug Fixes

The following bugs are now fixed:

• Each path scan now correctly has a unique scanId in logs and events.
• The Agent service now checks to see if an instance is already running and prevents multiple instances from running at the same time.
• Database locking has been reduced.
• Agent logs are now always retained locally for 72 hours, no matter how the Agent's TDP connection is configured.
• Tags on File Watcher paths are now limited to 114 characters or less so that they conform to TDP tag character limits.
• After a new Agent installation or upgrade, the Agent Management Console window now displays consistently.
• Files that the Agent service user does not have permissions to read are now correctly marked as Failed to Archive or Failed to Delete instead of assuming the file has already been deleted.
• Malformed event files no longer prevent other events from uploading.
• Timestamps in event files now use the UTC offset instead of the local offset of the agent machine.
• Files that have a .gz extension no longer have their Content-Encoding set to gzip incorrectly by the Agent.
• Modifying Agent connection information only now causes the Agent to upload its new configuration to the TDP.
• The Management Console no longer logs the original Agent Id if the ID is changed.
• The Management Console no longer crashes when there is insufficient disk space to save configuration changes to the local database.

Deprecated Features

There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

The following are known and possible issues for the Tetra File-Log Agent v4.4.0 release:

• Files uploading when a path is updated with an Archive configuration can sometimes get stuck in the Pending Archive status. A fix for this issue is in development and testing and is scheduled for a future patch release.
• For a path configured in File mode, if a glob pattern does not start with “\*\*/” and the path has system directories that the agent can't access, the path's scan process terminates. Check the logs and look for agents.filelog.scanStarted.v1, agents.filelog.scanError.v1, and agents.filelog.scanCompleted.v1 for a path to identify if the scan was terminated or completed.
• In the Service Settings section, if an existing path is highlighted and the DEL (Delete) keyboard key is pressed, the row is removed from the grid. However, this action doesn't actually remove the path from the agent. A fix for this issue is in development and testing and is planned for a future Tetra File-Log Agent release. To delete a path from the agent in the Agent Management Console, customers must use the Delete option in the Action column of the grid for each path, and then choose Save.

Upgrade Considerations

When upgrading to Tetra File-Log Agent v4.4.0, keep in mind the following:

• For installations that use a Tetra Data Hub, activating the Archive feature requires the following configurations in the agent's Advanced Settings so that the agent can authenticate with the TDP and retrieve verified checksums:
  • For TDP API URL Override, enter the TDP API endpoint for your deployment (for example, platform.tetrascience.com).
  • For Org Slug, enter the org slug for the organization that the agent is in.
  • For Authorization, enter the JSON Web Token that you're using.
• If a Tetra Data Hub acts as a proxy for the agent and the Archive feature is activated, the TDP API endpoint must be accessible from either the Agent host or from the Data Hub Host if using a Tetra L7 Proxy Connector. For more information, see Tetra Agent Allow List Endpoints.