Release Date: 22 February 2023 (Updated: 14 September 2023)

📘

Quality Report

TetraScience is committed to creating quality software. Software is developed and tested using our TetraScience Quality System. The Quality Report for this version is referenced here: TPM-140. Customers who have purchased the GxP package can request a copy of the Quality Report. If customers have further questions about the Quality Report or the GxP package, please contact the Customer Success Manager (CSM).

What's New?

TetraScience has released its next version of the Tetra Data Platform (TDP) version 3.4.0. Focuses for this release include enhancements to monitoring and system verification, supportability, usability and performance, and primary data access.

With TDP v3.4.0, customers can resolve issues faster with expanded visibility for Read-Only role access and quickly send File Log Agent diagnostic logs to TetraScience support. Customers can now prioritize pipeline execution and resource allocation, too. There is also improved performance on the File Processing Page and improved access to primary data files.

For details on specific feature and functionality improvements, review these sections:

• New Functionality
• Enhancements
• Bug Fixes
• Deprecated Features
• Known and Possible Issues
• Upgrade Considerations
• Security
• Miscellaneous
• Other Release Notes

Links to other Tetra Release Notes are in the Other Release Notes section at the end of this topic.

New Functionality

New Functionality includes features not previously available in TDP. These features may or may not affect "intended use" for GxP validation purposes, as indicated below. New functionality introduced in this version of the software includes:

Verification Report
TDP 3.4 has a new feature that allows GxP Package customers to download a verification report on demand. The report contains system information and the most recent results of automated verification tests that are run every three hours. For more information, see this link.

Pipeline Usability
Customers can set pipeline priorities for more intelligent workflow resource assignments to pipelines. To learn more, see this link.

Primary File Access
Bulk Downloads - Instead of downloading files one at a time or having to write a custom script to send several requests to the API, customers can now select up to 100 files from search results to download to their computer or device. More information is available here.

Supportability *

• Phone Home Telemetry - The Phone Home Telemetry feature periodically sends configuration and usage information from all deployments (ST/MT) back to a central TS-managed service (S3 bucket). The collected information is used by TetraScience support teams to support customers with debugging issues. TetraScience teams will also use this data to understand product usage in an effort to improve user experience. For more information, see this link.
• Collect File-Log Agent Logs - With the click of a button, site administrators can collect FLA Agent diagnostic logs and push them to a TetraScience-managed S3 bucket to streamline support and maintenance efforts. To learn more, see this link.

* These features are for supportability and troubleshooting, and do not affect "intended use" for validation purposes.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but do not alter the function or intended use of the system. Enhancements for this release appear below:

Pipeline Enhancements

• Workflow Detail Screen Improvements - The Workflow Details screen has been redesigned to make troubleshooting and finding statuses easier. To learn more, see this link.
• File Processing Screen Improvements - The File Processing Screen interface has been improved to provide easier visualization of multi-step pipeline progress. Performance improvements, which are in the Scaling and Performance Improvements section of these notes have also been made. More information is available here.
• Pipeline Is Not Trigger Value - The pipeline is not trigger value now verifies that there is a pipeline value and that it is not the value specified. pipeline is not no longer triggers unprocessed pipelines. More information is available here.

Scaling and Performance Improvements
Database usage has been improved and the current implementation simplified. These tables have also been restructured to improve workflow and report performance. There have been other improvements, such as making validation rules more consistent. For more information, see this link.

Additionally, there have been:

• 4-5x reduction in file registration times in DB performance
• 5-10x improvement in response times across the File Processing Page UI and API
• 2x improvement on high volume file processing time from S3 to Athena

Supportability

• Health Monitoring (File Processing Failures) - Additionally, in the Files tab’s File Processing Failures section, there are detailed logs for each failure. Logs can be searched for the selected file. More information is available here.
• Read-Only Access for Debugging Purposes - Read-Only access is now available in more areas of the TDP user interface to help support TetraScience support team for debugging purposes. Role privileges are listed here.

Other Improvements

• Several user interfaces were improved, including the Audit Trail and Service User Interfaces. To learn more, see this link.
• The SQL Search screen now adds a Copy Table Name functionality. Customers can copy the table name to their clipboards so that they can be used in SQL queries. For more information, see this link.
• The collection URL has been shortened and simplified; this makes it easier to copy the collection into an ELN/LIMS. More information is available here.
• Members can create collections for all users to access the saved search stored in a collection link. To learn more, see this link.
• Improved logging and added capability to decode XML messages for the Solace Connector. Solace Connector Documentation is here.
• S3 Intelligent Tiering Rules have been extended to also apply to previous, non-current versions of the files in TDP. For more information on S3 Intelligent Tiering, see https://aws.amazon.com/s3/storage-classes/intelligent-tiering/.
• To help customers avoid issues caused by the under-provisioning or over-provisioning of RDS resources, TetraScience has enabled RDS Storage Autoscaling by default. RDS Storage Autoscaling is an AWS feature that continuously monitors actual RDS storage consumption. It automatically scales up capacity when the actual utilization approaches provisioned storage capacity. Customers receive alerts when RDS storage reaches 75% usage. When RDS storage reaches 90% usage or more, autoscaling occurs. For more information on this feature, see https://aws.amazon.com/about-aws/whats-new/2019/06/rds-storage-auto-scaling/. Additional technical details can be found here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling.
• The SOURCE TYPE values listed on the TDP now appear in all lowercase with hyphens (-) for spaces, rather than in title case with blank spaces. This change makes the values consistent with the formatting that's required by Tetra Data Pipeline triggers and the Tetra File Log Agent.

Bug Fixes

Customer-reported bugs that have been fixed appear below:

• The workflow history for files now properly shows up in the detailed file view page. (Issue# 2010)

• Customers can now add and modify tags, metadata, and labels if they are not recorded in the Attribute Management screen. (Issue #2011)

• Fixed the issue with the mapping of property data.results changes. If customers can find the files using the manual filter, but not with the IDS custom filter, there might be a problem with how the IDS custom filter was cached. (Issue #1750)

• Displaying logs for failed workflows took an excessive amount of time (~1 min). The fix resolves log display latency. This issue is experienced when there are many historical workflows and the deployment was upgraded from a past release. This is the same fix as DP-603, which was reported in the 3.3.4 release notes. (Issue #1965)

• AWS recently made changes to their IAM role trust policy that breaks the functionality being used by Tetra Datahubs to get credentials that enable the Agent to upload data via S3. This change makes fixes so that the software now works with the new IAM role trust policy. This change impacts all organizations created after September 21, 2022. without this fix (which is also in TDP version 3.3.5), these organizations will not be able to use S3 Upload (or the Receive Commands feature) on any Agent connected to any Data Hub that tries to use S3 Upload. Organizations created before September 21, 2022 use the old IAM role trust policy method. Amazon has allowed any device using the old method a grace period until February 2023 to continue with that method but only for those devices which have done so since June 30th 2022. If any Agent on any Datahub in a given Organization has been online in S3 mode (or with the Receive Commands feature), with a valid connection, since June 30th it will continue to work as will all Agents attached to all Datahubs in that Organization. But, if no Agent has connected in that time then all Agents on all Datahubs in that Organization will no longer be able to work in S3 mode. At the end of Amazon’s grace period (June 30, 2023), all organizations will be affected.

• For Athena, all resources (where applicable) are now locked to the AWS region. (Issue #1048)

• A change reason is now required when customers manually upload a file or a new version of a file is created. Customers will be prompted for a change reason whenever they perform an action in the TDP that results in these things.

• Disabled user accounts are now recorded as having “disabled” status in the audit trail log. (Issue #1959)

• The platform version now loads more quickly. (Issue: #2040)

• The Pipeline Editor user interface now allows the user to deselect the Use Shared Setting value if the protocol does not require shared settings. (Issue: #2175)

Deprecated Features

The features that have been deprecated for this release or that are on a deprecation path appear below. For more deprecations, see Deprecation Notices.

Of particular interest is the deprecation of the Cloud Data Connector:

We are deprecating the Cloud Data Connector (CDC) to simplify agent configuration. As of 3.4, creating CDCs and adding agents to existing CDCs will be disabled. If an agent needs a Datahub and Generic Data Connector (GDC), agent creation will be unchanged. If not, new Agents can be created from the Agents page with ‘No Connector’ selected. The features that have been deprecated for this release or that are on a deprecation path appear below. For more deprecations, see TDP Deprecation Notices.

Known and Possible Issues

Known and possible issues for this release appear below.

• Issue: The File details page shows related workflows when a user opens it via a search page. It only shows details for IDS files. Workaround: To get more workflow information, visit the workflow processing page.
• Issue: For large systems, querying data via Athena might result in AWS limit rate exceeding errors. Workaround: To remediate this issue, specify additional query filters so that fewer results are returned.
• Issue: When you select a collection, click an option and then close it without making a change, the search filters reset.
• Issue: During very heavy pipeline processing (thousands of files per minute) the file processing page might return data slower than usual or might time out.
• Issue: During the 3.4.0 upgrade the Data Hub will go offline if customers have a Data Hub GDC that has a legacy configuration (HTTP or HTTPS ports with numbers below 1024). Note that editing the GDC or adding Agents will fail until the port is updated. Resolution: If possible, change the Data Hub's GDC insecure and secure port numbers to 1024 or higher, then change the connection URL for the agent and host machine to reflect the new GDC port number(s). If this is not possible, please contact TetraScience for further assistance.

Security

Various security improvements have been implemented.

Upgrade Considerations

During the upgrade, there might be a brief downtime when users will not be able to access the platform. This includes the TDP User Interface and the APIs. After the upgrade is complete, customers should check their pipelines for failures or cancellations and reprocess those if found. Customers should do a smoke test of their custom configurations as well to ensure there are no failures.

Post-upgrade, the TetraScience team ensures that the platform infrastructure is working as expected through a combination of manual and automated tests. If any failures are detected the issues are immediately addressed or the release is rolled back.

For additional information on the release schedule, including the GxP release schedule and timelines, please see: https://developers.tetrascience.com/docs/release-schedule.

For more details surrounding the timing of the upgrade, please see your CSM.

Miscellaneous

The Athena SQL number value is a 64-bit signed double-precision floating point number. The range is 4.94065645841246544e-324d to 1.79769313486231570e+308d, positive or negative.

Other Release Notes

Click here to view all Release Notes.