Release date: 30 May 2024

TetraScience has released its next version of the Tetra Data Platform (TDP), version 4.0.2. This release introduces a 10x performance improvement for query domain-specific language (DSL) queries and a new type of Embedded Data App based on Streamlit, the FPLC Data Explorer. It also includes enhancements to the TDP user interface and resolves several known issues.

Here are the details for what’s new in TDP v4.0.2.

📘

Security

TetraScience continually monitors and tests the TDP codebase to identify potential security issues. Various security updates are applied to the following areas on an ongoing basis:

• Operating systems
• Third-party libraries

📘

Quality Management

TetraScience is committed to creating quality software. Software is developed and tested following the ISO 9001-certified TetraScience Quality Management System. This system ensures the quality and reliability of TetraScience software while maintaining data integrity and confidentiality.

New Functionality

New functionalities are features that weren’t previously available in the TDP.

📘

GxP Impact Assessment

All new TDP functionalities go through a GxP impact assessment to determine validation needs for GxP installations. New Functionality items marked with an asterisk (*) address usability, supportability, or infrastructure issues, and do not affect Intended Use for validation purposes, per this assessment. Enhancements and Bug Fixes do not generally affect Intended Use for validation purposes, and items marked as beta release or early adopter program (EAP) are not suitable for GxP use.

Data Access and Management New Functionality

Embedded Data Apps Based on Streamlit (EAP)*

Updated to EAP from Beta Release on 4 October 2024

A new type of Embedded Data App based on Streamlit allows users to automatically authenticate into their Tetra Data & AI Workspace sessions through the TetraScience API.

The Fast Protein Liquid Chromatography (FPLC) Data Explorer introduced in TDP v4.0.2 is the first of many Embedded Data Apps based on Streamlit to come. It provides a dashboard for visualizing contents of AKTA Intermediate Data Schema (IDS) files. Customers can use the tool to quickly search for AKTA results, select and overlay chromatograms, and then export analysis results.

Embedded Data Apps based on Streamlit and the FPLC Data Explorer are available through an early adopter program (EAP) currently and may require changes in future TDP releases. If you are interested in using them, please contact your customer success manager (CSM).

For more information, see Embedded Data Apps.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.

Data Access and Management Enhancements

10x Performance Improvement for Query DSL Queries

Query domain-specific language (DSL) queries now return responses 10 times faster than they did in TDP v4.0.1 and earlier.

For more information, see Query DSL Queries.

UI Improvements for Embedded Data Apps Based on Windows

The following user interface (UI) improvements have been made for Embedded Data Apps based on Windows:

📘

NOTE

The Tetra FlowJo Data App is the only Embedded Data App based on Windows currently.

• A new Session Files sidebar displays all files that are part of the user’s current Tetra Data & AI Workspace session. Users can download files directly from the sidebar, or copy the file contents to their clipboard if the file is 15 MB or less.

• When adding a new Embedded Data App based on Windows, an Accept Terms dialog now appears that reminds Org Admin users that they must sign an addendum to their tenant’s end-user license agreement. The new dialog asks users to confirm that they agree to the terms and conditions, and that all agreements have been signed.

System Administration Enhancements

Improved UI for Adding and Removing Embedded Data Apps

A new Manage Data Apps dialog on the Tetra Data & AI Workspace page provides Org Admin users the ability to add or remove multiple Embedded Data Apps in one place.

For Embedded Data Apps based on Streamlit, Org Admin users can also now use the Manage Data Apps dialog to select which Streamlit container version they want their organization to use.

For more information, see Activate Embedded Data Apps.

Search System Logs by Partial File Names

Customers can now search for system logs by partial file name when using the Entity filter on the System Log page.

For more information, see System Log.

Security Enhancements

Database Instances Enforce Encryption In Transit

All communication between TDP services and database instances is now encrypted in transit.

Bug Fixes

The following bugs are now fixed.

Data Access and Management Bug Fixes

• The SQL Query Editor’s query generation feature now works with Intermediate Data Schemas (IDSs) in the private namespace. (Issue #3446)

System Administration Bug Fixes

• All TDP user authentication issues caused by the upgrade to TDP v4.0.0 and v4.0.1 are now resolved.
• The Go To Agent links on the Agents Health Monitoring dashboard now work for Agents configured with a subdomain.

Deprecated Features

There are no new deprecated features in this release.

For more information about TDP deprecations, see Tetra Product Deprecation Notices.

Known and Possible Issues

Last updated: 22 October 2024

The following are known and possible issues for TDP v4.0.2.

Data Harmonization and Engineering Known Issues

• File statuses on the File Processing page can sometimes display differently than the statuses shown for the same files on the Pipelines page in the Bulk Processing Job Details dialog. For example, a file with an Awaiting Processing status in the Bulk Processing Job Details dialog can also show a Processing status on the File Processing page. This discrepancy occurs because each file can have different statuses for different backend services, which can then be surfaced in the TDP at different levels of granularity. A fix for this issue is in development and testing.
• Logs don’t appear for pipeline workflows that are configured with retry settings until the workflows complete.
• Files with more than 20 associated documents (high-lineage files) do not have their lineage indexed by default. To identify and re-lineage-index any high-lineage files, customers must contact their CSM to run a separate reconciliation job that overrides the default lineage indexing limit.
• OpenSearch index mapping conflicts can occur when a client or private namespace creates a backwards-incompatible data type change. For example: If doc.myField is a string in the common IDS and an object in the non-common IDS, then it will cause an index mapping conflict, because the common and non-common namespace documents are sharing an index. When these mapping conflicts occur, the files aren’t searchable through the TDP UI or API endpoints. As a workaround, customers can either create distinct, non-overlapping version numbers for their non-common IDSs or update the names of those IDSs.
• File reprocessing jobs can sometimes show fewer scanned items than expected when either a health check or out-of-memory (OOM) error occurs, but not indicate any errors in the UI. These errors are still logged in Amazon CloudWatch Logs. A fix for this issue is in development and testing.
• File reprocessing jobs can sometimes incorrectly show that a job finished with failures when the job actually retried those failures and then successfully reprocessed them. A fix for this issue is in development and testing.
• On the Pipeline Manager page, pipeline trigger conditions that customers set with a text option must match all of the characters that are entered in the text field. This includes trailing spaces, if there are any.
• File edit and update operations are not supported on metadata and label names (keys) that include special characters. Metadata, tag, and label values can include special characters, but it’s recommended that customers use the approved special characters only. For more information, see Attributes.
• The File Details page sometimes displays an Unknown status for workflows that are either in a Pending or Running status. Output files that are generated by intermediate files within a task script sometimes show an Unknown status, too.

Data Access and Management Known Issues

• In the Filters dialog on the Search page, tag-related filters have two related known issues:

  • The Select a filter type dropdown doesn’t provide the option to filter search results by Tags.
  • The Tag Name filter’s dropdown doesn’t automatically populate with all of an organization’s available tags.

  As a workaround, customers can do the following:

  • To search for files with specific tags, customers should enter the exact tag value in the Tag Name filter field. (To retrieve a list of an organization’s existing tags, customers can reference the Tags filter dropdown on the Search (Classic) page.)
  • To search for files that exclude specific tags, customers can do either of the following:
    • Search by tags on the Search (Classic) page.
    • Add the tag to the Tags tab on the Attribute Management page before running the search. This action populates the Select a filter type dropdown with the added tags.

  A fix for this issue is in development and testing and is scheduled for a future TDP release. (Issue #3916) (Added on 22 October 2024)

• Query DSL queries run on indices in an OpenSearch cluster can return partial search results if the query puts too much compute load on the system. This behavior occurs because the OpenSearch search.default_allow_partial_result setting is configured as true by default. To help avoid this issue, customers should use targeted search indexing best practices to reduce query compute loads. A way to improve visibility into when partial search results are returned is currently in development and testing and scheduled for a future TDP release. (Added on 25 September 2024)

• Text within the context of a RAW file that contains escape (\) or other special characters may not always index completely in OpenSearch. A fix for this issue is in development and testing, and is scheduled for an upcoming release.

• If a data access rule is configured as [label] exists > OR > [same label] does not exist, then no file with the defined label is accessible to the Access Group. A fix for this issue is in development and testing and scheduled for a future TDP release.

• When using SAVED SEARCHES created with the Search Files page (Search (Classic)) prior to TDP v4.0.0, the new Search page can sometimes appear blank. A fix for this issue is in development and testing and planned for a future TDP release. As a workaround, customers should recreate the saved search by using the new Search page.

• File events aren’t created for temporary (TMP) files, so they’re not searchable. This behavior can also result in an Unknown state for Workflow and Pipeline views on the File Details page.

• File events aren’t created for temporary (TMP) files, so they’re not searchable. This behavior can also result in an Unknown state for Workflow and Pipeline views on the File Details page.

• When customers search for labels that include @ symbols in the TDP UI’s search bar, not all results are always returned.

• When customers search for some unicode character combinations in the TDP UI’s Search bar, not all results are always returned.

• If customers modify an existing collection of search queries by adding a new filter condition from one of the Options modals (Basic, Attributes, Data (IDS) Filters, or RAW EQL), but they don't select the Apply button, the previous, existing query is deleted. To modify the filters for an existing collection, customers must select the Apply button in the Options modal before you update the collection. For more information, see How to Save Collections and Shortcuts.

• The File Details page displays a 404 error if a file version doesn't comply with the configured Data Access Rules for the user.

TDP System Administration Known Issues

• The latest Connector versions incorrectly log the following errors in Amazon CloudWatch Logs:
  • Error loading organization certificates. Initialization will continue, but untrusted SSL connections will fail.
  • Client is not initialized - certificate array will be empty
    These organization certificate errors have no impact and shouldn’t be logged as errors. A fix for this issue is currently in development and testing, and is scheduled for an upcoming release. There is no workaround to prevent Connectors from producing these log messages. To filter out these errors when viewing logs, customers can apply the following CloudWatch Logs Insights query filters when querying log groups. (Issue #2818)
    CloudWatch Logs Insights Query Example for Filtering Organization Certificate Errors

  fields @timestamp, @message, @logStream, @log
  | filter message != 'Error loading organization certificates. Initialization will continue, but untrusted SSL connections will fail.'
  | filter message != 'Client is not initialized - certificate array will be empty'
  | sort @timestamp desc
  | limit 20
  

• If a reconciliation job, bulk edit of labels job, or bulk pipeline processing job is canceled, then the job’s ToDo, Failed, and Completed counts can sometimes display incorrectly.

Upgrade Considerations

During the upgrade, there might be a brief downtime when users won't be able to access the TDP user interface and APIs.

After the upgrade, the TetraScience team verifies that the platform infrastructure is working as expected through a combination of manual and automated tests. If any failures are detected, the issues are immediately addressed, or the release can be rolled back. Customers can also verify that TDP search functionality continues to return expected results, and that their workflows continue to run as expected.

For more information about the release schedule, including the GxP release schedule and timelines, see the Product Release Schedule.

For more details about the timing of the upgrade, customers should contact their CSM.

Other Release Notes

To view other TDP release notes, see Tetra Data Platform Release Notes.