Updated: 7 May 2024

The following deprecation notices are for TetraScience products, including the Tetra Data Platform, Tetra Agents, and more. For all deprecation notices prior to those listed on this page, see Tetra Data Platform (TDP) Release Notes.

Deprecations are components, features, or functions that are no longer supported or maintained by TetraScience. For more information about deprecations, see Tetra Product Deprecation Process Overview.

The following table summarizes the TetraScience product deprecations. To learn more about each deprecation, select the corresponding link for each product in the Details column.

Deprecation Notices

The following are deprecation notices for TetraScience products.

@idsConventionVersion Deprecation

The @idsConventionVersion field, which appears in many [Intermediate Data Schemas (IDSs)](https://developers.tetrascience.com/docs/intermediate-data-schema-ids-overview) developed by TetraScience, enforces schema standardization by using the field as a flag for schema validation. This field will no longer be used to enforce schema standardization in future versions of ts-ids-core and ts-ids-validator.

Starting in ts-ids-validator=v0.10.6 a deprecation notice will be included and in ts-ids-core=v0.6.2, @idsConventionVersion will no longer be included in the definition of @idsConventionVersion. A new schema metadata field, is_tetra_data_schema, will be used as the flag instead, and it will not be written to IDS JSON instances.

"Upload event bundles" Deprecation

The Upload event bundles option will no longer be available starting with Tetra Cellario Connector v2. The feature was used to test and troubleshoot Cellario connectivity and data upload. New features available in the latest Connector version make this option unnecessary.

context.run_command() Function Deprecation

The context.run_command() function is on a deprecation path to be replaced with the context.run_cmd function. This change removes the need to supply an org_slug value in the function.

File-Log Agent Global Path-Level Start Date and Interval Settings Deprecation

In TDP version 3.6.0, the global path-level Start Date and Interval (Seconds) settings for the File-Log Agent’s File Watcher service are being deprecated. These two global path-level settings are being deprecated because they’re not being used. Customers can still configure these settings at the individual path level.

Legacy Runtime Environment for Self-Service Tetra Data Pipelines Deprecation

The legacy runtime environment for self-service pipelines (SSPs) is scheduled for deprecation during the second half of 2024. A new, more secure SSP runtime environment will replace the legacy SSP runtime environment. Customers should plan on rebuilding and re-releasing their existing protocols to use the new SSP runtime environment before the legacy one is deprecated.

Existing SSPs and task scripts built with the previous design will continue to work during the depreciation period.

API_KEY Authentication Deprecation

Customers can’t use the API_KEY for authentication after the release of TDP version 3.6.0. To avoid breaking integrations, customers must switch to using a JSON Web Token (JWT) for authentication instead.

For more information, see Authentication in the TetraScience API documentation.

Search Files, Search Workflow, and Paginate through All Pipeline Details API Endpoints Deprecation

The following API endpoints are on a deprecation path:

• Search Files (GET)
• Search Files (POST)
• Search Workflows
• Paginate through All Pipeline Details

The Search Files endpoints search for files in the Data Lake by using Spleen query language and don't support the Access Groups feature introduced in TDP v4.0.0. Use the Search files via Elasticsearch Query Language endpoint instead.

The Paginate through All Pipeline Details endpoint lists all pipelines in a paginated list. Use the Search Pipelines endpoint instead.

The Search Workflows endpoint searches pipeline workflows. Use the Search Workflow endpoint instead.

Metadata and Tags Deprecation

Starting in TDP version 3.6.0, the metadata and tags attributes are on a deprecation path. Functionality for labels will be improved to further meet FAIR (findable, accessible, interoperable, and reusable) data principles and incorporate critical parts of existing metadata and tags functionality.

🚧

IMPORTANT

There is currently no metadata and tags deprecation date. Customers are not required to make any changes to existing configurations. In H1 2025, TetraScience will provide customers with a simple migration process to transfer their existing pipelines and searches to using labels. No breaking changes will occur with the metadata and tags deprecation and the TDP will continue supporting metadata and tag data for historical data. For more information, please contact your customer success manager (CSM).

The metadata and tags deprecation provides the following benefits:

• Simplifying the process of adding attributes to files
• Increasing general compliance with FAIR data principles
• Reducing human, processing, and data costs associated with attribute management
• Reducing the need for file reprocessing
• Increasing scalability and automation
• Providing more precise search results

For TDP v3.5.x

For any new use cases, customers should begin using labels as the only attribute to store each file’s contextual information. For existing use cases, it’s recommended that customers modify configurations to use labels instead of metadata and tags. The TDP will continue to support metadata and tags as read-only attributes after the deprecation. There is no need for customers to reprocess historical data.

For TDP v3.6.x

TetraScience plans to implement the following changes in TDP v3.6.x:

• Reducing the prominence of metadata and tags in the TDP user interface

For TDP v4.2.x

TetraScience plans to implement the following changes in TDP v4.2.x:

• Updating labels to maintain version history for better auditing and transparency, and to maintain the association of labels to versions of underlying files
• Helping to copy all existing metadata and tags to labels, so that all contexts of the primary data is searchable with labels.
  • Keep in mind that the context that is represented by metadata and tags may start to be out of sync with the consolidated context represented by labels.
  • Any exceptions will be automatically handled by the TDP during this migration.
  • Going forward, any metadata and tags that are created will also be created as labels in a dual-write approach
• Providing a pathway for customers to self-migrate existing metadata and tags to labels.
  • Labels will accommodate metadata and tags by using the following rules:
    • Metadata {"location": "Chicago"} will become label [{"name": "location", "value": "Chicago"}]
    • Tag ["validated", "decorated"] will become label [{"name": "tag", "value": "validated"}, {"name": "tag", "value": "decorated"}]
    • Any exceptions (namespace violations, character violations) will be identified for self-remediation.

📘

NOTE

No metadata or tags will be deleted.

For Future Versions of the TDP

TetraScience plans to implement the following changes in future versions of the TDP:

• Allowing metadata and tag context to be viewed through the TDP user interface, but no new metadata or tags to be created
• Preventing TDP user interface and TetraScience API users from creating or using metadata or tags.
  • Users will need to use labels for new configurations for agents, data sources, and pipeline triggers.
  • Existing pipelines and data sources will continue to function with existing metadata and tags.
  • Underlying data for metadata and tags won’t be removed.
  • EQL and SQL queries referencing metadata and tags will continue to work.
• Preventing TDP users from updating or creating new metadata and tags for files.
  • Users will still be able to add metadata and tags to specific files by using the TetraScience API to maintain customers' existing programmatic calls.
  • Metadata and tags will be hidden in the TDP user interface and TetraScience API.
  • Metadata and tags will remain as data for historical reference.

At the End of the Deprecation Process

• All existing metadata and tags data will be copied as labels and maintained as historical data.
• Metadata and tags will no longer be a concept for setup and management of data in the TDP and Tetra Agents.
• Existing pipelines and TetraScience API updates to files for metadata and tags set up prior to TDP v3.6.0 will continue to work. Any new metadata and tags will be written as labels.
• Programmatic search queries (EQL and SQL) that reference metadata and tags will continue to work, but will only return data from deprecated pipelines or data sources that were set up prior to TDP v3.6.0.

ES and FileInfo API Deprecation - Integration and Source.Integration

Cloud Data Connector Deprecation

We are deprecating the concept of Cloud Data Connector (CDC) to simplify Agent configuration. For TDP versions 3.4 and higher, CDC functionality will become a core component of the platform. CDCs will no longer be required to be set up as separate data sources, and you will no longer be able to create new CDCs or add Agents to them.

If your Agent needs a Data Hub and Generic Data Connector (GDC), Agent creation will be unchanged. If you don’t, new Agents can be created from the Agents page with No Connector selected.

Existing CDCs will be unaffected for TDP v3.4, but are scheduled to be removed in the TDP v3.5 release. All Agents without a Data Hub V1 + GDC will display ‘No Connector’ in the TDP user interface once the CDCs are removed.

Tetra Data Hub Deprecation

Starting with TDP v3.6.0, Tetra Data Hub is on a deprecation track. An improved design of Data Hub will be available, called Tetra Hub.

For new deployments, customers should install Hub, except for the following use cases:

• Tetra Solace Connector integrations
• Facilitating on-premises HTTP requests from pipelines if AWS PrivateLink or an API gateway doesn't meet customer requirements

For more information, see Tetra Hub Versions.

Athena metadata_v1 Table Deprecation

For version 3.3.0 of TDP, the Athena metadata_v1 table has been deprecated. Please transition queries and dashboards to use file_info_v1, which has the same fields and more attributes about all file categories. Tetrascience does not plan to release new versions of this table and starting from v3.6 (~June 2023) will remove this table from the TDP.

Full-Text Search Deprecation

For version 3.3.0 of the TDP, to improve performance, full-text search (FTS) for IDS files has been deprecated and we plan to deprecate the functionality for RAW data in a future release.

For customer-hosted environments:

• Deprecating FTS for IDS files provides additional computational and storage cost savings.
• To ensure particular FTS search use cases are still achievable, TetraScience has the ability to turn this on via an AWS parameter or to turn off FTS indexing for RAW data to further save indexing costs. Please contact support if you wish to do this.

Apps and Email Menu Options Removed from TDP

For version 3.3.0 of the TDP, the Apps and Email menu options have been removed from TDP. Existing links to Streamlit dashboards will continue to work with no functionality updates and are supported through TetraScience professional services team.

Pipeline Reserved Instances Deprecation

For version 3.3.0 of the TDP, the pipeline reserved instances have been removed. A new selection algorithm manages resources to make sure files are processed by pipelines in a timely manner and at scale.

Pipeline Dashboard Deprecation

For version 3.3.0 of the TDP, the Pipeline Dashboard has been removed from the main TDP Navigation. It is temporarily accessible via URL (https:<tdp-env>.com/dashboard), but will be removed in a future release. Its content has been integrated into the new Pipeline File Processing page.