Tetra Data Pipelines provide the ability to configure a set of actions to occur automatically each time new data is ingested into the Tetra Scientific Data and AI Cloud. They run automated data operations and transformations that can help you create queryable, harmonized Tetra Data, and then enrich and push that data to downstream systems.

For example, you can use Tetra Data Pipelines to do any of the following:

• Harmonize data: Parse proprietary instrument output files into a vendor-neutral and scientifically relevant Intermediate Data Schema (IDS), while also storing the data in SQL tables.
• Transform data: Add calculated fields to standard data fields.
• Contextualize files: Add attributes to files to improve how retrievable they are by search. For example, you can use Tetra Data Pipelines to programmatically add information about samples, experiment names, and laboratories.
• Enrich files: Get information from other files within the Tetra Scientific Data and AI Cloud to augment new data.
• Convert file formats: Convert IDS JSON files to other formats, such as ADF or CSV.
• Push data to third-party applications: Send data to an electronic lab notebook (ELN), laboratory information management system (LIMS), analytics application, or an AI/ML platform.

To create a pipeline, see Set Up and Edit Pipelines. To manage pipelines or compare and restore different pipeline versions, see Manage Pipelines. To extend the capabilities of Tetra Data Pipelines and the Tetra Data Platform (TDP), you can also create your own custom self-service Tetra Data pipelines (SSPs).

For more information and best practices, see Data Engineering and Tooling and Automation in the TetraConnect Hub. For access, see Access the TetraConnect Hub.

📘

NOTE

The Data Lakehouse Architecture early adopter program (EAP) introduced in TDP v4.1.0 uses a new Tetraflow Pipeline artifact that can be configured in a Tetra Data Pipeline instead of a protocol. Tetraflow pipelines run on infrastructure optimized for large-scale data processing and enable users to programmatically transform their Tetra Data in the Lakehouse into analytics-optimized datasets and aggregates.

Tetra Data Pipeline Architecture

The following diagram shows an example Tetra Data Pipeline workflow:

The diagram shows the following workflow:

1. When an event occurs, such as a file being uploaded to a specific location, all active pipelines determine if that event matches each pipeline's configured trigger conditions.
2. If the event matches the pipeline's trigger condition, the pipeline's protocol runs a predefined workflow. Protocols define the business logic of your pipeline by specifying the steps and the functions within task scripts that run those steps.
3. When processing is finished, output files are then indexed according to a predefined IDS and stored in the Tetra Scientific Data and AI Cloud. Through this process, the data becomes easily accessible through search in the TDP user interface, TetraScience API, and SQL queries. You can also push data to third-party applications, such as an ELN, LIMS, analytics application, or an AI/ML platform.

📘

NOTE

Keep in mind the following:

• Raw files that don't pass through a Tetra Data Pipeline are still searchable by using default and custom attributes, but aren't parsed.
• Pipelines can run on the latest version of a file only. This behavior ensures that previous file versions don't overwrite the latest data. If a pipeline tries to process an outdated or deleted file version, the workflow errors out and the TDP now displays the following error message on the Workflow Details page: "message":"file is outdated or deleted so not running workflow"

Tetra Data Pipeline Components

Tetra Data Pipelines consist of the following components.