Intermediate Data Schema (IDS) Overview

The "Intermediate Data Schema" (IDS) designed by TetraScience in collaboration with instrument manufacturers, scientists, and customers, is a schema that is applied to raw instrument data or report files. This schema is used to map vendor-specific information (like the name of a field) to vendor-agnostic information. The IDS standardizes naming, data type (whether the field is a string, an integer or a date), data range (for example, something needs to be a positive number) and data hierarchy.

By doing this, the IDS harmonizes different data sets in Life Science industry, such as instrument data, CRO assay data and software data. This allows Life Sciences companies to consume the data in their applications, build searches and aggregations and feed the data into visualization/analysis software seamlessly because the IDS generated JSON files are predictable, consistent and vendor-agnostic.

Raw data files are uploaded to the data lake and are then parsed by TetraScience data pipelines, which among other things, generates a JSON file using the IDS designed for that specific instrument as a guide. During processing, the data is validated and search indexes are also generated. The newly generated JSON file thus is an IDS JSON. The IDS JSON is a JSON representation of the RAW data. Other files are generated as well, such as logs and search mappings. The IDS JSON, as well as the other files, can be quickly and easily viewed in the TDP.

Once IDS JSONs are generated, you can query the data in TDP using SQL from the TDP or a tool of your choice, or with an API call using the tool of your choice. If you have questions as you review your data, you can refer to IDS examples and readme files in the TDP so that you can reference the structure to understand how the data maps to the original RAW file.

IDS Terminology

The following table lists terminology related to IDSs.