For an overview of pipelines, see the Pipeline Overview page.
The majority of Pipelines parse raw files, standardize the data to a pre-defined schema, and save the output file to the TetraScience data lake. While there are additional use cases, and in some cases multiple output files, in general all Pipeline outputs can be found in the data lake.
You can access the data lake by selecting "Files" in the main dropdown menu.
By default, the files in the data lake are sorted by date, so any new files can be found at the top of the page.
You can also use the quick filters on the lefthand side to easily filter down to a specific set of Pipelines.
Upon selecting a file in the data lake, additional file information will display in the file modal on the righthand side. You can use the additional information to validate the Pipeline information and click through to the raw file from the output. Only output files will display the link to the original raw file.
Updated almost 2 years ago