To search for data in the Tetra Scientific Data and AI Cloud, you can use any of the following methods:

You can also use the Tetra Data & AI Workspace to access and use scientific data with analysis applications and correlate analysis results through Tetra Data Apps. For more information and best practices, see Data Access in the TetraConnect Hub. For access, see Access the TetraConnect Hub.

The following procedure shows how to use the Search page.

📘

NOTE

The Search (Classic) page is designed for data engineering use cases, with default filter options for File Category, Source Type, and Pipeline.

File Types

There are three file types available through search: RAW, IDS, and PROCESSED.

RAW Files

RAW files are unprocessed files, such as the ones generated by instruments.

You can search for RAW files by filename, file metadata (such as creation time) and attributes.

IDS Files

IDS files are schemas that are applied to raw instrument data or report files. These schemas are used to map vendor-specific information (like the name of a field) to vendor-agnostic information. To learn more, see Intermediate Data Schemas.

To search for an IDS and view its visualization and associated artifacts, see View Artifact Information.

You can also list or find schemas by using the List schemas API endpoint, which returns the following information for each IDS:

  • The associated JSON schema
  • The associated OpenSearch mapping and also fields that are not indexed in OpenSearch

PROCESSED Files

PROCESSED files are derived from a RAW file and generated from a Tetra Data Pipeline. For example, a processed file could be a .zip file that was unzipped into many smaller files, or image thumbnails from a large microscopy image.

Access the Search Page

The Search page is designed for scientific users to be able to the following more efficiently:

You can also use the Projects feature to quickly organize and access data that’s associated with specific labs, experiments, instruments, or any other type of project through the Search page.

To view the Search page, do the following:

  1. Sign in to the TDP.
  2. In the left navigation menu, choose Search. A menu appears.
  3. From the menu, choose Search. The Search page appears.

Search page

Run a Search by Using the Search Bar

To run a search by using the Search bar, enter terms and fields that you want to search for in the upper left Search bar. Files that match the search criteria you enter display as results in the file list. By default, results are sorted by relevance instead of by upload date. However, you can organize the result set chronologically by selecting the Uploaded At column heading.

For example searches, see Search Query Examples and Results.

When running a search, keep in mind the following:

  • The Search feature filters and searches on terms similar to a common website search. You can enter both full-text queries (search all of the text associated with a file) and filtered queries.
  • Filters are case-insensitive and AND is the default Boolean operator.
  • To search and match for an exact phrase, enclose the text with "double quotes" (for example "lab123 experiment5").
  • Exact-match text searches and exact-match (is) filters return only exact matches to those search terms and filters.
  • Results from searches that don't require an exact match also include Intermediate Data Schema (IDS) output files that return the associated RAW input files.
  • Make sure that you don’t use wildcard searches.

📘

Fuzzy Search

A fuzzy search is done by means of a fuzzy matching program, which returns a list of results based on likely relevance even though search argument words and spellings may not exactly match. By adding a tilde (~) at the end of a search term, you can make a typo (maximum of two characters), and still return relevant results. However, make sure that you don't add a tilde (~) to the end of a keyword/value of a filter, or if the keyword it contains includes any of the following wildcard characters (*, ?, or !). If you do, the query fails.

IMPORTANT: Fuzzy searches don’t always return all possible related files. Incomplete results are sometimes returned when there is a high number of files indexed that have similar field values.

🚧

IMPORTANT

The following feature improvements are available for data that’s processed after the TDP v3.6.0 upgrade only. To apply these enhancements to historical data, customers must reindex the data by reconciling it, or contact their CSM for support. For more information, see Reprocess Files.

Improved Context Search Feature

An improved Context Search feature now displays results returned by the TDP’s Search bar for search terms and filters that don't require an exact match based on content in the primary (RAW) and schematized (IDS) versions of files. This new functionality allows for more powerful contextual search without metadata, tags, or labels. Now, when customers search for content found in an IDS file through the Search bar, the results show information from that IDS file and its related, source RAW file as well as any related IDS files.

Improved Broad Search Feature

An improved Broad Search feature provides customers the ability to enter a portion of a file path into either the TDP Search bar feature or TetraScience /searchEql endpoint search in the query_string to return results, rather than the entire file path. For example, if you were to search for part of a filename, such as lab123 experiment5, then a file with the following path would now also be returned in the search results: /lab123/instrumentB/user1_experiment5_20231212.dat.

Show Folders

To view search results by folder, rather than list view, choose the upper left Show Folders button. A FOLDERS panel appears that displays a folder view of your search results.

Show Folders search view

To show all of the search results in the folder view, select Files in the folder and child folders.

- or -

To show the current folder's files only, select Only files in the folder.

📘

NOTE

When you alter your search criteria, the folder view updates to display the most current search results automatically.

Filter Options for Search

The left filter pane on the Search page includes the following filter options.

Default Filter Options

The following filter options appear by default on every search on the Search page.

FilterDescription
SearchStandard search bar, which allows you to search for any keywords (for example, by file name or file contents)
Upload DateSearches by a date range, based on when files were last uploaded or modified in the TDP
File CategorySearches by the type of file: RAW, PROCESSED, or IDS (default option is RAW)
Labels*Searches by any populated Recommended Labels that are in your TDP environment, if there are any

*If you don't have labels in your TDP environment, no label search filters appear. Tool tips are available from the Description value in Attribute Management

More Filter Options

To add more filter options outside of the default filter options on the Search page, do the following:

  1. In the left filter pane, choose the Filters button. The Filters dialog appears.

Filters dialog

  1. To add a new filter type, select the upper right plus sign (+) button. A new row with a Select a filter type field appears. Then, select a filter type from the drop-down list in the new row and choose Update.

📘

NOTE

To remove a filter, choose the minus sign (-) button next to the filter that you want to remove from your search.

Optional Filter Types

The following are optional filter types that you can add to any search on the Search page.

FilterDescription
File PathSearches by the name of the file, and includes the ability to use contains, ends with and other operators. The File Path filter is case insensitive by default. To make the filter case sensitive, select the Case sensitive check box below the Any path... field in the Filter dialog.

Note: The filePath\* variable is determined by the TDP component that uploaded the file and has the following format: {orgSlug}/{sourceId}/{category}/{filePath\*}
Metadata, Tags, and LabelsSearches by specific metadata, tags, and labels as well as their corresponding values
Other OptionsSearches by Pipeline, Source Name, and/or Source Type, and includes the ability to do complex filters by using Boolean operators (AND/ OR)

Return to a Default Search

You can return to a default search, or access and manage saved searches from the Search page by using the actions at the top of the left filters pane.

Search actions

Start a New Search

To start a new search from the default search filters, select the Start a new search button (shown as an asterisk (*) icon).

Saved Searches

To access your personal, or your organization's saved searches from the Search page, select the Choose a saved search drop-down field from the top of the left filters pane.

To manage your personal, or your organization's saved searches, select the Manage Saved Searches icon to the right of the Choose a saved search drop-down.

Update an Existing Saved Search

To update the saved search that you're using, select the Update this saved search button (shown as a save icon). Then, choose one or both of the following options:

  • Add to Personal Favorites adds the updated search criteria as a saved search to your personal list of saved searches.
  • Share With the Entire Organization adds the updated search criteria as a saved search for every user in your organization.

To save the settings, choose OK.

Create a New Saved Search

To create a new saved search based on the filters that you've chosen, select the Create a new saved search icon (shown as a save icon that includes a pen).

Share a Saved Search with Your Organization

To share one of your personal saved searches with your organization, select the Share this saved search with your organization button (shown by a link icon).

Revert Filters to an Original Saved Search

To revert your search filters to the original saved search that you were using, select the Revert filters to the saved search button (shown as a curved-arrow icon).

Clear All Filters

To clear all filters from your search and revert to the default search filter options, select the Clear all filters button (shown as an X icon).

Refresh Search Results

To refresh your search results, choose the upper right Refresh Search option that's above your search results.

Edit Column Order and Width

To edit the order and width of the columns that display on the Search page based on the filters you choose, you can do any of the following:

  • To move a column, select the column's name. Then, drag and drop it where you want.
  • To change a column's width, hover over the bar that divides the column names. An expansion arrow appears. Then, select the expansion arrow and move your curser to adjust the column's width.

Edit Column Names and Create Custom Columns

To edit column names, or to create your own custom columns, do the following:

📘

NOTE

When you use recommended columns, the system will automatically generate columns based on the filters you are using. New columns will be automatically added or removed when the filters are changed. You can adjust the order and width of the columns, but you can't add or remove columns. When you use custom columns, you must manually add or remove columns. Columns will remain unchanged when filters are changed. You can specify the text of the column heading in addition to the order and width.

  1. Near the top of the SAVED SEARCHES pane, choose the Columns button. The Edit Columns dialog appears.

Edit Columns dialog

  1. (Optional) To create your own custom columns, move the Use Custom Columns toggle to the right.
  2. For each filter in the search, you can edit any of the following fields:
    • For TITLE, enter the name that you want the filter's column to display.
    • for WIDTH, enter how wide you want the column to be in pixels
  3. (Optional) To keep an applied filter's column from displaying in search results, select the minus (-) sign on the right of that column's row.
  4. Choose Update.

📘

NOTE

To change the order of the columns, select the far right three-dot icon in each column's row. Then, drag and drop it to the position where you want it.

Search Indexing

All of the data in Intermediate Data Schema (IDS) files is indexed and available through search.

The first 1 MB of the following RAW, unprocessed file types is also indexed for search:

Media (MIME) TypePossible File Extensions
text/plain["txt","text","conf","def","list","log","in","ini"]
application/json["json","map"]
text/csv["csv"]
application/xml["xml","xsl","xsd","rng"]

📘

NOTE

You can also search for RAW, unprocessed files by filename, file metadata (such as creation time) and attributes. The default 1 MB search indexing settings for RAW files can be adjusted on request.

View File Information

To view a specific file's information on the Search page, select a file from the list of files that are returned by your search. A pane appears on the right that lists optional actions the following information by default:

  • FILE NAME—shows the file's name
  • UPLOAD DATE—indicates the date and time when the file was last modified
  • SIZE—shows the compressed file size. The actual file size is larger when you download it.
  • CATEGORY—indicates the type of file (RAW, PROCESSED, or IDS)

Actions pane

Available Actions For Files

After you select a file on the Search page, you can run any of the following actions from the file's right Actions pane. The available actions are based on the logged-in user’s role.

ActionDescription
File DetailsOpens the File Details page for the file
DownloadDownloads the file to your computer
File PreviewOpens a preview of the file
More File Information*Shows JSON details
Upload New Version*Opens a dialog that allows you to upload a new version of the file
Edit Attributes*Opens a dialog that allows you to edit the file's attributes. For more information, see Apply Attributes to Specific Files.
Delete*Deletes the file permanently from the TDP

Note: A dialog appears that asks you to confirm if you want to delete the file or not.

📘

NOTE

Actions marked with an asterisk (*) are available for admin roles only.

View the File Details Page

To view additional file details, choose File Details in the right pane that appears for the selected file.

The File Details page appears and displays two tabs:

  • File info—shows additional file details
  • File Journey—shows the event history for a file after it's been uploaded to the Tetra Data Lake, including pipeline processing events

📘

NOTE

The File Journey tab displays Tetra File-Log Agent file events only. Incorporating events from other integration types is planned for a future TDP release. To view file events generated by the Tetra File-Log Agent outside of the TDP, see Monitor Events.

Link to a File in the TDP

To link to a specific file in the TDP outside of the platform, copy the File Details page's URL for the file. Then, use that URL to create a hyperlink.

📘

NOTE

To access a file in the TDP through an external link, users must still have the permissions required to access the file.

File Info

The following information is provided on the File info tab on the File Details page for each file in the Tetra Data Lake.

SectionDescription
FILE VERSIONS Lists the total number of file versions in chronological order with the most recent file displayed at the top of the list.
You can:
- Click a version to display its details in the File Details section.
- Hover over the file to displays its full name, date/time when it was uploaded, and its full ID.
- Copy the file ID to a clipboard.
FILE ACTIONSYou can:
- Click Download to download the file to your computer.
- Click View File Info Details to open a preview of the JSON file details.
- Click Add New Version to upload a new version of the file.
- Click Add Attributes to add or edit attributes (such as metadata, tags, and labels) to the file.
- Click Remove to remove the file and its subsequent versions. This action is only available for the most recent file version.
File InfoDisplays the following file details:
- VERSION—shows the file version number
- FILE NAME—shows the file's name
- FILE ID—shows the file's ID number from the Amazon Simple Storage Service (Amazon S3) bucket
- DATE CREATED—shows the date and time when the file was uploaded
- FILE PATH—shows the location of the file in the Amazon S3 bucket (S3 Object Key) in the following format: {orgSlug}/{sourceId}/{category}/{filePath\*}
Note: The filePath\* variable is determined by the TDP component that uploaded the file. For more information, see the documentation for the component that you’re using.
- ES INDEX TIME—shows the date and time the file was indexed in OpenSearch
- ATHENA ADD TIME—shows the date and time the file was added to Amazon Athena
- SOURCE TYPE—shows where the file came from
- SIZE—shows the compressed file size. The actual file size is larger when you download it.
AttributesDisplays any associated file attributes such as: metadata, tags, and labels.
INPUT FILEShows files from which the current file was derived. For example, for an IDS file, the RAW file would be the input file.
OUTPUT FILESShows files that the current file produced. For example, the IDS file typically produces a JSON file. This also indicates the pipeline that processed the file, the file name, the date and time that the file was created, the IDS file name, and the TMP file.
View all Files and WorkflowsShows files and workflows that are related to the current file.

File Journey

📘

NOTE

The File Journey tab displays Tetra File-Log Agent file events only. Incorporating events from other integration types is planned for a future TDP release.

The File Journey tab on the File Details page shows the event history for a file after it's been uploaded to the Tetra Data Lake, including pipeline processing events.

File Journey tab

File Journey Event Types

The File Journey tab on the File Details page reports the following event types.

Event TypeDescription
Connector Event
Connector file detectedShows when the system has detected a file for upload through a configured connector
Connector file processingShows when a file uploaded through a configured connector is processing
Connector file successShows when a file is uploaded to the TDP through a configured connector successfully
Data Lake Events
File UploadedShows when a file is uploaded to the Data Lake
Registered in TDPShows when a file is registered in the TDP
Indexed for searchShows when a file is indexed in OpenSearch
Indexed for SQLShows when a file is indexed in Amazon Athena
Pipeline Events
Pipeline TriggeredShows when a pipeline is triggered
Workflow FailedShows when an executed workflow that was triggered failed
Workflow CompletedShows when a workflow that was triggered completed
Workflow CancelledShows when a workflow was cancelled

View All Files and Workflows

On the File Details page, select the View All Files and Workflows button to display all of the current file's related files and workflows. The following All Files and Workflows page appears.

All Files and Workflows Page Section Overview

SectionDescription
CREATEDShows the date the file was created
KINDIndicates if a file is an input or output file
TYPEShows the file type
FILE NAMEShows the file's name
PIPELINEShows the pipeline that contains the workflow that produced the file
WORKFLOWShows the workflow ID of the workflow that produced the file
WORKFLOW STATUSIndicates if the workflow that created the file was successful or not

Download, Edit, or Delete Multiple Files

The BULK OPERATIONS options at the top right of the Search page allow you to select multiple files, and then either download, edit, or delete those files all at once.

📘

NOTE

You can also open a batch of selected files in a Data & AI Workspace by selecting the Data & AI Workspace icon next to BULK OPERATIONS.

Select Multiple Files

To select the files that you want to include in your bulk action, you can do any of the following:

  • Choose multiple, individual files by selecting the left check box in each file's row
    -or-
  • Choose all of the files returned by your search by selecting the top left check box in your search results (next to the NAME column)

Download Search Results as a CSV File

To download search results as a CSV file, do the following:

  1. Next to BULK OPERATIONS, select the Download search results as a CSV file option (shown as as a CSV icon). A Download CSV File dialog appears.
  2. (Optional) To include all platform metadata of the file as a JSON column in the results download, move the Include Full File Information (this will increase the size of the CSV file) toggle to the right. This option can be helpful for troubleshooting purposes.
  3. Choose Next. The CSV file builds.
  4. Choose Download.

Download Selected Files

🚧

IMPORTANT

The Download selected files action doesn’t work if you’re using Mozilla Firefox. You must use Google Chrome or Microsoft Edge instead.

To download all of your selected files to your computer, do the following:

  1. Next to BULK OPERATIONS, select the Download selected files option (shown as as an arrow pointing into a bin icon). A Bulk Download dialog appears.
Bulk Download dialog
  1. Under DESTINATION FOLDER, select Choose Folder. Then, select a local folder to download the files to.
  2. Choose Next. The dialog shows the bulk download job status as it's running, and then confirms when it's successful.

Bulk Edit Labels

To edit the labels of all of your selected files, do the following:

  1. Next to BULK OPERATIONS, select the Create a Bulk Label Change Job option (shown as a chevron icon). A Create Bulk Label Update Job dialog appears.
  2. Follow the instructions in steps 6-9 of Edit Labels in Bulk.
Create Bulk Label Update Job dialog

Create Bulk Label Update Job dialog

📘

NOTE

To monitor a Bulk Label Edit job's status, see Monitor a Bulk Label Edit Job's Status.

Bulk Delete

To delete all of your selected files from the TDP permanently, do the following:

  1. Next to BULK OPERATIONS, select the Delete Selected Files option. A Delete Files dialog appears.
  2. To delete the selected files, choose Delete Files. To cancel the operation, choose Cancel.

📘

NOTE

When you delete a file version, keep in mind the following:

  • Deleting a file version is a soft delete.
  • The file version remains in the Data Lake.
  • The file version is still displayed in file details (with a URL for the ID).
  • The file version is still available through the TetraScience API (with a URL for the ID).
  • The file version isn't available through search or SQL queries.

Upload a New File

📘

NOTE

When a new file version is uploaded, the TDP copies all file metadata from the previous file version, including workflow data. The system then uses this information to relate the new file version to the workflow that created the previous file version. This process results in the new file version being displayed on the File Details page and showing that the new, uploaded file version was produced by a workflow.

To upload a new file through the Search page, do the following:

  1. Select the upper right Upload File button. An Upload File dialog appears.
Upload File dialog

Upload File dialog

  1. For SOURCE TYPE, enter a source type for the uploaded file. Each newly uploaded file needs to be attributed to a source type.
  2. (Optional) For LABELS, select the labels that you want to add to the file.
  3. Select the file upload box to select a file to upload, or drag and drop a file into the box.
  4. Choose Upload.

📘

NOTE

The maximum file size you can upload through the TDP user interface is 200 MB. To upload larger files, use the TetraScience API or a Tetra Agent or Connector.