To search for data in the Tetra Scientific Data and AI Cloud, you can use any of the following methods:

• The Search page
• The Search (Classic) page
• SQL and Amazon Athena Queries
• TetraScience API
• Query domain-specific language (DSL) queries

You can also use the Tetra Data & AI Workspace to access and use scientific data with analysis applications and correlate analysis results through Tetra Data Apps. For more information and best practices, see Data Access in the TetraConnect Hub. For access, see Access the TetraConnect Hub.

The following procedure shows how to use the Search page.

📘

NOTE

The Search (Classic) page is designed for data engineering use cases, with default filter options for File Category, Source Type, and Pipeline.

File Types

There are three file types available through search: RAW, IDS, and PROCESSED.

RAW Files

RAW files are unprocessed files, such as the ones generated by instruments.

You can search for RAW files by filename, file metadata (such as creation time) and attributes.

IDS Files

IDS files are schemas that are applied to raw instrument data or report files. These schemas are used to map vendor-specific information (like the name of a field) to vendor-agnostic information. To learn more, see Intermediate Data Schemas.

To search for an IDS and view its visualization and associated artifacts, see View Artifact Information.

You can also list or find schemas by using the List schemas API endpoint, which returns the following information for each IDS:

• The associated JSON schema
• The associated OpenSearch mapping and also fields that are not indexed in OpenSearch

PROCESSED Files

PROCESSED files are derived from a RAW file and generated from a Tetra Data Pipeline. For example, a processed file could be a .zip file that was unzipped into many smaller files, or image thumbnails from a large microscopy image.

Access the Search Page

The Search page is designed for scientific users to be able to the following more efficiently:

• Find primary (RAW) data in the TDP by scientifically relevant criteria, including the following:
  • Text searches
  • File upload date
  • Any recommended labels where data exists
  • Saved searches (also referred to as Collections)
• Create, update, and manage saved searches
• Download or add labels to multiple files

You can also use the Projects feature to quickly organize and access data that’s associated with specific labs, experiments, instruments, or any other type of project through the Search page.

To view the Search page, do the following:

1. Sign in to the TDP.
2. In the left navigation menu, choose Search. A menu appears.
3. From the menu, choose Search. The Search page appears.

Run a Search by Using the Search Bar

To run a search by using the Search bar, enter terms and fields that you want to search for in the upper left Search bar. Files that match the search criteria you enter display as results in the file list. By default, results are sorted by relevance instead of by upload date. However, you can organize the result set chronologically by selecting the Uploaded At column heading.

For example searches, see Search Query Examples and Results.

When running a search, keep in mind the following:

• The Search feature filters and searches on terms similar to a common website search. You can enter both full-text queries (search all of the text associated with a file) and filtered queries.
• Filters are case-insensitive and AND is the default Boolean operator.
• To search and match for an exact phrase, enclose the text with "double quotes" (for example "lab123 experiment5").
• Exact-match text searches and exact-match (is) filters return only exact matches to those search terms and filters.
• Results from searches that don't require an exact match also include Intermediate Data Schema (IDS) output files that return the associated RAW input files.
• Make sure that you don’t use wildcard searches.

📘

Fuzzy Search

A fuzzy search is done by means of a fuzzy matching program, which returns a list of results based on likely relevance even though search argument words and spellings may not exactly match. By adding a tilde (~) at the end of a search term, you can make a typo (maximum of two characters), and still return relevant results. However, make sure that you don't add a tilde (~) to the end of a keyword/value of a filter, or if the keyword it contains includes any of the following wildcard characters (*, ?, or !). If you do, the query fails.

IMPORTANT: Fuzzy searches don’t always return all possible related files. Incomplete results are sometimes returned when there is a high number of files indexed that have similar field values.

🚧

IMPORTANT

The following feature improvements are available for data that’s processed after the TDP v3.6.0 upgrade only. To apply these enhancements to historical data, customers must reindex the data by reconciling it, or contact their CSM for support. For more information, see Reprocess Files.

Improved Context Search Feature

An improved Context Search feature now displays results returned by the TDP’s Search bar for search terms and filters that don't require an exact match based on content in the primary (RAW) and schematized (IDS) versions of files. This new functionality allows for more powerful contextual search without metadata, tags, or labels. Now, when customers search for content found in an IDS file through the Search bar, the results show information from that IDS file and its related, source RAW file as well as any related IDS files.

Improved Broad Search Feature

An improved Broad Search feature provides customers the ability to enter a portion of a file path into either the TDP Search bar feature or TetraScience /searchEql endpoint search in the query_string to return results, rather than the entire file path. For example, if you were to search for part of a filename, such as lab123 experiment5, then a file with the following path would now also be returned in the search results: /lab123/instrumentB/user1_experiment5_20231212.dat.

Show Folders

To view search results by folder, rather than list view, choose the upper left Show Folders button. A FOLDERS panel appears that displays a folder view of your search results.

To show all of the search results in the folder view, select Files in the folder and child folders.

- or -

To show the current folder's files only, select Only files in the folder.

📘

NOTE

When you alter your search criteria, the folder view updates to display the most current search results automatically.

Filter Options for Search

The left filter pane on the Search page includes the following filter options.

Default Filter Options

The following filter options appear by default on every search on the Search page.

More Filter Options

To add more filter options outside of the default filter options on the Search page, do the following:

1. In the left filter pane, choose the Filters button. The Filters dialog appears.

2. To add a new filter type, select the upper right plus sign (+) button. A new row with a Select a filter type field appears. Then, select a filter type from the drop-down list in the new row and choose Update.

📘

NOTE

To remove a filter, choose the minus sign (-) button next to the filter that you want to remove from your search.

Optional Filter Types

The following are optional filter types that you can add to any search on the Search page.

Return to a Default Search

You can return to a default search, or access and manage saved searches from the Search page by using the actions at the top of the left filters pane.

Start a New Search

To start a new search from the default search filters, select the Start a new search button (shown as an asterisk (*) icon).

Saved Searches

To access your personal, or your organization's saved searches from the Search page, select the Choose a saved search drop-down field from the top of the left filters pane.

To manage your personal, or your organization's saved searches, select the Manage Saved Searches icon to the right of the Choose a saved search drop-down.

Update an Existing Saved Search

To update the saved search that you're using, select the Update this saved search button (shown as a save icon). Then, choose one or both of the following options:

• Add to Personal Favorites adds the updated search criteria as a saved search to your personal list of saved searches.
• Share With the Entire Organization adds the updated search criteria as a saved search for every user in your organization.

To save the settings, choose OK.

Create a New Saved Search

To create a new saved search based on the filters that you've chosen, select the Create a new saved search icon (shown as a save icon that includes a pen).

Share a Saved Search with Your Organization

To share one of your personal saved searches with your organization, select the Share this saved search with your organization button (shown by a link icon).

Revert Filters to an Original Saved Search

To revert your search filters to the original saved search that you were using, select the Revert filters to the saved search button (shown as a curved-arrow icon).

Clear All Filters

To clear all filters from your search and revert to the default search filter options, select the Clear all filters button (shown as an X icon).

Refresh Search Results

To refresh your search results, choose the upper right Refresh Search option that's above your search results.

Edit Column Order and Width

To edit the order and width of the columns that display on the Search page based on the filters you choose, you can do any of the following:

• To move a column, select the column's name. Then, drag and drop it where you want.
• To change a column's width, hover over the bar that divides the column names. An expansion arrow appears. Then, select the expansion arrow and move your curser to adjust the column's width.

Edit Column Names and Create Custom Columns

To edit column names, or to create your own custom columns, do the following:

📘

NOTE

When you use recommended columns, the system will automatically generate columns based on the filters you are using. New columns will be automatically added or removed when the filters are changed. You can adjust the order and width of the columns, but you can't add or remove columns. When you use custom columns, you must manually add or remove columns. Columns will remain unchanged when filters are changed. You can specify the text of the column heading in addition to the order and width.

1. Near the top of the SAVED SEARCHES pane, choose the Columns button. The Edit Columns dialog appears.

2. (Optional) To create your own custom columns, move the Use Custom Columns toggle to the right.
3. For each filter in the search, you can edit any of the following fields:
   • For TITLE, enter the name that you want the filter's column to display.
   • for WIDTH, enter how wide you want the column to be in pixels
4. (Optional) To keep an applied filter's column from displaying in search results, select the minus (-) sign on the right of that column's row.
5. Choose Update.

📘

NOTE

To change the order of the columns, select the far right three-dot icon in each column's row. Then, drag and drop it to the position where you want it.

Search Indexing

All of the data in Intermediate Data Schema (IDS) files is indexed and available through search.

The first 1 MB of the following RAW, unprocessed file types is also indexed for search:

📘

NOTE

You can also search for RAW, unprocessed files by filename, file metadata (such as creation time) and attributes. The default 1 MB search indexing settings for RAW files can be adjusted on request.

View File Information

To view a specific file's information on the Search page, select a file from the list of files that are returned by your search. A pane appears on the right that lists optional actions the following information by default:

• FILE NAME—shows the file's name
• UPLOAD DATE—indicates the date and time when the file was last modified
• SIZE—shows the compressed file size. The actual file size is larger when you download it.
• CATEGORY—indicates the type of file (RAW, PROCESSED, or IDS)

Available Actions For Files

After you select a file on the Search page, you can run any of the following actions from the file's right Actions pane. The available actions are based on the logged-in user’s role.

📘

NOTE

Actions marked with an asterisk (*) are available for admin roles only.

View the File Details Page

To view additional file details, choose File Details in the right pane that appears for the selected file.

The File Details page appears and displays two tabs:

• File info—shows additional file details
• File Journey—shows the event history for a file after it's been uploaded to the Tetra Data Lake, including pipeline processing events

📘

NOTE

The File Journey tab displays Tetra File-Log Agent file events only. Incorporating events from other integration types is planned for a future TDP release. To view file events generated by the Tetra File-Log Agent outside of the TDP, see Monitor Events.

Link to a File in the TDP

To link to a specific file in the TDP outside of the platform, copy the File Details page's URL for the file. Then, use that URL to create a hyperlink.

📘

NOTE

To access a file in the TDP through an external link, users must still have the permissions required to access the file.

File Info

The following information is provided on the File info tab on the File Details page for each file in the Tetra Data Lake.

File Journey

📘

NOTE

The File Journey tab displays Tetra File-Log Agent file events only. Incorporating events from other integration types is planned for a future TDP release.

The File Journey tab on the File Details page shows the event history for a file after it's been uploaded to the Tetra Data Lake, including pipeline processing events.

File Journey Event Types

The File Journey tab on the File Details page reports the following event types.

View All Files and Workflows

On the File Details page, select the View All Files and Workflows button to display all of the current file's related files and workflows. The following All Files and Workflows page appears.

All Files and Workflows Page Section Overview

Download, Edit, or Delete Multiple Files

The BULK OPERATIONS options at the top right of the Search page allow you to select multiple files, and then either download, edit, or delete those files all at once.

📘

NOTE

You can also open a batch of selected files in a Data & AI Workspace by selecting the Data & AI Workspace icon next to BULK OPERATIONS.

Select Multiple Files

To select the files that you want to include in your bulk action, you can do any of the following:

• Choose multiple, individual files by selecting the left check box in each file's row
  -or-
• Choose all of the files returned by your search by selecting the top left check box in your search results (next to the NAME column)

Download Search Results as a CSV File

To download search results as a CSV file, do the following:

1. Next to BULK OPERATIONS, select the Download search results as a CSV file option (shown as as a CSV icon). A Download CSV File dialog appears.
2. (Optional) To include all platform metadata of the file as a JSON column in the results download, move the Include Full File Information (this will increase the size of the CSV file) toggle to the right. This option can be helpful for troubleshooting purposes.
3. Choose Next. The CSV file builds.
4. Choose Download.

Download Selected Files

🚧

IMPORTANT

The Download selected files action doesn’t work if you’re using Mozilla Firefox. You must use Google Chrome or Microsoft Edge instead.

To download all of your selected files to your computer, do the following:

1. Next to BULK OPERATIONS, select the Download selected files option (shown as as an arrow pointing into a bin icon). A Bulk Download dialog appears.

2. Under DESTINATION FOLDER, select Choose Folder. Then, select a local folder to download the files to.
3. Choose Next. The dialog shows the bulk download job status as it's running, and then confirms when it's successful.

Bulk Edit Labels

To edit the labels of all of your selected files, do the following:

1. Next to BULK OPERATIONS, select the Create a Bulk Label Change Job option (shown as a chevron icon). A Create Bulk Label Update Job dialog appears.
2. Follow the instructions in steps 6-9 of Edit Labels in Bulk.

📘

NOTE

To monitor a Bulk Label Edit job's status, see Monitor a Bulk Label Edit Job's Status.

Bulk Delete

To delete all of your selected files from the TDP permanently, do the following:

1. Next to BULK OPERATIONS, select the Delete Selected Files option. A Delete Files dialog appears.
2. To delete the selected files, choose Delete Files. To cancel the operation, choose Cancel.

📘

NOTE

When you delete a file version, keep in mind the following:

• Deleting a file version is a soft delete.
• The file version remains in the Data Lake.
• The file version is still displayed in file details (with a URL for the ID).
• The file version is still available through the TetraScience API (with a URL for the ID).
• The file version isn't available through search or SQL queries.

Upload a New File

📘

NOTE

When a new file version is uploaded, the TDP copies all file metadata from the previous file version, including workflow data. The system then uses this information to relate the new file version to the workflow that created the previous file version. This process results in the new file version being displayed on the File Details page and showing that the new, uploaded file version was produced by a workflow.

To upload a new file through the Search page, do the following:

1. Select the upper right Upload File button. An Upload File dialog appears.

2. For SOURCE TYPE, enter a source type for the uploaded file. Each newly uploaded file needs to be attributed to a source type.
3. (Optional) For LABELS, select the labels that you want to add to the file.
4. Select the file upload box to select a file to upload, or drag and drop a file into the box.
5. Choose Upload.

📘

NOTE

The maximum file size you can upload through the TDP user interface is 200 MB. To upload larger files, use the TetraScience API or a Tetra Agent or Connector.