How to Search Files in the Tetra Data Lake

📘

Tetra Data Platform (TDP) Versions

The following procedures apply to TDP versions 3.2.0 and higher. For earlier TDP versions, see (Version 3.1.x) Searching the Tetra Data Lake.

You can search for data in the Tetra Data Lake in the following ways.

TDP User Interface

• The Search Files page in the TDP UI includes the following:
  • A List view with a search bar (searches can be saved as Collections and saved across an organization).
  • A Label & Advanced Filters dialog where you can configure a custom query using a query builder or enter a RAW EQL query.
  • A Browse view that provides a traditional tree-style view of your files as you would see in Windows Explorer or macOS Finder apps. (Shortcuts can be saved for frequently visited locations in this view.)
• The SQL Search page helps you build and run SQL statements.

Amazon Athena Queries

For more information, see Use SQL and Athena to Query Data.

RAW EQL Queries

For more information, see How to Search Using RAW EQL.

TetraScience API

For more information, see the TetraScience API Documentation.

You can easily apply filters to create complex searches and fine-tune queries. The search feature in the TDP behaves similarly to a common website search. Each field you enter is analyzed as both keyword (default) and text (field.text). The default behavior for Search sorts results by relevance or score.

📘

NOTE

The TDP Search feature uses a search engine to help you look for specific datasets and corresponding files. To explore or analyze tabular data across one or more datasets, see Use SQL and Athena to Query Data.

What is Indexed for Search?

The first 1MB is all files is indexed for search. All of the data in harmonized (IDS) files is also indexed and available in search.

📘

NOTE

You can turn on the Highlight option to have TDP UI highlight where in the text the search string is located.

How is Search Used?

You can use the TDP search feature to assist in the following:

• View and sort search results by name, source type, or upload date
• Browse files in specific folders and subfolders
• Filter search results by entering text in the Search box
• Save a query or grouping of results as a collection using the List view
• Save file path locations as shortcuts using the Browse view
• View files from specific file categories, sources, or pipelines
• Upload, download, preview, and delete files
• Open the file page to view its details
• View JSON files
• Add or edit attributes (metadata, tags, or labels)

Access the Search Feature

To access the Search feature, do the following:

1. In the Tetra Data Platform, click the Hamburger icon at the top left corner of the page to expand the TDP menu options (or hover over the list of icons to display the menu options):

2. Select Search Files from the list of menu options that appears on the left side of the page.

The Search page displays and enables you to:

• (In List view) Create, save, and manage search queries (grouping of results) as a collection to display at the top of the Search page
• (In Browse view) Create, save, and manage file path locations as a shortcut to display at the top of the My Home page
• Use quick filters where you can:
  • List files by category (RAW, PROCESSED, or IDS), source, or pipeline.
  • Browse files by your organization's folders in the Tetra Lake.
• Easily search files by entering any text in a Search box and view which filters have been applied.
• Conduct advanced searching and file uploading using additional filters and features.
• Review the file search results in a display area sorted by relevance (by default).

This table provides a list of the Search page items and their descriptions:

Perform a Basic File Search

To perform a basic file search, do the following:

1. From the Search page, click List. The file name, source type, and the date/time that each file was last modified displays.
2. Enter terms and fields that you want to search in the Search box. To search and match for an exact phrase, enclose the text with "double quotes". The Search feature filters and searches on terms similar to a common website search. You can enter both full-text queries (search all of the text associated with a file) and filtered queries. Filters are case-insensitive and AND is the default Boolean operator.

📘

Fuzzy Search

A fuzzy search is done by means of a fuzzy matching program, which returns a list of results based on likely relevance even though search argument words and spellings may not exactly match. By adding the ~ after a keyword, you can make a typo (maximum of two characters), and still return relevant results.

However, do not add a ~ to the end of a keyword/value of a filter, or if the keyword it contains these wildcard characters (*, ?, or !) because the query will fail.

3. Click Search. Files that match the search criteria you entered in the Search box display as results in the file list. The default behavior for Search is to sort results by relevance (or score) instead of by Upload Date. However, you can organize the result set chronologically by sorting on Upload Date. Click here for basic search examples.
4. To perform additional filtering, you can select a source or pipeline from the list. To avoid unnecessary scrolling, you can expand and collapse the Source Types and Pipelines from the side panel of the Search page. Additionally, you can search within the filter or facet values. For example, you can click Source Type and start typing “humidity sensor”. You can also create and save a search query as a collection. For more details, see How to Save Collections and Shortcuts.
5. To further filter your search results, you can select filters from the Label & Advanced Filters tabs.
6. To clear the existing search criteria you entered in the Search box, click Clear.

View a File Summary

To view a summary of the file details, click the file from the list of files.

This table describes the list of File Summary items:

At the upper right corner of the File Summary panel, the following icons and the More option display:

• Open File Page
• Download File
• View JSON Details
• File Preview

View Additional File Details

To view additional file details, choose Open File Page for the selected file. The File Details page appears.

Files Details page section overview

View All Files and Workflows

On the File Details page, select the View All Files and Workflows button to display all of the current file's related files and workflows. The following All Files and Workflows page appears.

All Files and Workflows Page Section Overview

Download a File

To download a file to your computer, you can select either of the following:

• The Download File icon from the File Summary page.
• The Download Version File Action from the File Details page.

Bulk File Downloads (Downloading Several Files at Once)

Bulk download allows you to quickly retrieve files sent to TDP. Instead of downloading files one at a time or having to write a custom script to send several requests to our API, you can now select up to 100 files from search results to download to your computer or device. Typically files are downloaded to the default location set in your web browser. If you want to change that location, see the documentation for your web browser.

1. In the search page, select the files that you want to download. You can select up to 100 files to download. Note that you can select files on different search result pages.

2. Select Bulk Actions from the top of the screen.
3. Select Download ## Files. (## is replaced by the number of files that you selected. If you select more than 100 files - which is the limit – the number is grayed out.)

4. A message appears asking if you are sure that you want to download the files. Note that the minimum size estimate of the download is displayed. Note that you’ll probably need more storage space for the files than is indicated by the minimum size estimate. This is because the minimum size estimate is the size of the stored compressed files: the actual size of all of the downloaded, uncompressed files is likely to be larger than that. Click Download Files to continue.

5. Your browser might prompt to you to allow the downloading of multiple files. The files are downloaded as separate files in your default download area on your computer or device. If you want to change this location or whether you will need to be prompted to download multiple files, see your web browser’s documentation. If the download fails, try the download again.

View JSON File Details

To preview the JSON file details, you can select either of the following:

• The View JSON icon from the from the File Summary page.
• The View File Info Details File Action from the File Details page.

From the JSON preview window, you can view details such as: total number of items, source type, when the file was created, the location of the file (bucket), source, category, and so on.

Add and Edit Attributes

To add or edit attributes (such as metadata, tags, and labels), you can do either of the following:

• Choose More from the File Summary page, and then select Add/Edit Attributes.
• Select the Add Attributes File Action from the File Details page.

Follow the instructions in this topic. When you have finished adding or editing attributes, click Save.

Upload a New File or New Version of the File

📘

NOTE

When a new file version is uploaded, the TDP copies all file metadata from the previous file version, including workflow data. The system then uses this information to relate the new file version to the workflow that created the previous file version. This process results in the new file version being displayed on the File Details page and showing that the new, uploaded file version was produced by a workflow.

To upload a new version of the file, you can select any of the following:

• Upload File at the top right of the Search page.
• More from the File Summary page, and then select Upload New Version.
• The Add New Version File Action from the File Details page.

1. For a brand new file, you must select a source type for the uploaded file. Each newly uploaded file needs to be attributed to a source type.

2. You can add a new label, if desired. Labels are applied to an existing file without creating new versions. For details, see this topic. To add metadata or tags, click Advanced Fields. These fields create new file versions and trigger new workflows when modified. Be aware that the contents of these files may be versioned across edits.

3. Click the file upload box to select a file to upload, or drag and drop the file into the box.

4. To preview an uploaded file, click the Preview icon. Preview is available for these valid files and file types:

   • Images with file type: .png, .jpeg, .gif, or .bmp (including 360-degree images)
   • .pdf
   • .csv
   • .xlsx
   • .docx
   • Video with file type: .mp4 or .webm
   • Audio with file type .mp3

   If any valid file is greater than 50 MB, then a warning message displays indicating that the file size is too large to display in a preview.

5. When complete, click Upload.

📘

File Size Limitation

The maximum file size you can upload through the TDP UI is 200 MB. To upload larger files, use the TDP API or a Tetra Agent or Connector.

Delete a File

To delete a file, you can do any of the following:

• Select one or more file(s) from list of files on the Search page, then click Bulk Actions, and Delete Selected.

📘

NOTE

You can select all of the files on a single Search Files results page only (up to 20 at a time). To select more files, you must open each search results page individually.

• Click More from the File Summary page, and then select Delete.
• Click the Remove File Action from the File Details page. This action is only available for the most recent file version.

Click OK to confirm deletion of the file(s) and its subsequent versions; or click Cancel to retain the file(s).

📘

NOTE

When you delete a file version, keep in mind the following:

• Deleting a file version is a soft delete.
• The file version remains in the Data Lake.
• The file version is still displayed in file details (with a URL for the ID).
• The file version is still available through the TetraScience API (with a URL for the ID).
• The file version isn't available through search or SQL queries.

Browse Files in Folders

To browse files in folders, do the following:

1. From the Search page, click Browse. The Tetra Lake folders display as the source instead of the Source Type and Pipeline.

2. Select the folder to browse the Tetra Lake's folder hierarchy based on your organization. You can continue to select subfolders until the files you are searching for display in the results section of the page.

Your current file path location displays at the top of the page. Additionally, you can save file path searches and add as shortcuts to the top of the My Home page. To quickly return to your home directory (and your shortcuts), you can click My Home at the top of the folder list. For more details, see How to Save Collections and Shortcuts. Any files that you removed display under the Removed Sources section at the end of the folder list indicated with the red trash icon.

3. To further filter your search results, you can select filters from the Label & Advanced Filters tabs, or manually create a search query.