API DocumentationProduct Documentation
Product Documentation

Query DSL Queries

Suggest Edits

Running direct query domain-specific language (DSL) searches in the Tetra Data Platform (TDP) user interface can help you test your search syntax before applying it programmatically in another system.

There are three ways to run Query DSL queries in the TDP:

To run searches programmatically, see the TetraScience API documentation. For more information, see Query DSL in the OpenSearch documentation.

📘

NOTE

When using the OpenSearch Query Validate API, errors appear in an explanation field, rather than an error field.

Run a Query DSL Query by Using the Query API Sandbox

To run a query using the Query API Sandbox, do the following:

  1. In left navigation menu, choose Search.
  2. Choose Query API Sandbox. The Query API Sandbox page appears and displays an example Query DSL query.
  3. Enter a Query DSL query. To upload a query from an existing file, choose Upload File.
  4. Choose Run. The query results display in a RESULTS pane next to the query.
  5. (Optional) To download the query as JSON file, choose Download File.

📘

NOTE

You can also use the Query API Sandbox to run filter-based queries by selecting Filters from the LANGUAGE drop-down list at the top of the page.

Run a Query DSL Query by Using the Search Bar

📘

NOTE

The Search bar uses the query_string query, which parses the input and splits text around operators. Each textual part is analyzed independently of each other.

To run a query using the Search bar, do the following:

  1. Open either the Search or Search (Classic) page.
  2. In the Search bar, enter either parts of a file path (separated by spaces), or a complete file path. Don’t use wildcard searches.
  3. Choose Search. Files that match the search criteria you entered display as results in the file list. Results are sorted by relevance instead of by upload date by default.

📘

NOTE

Search results that include Intermediate Data Schema (IDS) output files also return the associated RAW input files.

Run a Query DSL Query by Using the Label & Advanced Filters Menu

To run a direct Query DSL query by using the Label & Advanced Filters menu on the Search (Classic) page, do the following:

  1. Open the Search (Classic) page).
  2. Choose Label & Advanced Filters. A dialog appears that includes filter options as different tabs.
  3. Select the Raw EQL tab. The /searchEql endpoint request displays.
  4. Edit the query as needed for your use case. (See Query DSL Best Practices).
  5. (Optional) To run the query without validating the query structure, select the No validation check box.
  6. Choose Run EQL. The query runs and displays the Response.

🚧

IMPORTANT

To search for a specific file path use the filePath field. To search for a part of the file path, use the enhancedSearchContext.filePathParts field. This field allows you to target specific tokens in the file path. Don’t use a wildcard prefix (*). Queries that include wildcards aren’t as effective, take longer to run, and require more computing resources. For more information, see Wildcard Searches.

Query DSL Best Practices

When creating Query DSL queries, keep in mind the following:

  • Search terms must be specified in lowercase, unless you want to target an exact match for a term. To target an exact match for a search term, place the term in quotes (for example, "My Specifically Cased Phrase").
  • For programmatic use cases, make sure that you use pagination features and target only the information you need.
  • To limit the amount of data returned from the data lake, it’s a best practice to do the following:
    • Specify the following fields in your queries:size (determines the maximum number of hits to return) and _source (determines the source fields to return)
    • Specify an indexparameter in the /searchEql endpoint's URL query string to determine one or more specific indexes to return results from (for more information, see Target Your Search by index)
    • Outside of the recommended query fields listed here, make sure that you select only the fields required in your query response. For example, it's generally recommended to filter out the filePathHierarcy field, since it only exists to support the Folder view feature on the Search page.

Target Your Search by index

By default, each /searchEql call searches all of an organization's OpenSearch indexes, which can put a high compute load on the system. To help reduce this load and optimize your queries, it's recommended that you specify an index URL query string parameter to determine the specific indexes that you want to return results from.

The index URL query string parameter has four possible values that you can either list multiple times in each query or as a comma-separated string:

  • raw targets all files indexed as RAW documents
  • processed targets all files indexed as PROCESSED documents
  • ids targets all files indexed as IDS documents
  • <idsName> targets all files associated with a specific IDS
  • <idsName:v.x.x> targets all files associated with a specific IDS version (you can also use an underscore (_) instead of a colon (:) to separate the IDS name from its version number)

index Field Example: Listed Multiple Times

  POST https://{{dlsvc_host}}/v1/datalake/searchEql
  ?index=raw
  &index=lcuv-empower:v12*

index Field Example: Comma-Separated List

  POST https://{{dlsvc_host}}/v1/datalake/searchEql
  ?index=raw,processed,lcuv-empower:v12*

📘

NOTE

Wildcard (*) searches are supported but not recommended. For more information, see Wildcard Searches.

Targeting More than One index

Query DSL queries that target multiple indexes will not cause an HTTP 400 request failure when individual index type failures occur. The query will still return an HTTP 200 response. If an index type failure occurs, it will appear in a _shards.failed field within the response.

For example, consider if two targeted indexes include a fieldA, but in index1, it’s a boolean value, and in index2, it’s a text value. If you write a valid Query DSL query that searches fieldA with a text value, then the query will return an HTTP 200 response that includes information from index2, but nothing from index1. In this case, the response will also contain a reference to the failure to search index1 for that field within the _shards.failed field.

Query DSL Query Examples

There are many optional queries and fields that you can use to create different search behaviors when running Query DSL queries. The following are examples of two common Query DSL use cases in the TDP.

For more example queries, see Query DSL in the OpenSearch documentation.

Search a File Path by Multiple, Case-Sensitive Terms

The following example Boolean query combines several query clauses into one query that searches for specific, case-sensitive phrases in a file path.

This example includes the following variables:

  • Example files that have the following case-sensitive filePath value: chromeleon/ChromeleonLocal/MYUSER/HPLC_Run_20230817_myuser.seq
  • Query parameters that specify that the file path must match the the following values: hplc, myuser, and 20230817
  • Query parameters that specify that the files returned must be indexed in OpenSearch with the following values: raw, processed, and chromeleon-raw-to-ids:v6.0.0
{
  "size": 100,
  "_source": {
      "includes": ["filePath", "fileId", "category", "createdAt", "labels"]
  },
  "query": {
    "bool": {
      "must": [
        { "match": { "enhancedSearchContext.filePathParts": "hplc" } },
        { "match": { "enhancedSearchContext.filePathParts": "myuser" } },
        { "match": { "enhancedSearchContext.filePathParts": "20230817" } }
      ]
    }
  }
}

Search a File Path for a Case-Insensitive Term

📘

NOTE

You can also use the following prefix query in a Boolean query to search for more than one case-insensitive term.

The following example Prefix query searches for a specified field that begins with a given string (prefix) and returns results that are case-insensitive by using a case_insensitive parameter that's set to true. This query will return results from the same filePath as the one used in the Search a File Path by Multiple, Case-Sensitive Terms example query, (chromeleon/ChromeleonLocal/MYUSER/HPLC_Run_20230817_myuser.seq). If the case_insensitive parameter were set to false (the default behavior), or not included, the query would not match because of the case differences between the search value and the filePath value.

This example includes the following variable:

  • Example files that have the following case-insensitive filePath value: chromeleon/chromeleon
  • A case_insensitive parameter that's set to true. This setting makes it so that case-insensitive matches for the filePath parameter's value are still returned.
{
  "query": {
    "prefix": {
      "filePath": {
        "value": "chromeleon/chromeleonlocal",
        "case_insensitive": true
      }
    }
  }
}

Updated 4 months ago