IDS Conventions - elasticsearch.json

❗️

Location Changed

The content of this page has been moved to TetraScience Confluence and it's not public-facing anymore. We will revise this page once self-service IDS is available.

The elasticsearch.json file is a component of the IDS artifact, and is used to define:

  • IDS attributes that will be of type nested in the Elasticsearch document
  • IDS attributes that will not be included in the Elasticsearch document
  • Additional dynamic mappings that can be included

How to Generate the File

TetraScience has created a elasticsearch.json generating script. Click here for our public repository.
To create elasticsearch.json, run the generator on your schema.json. By default, all arrays of objects are created as type nested, and datacubes are excluded from search. To customize after generation, you can:

  1. Remove the properties from the mapping for those properties that you do not want to be of type nested.
  2. Add paths to the "nonSearchableFields" for those fields you want to exclude from the Elasticsearch document.

Platform Requirements

#RuleChecked by IDS Validator
1datacubes, if defined in schema.json, should be in nonSearchableFieldsYes

TS Convention

#RuleChecked by IDS Validator (only for IDS designed by TS)
1All object-type fields in schema.json should be defined as nestedYes, warning if not followed

In-Depth Explanation

Elasticsearch.json is composed of three main fields:

  • IDS attributes that will be of type "nested" in the Elasticsearch document
  • IDS attributes that will not be included in the Elasticsearch document
  • Additional dynamic mappings that can be included

For an example elasticsearch.json, expand the following nested example:

Nested - Example

If you have the following expected.json:

{
  "results": [{
    "peaks": [{
      "number": 1,
      "name": "a" 
    },{
      "number": 2,
      "name": "b"
    }]
  }],
  "my_really_long_array": [{
    "foo": "hello",
    "bar": "world"
  }],
}

Then, your elasticsearch.json should look like the following:

{
  "mapping": {
    "properties": {
      "result": {
        "type": "nested",
        "properties": {
          "peaks": {
            "type": "nested"
          }
        }
      }
    },
    "dynamic_templates": []
  },
  "nonSearchableFields": ["my_really_long_array"]
}

Mapping and Defining nested Attributes

Click here for the Elasticsearch official documentation describing the nested type.

You should define all fields within an array of objects in schema.json as nested. You use the "mappings" property of elasticsearch.json to define nested types. To review how properties within an array of objects is assigned as nested in elasticsearch.json, review the previous example.

Do not define a field to be nested if you have any of these conditions:

nonSearchableFields

Fields defined here will not be indexed into Elasticsearch. You want to exclude fields when:

  • You will most likely not search this field
  • Save some space
  • Elasticsearch indexing will be faster
  • Your file is larger than 100 MB. Elasticsearch indexing currently has a 100 MB limit.

nonSearchableFields uses lodash omit to exclude. Use the official lodash syntax here: https://lodash.com/docs/4.17.15#omit

Dynamic Mapping

It is theoretically possible to include a dynamic type mapping based on either path pattern or data type. Currently, this feature is not used extensively. For details, please contact your Customer Success representative.