IDS Conventions - elasticsearch.json

TetraScience has created a elasticsearch.json generating script. You can find our public repo here: https://github.com/tetrascience/ts-ids-es-json-generator

Platform Requirements

#RuleChecked by IDS Validator
1datacubes, if defined in schema.json, should be in nonSearchableFieldsYes

Design Guideline

#RuleChecked by IDS Validator (only for IDS designed by TS)
1All object-type fields in schema.json should be defined as nestedYes, warning if not followed

NonSearchableFields

Fields defined here will not be indexed into Elasticsearch. You want to exclude fields when:

  • You will not likely to search this field
  • Save some space
  • ES indexing will be faster
  • Your file is larger than 100MB. ES indexing currently has a hard limit on 100MB.

nonSearchableFields uses lodash omit to exclude, so follow the official lodash syntax here: https://lodash.com/docs/4.17.15#omit

Elasticsearch Concept - Nested

Elasticsearch official documentation on nested: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

All object-type fields in schema.json should be defined as nested.

There are situations where you don’t want to define a field to be nested:

Example:

Nested - Example

If you have the following expected.json:

{
  "results": [{
    "peaks": [{
      "number": 1,
      "name": "a" 
    },{
      "number": 2,
      "name": "b"
    }]
  }]
}

Then, your elasticsearch.json should look like the following:

{
  "mapping": {
    "properties": {
      "result": {
        "type": "nested",
        "properties": {
          "peaks": {
            "type": "nested"
          }
        }
      }
    },
    "dynamic_templates": []
  },
  "nonSearchableFields": []
}

Did this page help you?