Edge Delta Elastic Output

Stream data to Elastic.

13 minute read

See the latest version here.

You need to configure Elastic to use it as a data destination in Edge Delta. To do this you create a lifecycle policy and an index template. Then you can update the Edge Delta agent configuration to send data to Elastic.

Step 1: Create a Lifecycle Policy

Index lifecycle policies manage indices based on your performance, resiliency, and retention requirements. You can use the following sample lifecycle policy to start, which creates a new index every day and maintains data from the last 15 days. While you can run the command with the pre-populated settings, you can also change the retention period, as well as other fields.

In your Elastic’s dev console, run this command.

PUT _ilm/policy/ed-agent-log-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "5gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

After you run the command, you should have an index lifecycle policy named ed-agent-log-policy.

Step 2: Create an Index Template

An index template is useful to configure Elastic indices before they are created. While the Edge Delta agent can be configured to stream various types of observations to the Elasticsearch destination, we recommend that you create the target index with the recommend index template. The following sample will create an index template named ed-agent-log with field mappings to the ed-agent-log-policy lifecycle policy.

Too many unique field mappings in Elastic can cause out of memory errors and difficult situations to recover from. Edge Delta supports aggregation of custom fields in Elastic by using a field called “custom_labels_flattened” to index mapping and it utilizes that field for metric and log entries. This field is used by default to send custom labels to Elastic.

In your Elastic’s dev console, run one of the following commands depending on your version of Elastic. For Elasticsearch 8.x, use the following command:

PUT _index_template/ed-agent-log
{
  "index_patterns": [
    "ed-agent-log-*"
  ],    
      
  "template":{  
    "settings": {
    "number_of_shards": "1",
    "number_of_replicas": "1",
    "lifecycle": {
        "name": "ed-agent-log-policy",
        "rollover_alias": "ed-agent-log"
        },

    "analysis": {
      "analyzer": {
        "custom_label_analyzer": {
          "tokenizer": "custom_label_tokenizer"
        }
      },
      "tokenizer": {
        "custom_label_tokenizer": {
          "type": "pattern",
          "pattern": "\\$"
        }
      }
    }
  },
  "mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": [],
      "properties": {
        "msg": {
          "type": "text"
        },
        "alert_def_id": {
          "type": "keyword"
        },
        "k8s_namespace": {
          "type": "keyword"
        },
        "merge_level": {
          "type": "keyword"
        },
        "ecs_task_family": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "k8s_controller_kind": {
          "type": "keyword"
        },
        "k8s_container_image": {
          "type": "keyword"
        },
        "title": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "type": {
          "type": "keyword"
        },
        "src_name": {
          "type": "keyword"
        },
        "k8s_container_name": {
          "type": "keyword"
        },
        "score": {
          "type": "double"
        },
        "sub_type": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "capture_flush_mode": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "tag": {
          "type": "keyword"
        },
        "k8s_controller_logical_name": {
          "type": "keyword"
        },
        "timestamp_end": {
          "type": "date"
        },
        "value": {
          "type": "double"
        },
        "timestamp": {
          "index": true,
          "ignore_malformed": false,
          "store": false,
          "type": "date",
          "doc_values": true
        },
        "app": {
          "type": "keyword"
        },
        "capture_size": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "ecs_task_version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "index_options": "docs",
          "doc_values": true
        },
        "stat_type": {
          "type": "keyword"
        },
        "docker_container_name": {
          "type": "keyword"
        },
        "conf_id": {
          "type": "keyword"
        },
        "edac_id": {
          "type": "keyword"
        },
        "ip": {
          "type": "ip"
        },
        "k8s_pod_name": {
          "type": "keyword"
        },
        "logical_source": {
          "type": "keyword"
        },
        "environment": {
          "type": "keyword"
        },
        "event_id": {
          "type": "keyword"
        },
        "capture_duration": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "ecs_container": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "capture_bytesize": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "group_id": {
          "type": "keyword"
        },
        "org_id": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "alert_def_name": {
          "type": "keyword"
        },
        "ecs_cluster": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "threshold_description": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "threshold_type": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "src_type": {
          "type": "keyword"
        },
        "region": {
          "type": "keyword"
        },
        "custom_labels": {
            "type": "text",
            "analyzer": "custom_label_analyzer"
        },
        "properties": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "docker_image": {
          "type": "keyword"
        },
        "custom_labels_flattened": {
          "type": "flattened"
        }
      }
    }
  }
}

For Elasticsearch 7.x, use the following command:

PUT _template/ed-agent-log?
{
  "order": 0,
  "index_patterns": [
    "ed-agent-log-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "ed-agent-log-policy",
        "rollover_alias": "ed-agent-log"
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
        },
    "analysis": {
      "analyzer": {
        "custom_label_analyzer": {
          "tokenizer": "custom_label_tokenizer"
        }
      },
      "tokenizer": {
        "custom_label_tokenizer": {
          "type": "pattern",
          "pattern": "\\$"
        }
      }
    }
  },
  "mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": [],
      "properties": {
        "msg": {
          "type": "text"
        },
        "alert_def_id": {
          "type": "keyword"
        },
        "k8s_namespace": {
          "type": "keyword"
        },
        "merge_level": {
          "type": "keyword"
        },
        "ecs_task_family": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "k8s_controller_kind": {
          "type": "keyword"
        },
        "k8s_container_image": {
          "type": "keyword"
        },
        "title": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "type": {
          "type": "keyword"
        },
        "src_name": {
          "type": "keyword"
        },
        "k8s_container_name": {
          "type": "keyword"
        },
        "score": {
          "type": "double"
        },
        "sub_type": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "capture_flush_mode": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "tag": {
          "type": "keyword"
        },
        "k8s_controller_logical_name": {
          "type": "keyword"
        },
        "timestamp_end": {
          "type": "date"
        },
        "value": {
          "type": "double"
        },
        "timestamp": {
          "index": true,
          "ignore_malformed": false,
          "store": false,
          "type": "date",
          "doc_values": true
        },
        "app": {
          "type": "keyword"
        },
        "capture_size": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "ecs_task_version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "index_options": "docs",
          "doc_values": true
        },
        "stat_type": {
          "type": "keyword"
        },
        "docker_container_name": {
          "type": "keyword"
        },
        "conf_id": {
          "type": "keyword"
        },
        "edac_id": {
          "type": "keyword"
        },
        "ip": {
          "type": "ip"
        },
        "k8s_pod_name": {
          "type": "keyword"
        },
        "logical_source": {
          "type": "keyword"
        },
        "environment": {
          "type": "keyword"
        },
        "event_id": {
          "type": "keyword"
        },
        "capture_duration": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "ecs_container": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "capture_bytesize": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "group_id": {
          "type": "keyword"
        },
        "org_id": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "alert_def_name": {
          "type": "keyword"
        },
        "ecs_cluster": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "threshold_description": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "threshold_type": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "src_type": {
          "type": "keyword"
        },
        "region": {
          "type": "keyword"
        },
        "custom_labels": {
            "type": "text",
            "analyzer": "custom_label_analyzer"
        },
        "properties": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "docker_image": {
          "type": "keyword"
        },
        "custom_labels_flattened": {
          "type": "flattened"
        }
      }
    }
  }

Step 3: Create the First Index

To generate a daily index, you must create the first index. This first index will inherit field mappings and policies from the template. In your Elastic’s dev console, run one of the following commands depending on your version of Elastic.

Run one of the following commands depending on your Elastic version:

For Elasticsearch 8.x, use the following command in the Elastic dev console:

PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "ed-agent-log": {
     "is_write_index": true
   }
 }
}

For Elasticsearch 7.x, use the following command:

PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "ed-agent-log": {
     "is_write_index": true
   }
 }
}

Click Index Management > Indices, and then locate a new index, similar to ed-agent-log-2025.10.22-000000 with the current date.

Your Elastic environment is ready for Edge Delta data. Next you configure the Edge Delta agent to flush data to your Elastic index.

Step 4: Configure the Edge Delta Agent

Finally, you configure a streaming output that points to the new index in your Edge Delta agent configuration. The following example illustrates Elastic data destination configurations:

outputs:
  streams:
    - name: elastic-cloud
      type: elastic
      index: "<index name>"
      cloud_id: "<cloud ID>"
      token: "<elastic search token>"
      worker_count: 3
      features: metric,cluster,context
    - name: elastic-local
      type: elastic
      index: "index name"
      user: elastic
      password: '{{ Env "ELASTIC_PWD" }}'
      address:
        - elasticnode1
    - name: elastic-send-as-is
      type: elastic
      index: "index name"
      user: elastic
      password: '{{ Env "ELASTIC_PWD" }}'
      address:
        - elasticnode1
      features: edac
      send_as_is: true
      edac_enrichment:
        edac_id_field: "edac_id" 
        metric_name_field: "name" 
    - name: elastic-disable-verify-certificate
      type: elastic
      index: "index name"
      user: elastic
      password: '{{ Env "ELASTIC_PWD" }}'
      address:
        - elasticnode1
      tls:
        disable_verify: true

Required Parameters

name

The name parameter specifies a name for the data destination. You refer to this name in other places, for example to refer to a specific destination in a workflow. Names must be unique within the outputs section. It is a yaml list element so it begins with a - and a space followed by the string. A name is required for a data destinations.

outputs:
  streams:
    - name: <data destination name>

type: elastic

The type parameter specifies a vendor or technology for the streaming data destination. It is a closed list element that requires one of the options. See the supported types here{target="_blank"}. A type is required for a streaming data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <destination type>

index

The index parameter specifies which index to send data to in Elastic or OpenSearch. It is written as a string. An index is required for an Elastic or OpenSearch data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: "<index name>"

Optional Parameters

address

The address parameter specifies the endpoint for an Elastic data destination: either an Elastic node or an OpenSearch domain endpoint. An address is a required parameter for an elastic type data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: "<index name>"
      address: 
        - <opensearch domain endpoint | elastic node>

cloud_id

The cloud_id parameter specifies a unique identifier for authentication to hosted elastic instances. It is used with the token parameter. It is written as a string. A cloud_id is an optional parameter for a data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      cloud_id: "<cloud ID>"
      token: "<cloud token>"

edac_enrichment

The edac_enrichment parameter specifies the EDAC ID and a custom metric name used to enrich the original JSON payload when sending to an Elastic output. It is used when the send_as_is parameter is set to true. It takes two child parameters:

edac_id_field is used when the features parameter includes edac. It specifies the EDAC ID to be used as an additional field in the final JSON payload.
metric_name_field is used when the features parameter includes metric. It specifies a custom name to create or to override an exiting metric name.

The edac_enrichment parameter is optional.

outputs:
  streams:
    - name: elastic-send-as-is
      type: elastic
      index: "index name"
      features: edac
      send_as_is: true
      edac_enrichment:
        edac_id_field: "edac_id"
        metric_name_field: "name"

features

The features parameter specifies which types of data collected or generated by the agent to send to the output. It is written as a comma separated list. All streaming destinations support a features field but not all of them support the full list of datasets. For example, some destinations only support metrics. The features you can include are listed here. A feature is optional for a data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      features: <feature 1>, <feature 2>

password

The password parameter is used with the user parameter to authenticate and authorize access to the streaming destination, depending on how access has been configured. It should refer to a secret environment variable. A password is optional for a data destination, depending on the access configuration.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: <index name>
      user: <user name>
      password: '{{ Env "ELASTIC_PWD" }}'

send_as_is

The send_as_is parameter is used to configure child options for JSON message formats. It is a Boolean parameter. If send_as_is is enabled, you can use send_as_is_options child parameters:

nest_under can be used to nest all the content of JSON log under a custom field. For example, if set to msg, a log message like this: {pid: 1223, pname: os_stat_check} would be sent to elastic like this: {tag: "prod", src_type: "File",..., msg.pid: 1223, msg.pname: os_stat_check}. Top level fields are ED metadata fields and msg.* contains the log JSON
include_ed_metadata is a Boolean used to send all ed metadata fields at top level fields in JSON, the default is false.
on_failure_options is used to handle incoming raw data not in JSON format. It specifies a sub_field_name under which a JSON object is created and the raw message is populated

A send_as_is parameter is optional for a data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: <index name>
      send_as_is: true
      send_as_is_options:
        nest_under: msg
        include_ed_metadata: true
        on_failure_options:
          sub_field_name: "<nested field name>"

tls

The tls parameter specifies whether to perform TLS verification of certificates. It takes a child parameter disable_verify which is a Boolean. The default value is false. The tls parameter is optional.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      tls:
        disable_verify: true

token

The token parameter provides authentication to hosted elastic instances. It is used with the cloud_id parameter. It is written as a string. A token is optional for a data destination.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      cloud_id: "<cloud ID>"
      token: "<cloud token>"

user

The user parameter is used with the password parameter to authenticate and authorize access to the streaming destination, depending on how access has been configured. It is written as a string. A user is optional for a data destination depending on the access configuration.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: <index name>
      user: <user name>
      password: '{{ Env "ELASTIC_PWD" }}'

worker_count

The worker_count parameter is used to specify the number of worker nodes to use for processing traffic. It is written as an integer. A worker_count is optional and the default is 2.

outputs:
  streams:
    - name: <data destination name>
      type: <data destination type>
      index: <index name>
      worker_count: 3

Troubleshooting Elastic

Full Agent Buffer

The Edge Delta agent’s Elastic buffer can become saturated due to high volumes. For example, this may occur when there is a heavy re-try load while still processing current logs, or when the incoming data volume overwhelms the number of agents. In this instance you may receive an error about the output queue:

"Elasticsearch-Streamer-AA-BBBB-CCC-DDDD-LOGS" streamer's output queue of size 1000 is full; consider increasing worker_count (current: 1) for the destination

In this instance, you can increase the worker_count parameter in the output configuration for the integration named in the error.

outputs:
  streams:
  - integration_name: AA-BBBB-CCC-DDDD-LOGS
    worker_count: 2

You can monitor the queue buffer with the following health metrics on the Pipeline Status page:

<tag>.filled_buffer.count.one_minute
<tag>.filled_buffer.count.ten_minutes

Time Format

Check that the correct time format has been configured or the timestamp will not be parsed correctly:

bulk add custom entry operation failed, error type: mapper_parsing_exception, reason: failed to parse field [timestamp] of type [date] in document

See the Elastic documentation for configurable time formats.

The template example Edge Delta provides uses the strict format. If you use the basic format, change the date format as follows:

"mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "basic_date_time",
        "yyyyMMdd'T'HHmmss.SSSZ"

If you have multiple formats for timestamps in your logs, the basic_date_time may break those records that are being accepted with the strict format. In that instance, combine the formats with an OR operator to have Elastic accept multiple formats:

"mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "basic_date_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z||yyyyMMdd'T'HHmmss.SSSZ"