Edge Delta Elastic Output
13 minute read
See the latest version here.
You need to configure Elastic to use it as a data destination in Edge Delta. To do this you create a lifecycle policy and an index template. Then you can update the Edge Delta agent configuration to send data to Elastic.
Step 1: Create a Lifecycle Policy
Index lifecycle policies manage indices based on your performance, resiliency, and retention requirements. You can use the following sample lifecycle policy to start, which creates a new index every day and maintains data from the last 15 days. While you can run the command with the pre-populated settings, you can also change the retention period, as well as other fields.
In your Elastic’s dev console, run this command.
PUT _ilm/policy/ed-agent-log-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1d",
"max_size": "5gb"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "15d",
"actions": {
"delete": {}
}
}
}
}
}
After you run the command, you should have an index lifecycle policy named ed-agent-log-policy.
Step 2: Create an Index Template
An index template is useful to configure Elastic indices before they are created. While the Edge Delta agent can be configured to stream various types of observations to the Elasticsearch destination, we recommend that you create the target index with the recommend index template. The following sample will create an index template named ed-agent-log with field mappings to the ed-agent-log-policy lifecycle policy.
Too many unique field mappings in Elastic can cause out of memory errors and difficult situations to recover from. Edge Delta supports aggregation of custom fields in Elastic by using a field called “custom_labels_flattened” to index mapping and it utilizes that field for metric and log entries. This field is used by default to send custom labels to Elastic.
In your Elastic’s dev console, run one of the following commands depending on your version of Elastic. For Elasticsearch 8.x, use the following command:
PUT _index_template/ed-agent-log
{
"index_patterns": [
"ed-agent-log-*"
],
"template":{
"settings": {
"number_of_shards": "1",
"number_of_replicas": "1",
"lifecycle": {
"name": "ed-agent-log-policy",
"rollover_alias": "ed-agent-log"
},
"analysis": {
"analyzer": {
"custom_label_analyzer": {
"tokenizer": "custom_label_tokenizer"
}
},
"tokenizer": {
"custom_label_tokenizer": {
"type": "pattern",
"pattern": "\\$"
}
}
}
},
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic_templates": [],
"properties": {
"msg": {
"type": "text"
},
"alert_def_id": {
"type": "keyword"
},
"k8s_namespace": {
"type": "keyword"
},
"merge_level": {
"type": "keyword"
},
"ecs_task_family": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"k8s_controller_kind": {
"type": "keyword"
},
"k8s_container_image": {
"type": "keyword"
},
"title": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": false
},
"type": {
"type": "keyword"
},
"src_name": {
"type": "keyword"
},
"k8s_container_name": {
"type": "keyword"
},
"score": {
"type": "double"
},
"sub_type": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"capture_flush_mode": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"tag": {
"type": "keyword"
},
"k8s_controller_logical_name": {
"type": "keyword"
},
"timestamp_end": {
"type": "date"
},
"value": {
"type": "double"
},
"timestamp": {
"index": true,
"ignore_malformed": false,
"store": false,
"type": "date",
"doc_values": true
},
"app": {
"type": "keyword"
},
"capture_size": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"ecs_task_version": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"index_options": "docs",
"doc_values": true
},
"stat_type": {
"type": "keyword"
},
"docker_container_name": {
"type": "keyword"
},
"conf_id": {
"type": "keyword"
},
"edac_id": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"k8s_pod_name": {
"type": "keyword"
},
"logical_source": {
"type": "keyword"
},
"environment": {
"type": "keyword"
},
"event_id": {
"type": "keyword"
},
"capture_duration": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"ecs_container": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"capture_bytesize": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"group_id": {
"type": "keyword"
},
"org_id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"alert_def_name": {
"type": "keyword"
},
"ecs_cluster": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"threshold_description": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"threshold_type": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"src_type": {
"type": "keyword"
},
"region": {
"type": "keyword"
},
"custom_labels": {
"type": "text",
"analyzer": "custom_label_analyzer"
},
"properties": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"docker_image": {
"type": "keyword"
},
"custom_labels_flattened": {
"type": "flattened"
}
}
}
}
}
For Elasticsearch 7.x, use the following command:
PUT _template/ed-agent-log?
{
"order": 0,
"index_patterns": [
"ed-agent-log-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "ed-agent-log-policy",
"rollover_alias": "ed-agent-log"
},
"number_of_shards": "1",
"number_of_replicas": "1"
},
"analysis": {
"analyzer": {
"custom_label_analyzer": {
"tokenizer": "custom_label_tokenizer"
}
},
"tokenizer": {
"custom_label_tokenizer": {
"type": "pattern",
"pattern": "\\$"
}
}
}
},
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic_templates": [],
"properties": {
"msg": {
"type": "text"
},
"alert_def_id": {
"type": "keyword"
},
"k8s_namespace": {
"type": "keyword"
},
"merge_level": {
"type": "keyword"
},
"ecs_task_family": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"k8s_controller_kind": {
"type": "keyword"
},
"k8s_container_image": {
"type": "keyword"
},
"title": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": false
},
"type": {
"type": "keyword"
},
"src_name": {
"type": "keyword"
},
"k8s_container_name": {
"type": "keyword"
},
"score": {
"type": "double"
},
"sub_type": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"capture_flush_mode": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"tag": {
"type": "keyword"
},
"k8s_controller_logical_name": {
"type": "keyword"
},
"timestamp_end": {
"type": "date"
},
"value": {
"type": "double"
},
"timestamp": {
"index": true,
"ignore_malformed": false,
"store": false,
"type": "date",
"doc_values": true
},
"app": {
"type": "keyword"
},
"capture_size": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"ecs_task_version": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"index_options": "docs",
"doc_values": true
},
"stat_type": {
"type": "keyword"
},
"docker_container_name": {
"type": "keyword"
},
"conf_id": {
"type": "keyword"
},
"edac_id": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"k8s_pod_name": {
"type": "keyword"
},
"logical_source": {
"type": "keyword"
},
"environment": {
"type": "keyword"
},
"event_id": {
"type": "keyword"
},
"capture_duration": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"ecs_container": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"capture_bytesize": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"group_id": {
"type": "keyword"
},
"org_id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"alert_def_name": {
"type": "keyword"
},
"ecs_cluster": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"threshold_description": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"threshold_type": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"src_type": {
"type": "keyword"
},
"region": {
"type": "keyword"
},
"custom_labels": {
"type": "text",
"analyzer": "custom_label_analyzer"
},
"properties": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"docker_image": {
"type": "keyword"
},
"custom_labels_flattened": {
"type": "flattened"
}
}
}
}
Step 3: Create the First Index
To generate a daily index, you must create the first index. This first index will inherit field mappings and policies from the template. In your Elastic’s dev console, run one of the following commands depending on your version of Elastic.
- Run one of the following commands depending on your Elastic version:
For Elasticsearch 8.x, use the following command in the Elastic dev console:
PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"ed-agent-log": {
"is_write_index": true
}
}
}
For Elasticsearch 7.x, use the following command:
PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"ed-agent-log": {
"is_write_index": true
}
}
}
- Click Index Management > Indices, and then locate a new index, similar to ed-agent-log-2025.10.22-000000 with the current date.
Your Elastic environment is ready for Edge Delta data. Next you configure the Edge Delta agent to flush data to your Elastic index.
Step 4: Configure the Edge Delta Agent
Finally, you configure a streaming output that points to the new index in your Edge Delta agent configuration. The following example illustrates Elastic data destination configurations:
outputs:
streams:
- name: elastic-cloud
type: elastic
index: "<index name>"
cloud_id: "<cloud ID>"
token: "<elastic search token>"
worker_count: 3
features: metric,cluster,context
- name: elastic-local
type: elastic
index: "index name"
user: elastic
password: '{{ Env "ELASTIC_PWD" }}'
address:
- elasticnode1
- name: elastic-send-as-is
type: elastic
index: "index name"
user: elastic
password: '{{ Env "ELASTIC_PWD" }}'
address:
- elasticnode1
features: edac
send_as_is: true
edac_enrichment:
edac_id_field: "edac_id"
metric_name_field: "name"
- name: elastic-disable-verify-certificate
type: elastic
index: "index name"
user: elastic
password: '{{ Env "ELASTIC_PWD" }}'
address:
- elasticnode1
tls:
disable_verify: true
Required Parameters
name
The name parameter specifies a name for the data destination. You refer to this name in other places, for example to refer to a specific destination in a workflow. Names must be unique within the outputs section. It is a yaml list element so it begins with a - and a space followed by the string. A name is required for a data destinations.
outputs:
streams:
- name: <data destination name>
type: elastic
The type parameter specifies a vendor or technology for the streaming data destination. It is a closed list element that requires one of the options. See the supported types here{target="_blank"}. A type is required for a streaming data destination.
outputs:
streams:
- name: <data destination name>
type: <destination type>
index
The index parameter specifies which index to send data to in Elastic or OpenSearch. It is written as a string. An index is required for an Elastic or OpenSearch data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: "<index name>"
Optional Parameters
address
The address parameter specifies the endpoint for an Elastic data destination: either an Elastic node or an OpenSearch domain endpoint. An address is a required parameter for an elastic type data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: "<index name>"
address:
- <opensearch domain endpoint | elastic node>
cloud_id
The cloud_id parameter specifies a unique identifier for authentication to hosted elastic instances. It is used with the token parameter. It is written as a string. A cloud_id is an optional parameter for a data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
cloud_id: "<cloud ID>"
token: "<cloud token>"
edac_enrichment
The edac_enrichment parameter specifies the EDAC ID and a custom metric name used to enrich the original JSON payload when sending to an Elastic output. It is used when the send_as_is parameter is set to true. It takes two child parameters:
edac_id_fieldis used when thefeaturesparameter includesedac. It specifies the EDAC ID to be used as an additional field in the final JSON payload.metric_name_fieldis used when thefeaturesparameter includesmetric. It specifies a custom name to create or to override an exiting metric name.
The edac_enrichment parameter is optional.
outputs:
streams:
- name: elastic-send-as-is
type: elastic
index: "index name"
features: edac
send_as_is: true
edac_enrichment:
edac_id_field: "edac_id"
metric_name_field: "name"
features
The features parameter specifies which types of data collected or generated by the agent to send to the output. It is written as a comma separated list. All streaming destinations support a features field but not all of them support the full list of datasets. For example, some destinations only support metrics. The features you can include are listed here. A feature is optional for a data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
features: <feature 1>, <feature 2>
password
The password parameter is used with the user parameter to authenticate and authorize access to the streaming destination, depending on how access has been configured. It should refer to a secret environment variable. A password is optional for a data destination, depending on the access configuration.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: <index name>
user: <user name>
password: '{{ Env "ELASTIC_PWD" }}'
send_as_is
The send_as_is parameter is used to configure child options for JSON message formats. It is a Boolean parameter. If send_as_is is enabled, you can use send_as_is_options child parameters:
nest_undercan be used to nest all the content of JSON log under a custom field. For example, if set tomsg, a log message like this:{pid: 1223, pname: os_stat_check}would be sent to elastic like this:{tag: "prod", src_type: "File",..., msg.pid: 1223, msg.pname: os_stat_check}. Top level fields are ED metadata fields andmsg.*contains the log JSONinclude_ed_metadatais a Boolean used to send all ed metadata fields at top level fields in JSON, the default isfalse.on_failure_optionsis used to handle incoming raw data not in JSON format. It specifies asub_field_nameunder which a JSON object is created and the raw message is populated
A send_as_is parameter is optional for a data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: <index name>
send_as_is: true
send_as_is_options:
nest_under: msg
include_ed_metadata: true
on_failure_options:
sub_field_name: "<nested field name>"
tls
The tls parameter specifies whether to perform TLS verification of certificates. It takes a child parameter disable_verify which is a Boolean. The default value is false. The tls parameter is optional.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
tls:
disable_verify: true
token
The token parameter provides authentication to hosted elastic instances. It is used with the cloud_id parameter. It is written as a string. A token is optional for a data destination.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
cloud_id: "<cloud ID>"
token: "<cloud token>"
user
The user parameter is used with the password parameter to authenticate and authorize access to the streaming destination, depending on how access has been configured. It is written as a string. A user is optional for a data destination depending on the access configuration.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: <index name>
user: <user name>
password: '{{ Env "ELASTIC_PWD" }}'
worker_count
The worker_count parameter is used to specify the number of worker nodes to use for processing traffic. It is written as an integer. A worker_count is optional and the default is 2.
outputs:
streams:
- name: <data destination name>
type: <data destination type>
index: <index name>
worker_count: 3
Troubleshooting Elastic
Full Agent Buffer
The Edge Delta agent’s Elastic buffer can become saturated due to high volumes. For example, this may occur when there is a heavy re-try load while still processing current logs, or when the incoming data volume overwhelms the number of agents. In this instance you may receive an error about the output queue:
"Elasticsearch-Streamer-AA-BBBB-CCC-DDDD-LOGS" streamer's output queue of size 1000 is full; consider increasing worker_count (current: 1) for the destination
In this instance, you can increase the worker_count parameter in the output configuration for the integration named in the error.
outputs:
streams:
- integration_name: AA-BBBB-CCC-DDDD-LOGS
worker_count: 2
You can monitor the queue buffer with the following health metrics on the Pipelines Dashboard page:
<tag>.filled_buffer.count.one_minute<tag>.filled_buffer.count.ten_minutes
Time Format
Check that the correct time format has been configured or the timestamp will not be parsed correctly:
bulk add custom entry operation failed, error type: mapper_parsing_exception, reason: failed to parse field [timestamp] of type [date] in document
See the Elastic documentation for configurable time formats.
The template example Edge Delta provides uses the strict format. If you use the basic format, change the date format as follows:
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"basic_date_time",
"yyyyMMdd'T'HHmmss.SSSZ"
If you have multiple formats for timestamps in your logs, the basic_date_time may break those records that are being accepted with the strict format. In that instance, combine the formats with an OR operator to have Elastic accept multiple formats:
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"basic_date_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z||yyyyMMdd'T'HHmmss.SSSZ"