Edge Delta Elastic Output
14 minute read
Overview
The Elastic Output Node send items to an Elastic destination. It sends raw bytes that are generated via marshaling items as JSON. Before marshaling, the _type
field is changed to __type
and _timestamp
is changed into @timestamp
.
Configuring Elastic
You need to configure Elastic to use it as a data destination in Edge Delta. To do this you create a lifecycle policy and an index template. Then you can update the Edge Delta Pipeline configuration to send data to Elastic.
Step 1: Create a Lifecycle Policy
Index lifecycle policies manage indices based on your performance, resiliency, and retention requirements. You can use the following sample lifecycle policy to start, which creates a new index every day and maintains data from the last 15 days. While you can run the command with the pre-populated settings, you can also change the retention period, as well as other fields.
In your Elastic’s dev console, run this command:
PUT _ilm/policy/ed-agent-log-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1d",
"max_size": "5gb"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "15d",
"actions": {
"delete": {}
}
}
}
}
}
After you run the command, you should have an index lifecycle policy named ed-agent-log-policy.
Step 2: Create an Index Template
An index template is useful to configure Elastic indices before they are created. While the Edge Delta agent can be configured to stream various types of observations to the Elasticsearch destination, we recommend that you create the target index with the recommend index template. The following sample will create an index template named ed-agent-log with field mappings to the ed-agent-log-policy lifecycle policy.
Too many unique field mappings in Elastic can cause out of memory errors and difficult situations to recover from. Edge Delta supports aggregation of custom fields in Elastic by using a field called “custom_labels_flattened” to index mapping and it utilizes that field for metric and log entries. This field is used by default to send custom labels to Elastic.
In your Elastic’s dev console, run one of the following commands depending on your version of Elastic. For Elasticsearch 8.x, use the following command:
PUT _index_template/ed-agent-log
{
"index_patterns": [
"ed-agent-log-*"
],
"template":{
"settings": {
"number_of_shards": "1",
"number_of_replicas": "1",
"lifecycle": {
"name": "ed-agent-log-policy",
"rollover_alias": "ed-agent-log"
},
"analysis": {
"analyzer": {
"custom_label_analyzer": {
"tokenizer": "custom_label_tokenizer"
}
},
"tokenizer": {
"custom_label_tokenizer": {
"type": "pattern",
"pattern": "\\$"
}
}
}
},
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic_templates": [],
"properties": {
"msg": {
"type": "text"
},
"alert_def_id": {
"type": "keyword"
},
"k8s_namespace": {
"type": "keyword"
},
"merge_level": {
"type": "keyword"
},
"ecs_task_family": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"k8s_controller_kind": {
"type": "keyword"
},
"k8s_container_image": {
"type": "keyword"
},
"title": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": false
},
"type": {
"type": "keyword"
},
"src_name": {
"type": "keyword"
},
"k8s_container_name": {
"type": "keyword"
},
"score": {
"type": "double"
},
"sub_type": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"capture_flush_mode": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"tag": {
"type": "keyword"
},
"k8s_controller_logical_name": {
"type": "keyword"
},
"timestamp_end": {
"type": "date"
},
"value": {
"type": "double"
},
"timestamp": {
"index": true,
"ignore_malformed": false,
"store": false,
"type": "date",
"doc_values": true
},
"app": {
"type": "keyword"
},
"capture_size": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"ecs_task_version": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"index_options": "docs",
"doc_values": true
},
"stat_type": {
"type": "keyword"
},
"docker_container_name": {
"type": "keyword"
},
"conf_id": {
"type": "keyword"
},
"edac_id": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"k8s_pod_name": {
"type": "keyword"
},
"logical_source": {
"type": "keyword"
},
"environment": {
"type": "keyword"
},
"event_id": {
"type": "keyword"
},
"capture_duration": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"ecs_container": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"capture_bytesize": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"group_id": {
"type": "keyword"
},
"org_id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"alert_def_name": {
"type": "keyword"
},
"ecs_cluster": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"threshold_description": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"threshold_type": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"src_type": {
"type": "keyword"
},
"region": {
"type": "keyword"
},
"custom_labels": {
"type": "text",
"analyzer": "custom_label_analyzer"
},
"properties": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"docker_image": {
"type": "keyword"
},
"custom_labels_flattened": {
"type": "flattened"
}
}
}
}
}
For Elasticsearch 7.x, use the following command:
PUT _template/ed-agent-log?
{
"order": 0,
"index_patterns": [
"ed-agent-log-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "ed-agent-log-policy",
"rollover_alias": "ed-agent-log"
},
"number_of_shards": "1",
"number_of_replicas": "1"
},
"analysis": {
"analyzer": {
"custom_label_analyzer": {
"tokenizer": "custom_label_tokenizer"
}
},
"tokenizer": {
"custom_label_tokenizer": {
"type": "pattern",
"pattern": "\\$"
}
}
}
},
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic_templates": [],
"properties": {
"msg": {
"type": "text"
},
"alert_def_id": {
"type": "keyword"
},
"k8s_namespace": {
"type": "keyword"
},
"merge_level": {
"type": "keyword"
},
"ecs_task_family": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"k8s_controller_kind": {
"type": "keyword"
},
"k8s_container_image": {
"type": "keyword"
},
"title": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": false
},
"type": {
"type": "keyword"
},
"src_name": {
"type": "keyword"
},
"k8s_container_name": {
"type": "keyword"
},
"score": {
"type": "double"
},
"sub_type": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"capture_flush_mode": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"tag": {
"type": "keyword"
},
"k8s_controller_logical_name": {
"type": "keyword"
},
"timestamp_end": {
"type": "date"
},
"value": {
"type": "double"
},
"timestamp": {
"index": true,
"ignore_malformed": false,
"store": false,
"type": "date",
"doc_values": true
},
"app": {
"type": "keyword"
},
"capture_size": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"ecs_task_version": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"index_options": "docs",
"doc_values": true
},
"stat_type": {
"type": "keyword"
},
"docker_container_name": {
"type": "keyword"
},
"conf_id": {
"type": "keyword"
},
"edac_id": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"k8s_pod_name": {
"type": "keyword"
},
"logical_source": {
"type": "keyword"
},
"environment": {
"type": "keyword"
},
"event_id": {
"type": "keyword"
},
"capture_duration": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"ecs_container": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"capture_bytesize": {
"coerce": true,
"index": false,
"ignore_malformed": false,
"store": false,
"type": "long",
"doc_values": false
},
"group_id": {
"type": "keyword"
},
"org_id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"alert_def_name": {
"type": "keyword"
},
"ecs_cluster": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"threshold_description": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"threshold_type": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"src_type": {
"type": "keyword"
},
"region": {
"type": "keyword"
},
"custom_labels": {
"type": "text",
"analyzer": "custom_label_analyzer"
},
"properties": {
"eager_global_ordinals": false,
"norms": false,
"index": false,
"store": false,
"type": "keyword",
"split_queries_on_whitespace": false,
"doc_values": false
},
"docker_image": {
"type": "keyword"
},
"custom_labels_flattened": {
"type": "flattened"
}
}
}
}
Step 3: Create the First Index
To generate a daily index, you must create the first index. This first index will inherit field mappings and policies from the template. In your Elastic’s dev console, run one of the following commands depending on your version of Elastic.
- Run one of the following commands depending on your Elastic version:
For Elasticsearch 8.x, use the following command in the Elastic dev console:
PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"ed-agent-log": {
"is_write_index": true
}
}
}
For Elasticsearch 7.x, use the following command:
PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"ed-agent-log": {
"is_write_index": true
}
}
}
- Click Index Management > Indices, and then locate a new index, similar to ed-agent-log-2025.10.22-000000 with the current date.
Your Elastic environment is ready for Edge Delta data. Next you configure the Edge Delta agent to flush data to your Elastic index.
Step 4: Configure the Edge Delta Agent
Use Visual Pipelines or the agent YAML to configure the Elastic Output node.
Example Configuration
nodes:
- name: my_elastic
type: elastic_output
index: <REDACTED>
user: elastic
password: <REDACTED>
address:
- <REDACTED>
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: elastic_output
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
Optional Parameters
address
The address
parameter specifies the address list for the Elastic backend. It is specified as a string. It is required unless cloud_id
is specified.
nodes:
- name: <node name>
type: elastic_output
address: <string>
token: <token>
buffer_max_bytesize
The buffer_max_bytesize
parameter configures the maximum byte size for total unsuccessful items. If the limit is reached, the remaining items are discarded until the buffer space becomes available. It is specified as a datasize.Size, has a default of 0
indicating no size limit, and it is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
buffer_max_bytesize: 2048
buffer_path
The buffer_path
parameter configures the path to store unsuccessful items. Unsuccessful items are stored there to be retried back (exactly once delivery). It is specified as a string and it is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
buffer_path: <path to unsuccessful items folder>
buffer_ttl
The buffer_ttl
parameter configures the time-to-Live for unsuccessful items, which indicates when to discard them. It is specified as a duration, has a default of 10m
, and it is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
buffer_ttl: 20m
cloud_id
The cloud_id
parameter specifies the authentication ID for the Elastic backend. It is specified as a string. It is required unless address
is specified.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
external_id
The external_id
parameter is a unique identifier to avoid a confused deputy attack. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
external_id: <ID>
features
The features parameter defines which data types the agent sends to a streaming destination. You can specify one or more of the following, some of which are enabled by default if specific features are not set:
- metric (default)
- edac (default)
- cluster (default)
- cluster_pattern
- cluster_sample
- log (default)
- topk (default)
- alert (default)
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
features: metric
index
The index
parameter defines which index the node should flush data into. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
index: <index>
password
The password
parameter specifies the password for authentication if user
has been specified instead of a token
. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
user: <username>
password: <password>
region
The region
parameter specifies the region where the OpenSearch cluster is found. It is used with user
and password
. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
user: <username>
password: <password>
region: <region>
role_arn
The role_arn
parameter is used if authentication and authorization is performed using an assumed AWS IAM role. It should consist of the account ID and role name. A role_arn
is optional for a data destination depending on the access configuration.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
user: <username>
password: <password>
region: <region>
role_arn: <role ARN>
tls
ca_file
The ca_file
parameter is a child of the tls
parameter. It specifies the CA certificate file. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
tls:
ca_file: /certs/ca.pem
ca_path
The ca_path
parameter is a child of the tls
parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
tls:
ca_path: /var/etc/kafka
client_auth_type
The client_auth_type
parameter is a child of the tls
parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.
The following authentication methods are available:
- noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
- requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
- requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
- verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
- requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client
nodes:
- name: <node name>
type: elastic_output
tls:
client_auth_type: <auth type>
crt_file
The crt_file
parameter is a child of the tls
parameter. It specifies the certificate file. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
tls:
crt_file: /certs/server-cert.pem
ignore_certificate_check
The ignore_certificate_check
parameter is a child of the tls
parameter. When set to true
, it ignores certificate checks for the remote endpoint. It is specified as a Boolean value and the default is false
, indicating that TLS verification will be performed. This is an optional parameter.
nodes:
- name: <node name>
type: elastic_output
tls:
ignore_certificate_check: true
key_file
The key_file
parameter is a child of the tls
parameter. It specifies the key file. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
tls:
key_file: /certs/server-key.pem
key_password
The key_password
parameter is a child of the tls
parameter. It specifies the key password. When the private key_file
location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
tls:
key_password: <password>
max_version
The max_version
parameter is a child of the tls
parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.
You can select one of the following options:
TLSv1_0
TLSv1_1
TLSv1_2
TLSv1_3
nodes:
- name: <node name>
type: elastic_output
tls:
max_version: <TLS version>
min_version
The min_version
parameter is a child of the tls
parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2
.
You can select one of the following options:
TLSv1_0
TLSv1_1
TLSv1_2
TLSv1_3
nodes:
- name: <node name>
type: elastic_output
tls:
min_version: <TLS version>
token
The token
parameter provides authentication to hosted elastic instances. It is used with the cloud_id
parameter. It is written as a string. A token
is optional for a data destination.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
token: <token>
user
The user
parameter specifies the username for authentication if password
has been specified instead of a token
. It is specified as a string and is optional.
nodes:
- name: <node name>
type: elastic_output
cloud_id: <Cloud ID>
user: <username>
password: <password>
Troubleshooting Elastic
Time Format
Check that the correct time format has been configured or the timestamp will not be parsed correctly:
bulk add custom entry operation failed, error type: mapper_parsing_exception, reason: failed to parse field [timestamp] of type [date] in document
See the Elastic documentation for configurable time formats.
The template example Edge Delta provides uses the strict format. If you use the basic format, change the date format as follows:
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"basic_date_time",
"yyyyMMdd'T'HHmmss.SSSZ"
If you have multiple formats for timestamps in your logs, the basic_date_time
may break those records that are being accepted with the strict format. In that instance, combine the formats with an OR
operator to have Elastic accept multiple formats:
"mappings": {
"_meta": {},
"_routing": {
"required": false
},
"dynamic": true,
"numeric_detection": false,
"date_detection": true,
"dynamic_date_formats": [
"basic_date_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z||yyyyMMdd'T'HHmmss.SSSZ"