Edge Delta Elastic Destination

Send logs to Elastic.

  14 minute read  

Overview

The Elastic destination node send items to an Elastic destination. It sends raw bytes that are generated via marshaling items as JSON. Before marshaling, the _type field is changed to __type and _timestamp is changed into @timestamp.

Configuring Elastic

You need to configure Elastic to use it as a data destination in Edge Delta. To do this you create a lifecycle policy and an index template. Then you can update the Edge Delta Pipeline configuration to send data to Elastic.

Step 1: Create a Lifecycle Policy

Index lifecycle policies manage indices based on your performance, resiliency, and retention requirements. You can use the following sample lifecycle policy to start, which creates a new index every day and maintains data from the last 15 days. While you can run the command with the pre-populated settings, you can also change the retention period, as well as other fields.

In your Elastic’s dev console, run this command:

PUT _ilm/policy/ed-agent-log-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "5gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

After you run the command, you should have an index lifecycle policy named ed-agent-log-policy.

Step 2: Create an Index Template

An index template is useful to configure Elastic indices before they are created. While the Edge Delta agent can be configured to stream various types of observations to the Elasticsearch destination, we recommend that you create the target index with the recommend index template. The following sample will create an index template named ed-agent-log with field mappings to the ed-agent-log-policy lifecycle policy.

Too many unique field mappings in Elastic can cause out of memory errors and difficult situations to recover from. Edge Delta supports aggregation of custom fields in Elastic by using a field called “custom_labels_flattened” to index mapping and it utilizes that field for metric and log entries. This field is used by default to send custom labels to Elastic.

In your Elastic’s dev console, run one of the following commands depending on your version of Elastic. For Elasticsearch 8.x, use the following command:

PUT _index_template/ed-agent-log
{
  "index_patterns": [
    "ed-agent-log-*"
  ],    
      
  "template":{  
    "settings": {
    "number_of_shards": "1",
    "number_of_replicas": "1",
    "lifecycle": {
        "name": "ed-agent-log-policy",
        "rollover_alias": "ed-agent-log"
        },

    "analysis": {
      "analyzer": {
        "custom_label_analyzer": {
          "tokenizer": "custom_label_tokenizer"
        }
      },
      "tokenizer": {
        "custom_label_tokenizer": {
          "type": "pattern",
          "pattern": "\\$"
        }
      }
    }
  },
  "mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": [],
      "properties": {
        "msg": {
          "type": "text"
        },
        "alert_def_id": {
          "type": "keyword"
        },
        "k8s_namespace": {
          "type": "keyword"
        },
        "merge_level": {
          "type": "keyword"
        },
        "ecs_task_family": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "k8s_controller_kind": {
          "type": "keyword"
        },
        "k8s_container_image": {
          "type": "keyword"
        },
        "title": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "type": {
          "type": "keyword"
        },
        "src_name": {
          "type": "keyword"
        },
        "k8s_container_name": {
          "type": "keyword"
        },
        "score": {
          "type": "double"
        },
        "sub_type": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "capture_flush_mode": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "tag": {
          "type": "keyword"
        },
        "k8s_controller_logical_name": {
          "type": "keyword"
        },
        "timestamp_end": {
          "type": "date"
        },
        "value": {
          "type": "double"
        },
        "timestamp": {
          "index": true,
          "ignore_malformed": false,
          "store": false,
          "type": "date",
          "doc_values": true
        },
        "app": {
          "type": "keyword"
        },
        "capture_size": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "ecs_task_version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "index_options": "docs",
          "doc_values": true
        },
        "stat_type": {
          "type": "keyword"
        },
        "docker_container_name": {
          "type": "keyword"
        },
        "conf_id": {
          "type": "keyword"
        },
        "edac_id": {
          "type": "keyword"
        },
        "ip": {
          "type": "ip"
        },
        "k8s_pod_name": {
          "type": "keyword"
        },
        "logical_source": {
          "type": "keyword"
        },
        "environment": {
          "type": "keyword"
        },
        "event_id": {
          "type": "keyword"
        },
        "capture_duration": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "ecs_container": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "capture_bytesize": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "group_id": {
          "type": "keyword"
        },
        "org_id": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "alert_def_name": {
          "type": "keyword"
        },
        "ecs_cluster": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "threshold_description": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "threshold_type": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "src_type": {
          "type": "keyword"
        },
        "region": {
          "type": "keyword"
        },
        "custom_labels": {
            "type": "text",
            "analyzer": "custom_label_analyzer"
        },
        "properties": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "docker_image": {
          "type": "keyword"
        },
        "custom_labels_flattened": {
          "type": "flattened"
        }
      }
    }
  }
}

For Elasticsearch 7.x, use the following command:

PUT _template/ed-agent-log?
{
  "order": 0,
  "index_patterns": [
    "ed-agent-log-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "ed-agent-log-policy",
        "rollover_alias": "ed-agent-log"
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
        },
    "analysis": {
      "analyzer": {
        "custom_label_analyzer": {
          "tokenizer": "custom_label_tokenizer"
        }
      },
      "tokenizer": {
        "custom_label_tokenizer": {
          "type": "pattern",
          "pattern": "\\$"
        }
      }
    }
  },
  "mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": [],
      "properties": {
        "msg": {
          "type": "text"
        },
        "alert_def_id": {
          "type": "keyword"
        },
        "k8s_namespace": {
          "type": "keyword"
        },
        "merge_level": {
          "type": "keyword"
        },
        "ecs_task_family": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "k8s_controller_kind": {
          "type": "keyword"
        },
        "k8s_container_image": {
          "type": "keyword"
        },
        "title": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "type": {
          "type": "keyword"
        },
        "src_name": {
          "type": "keyword"
        },
        "k8s_container_name": {
          "type": "keyword"
        },
        "score": {
          "type": "double"
        },
        "sub_type": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "capture_flush_mode": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "tag": {
          "type": "keyword"
        },
        "k8s_controller_logical_name": {
          "type": "keyword"
        },
        "timestamp_end": {
          "type": "date"
        },
        "value": {
          "type": "double"
        },
        "timestamp": {
          "index": true,
          "ignore_malformed": false,
          "store": false,
          "type": "date",
          "doc_values": true
        },
        "app": {
          "type": "keyword"
        },
        "capture_size": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "ecs_task_version": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "index_options": "docs",
          "doc_values": true
        },
        "stat_type": {
          "type": "keyword"
        },
        "docker_container_name": {
          "type": "keyword"
        },
        "conf_id": {
          "type": "keyword"
        },
        "edac_id": {
          "type": "keyword"
        },
        "ip": {
          "type": "ip"
        },
        "k8s_pod_name": {
          "type": "keyword"
        },
        "logical_source": {
          "type": "keyword"
        },
        "environment": {
          "type": "keyword"
        },
        "event_id": {
          "type": "keyword"
        },
        "capture_duration": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "ecs_container": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "capture_bytesize": {
          "coerce": true,
          "index": false,
          "ignore_malformed": false,
          "store": false,
          "type": "long",
          "doc_values": false
        },
        "group_id": {
          "type": "keyword"
        },
        "org_id": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "alert_def_name": {
          "type": "keyword"
        },
        "ecs_cluster": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "threshold_description": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "threshold_type": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "src_type": {
          "type": "keyword"
        },
        "region": {
          "type": "keyword"
        },
        "custom_labels": {
            "type": "text",
            "analyzer": "custom_label_analyzer"
        },
        "properties": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "docker_image": {
          "type": "keyword"
        },
        "custom_labels_flattened": {
          "type": "flattened"
        }
      }
    }
  }

Step 3: Create the First Index

To generate a daily index, you must create the first index. This first index will inherit field mappings and policies from the template. In your Elastic’s dev console, run one of the following commands depending on your version of Elastic.

  1. Run one of the following commands depending on your Elastic version:

For Elasticsearch 8.x, use the following command in the Elastic dev console:

PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "ed-agent-log": {
     "is_write_index": true
   }
 }
}

For Elasticsearch 7.x, use the following command:

PUT /%3Ced-agent-log-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "ed-agent-log": {
     "is_write_index": true
   }
 }
}
  1. Click Index Management > Indices, and then locate a new index, similar to ed-agent-log-2025.10.22-000000 with the current date.

Your Elastic environment is ready for Edge Delta data. Next you configure the Edge Delta agent to flush data to your Elastic index.

Step 4: Configure the Edge Delta Agent

Use Visual Pipelines or the agent YAML to configure the Elastic destination node.

Example Configuration

nodes:
  - name: my_elastic
    type: elastic_output
    index: <REDACTED>
    user: elastic
    password: <REDACTED>
    address:
      - <REDACTED>

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: elastic_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Optional Parameters

address

The address parameter specifies the address list for the Elastic backend. It is specified as a string. It is required unless cloud_id is specified.

nodes:
  - name: <node name>
    type: elastic_output
    address: <string>
    token: <token>

buffer_max_bytesize

The buffer_max_bytesize parameter configures the maximum byte size for total unsuccessful items. If the limit is reached, the remaining items are discarded until the buffer space becomes available. It is specified as a datasize.Size, has a default of 0 indicating no size limit, and it is optional.

nodes:
 - name: <node name>
   type: elastic_output
   cloud_id: <Cloud ID>
   token: <token>
   buffer_max_bytesize: 2048

buffer_path

The buffer_path parameter configures the path to store unsuccessful items. Unsuccessful items are stored there to be retried back (exactly once delivery). It is specified as a string and it is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>
    buffer_path: <path to unsuccessful items folder>

buffer_ttl

The buffer_ttl parameter configures the time-to-Live for unsuccessful items, which indicates when to discard them. It is specified as a duration, has a default of 10m, and it is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>
    buffer_ttl: 20m

cloud_id

The cloud_id parameter specifies the authentication ID for the Elastic backend. It is specified as a string. It is required unless address is specified.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>

external_id

The external_id parameter is a unique identifier to avoid a confused deputy attack. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>
    external_id: <ID>

features

YAML Only

The features parameter defines which data types the agent sends to a streaming destination. You can specify one or more of the following, some of which are enabled by default if specific features are not set:

  • metric (default)
  • edac (default)
  • cluster (default)
  • cluster_pattern
  • cluster_sample
  • log (default)
  • topk (default)
  • alert (default)
nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>
    features: metric

index

The index parameter defines which index the node should flush data into. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>
    index: <index>

password

The password parameter specifies the password for authentication if user has been specified instead of a token. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    user: <username>
    password: <password>

region

The region parameter specifies the region where the OpenSearch cluster is found. It is used with user and password. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    user: <username>
    password: <password>
    region: <region>

role_arn

The role_arn parameter is used if authentication and authorization is performed using an assumed AWS IAM role. It should consist of the account ID and role name. A role_arn is optional for a data destination depending on the access configuration.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    user: <username>
    password: <password>
    region: <region>
    role_arn: <role ARN>

tls

ca_file The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      ca_file: /certs/ca.pem

ca_path The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      ca_path: /var/etc/kafka

client_auth_type The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

  • noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
  • requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
  • requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
  • verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
  • requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client
nodes:
  - name: <node name>
    type: elastic_output
    tls:
      client_auth_type: <auth type>

crt_file The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      crt_file: /certs/server-cert.pem

ignore_certificate_check The ignore_certificate_check parameter is a child of the tls parameter. When set to true, it ignores certificate checks for the remote endpoint. It is specified as a Boolean value and the default is false, indicating that TLS verification will be performed. This is an optional parameter.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      ignore_certificate_check: true

key_file The key_file parameter is a child of the tls parameter. It specifies the key file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      key_file: /certs/server-key.pem

key_password The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    tls:
      key_password: <password>

max_version The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: elastic_output
    tls:
      max_version: <TLS version>

min_version The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: elastic_output
    tls:
      min_version: <TLS version>

token

The token parameter provides authentication to hosted elastic instances. It is used with the cloud_id parameter. It is written as a string. A token is optional for a data destination.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    token: <token>

user

The user parameter specifies the username for authentication if password has been specified instead of a token. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: elastic_output
    cloud_id: <Cloud ID>
    user: <username>
    password: <password>

Troubleshooting Elastic

Time Format

Check that the correct time format has been configured or the timestamp will not be parsed correctly:

bulk add custom entry operation failed, error type: mapper_parsing_exception, reason: failed to parse field [timestamp] of type [date] in document

See the Elastic documentation for configurable time formats.

The template example Edge Delta provides uses the strict format. If you use the basic format, change the date format as follows:

"mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "basic_date_time",
        "yyyyMMdd'T'HHmmss.SSSZ"

If you have multiple formats for timestamps in your logs, the basic_date_time may break those records that are being accepted with the strict format. In that instance, combine the formats with an OR operator to have Elastic accept multiple formats:

"mappings": {
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "basic_date_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z||yyyyMMdd'T'HHmmss.SSSZ"