Agent v0.1.21

April 27, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Output In the Edge Delta App, the Splunk streaming output has been updated with the ability to send data in a JSON format.

Specifically, to use this option, you must update the endpoint parameter to point to Splunk’s API services/collector/raw, instead of services/collector/event.

    * name: splunk-integration
      type: splunk
      endpoint: ..../services/collector/raw
      token: ....
      features: log,metric,edac,cluster,alert
      index: rehydration

New Filter Type In the Edge Delta App, you can use the newly created JSON Field Extractor filter to extract a field’s value and replace the whole JSON content with the field’s value.

  * name: extract_severity
    type: extract-json-field
    field_path: "severity"
  * name: extract_first_data
    type: extract-json-field
    field_path: "records.[0].data"

To learn more, see JSON Field Extractor Filters.

Updated Enrichments for AWS ECS Inputs The configurations to enrich input data have been updated.

Specifically, you can use the dynamic parameter to enrich input data from AWS ECS.

In the agent configuration, you can

To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            * field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            * field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'
            * field_name: "cluster_name"
              value: '{{".aws-instance.cluster-name"}}'
            * field_name: "ec2launchtemplate_id"
              value: '{{".aws-instance.ec2launchtemplate-id"}}'
            * field_name: "ec2launchtemplate_version"
              value: '{{".aws-instance.ec2launchtemplate-version"}}'
            * field_name: "inspector_enabled"
              value: '{{".aws-instance.inspector-enabled"}}'
            * field_name: "cluster_autoscaler_enabled"
              value: '{{".aws-instance.cluster-autoscaler-enabled"}}'
            * field_name: "autoscaling_groupName"
              value: '{{".aws-instance.autoscaling-groupName"}}'
            * field_name: "nodegroup_name"
              value: '{{".aws-instance.nodegroup-name"}}'
            * field_name: "ec2_fleet_id"
              value: '{{".aws-instance.ec2-fleet-id"}}'

To learn how to enrich input, see Enrich Input Data.

To learn how to retrieve instance metadata, review this document from AWS.

New Filter Type In the Edge Delta App, you can use the newly createdSplit with Delimiter filter to match, then split a single log into multiple logs.

For example, the abc\n\ndef\nxyz\n log would split into 3 separate logs (abc , def , xyz ), based on the configured delimiter, (newline character ( \n )).

 - name: split_logs_using_specified_delimiter
    type: split-with-delimiter
    delimiter: ","

To learn more, see Split Lines Filters.

Updated Archiving Outputs In the Edge Delta App, archiving outputs have been updated.

Specifically, you can use the new use_native_compression option to compress data, but not metadata.

This option can be useful with big data cloud applications, such as AWS Athena and Google BigQuery.

To use this parameter, you must set the encoding parameter to parquet.

    * name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      s3_force_path_style: true
      encoding: parquet
      compression: zstd
      use_native_compression: true

Updated Cluster Processors In the Edge Delta App, cluster processors have been updated with a new configuration.

Specifically, you can use the newly created include_pattern_info_in_samples parameter to include pattern information in a cluster sample, such as patterns, pattern counts, and sentiment scores.

processors:
  cluster:
    name: clustering
    num_of_clusters: 100
    samples_per_cluster: 20
    reporting_frequency: 30s
    retention: 10m
    cpu_friendly: true
    throttle_limit_per_sec: 200
    include_pattern_info_in_samples: true

To learn more, see Cluster Processors.

New Input type In the Edge Delta App, NATS JetStream is now a supported input type.

This input type allows you to specify a NATS stream subscription for Edge Delta to monitor.

  nats:
    * labels: "my-nats-normal"
      input_mode: "normal"
      consumer_mode: "pull"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject: "example-subject-1"
      timeout: 1m
      ack_wait_duration: 10s
    * labels: "my-nats-distributed"
      input_mode: "distributed"
      consumer_mode: "push"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject_prefix: "example-subject"
      total_agent_count: 5
      total_subject_count: 10
      should_split_lines: true
      timeout: 1m
      disable_acks: true

To learn more, see NATS JetStream Inputs.

Updated Datadog Streaming Output In the Edge Delta App, the Datadog Streaming Output has been updated with buffered-related options.

Specifically, you can use the following, newly created parameters to configure the output’s buffering behavior:

Parameter Description buffer_ttl

Enter a length of time to retry failed streaming data.

After this length of time is reached, the failed streaming data will no longer be tried.

This parameter is optional.

buffer_ttl: 2h

buffer_path

Enter a folder path to temporarily store failed streaming data.

The failed streaming data will be retried until the data reaches its destinations or until the Buffer TTL value is reached.

If you enter a path that does not exist, then the agent will create directories, as needed.

This parameter is optional.

buffer_path: /var/log/edgedelta/pushbuffer/

buffer_max_bytesize

Enter the maximum size of failed streaming data that you want to retry.

If the failed streaming data is larger than this size, then the failed streaming data will not be retried.

This parameter is optional.

buffer_max_bytesize: 100MB