Edge Delta Release Notes

Release Notes for Edge Delta.

March 26, 2024 - Agent v0.1.97

This agent release

  • introduces the OTLP input node to collect raw logs or metrics in the OTEL format,
  • improves the Demo input node by adding additional log types the node can emit,

Note: after upgrading to this version, existing demo nodes need to be removed and replaced with the new demo node.

  • improves validation on the Enrich and Log Transform nodes,
  • adds support for nested Compound Nodes, and
  • improves validation for compound nodes.

Breaking Change This update contains a breaking change: Customers with HTTP Input, TCP Input or UDP Input nodes should not use or update to Agent version v0.1.97.

March 12, 2024 - Agent v0.1.96

This agent release continued to deprecate the custom Edge Delta schema in favor of standardization on the OTEL schema for v3 configurations. Patterns have been enriched with ed.team and ed.source fields. In addition, the default for the TCP Output and FluentD nodes Host name was changed to 127.0.0.1.

This agent release resolved technical debt related to multiple metrics ingestion paths. It contains a breaking change for existing customers with any agents older than v0.1.63. It can be mitigated as follows:

  1. Update all agents to v0.1.95.
  2. Notify Edge Delta to migrate ingestion settings to use the new metric provider.
  3. After 7 days, update to v0.1.96 or above.

February 26, 2024 - Agent v0.1.95

The agent release introduces Compound Nodes - an aggregation of pipeline nodes that you can add to a pipeline as a single object - as a beta feature. In addition, the first_non_empty() CEL macro was updated to handle nested functions within it. It extended the Log to Metrics node to allow CEL fields to be used to define paths. In addition, it improved adoption of the OTEL schema along with other performance and stability improvements.

February 9, 2024 - Agent v0.1.94

In this agent release the Output Transform node was upgraded to support removal of metric attributes to improve pipeline optimization. Among other fixes and optimizations, the Route node user experience has been improved and node self-logs have been optimized.

February 9, 2024 - Agent v0.1.93

This agent release contained a critical bug. Please upgrade to v0.1.94.

January 30, 2024 - Agent v0.1.92

This agent release introduced the beta Prometheus output node to enable the Edge Delta pipeline to send metrics to Prometheus. In addition, the Kubernetes Events input node entered beta. This node enables ingestion of kubernetes events into Edge Delta v3 pipelines.

January 16, 2024 - Agent v0.1.91

This agent release improved back end stability and performance. In addition, it improved the Datadog Mapper node by allowing a custom dd_hostname and it contains CEL improvements such as the introduction of a CEL macro for converting timestamps.

January 3, 2024 - Agent v0.1.90

This agent released added support for case lower/upper control in data streams. In addition, there were updates and bug fixes to the Log Compactor agent.

December 19, 2023 - Agent v0.1.89

This agent release added name validation to integrations. As with node names, integration names containing a period character (.) will not pass validation. In addition it improved memory usage of the Agent’s Aggregator component.

November 30, 2023 - Agent v0.1.88

This agent release improved configuration validation for duration and CEL macro fields. In addition it improved agent health monitoring.

November 20, 2023 - Agent v0.1.87

This agent release improved parsing of JSON attributes to allow for more precise data extraction from JSON logs. It added support for environment variables in the agent configuration. In addition, a new agent manifest includes the new compactor.

November 6, 2023 - Agent v0.1.86

This agent release improved agent metric reporting including agent health among other stability and performance improvements.

November 2, 2023 - Agent v0.1.85

This agent release improved log to pattern node sampling and the Slack integration payload.

October 20, 2023 - Agent v0.1.84

This agent release fixed a log transform upsert bug. It improved pipeline reliability for metrics collection and it reduced error suppression for agent start logs.

October 17, 2023 - Agent v0.1.83

This agent release improved log transform upsert functionality and validation. It also reduced alert noise caused by the Webhook output node.

October 8, 2023 - Agent v0.1.82

This agent release improved collection of health and I/O metrics from visual pipeline nodes.

October 5, 2023 - Agent v0.1.81

This agent release improved performance and error handling of parsing, transformation and mask type nodes. In addition, Visual Pipelines moved out of beta, while the following nodes were introduced as beta:

September 20, 2023 - Agent v0.1.80

This agent release fixed a trace log issue that caused higher than normal trace traffic.

September 18, 2023 - Agent v0.1.79

This agent release defaults to the OTEL schema for Patterns and Metrics when using nodes with the v3 configuration. It also introduces the following nodes:

August 25, 2023 - Agent v0.1.78

This agent release defaults to the OTEL schema for log data types. This aligns data in the pipeline with data discovered using log search. In addition, this release enables Edge Delta agents to detect and capture HTTP2/GRPC connection information using EBPF.

August 9, 2023 - Agent v0.1.77

This agent release reduced memory usage when handling EBPF packets. It introduced the Pipeline IO Stats input node to enable IO stats reporting, and it exposed the item_schema agent setting to enable schema switching.

August 3, 2023 - Agent v0.1.76

This agent release introduced aggregator agent support for the Top-K and Log to Pattern nodes.

July 30, 2023 - Agent v0.1.75

This agent release includes the beta Docker Input Node for Visual Pipelines. It captures log input from Docker containers.

July 26, 2023 - Agent v0.1.74

This agent release improved Prometheus integration behavior in the event of a source change. In addition, the Log Transform Node entered beta.

July 20, 2023 - Agent v0.1.73

This agent release exposes bulk indexer configurations to help debug issues and enable better configuration of elastic clients.

July 17, 2023 - Agent v0.1.72

This agent release improved a number of configuration v3 nodes. The following Visual Pipeline nodes entered beta:

  • TCP Output
  • HTTP Output
  • Microsoft Teams Output
  • Webhook Output
  • Ratio Processor
  • TopK Processor
  • OpenTSDB parser

In addition, parent source fields were added to the metrics payload sent to s3 to enable downstream processing, and the FluentD input node and log-to-pattern processor node reliability was improved.

June 21, 2023 - Agent v0.1.71

This agent release supports a new agent configuration format (v3) in addition to version 2. Agents with the version 3 format can be configured using either the normal YAML text editor or the new Visual Pipelines interface. There are a number of v3 nodes that can be configured using Visual Pipelines, for example,

See the full list of input nodes, processor nodes, and output nodes.

June 16, 2023 - Agent v0.1.70

This agent release improved back end stability and performance.

May 31, 2023 - Agent v0.1.69

This agent release improved back end stability and performance.

May 26, 2023 - Agent v0.1.68

This agent release improved enrichment logic to deal with aliases and reserved keywords. It also optimized backend performance for metrics and archiving and improved helm chart customization.

May 15, 2023 - Agent v0.1.67

This version improved log search and hosted agent performance as well as backend stability.

May 10, 2023 - Agent v0.1.66

This agent release introduced support for a specifying a worker count for Elastic and OpenSearch data destinations to improve peak traffic performance.

May 8, 2023 - Agent v0.1.65

This agent release improved back end stability and performance.

May 1, 2023 - Agent v0.1.64

This agent release improves Elastic integration performance by adding validation for send as is configurations.

April 27, 2023 - Agent v0.1.63

This agent release improved retry performance for S3 archiving and it adds the option to remove root names for Wavefront.

April 21, 2023 - Agent v0.1.62

This agent release improved agent performance with pre-start agent validation. It also removed the PVC dependency from aggregators and it added a more granular drop metric column option.

April 14, 2023 - Agent v0.1.61

This agent release improved the Helm template and added template validation.

April 12, 2023 - Agent v0.1.60

This agent release improved essential metric reporting and optimized S3 usage by reducing the file count.

April 10, 2023 - Agent v0.1.59

This agent release improved handling of agent health data as well as improved Elastic integration. It added support for creating AVRO formatted archive files and it added a distinct count type regex processor.

March 28, 2023 - Agent v0.1.58

This agent release improved handling of agent health data and it implemented a number of performance fixes.

March 16, 2023 - Agent v0.1.57

This agent release improved regex processor and log to metric performance. It also added options for handling internal certificates.

February 15, 2023 - Agent v0.1.56

This agent release improved back end stability and performance.

February 9, 2023 - Agent v0.1.55

This agent release added an option to disable TLS certificate verification for Elastic Streaming destinations. This may be useful for self-signed certificates.

February 1, 2023 - Agent v0.1.54

This agent release improved agent and back end stability and performance.

January 23, 2023 - Agent v0.1.53

This agent release improved stability and performance. In addition, Edge Delta Anomaly Context (EDAC) logs were improved for Amazon S3.

December 23, 2022 - Agent v0.1.52

This agent release made particular improvements to filter error handling and improved hosted agent environment performance among other backend stability and performance improvements.

December 6, 2022 - Agent v0.1.51

This agent release improved regex processors by making it easier to identify dimension group metrics using a custom suffix. In addition to agent and backend stability and performance improvements, script-based filters were updated to handle errors more effectively.

November 30, 2022 - Agent v0.1.50

This agent release improved agent and back end stability and performance.

November 28, 2022 - Agent v0.1.49

This agent release improved the querying experience for customers who use Elastic destinations.

November 24, 2022 - Agent v0.1.48

This agent release added support for enriching logs using K8s controller information. In addition attribute mode was added to the source detection filter to enable field mapping using keys from the source attributes rather than the log body. A number of stability and performance improvements were also deployed.

November 15, 2022 - Agent v0.1.47

This agent release improved agent functionality and performance particularly for Elastic and Datadog. In addition, support was added for Prometheus to scrape metrics with dimensions as attributes.

November 9, 2022 - Agent v0.1.46

This agent release improved stability and performance. In addition, an Enrichment filter that uses a script function was released.

October 11, 2022 - Agent v0.1.45

This agent release improved Prometheus integration. Rule metrics in Prometheus now work end to end:

You can now configure an agent to expose regex processor rule metrics at the metric endpoint for Prometheus scraping. The agent can now handle late-coming source tags for rule metrics. Rule metrics that are exposed from regex processors in the Prometheus format now support aggregator agents. Regex processor reporting now aligns with Prometheus 1 minute scraping intervals. In addition, pipeline performance has been improved with better handling of Elastic destinations not being available.

October 6, 2022 - On Prem UI v0.1.13

This release enabled the Observability - Metrics and the Data Pipeline - Pipeline Status pages for the self-hosted user interface. In addition, users of the self-hosted user interface can now access Observability, Data Pipeline and Management features without any 3rd party identity provider integrations, for example, for internal demo purposes:

October 3, 2022 - Agent v0.1.44

This agent release improved agent and back end stability and performance.

September 27, 2022 - Agent v0.1.43

This agent release added a new filter that can perform JavaScript enrichment and log transformation. In addition, the log files of newly created Kubernetes pods are now scraped from when they are created, including startup logs. The agent can also resume scraping from the previous position in a pod log file if the pod is restarted.

September 19, 2022 - Agent v0.1.42

This agent release added support in the numeric capture regex processor for multiple dimension groups with a range of metric types. In addition, the agent can now be configured to conform with the Amazon Web Services CloudWatch log quota limits.

September 9, 2022 - Agent v0.1.41

This agent release added Transport Layer Security (TLS) configuration support for S3-compatible archive destinations. The config wizard was updated and TLS support was added for AWS session settings.

September 5, 2022 - Agent v0.1.40

This agent release improved the Ratio Processor which is now out of Beta.

August 29, 2022 - Agent v0.1.39

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

On Demand Log Forwarding Log forwarding can now be triggered via an API call. It is used to temporarily forward specific sets of raw data to streaming destinations for a given time period. Both the duration and the log sources can be granularly defined to meet a number of use cases such as forwarding for 30 minutes following a deployment or when an alert triggers. The API call can be automated with integration into CI/CD tooling or third party alerting systems.

For more information, see On Demand Log Forwarding.

Prometheus Integration The Edge Delta agent can now be configured with a service monitor to expose metrics on an endpoint for Prometheus to scrape. The following metrics are exposed:

Count of incoming lines Sum of incoming bytes Count of outgoing lines Sum of outgoing bytes Count of successful outgoing streams Count of failed outgoing streams Open file status To learn more, see Prometheus Integration.

Updated Azure AppInsight Streaming Output The Azure AppInsight streaming output has been updated to allow you to specify where to funnel data. Previously, you could only funnel data to an event index.

Now, you can use the newly published base_type parameter to specify where to funnel data.

To funnel data into a tracing index, enter MessageData.

To funnel data into an event index, enter EventData.

To learn more, see Azure AppInsight.

New Filter Type You can use the newly created Log Transformer Javascript filter to transform specified log messages. Specifically, this filter uses Goja, a type of script in JavaScript. When a log matches the criteria, it will be transformed, and then passed through the filter.

To learn more, see Log Transformer Javascript.

Updated agent_settings For agent configuration, under agent_settings, you can use the newly created agent_stats_enabled parameter to display agent-related information (such as CPU and memory usage) in the Metrics page in the Edge Delta App.

To learn more, see Agent Settings.

June 6, 2022 - Agent v0.1.25

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updates Agent Installation The logging directory for the Edge Delta agent has been updated.

Specifically, if you install the agent via the installation script, then:

For Windows agents, logs can be found under %AppData%/edgedelta.

For all other operating systems, logs can be found under /var/logs/edgedelta.

Updated Input Enrichment In the Edge Delta App, input enrichment has been updated. Specifically, thefield_name parameter under dynamic enrichment can be applied to a template.

      enrichments:
        dynamic:
          field_mappings:
            - field_name: `{{if eq .controllerKind "replicaSet"}}kube_deployment{{else}}kube_{{.controllerKind}}{{end}}`
              value: "{{.controllerName}}"

For more complicated templates that include if / else statements or range statements, you must use bracket as a delimiter.

Updated source_detection Parameter The source_detection parameter for inputs has been updated.

Specifically, source_detection now supports custom as a source_type. When you enter custom , you must configure thefield_mappings parameter with a key-value pair.

    - labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Additionally, source_detection now supports regex as a processing_mode.

    - labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

May 10, 2022 - Agent v0.1.24

This agent release contains backend enhancements and bug fixes.

May 6, 2022 - Agent v0.1.23

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Streaming Output In the Edge Delta App, the Splunk streaming output has been updated to support custom tags via the custom_tags parameter.

You can use this parameter to define key-value pairs that are streamed with every request.


- name: my-splunk
      type: splunk
      endpoint: "://:/"
      token: "32-character GUID token"
      custom_tags:
        "app": "test"
        "region": "us-west-2"
        "File Path": "{{.FileGlobPath}}"
        "K8s PodName": "{{.K8sPodName}}"
        "K8s Namespace": "{{.K8sNamespace}}"
        "K8s ControllerKind": "{{.K8sControllerKind}}"
        "K8s ContainerName": "{{.K8sContainerName}}"
        "K8s ContainerImage": "{{.K8sContainerImage}}"
        "K8s ControllerLogicalName": "{{.K8sControllerLogicalName}}"
        "ECSCluster": "{{.ECSCluster}}"
        "ECSContainerName": "{{.ECSContainerName}}"
        "ECSTaskVersion": "{{.ECSTaskVersion}}"
        "ECSTaskFamily": "{{.ECSTaskFamily}}"
        "DockerContainerName": "{{.DockerContainerName}}"
        "ConfigID": "{{.ConfigID}}"
        "Host": "{{.Host}}"
        "Source": "{{.Source}}"
        "SourceType": "{{.SourceType}}"
        "Tag": "{{.Tag}}"

Updated Agent Settings In the Edge Delta App, the Agent Settings section has been updated with new parameters.


  max_file_per_glob_path: 100
  forget_file_after: 1h
  total_seek_capacity: "5 MB"
  max_seek_size: "4 MB"
  source_discovery_interval: 5s
  file_tailer_buffer_size: 1000
  router_per_source_buffer_size: 1000
  archive_flush_interval: 5m
  archive_max_byte_limit: "16MB"

To learn more, see Agent Settings.

May 2, 2022 - Agent v0.1.22

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Enrichment Options In the Edge Delta App, data enrichment options have been updated to address failed or failing sources.

To troubleshoot potential mapping failures, you can configure the failure_behavior parameter.

Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed.

      enrichments:
        failure_behavior: stop_enrichment
        dynamic:
          field_mappings:
            - field_name: "service"
              value: '{{".labels.service"}}'
            - field_name: "source"
              value: '.annotations.kubernetes.io/{{.container_name}}.logs'
              json_path: "[0].source"
              fallback_value: '{{".short_container_image"}}'

To learn more, see Enrich Input Data.

Updated Source Types In the Edge Delta App, streaming outputs have been updated.

Specifically, for the source_type parameter, you can now enter custom.

Previously, this parameter only supported K8s, Docker, ECS, and File.

When you enter custom, you must add field_mappings parameters to indicate the file source.

- labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Updated File Inputs In the Edge Delta App, the file input type has been updated.

Specifically, you can use the newly created exclude parameter to enter a glob path to exclude matched patterns.

files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"

April 27, 2022 - Agent v0.1.21

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Output In the Edge Delta App, the Splunk streaming output has been updated with the ability to send data in a JSON format.

Specifically, to use this option, you must update the endpoint parameter to point to Splunk’s API services/collector/raw, instead of services/collector/event.

    - name: splunk-integration
      type: splunk
      endpoint: ..../services/collector/raw
      token: ....
      features: log,metric,edac,cluster,alert
      index: rehydration

New Filter Type In the Edge Delta App, you can use the newly created JSON Field Extractor filter to extract a field’s value and replace the whole JSON content with the field’s value.

  - name: extract_severity
    type: extract-json-field
    field_path: "severity"
  - name: extract_first_data
    type: extract-json-field
    field_path: "records.[0].data"

To learn more, see JSON Field Extractor Filters.

Updated Enrichments for AWS ECS Inputs The configurations to enrich input data have been updated.

Specifically, you can use the dynamic parameter to enrich input data from AWS ECS.

In the agent configuration, you can

To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance.


      enrichments:
        dynamic:
          field_mappings:
            - field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            - field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'
            - field_name: "cluster_name"
              value: '{{".aws-instance.cluster-name"}}'
            - field_name: "ec2launchtemplate_id"
              value: '{{".aws-instance.ec2launchtemplate-id"}}'
            - field_name: "ec2launchtemplate_version"
              value: '{{".aws-instance.ec2launchtemplate-version"}}'
            - field_name: "inspector_enabled"
              value: '{{".aws-instance.inspector-enabled"}}'
            - field_name: "cluster_autoscaler_enabled"
              value: '{{".aws-instance.cluster-autoscaler-enabled"}}'
            - field_name: "autoscaling_groupName"
              value: '{{".aws-instance.autoscaling-groupName"}}'
            - field_name: "nodegroup_name"
              value: '{{".aws-instance.nodegroup-name"}}'
            - field_name: "ec2_fleet_id"
              value: '{{".aws-instance.ec2-fleet-id"}}'

To learn how to enrich input, see Enrich Input Data.

To learn how to retrieve instance metadata, review this document from AWS.

New Filter Type In the Edge Delta App, you can use the newly createdSplit with Delimiter filter to match, then split a single log into multiple logs.

For example, the abc\n\ndef\nxyz\n log would split into 3 separate logs (abc , def , xyz ), based on the configured delimiter, (newline character ( \n )).


 - name: split_logs_using_specified_delimiter
    type: split-with-delimiter
    delimiter: ","

To learn more, see Split Lines Filters.

Updated Archiving Outputs In the Edge Delta App, archiving outputs have been updated.

Specifically, you can use the new use_native_compression option to compress data, but not metadata.

This option can be useful with big data cloud applications, such as AWS Athena and Google BigQuery.

To use this parameter, you must set the encoding parameter to parquet.


    - name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      s3_force_path_style: true
      encoding: parquet
      compression: zstd
      use_native_compression: true

Updated Cluster Processors In the Edge Delta App, cluster processors have been updated with a new configuration.

Specifically, you can use the newly created include_pattern_info_in_samples parameter to include pattern information in a cluster sample, such as patterns, pattern counts, and sentiment scores.


processors:
  cluster:
    name: clustering
    num_of_clusters: 100
    samples_per_cluster: 20
    reporting_frequency: 30s
    retention: 10m
    cpu_friendly: true
    throttle_limit_per_sec: 200
    include_pattern_info_in_samples: true

To learn more, see Cluster Processors.

New Input type In the Edge Delta App, NATS JetStream is now a supported input type.

This input type allows you to specify a NATS stream subscription for Edge Delta to monitor.


  nats:
    - labels: "my-nats-normal"
      input_mode: "normal"
      consumer_mode: "pull"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject: "example-subject-1"
      timeout: 1m
      ack_wait_duration: 10s
    - labels: "my-nats-distributed"
      input_mode: "distributed"
      consumer_mode: "push"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject_prefix: "example-subject"
      total_agent_count: 5
      total_subject_count: 10
      should_split_lines: true
      timeout: 1m
      disable_acks: true

To learn more, see NATS JetStream Inputs.

Updated Datadog Streaming Output In the Edge Delta App, the Datadog Streaming Output has been updated with buffered-related options.

Specifically, you can use the following, newly created parameters to configure the output’s buffering behavior:

Parameter Description buffer_ttl

Enter a length of time to retry failed streaming data.

After this length of time is reached, the failed streaming data will no longer be tried.

This parameter is optional.

buffer_ttl: 2h

buffer_path

Enter a folder path to temporarily store failed streaming data.

The failed streaming data will be retried until the data reaches its destinations or until the Buffer TTL value is reached.

If you enter a path that does not exist, then the agent will create directories, as needed.

This parameter is optional.

buffer_path: /var/log/edgedelta/pushbuffer/

buffer_max_bytesize

Enter the maximum size of failed streaming data that you want to retry.

If the failed streaming data is larger than this size, then the failed streaming data will not be retried.

This parameter is optional.

buffer_max_bytesize: 100MB

April 12, 2022 - Agent v0.1.20

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Log Enrichment In the Edge Delta App, log enrichment features have been updated to now support enrichment from Kubernetes annotations.

Specifically, you can use the from_k8s parameter to enrich streaming data with K8s attributes.

You can enter a pod, namespace, or node attributes.

        from_k8s:
          pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.*
          field_mappings:
            - field_name: instance_id
              pod_attribute: pod
              transformers:
                # replace all "source" matches with "target"
                - source: "-"
                  target: "_"
                  type: "replace"
                # remove all "test" words
                - source: "test*"
                  target: ""
                  type: "regex"
            - field_name: namespace
              pod_attribute: namespace
            # fields from labels should have pod_attribute start with "labels."
            - field_name: service
              pod_attribute: labels.service

To learn more, review the Enrich Input Data section in the Inputs document.

Updated Numeric Capture (Regexes) Processors In the Edge Delta App, the Numeric Capture (Regexes) processor has been updated to support multiplication and division for numeric value captures.

Specifically, you can use the newly created value_adjustment_rules parameter to create a rule per capture group.

The rule must follow the “(*|/)” format where:

An asterisk ( * ) represents multiplication A slash ( / ) represents division

    - name: "flog"
      pattern: " (?P\\d+) (?P\\d+)$"
      value_adjustment_rules:
        responsesize:
          operator: "/"
          operand: 1000.0

To learn more, review theNumeric Capture (Regexes) Processor section in the Processors document.

Updated Edge Delta Agent In an effort to improve security during agent installation, makeself –sha256 option has been enabled on agent deployments.

Specifically, makeself performs md5 and crc checks for content integrity.

Starting with version 0.1.20 of the agent, this upgraded security measure update will be included in all agent deployments.

Updated Sumo Logic Output In the Edge Delta App, the Sumo Logic streaming output has been updated.

Specifically, you can use the newly created send_as_json parameter to send data in a JSON format, which allows the fields to be auto-parsed and extracted in Sumo.

    - name: sumo-us-2
      type: sumologic
      endpoint: '{{ Env "EMPTY" "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/XYZ" }}'
      send_as_json: true

Updated Enriched Data for AWS In the Edge Delta App, you can use the dynamic enrichment feature to obtain data from an AWS EC2 instance.

Specifically, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            # if the field value starts with "aws-instance" then instance metadata is get from aws ec2 instance.
            # for more info ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
            - field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            - field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'

To learn more, review the Enrich Input Data section of the Inputs document.

For additional information, please review the Retrieve instance metadata document from Amazon.

Updated Enriched Data for JSON In the Edge Delta App, the from_logs enrichment feature has been with the json_path parameter.

You can use thejson_path parameter to enrich data with fields extracted from JSON logs.

        # from_logs is used to enrich data with fields extracted from logs
        from_logs:
          field_mappings:
            - field_name: component
              # extracting using json_path is also supported
              json_path: fields.[1].component

To learn more, review the Enrich Input Data section of the Inputs document.

Updated Workflows In the Edge Delta App, you can use the newly created enabled_hosts parameter to limit the workflow to specific hosts.

With this parameter, the workflow will only run for specified agent hosts.

  enabled_hosts_workflow:
    description: "runs only specified hosts"
    input_labels:
      - system
      - docker
      - agent
      - infa-processes
    filters:
      - info
    destinations:
      - '{{ Env "TEST_SUMO" }}'
    enabled_hosts:
      - my.host.us1
      - my.host.us2

To learn more, see Workflows.

New Filter / Process Type In the Edge Delta App, you can use the newly created OTLP filter / processor to process OTLP (Open Telemetry) logs.

  - name: opentelemetry_trace_filter
    type: buffered-otlp-trace-processor
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "attributes.result_code"
    failure_value_pattern: "(4|5)xx"
    latency_threshold: 2500.0
    success_sample_rate: 0.1

To learn more, see Filters.

March 21, 2022 - Agent v0.1.19

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - GCP Cloud Monitoring In the Edge Delta App, GCP Cloud Monitoring is now a supported streaming output.

The GCP Cloud Monitoring output will stream custom Google Cloud metrics to a Cloud project.

In the app, you can use the visual editor or YAML file to add GCP Cloud Monitoring to an agent configuration.

GCP Cloud Monitoring was previously known as GCP Stackdriver.

New Flush Mode In the Edge Delta App, you can use the newly created custom_local_per_group flush mode to specify custom groups that should flush together if one of the groups triggers an alert.

In other words, if you set up multiple inputs, and one input triggers an alert, then all (or selected) inputs will flush.

agent_settings:
  tag: sett_test_custom
  log:
    level: debug
  capture_flush_mode: custom_local_per_group
  capture_flush_custom:
    label_grouping:
      group1:
        - file1
        - file2
      group2:
        - file1
        - file3

inputs:
  files:
    - labels: "file1"
      path: "test1.log"
    - labels: "file2"
      path: "test2.log"
    - labels: "file3"
      path: "test3.log"
    - labels: "file4"
      path: "test4.log"

outputs:
  streams:
    - name: sumo
      type: sumologic
      endpoint: "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1ozOeONNQ8LuYTYUj7SaKgr6dt1ueSTOc6mMS2pQz9BM169sb8_UQs5IRaqaRcbpKdI4Tms9S5La9ZFRTL_bf-Ptf_I5ICXcQz2WEQg0fNfA=="
      features: alert

processors:
  regexes:
    - name: "error-regex"
      pattern: "error|ERROR|problem|ERR|Err|POST|hostname|GET"
      interval: 10s
      retention: 1h
      trigger_thresholds:
        upper_limit_per_interval: 3

workflows:
  error-anomaly-workflow:
    input_labels:
      - file1
      - file2
      - file3
      - file4
    processors:
      - error-regex
    destinations:
      - sumo

The following actions will take place:

If an alert is triggered for test1.log, then the file1, file2, and file3 sources will be flushed together. If an alert is triggered for test2.log, then the file1 and file2 sources will be flushed together. If an alert is triggered for test3.log, then the file1 and file3 sources will be flushed together. If an alert is triggered for test4.log, then the file4 source will be flushed. Since file4 was not specified in any group and a fallback_mode was not provided, the agent will use the default fallback_mode local_per_source and only flush file4. When fallback_mode: local_all is added, and alert is triggered for test4.log, then all sources will be flushed. To learn more about filters, see Agent Settings.

New Filter / Processor Type In the Edge Delta App, you can use the newly created buffered-elastic-apm filter / processor to process Elastic APM logs.

  - name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor

To learn more about processors, see Processors.

New Filter Type - base64 decoder In the Edge Delta App, you can use the newly created base64 decoder filter type to decode base64 encoding.

This update helps to support a base64 encoded input on Edge Delta’s hosted collector. In other words, you can attach this filter to a source to display logs that are base64 encoded.

  - name: base64_decoder
    type: base64-decode

To learn more about filters, see Filters.

New Monitor Types You can use the new Pattern Alert and Skyline Alert monitors to trigger an alert for negative patterns.

If an alert is triggered, then the monitor will create a finding.

To learn more, see Patterns.

New feature - Suppress Notifications In the Edge Delta App, you can use the newly created Finding Status option to suppress notifications for a specific finding.

When you suppress a finding, the finding will no longer be displayed in the Insights page. Additionally, any future detection of the finding will not be displayed.

To learn more, see Patterns.

New Filter Type - APM In the Edge Delta App, you can use the newly created APM filter to process Elastic APM logs.

This filter type samples failed and high-latency traces with successful traces and a sampling probability.


  - name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor
    payload_separator: "-----------------"
    enabled_types: "transaction,span,error,metricset"
    optimize_types: "transaction,span"
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "transaction.result"
    failure_value_pattern: "HTTP (4|5)xx"
    transaction_latency_path: "transaction.duration"
    span_latency_path: "span.duration"
    latency_threshold: 35.5
    success_sample_rate: 0.2

To learn more, see Filters.

March 24, 2022 - New Organizations Feature

In the Edge Delta App, you can now create and join different organizations for you and your users.

At a high level, organizations can be considered as different environments, platforms, or sub-accounts that live within your main Edge Delta account.

You can create and belong to multiple organizations, as well as invite users to join various organizations.

Additionally, you can switch between organizations without the need to log off and log back into the app.

To learn more, see Invite Users, Manage Permissions, Access Organizations.

March 21, 2022 - Agent v0.1.18

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - ObserveInc In the Edge Delta App, ObserveInc is now a supported streaming output.

The ObserveInc output will stream analytics and insights to your ObserveInc endpoint.

In the app, you can use the visual editor or YAML file to add ObserveInc to an agent configuration.

New Filter - Custom Attributes In the Edge Delta App, you can use the newCustom Attributes filter to filter for custom attributes.

Specifically, you can use the Attribute Key andAttribute Value parameters to filter for custom attributes.

  # Custom attribute filter do all the log filtering with given key-value of the attribute
  - name: custom_attributes_filter
    type: custom-attributes
    key: service
    value: billing
  - name: negate_custom_attributes_filter
    type: custom-attributes
    key: component
    # A comma separated values to match. If any of them matches the given attribute's value then the log will be pass through
    value: credithandler,debithandler
    # Negate is also supported for attribute filter
    negate: true
  # Filtering custom attributes also support regex matching
  - name: regex_custom_attributes_filter
    type: custom-attributes
    key: level
    pattern: "error|ERROR|problem|ERR|Err"

To learn more, see Filters.

Updated Agent Settings - Log In the Edge Delta App, the Log parameter in the Agent Settings has been updated.

Specifically, you can use theSecure Logging option to hide sensitive data from the specified agent logs, such as API keys, secrets, and authentication information.

To learn more, see Agent Settings.

Updated Integration - Loki In the Edge Delta App, the Loki integration has been updated with a new option.

Specifically, the Send Alert As Loki Log option allows you to send alerts as a log to a Loki endpoint.

New Input Type - Google Pub/Sub In the Edge Delta App, Pub/Sub is now a supported input type.

This input type allows you to specify a Pub/Sub project for Edge Delta to monitor. Specifically, EdgeDelta will consume messages from Pub/Sub subscriptions.

In the app, you can use the visual editor or YAML file to add PubSub to an agent configuration.

To learn more, see Inputs.

Updated Input - File In the Edge Delta App, the File input type has been updated. Specifically, there are 2 new parameters:

Add Ingestion Timestamp

You can use this parameter to ingest a timestamp if the input format is in JSON. Skip Ingestion Timestamp On Failure

You can use this parameter to skip the ingestion of the timestamp when the input is broken or in an invalid format.

files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      # ingest timestamp if input is JSON format.
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true # skip ingestion time when the input is broken or invalid format.

New Filter - Combinations In the Edge Delta App, you can use the newly created combination filter to combine with other, existing filters. Specifically, you can use and or or terms to combine filters to create a more customized filter.

filters:
  - name: combine_two_filters
    type: combination
    operator: or
    filters_list:
      - pattern: "INFO"
      - filter_name: error

To learn more, see Filters.

New Filter Type - Drop Json Fields In the Edge Delta App, you can use the newly created drop-json-fields filter to filter and drop specified JSON fields.

filters:
  - name: drop_some_fields
    type: drop-json-fields
    field_paths: # Each field path is a dot separated path of the field (i.e. "log.source")
      - "level"
      - "details"
      - "log.source"

New Input Type - EDPort In the Edge Delta App, you can use the newly created EDPort Collector Inputs input type to specify a set of ports and protocols for the agent to listen on for incoming traffic.

inputs:
  ed_ports:
    - labels: request
      port: 9000
      protocol: tcp
      read_size: 1
      read_timeout: 30s
      source_detection:
        source_type: "K8s"
        optional: false
        field_mappings:
          k8s_namespace: "kubernetes.namespace"
          k8s_pod_name: "kubernetes.pod.name"
          k8s_container_name: "kubernetes.container.name"
          k8s_container_image: "kubernetes.container.image"
      enrichments:
        from_logs:
          field_mappings:
            - field_name: environment
              json_path: kubernetes.tags.env

To learn more, see Inputs.

February 11, 2022 - Updated Edge Delta App Design

In order to provide a better user experience, the overall look and feel of the Edge Delta App has been updated.

February 1, 2022 - Agent v0.1.16

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Multi-Threshold Support In the Edge Delta App, you can add multi-threshold settings to an agent configuration file.

For example, you can set a threshold to generate an alert when the following conditions are met:

anomaly score is > 90 response_time_ms.avg is > 250 To set this configuration, use the type parameter, specifically set to AND.

  - name: cluster-errors-multi-threshold
    type: and
    interval: 1m
    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

To learn more, see Thresholds.

Graylog Integration In the Edge Delta App, Graylog is now a supported streaming output.

The Graylog output will stream analytics and insights to your Graylog endpoint. In the app, you can use a YAML file to add Graylog to a configuration.

Dynatrace Integration In the Edge Delta App, Dynatrace is now a supported streaming output.

The Dynatrace output will stream analytics and insights to a Dynatrace environment. In the app, you can use the visual editor or YAML file to add Dynatrace to a configuration.

Updates to Datadog Integration The Datadog Integration has been updated with a new parameter called Send Alert As Datadog Log. With this update, you can now send alerts as logs.

Additional Information This agent release contains additional enhancements and bug fixes.

To see the complete list of changes, please visit the Changelog - Agent Releases page.

January 31, 2022 - Notification for Throttled Data

In the Edge Delta App, a notification has been created to let users know when data ingestion is being throttled.

To learn more about these limits, please contact Edge Delta Support.

December 15, 2021 - VictorOps Integration

In the Edge Delta App, VictorOps is now a supported triggering output.

The VictorOps output streams notifications and alerts to a VictorOps endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

VictorOps is also known as Splunk On-Call; however, the app will refer to this output as simply VictorOps.

December 30, 2021 - AppDynamics Integration

In the Edge Delta App, AppDynamics is now a supported streaming output.

The AppDynamics output will stream analytics and insights to an AppDynamics environment. In the app, you can use the visual editor or YAML file to add AppDynamics to a configuration.

December 17, 2021 - OpenMetrics Integration

In the Edge Delta App, OpenMetrics is now a supported streaming output.

The OpenMetrics output will stream analytics and insights to an OpenMetrics endpoint. In the app, you can use a YAML file or visual editor to add OpenMetrics to a configuration.

December 16, 2021 - S3 Integration

In the Edge Delta App, S3 is now a supported streaming output.

The S3 output will stream analytics and insights to an S3 bucket. In the app, you can use a YAML file or visual editor to add S3 to a configuration.

December 15, 2021 - Cribl Integration

In the Edge Delta App, Cribl is now a supported streaming output.

The Cribl output streams analytics and insights to a Cribl endpoint. In the app, you can use the visual editor or YAML file to add Cribl to a configuration.

December 4, 2021 - Moogsoft Integration

In the Edge Delta App, Moogsoft is now a supported triggering output.

The Moogsoft output will stream notifications and alerts to a specified Moogsoft URL. In the app, you can use the visual editor or YAML file to add Moogsoft to a configuration.

December 1, 2021 - Honeycomb Integration

In the Edge Delta App, Honeycomb is now a supported streaming output.

The Honeycomb output will stream analytics and insights to a Honeycomb environment. In the app, you can use the visual editor or YAML file to add Honeycomb to a configuration.

December 1, 2021 - FluentD Integration

In the Edge Delta App, FluentD is now a supported streaming output.

The FluentD output will stream analytics and insights to your FluentD endpoint. In the app, you can use the visual editor or YAML file to add FluentD to a configuration.

December 1, 2021 - AWS CloudWatch Event Logs Input

In the Edge Delta App, Cloudwatch Event Logs is now a supported input.

The Cloudwatch Event Logs input type allows you to specify a set of AWS CloudWatch Log Events for Edge Delta to monitor. With this input, you can monitor multiple regions and log streams.

In the app, you can use the visual editor or YAML file to add Cloudwatch Event Logs to a configuration.

To learn more, see Inputs.

December 1, 2021 - Big Panda Integration

In the Edge Delta App, Big Panda is now a supported triggering output.

The Big Panda output will stream notifications and alerts to a specified BigPanda endpoint. In the app, you can use the visual editor or YAML file to add Big Panda to a configuration.

December 1, 2021 - Loki Integration

In the Edge Delta App, Loki is now a supported streaming output.

The Loki output streams analytics and insights to your Loki endpoint. In the app, you can use the visual editor or YAML file to add Loki to a configuration.

December 1, 2021 - Logz.io Integration

In the Edge Delta App, Logz.io is now a supported streaming output.

The Logz.io output will stream analytics and insights to your Logz.io endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

November 30, 2021 - Child Configurations

In the Edge Delta App, you can add a configuration into another, existing configuration. With this action, the existing configurations will convert into a parent and child configuration.

To learn more, review the Create and Add a Child Configuration section of the Cloud Configuration Backend (CCB) document.

November 30, 2021 - New Compression and Encoding Settings

In the Edge Delta App, you can change compression and encoding settings for Outputs - Archives.

For encoding, Edge Delta now supports Parquet. For compression, Edge Delta now supports zstd and Snappy. Previously, Edge Delta only offered gzip compression and JSON encoding.

To update these settings, you must access the configuration’s YAML file.

archives:
    - name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      # Force archive destination to use {endpoint}/{bucket} format instead of {bucket}.{endpoint}/ when reaching buckets.
      s3_force_path_style: true
      encoding: parquet # supported ones: json, parquet
      compression: zstd # supported ones: gzip, zstd, snappy, uncompressed

November 23, 2021 - EDPort Integration

The EDPort streaming integration has been updated to offer JSON as a schema option for sending data.

November 22, 2021 - Local Storage

In the Edge Delta App, Local Storage is now a supported archiving output.

The Local Storage output will send logs to a file on your local machine.

November 18, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported streaming output.

The Azure Event Hubs output will stream analytics and insights to an Azure Event Hubs endpoint. In the app, you can use a YAML file to add Azure Event Hubs to a configuration.

November 12, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported triggering output.

The Azure Event Hubs output will stream notifications and alerts to a specified Event Hub URL. In the app, you can use the visual editor or YAML file to add Azure Event Hubs to a configuration.

November 9, 2021 - Create a Token

In the Edge Delta App, you can create a token to give your users specific access to the Edge Delta API system.

With tokens, you can specify read / write access for specific backend functionality for your users. In other words, you can create a token to give your users specific read / write access to the Edge Delta API system.