Edge Delta Release Notes

October 9, 2024 - Agent v1.20.0

Updates

• MicroK8s Compatibility: MicroK8s clusters now use Calico CNI by default with the vxlan backend. This update introduces eBPF tracing support for Calico CNI-generated virtual interfaces, enhancing MicroK8s integration.

Bug Fixes

• Agent Shutdown Stability: Investigated and resolved an issue causing the agent to not shut down gracefully due to stuck sample collectors.
• Kubernetes Parsing Correction: Addressed a bug with the Parsing Pattern in the Discovery section for Kubernetes input.

Enhancements

• Expanded Log Output Formats: Enhanced the demo input to support additional output formats, including CEF without a syslog header, CEF with a syslog header, Fortigate Traffic logs, and Fortigate UPM DNS logs.
• Security Validation: Added validation for the HMAC access key and secret.

October 1, 2024 - Agent v1.19.0

Enhancements

• Optimized Kubernetes API Interaction: Redesigned the leader election lease process to minimize interactions with the Kubernetes API.
• Wildcard Field Support: Enhanced the agent log threshold visitor to support wildcard fields.
• Cached Environment Variables: Introduced a cache for the CEL environment function to reduce unnecessary system calls.

New Features

• Map Folding Functions: Added new CEL functions fold_left and fold_right to facilitate in-place map attribute merging.

Bug Fixes

• Diagnostic Upload Timeout: Increased the admin client timeout to resolve issues reported with uploading diagnostics to S3.
• Kubernetes Input Error: Resolved error while editing Kubernetes input node.
• OTLP Input Deadlock: Fixed a deadlock during the stop procedure by ensuring OTLP input channels are properly closed post-stop signal.
• Logging Level Adjustment: Downgraded tracer/connection_monitor logs from ERROR level for non-fatal HTTP2 header decoding mismatches.
• Kafka Source Node Fix: Corrected an issue where Kafka brokers were not correctly passed to the agent.
• Elastic Cloud ID Format Validation: Updated validation messages to correctly reflect expected formats for Elastic Cloud IDs.
• Kubernetes Consistency Adjustments: Adjusted K8s-related inputs to consistently use controller as the service.name field.
• System Stats CPU Metrics on MacOS: Fixed CPU metric collection for MacOS by ensuring compatibility for non-CGO environments.

September 23, 2024 - Agent v1.18.0

Enhancements

• New CEL Macros: Introduced two additional CEL macros (iterate and reduce) enhancing the agent’s capabilities.
• HTTP Output Batching: Added batching support for HTTP output to improve data transmission efficiency.
• Custom File Discovery under Kubernetes Input: Introduced support for custom file discovery paths with Kubernetes input, allowing users to collect data from specific glob paths and filter undesired files. Additionally, added regex capabilities to capture pod UID and container names.
• Demo Log: Added ArgoCD and Istio Access log types to the Demo node, along with their respective Grok patterns, facilitating easier log analysis in these environments.

Bug Fixes

• New Relic Output: Resolved an issue where the whole resource and attributes map were being removed when pushing to New Relic. Now, attributes and resource fields are appropriately included.

Deprecated Features

• Compactor Incoming Stats: Removed dependency on incoming stats for compactor metrics in favor of heartbeat-based representation.

September 9, 2024 - Agent v1.17.0

Enhancements

• Enhanced Kubernetes Input Source Detection: The source detection mechanism for Kubernetes inputs has been improved. Previously, it utilized the first parent of a pod, such as a ReplicaSet, to create the k8s.pod.name filter. The update now ensures that the root controller is used for creating the k8s.pod.name filter, providing a more stable and reliable source reference. Additionally, the unused attribute ed.k8s.pods has been removed to streamline the system.

• Debug Output Stability: A bug was fixed in the Debug output that caused a crashloop issue due to undefined streamer and integration names. The fix involved ensuring that both names are appropriately set, thus stabilizing the Debug output functionality and preventing further crashes.

• Refinement in Metrics Collection: The default logic for collecting pod labels and annotations for cAdvisor and Kube-state metrics has been overhauled. The old system, which automatically gathered these by default, has been replaced with a new resource field selection logic. This new logic mirrors the approach used for the Kubernetes Input node, ensuring a more targeted and efficient collection of metrics which improves performance and accuracy.

Fixes

• Google Cloud Logging API Limitations: An issue with the Google Cloud Logging API’s limit of 256KB per log entry has been addressed. Log entries will now be split correctly to adhere to this size limit while maintaining their labels. Furthermore, a new label called edgedelta_GCL_split_id has been introduced to store the UUID of split messages. This ensures proper handling of logs without exceeding API restrictions, which include no more than 64 labels, label keys being at most 512 bytes, and label values being at most 64KB.

September 2, 2024 - Agent v1.16.0

Enhancements

• Conversion to OTEL Definition: Attributes’ data types have been converted to more closely follow the OpenTelemetry (OTEL) specifications. This ensures better compliance and integration with the OTEL ecosystem.
• CEL Evaluations Optimization: Enhanced ordered evaluations in the Common Expression Language (CEL) to optimize performance, particularly for evaluating multiple data types sequentially (e.g., string, float, integer, map).
• Pod Listener Caching: Introduced a caching mechanism for non-existent pods in the pod listener, leading to a reduction in redundant Kubernetes calls for ephemeral containers.
• CEL Version Update: Updated to the latest CEL version and introduced benchmarks for evaluating CEL performance. Added external libraries from the CEL official repository to support this enhancement.
• Support for Granular Stream Stats: Updated ED Log Output, Loki, Google Cloud Logging (GCL), and New Relic to support granular stream statistics in the OpenTelemetry logs format.

New Features

• Updated Inputs and Services: Enhanced service name collection across multiple inputs including Kafka, S3 SQS, Docker, file input, demo, Exec, OTLP, ports, Kubernetes, Edge Delta-generated inputs, and container stats inputs.
• Null Value Option: Added an option (ignore_if_null: true) to transforms, allowing the system to ignore null values during transformations, thus preventing them from being added to the final output.

Bug Fixes

• Push Processor Error Handling: Improved handling of errors in the push processor during the agent shutdown procedure to ensure smoother termination.
• Deprecated Data Type Dependency in OTLP Input: Removed dependency on the data_type in the OTLP input configuration, streamlining data handling and reducing configuration complexity.

August 26, 2024 - Agent v1.15.0

New Features

• S3 Input Compression Selection Support: Compression options for the S3 Input node have been expanded. You can now select from “gzip”, “zstd”, “snappy”, and “uncompressed” compression types. This update allows for better support of AWS logs such as ALB and CloudTrail.
• HTTP Protocol Support for OTLP Input Nodes: The OTLP Input node now supports HTTP in addition to gRPC, providing more flexibility in how you can send OTLP data to the Edge Delta agent.

Security Fixes

• Environment Variable Masking: Environment variables loaded by the agent are now masked before being written to logs, ensuring sensitive information is protected.
• gRPC Metadata Security Fix: Addressed a potential PII concern where gRPC metadata, which may include private information, was being logged. The gRPC version has been updated to address this.

Improvements

• Granular Stream Stats for Port-based Outputs: Enhanced metrics for outputs using ports now include more granular stream stats, providing detailed insights into data streams.

Bug Fixes

• JSON Unroll Casting and Validation Fix: Fixed an unexpected behavior in JSON Unroll casting. Also corrected JSON Unroll config validation.
• ED_ENV_VARS Functionality Fix: Resolved an issue where the ED_ENV_VARS functionality was not working for users trying to pass their variables to the Agent on Linux and MacOS.
• Kubernetes Short-lived Container Status Fix: Fixed an out-of-bounds error in Source Detection.
• Improved Grok Processor Handling of Redis Logs: Enhanced the Grok Parsing node to better handle Redis logs, improving log parsing accuracy.
• Support for ed_logs Node Type: The agent now accepts ed_logs as a valid node type.

August 20, 2024 - Agent v1.14.0

Critical Fix

• Health Data Upload Fix: Resolved an issue with health data uploads causing throttling. Now, health and diagnostic data are buffered to ensure they are uploaded as a single file, significantly reducing upload frequency. All agents running version 1.13.0 must upgrade to 1.14.0.

New Features

• Granular Stream Stats: Added support for granular stream statistics in S3 and Azure Blob Storage, with similar functionality extended to the ED Archive when metadata is enabled.
• CEL Function: Introduced the to_json CEL macro.
• Health Data Debugging: Allowed ingestion of health data by the Debug output node.
• Datadog and Splunk Mapper Updates: Allowed ingestion of metrics by Datadog and Splunk mappers.
• Cluster-Pattern Item Manipulation: Enabled Datadog and Splunk mappers as well as Output Transform nodes to ingest the cluster-pattern data type.

Enhancements

• Agent handling of large data items: To improve agent performance, the agent will split any incoming message larger than 1Mb into individual messages. In addition, the Edge Delta archive will not ingest telemetry messages larger than 2Mb.
• Improved Kubernetes CEL Function: Added the GetPod function to improve use of the from_k8s CEL macro. Introduced a Kubernetes API fetch step if the pod is not found in the cache.
• OTEL Log Ingestion: Made OpenTelemetry (OTEL) log ingestion the default path and removed the old ingestion path in the v3 codebase.
• Rename ed_archive_output to ed_logs_output: The node type for ed_archive_output will be changed to ed_logs_output. To ensure backward compatibility, both of these node types are supported.
• Data Type Validation in OTLP Input: Introduced stricter string data type validation for OTLP input and changed the input field to a dropdown menu instead of a freeform text field.
• Display Name Consistency: Updated the display name for the unescape JSON node to JSON Unescape.
• The following advanced firewall rules are no longer required:
  • ed-agent-log.s3.us-west-2.amazonaws.com
  • ed-overflow-agent-log.s3.us-west-2.amazonaws.com
  • agent-pprof.s3.us-west-2.amazonaws.com
• Kubernetes Pod Topology Spread Constraints: Introduced pod topology spread constraints to our Helm chart. This feature helps control how Pods are spread across your cluster among failure domains such as regions, zones, nodes, and other user-defined topology domains, improving operability with KaaS and overall K8s scheduling.
• Cache Health Observability: Added health data to the pod listener component, allowing better observation of cache contents over time.
• OTLP Traces: Added support for OTLP traces to the OTLP input node. This enhancement, data_type: trace, improves tracing capabilities.

Fixes

• Docker Library Update: Updated the Docker library from version 24.0.9 to 26.1.5 to address critical CVEs including CVE-2024-41110.
• Remove Config Content ID: Removed all references to the now-unsupported config content ID.
• Stream Stats Calculation: Fixed potential divide-by-zero panics by adding length checks before performing average calculations on metadata.
• Resource Flexibility: Made source attributes more flexible by removing mappings that prevented certain labels from propagating downstream when added by users.
• Large Stack Trace Handling: Increased minimum seek size to handle large stack traces more effectively.
• Nested Compound Nodes: Resolved issues with compound nodes having the same name as their parent compound node, ensuring correct pipeline imports.
• Node Creation for Rollup: Reduced memory consumption on rollup agents by limiting the creation of unnecessary components.
• Sample Collection Time: Increased the default sample collection time from 1 minute to 15 minutes to ensure coverage for lower volume sources.
• Compactor Service DNS Resolver: Fixed an issue where the compactor service’s DNS resolver watched for changes in all services in a K8s cluster. The DNS resolver now only monitors the compactor service, reducing unnecessary load. Also fixed the deregistration of the pod listener from the health manager.
• K8s Metrics Collection: Corrected an issue where some metrics collectors did not check if metric items were nil, causing errors during K8s metrics collection.
• Health Endpoints in HTTP(S) Input: Removed constraints on health endpoints.
• Transform Node Updates: Fixed on-screen wording and updated examples for transform nodes.
• Ingest Health Data Type: Allowed health data type ingestion by debug output.

Stability and Performance Improvements:

• Several stability and performance improvements have been made, including Zstd encoder thread safety. Additionally, error logging for output nodes has been added, dependencies on deprecated ingest configuration fields removed, transformation nodes set to use a no-op poder, and a feedback channel added for the health manager to ensure proper stop procedures.

Maintenance

• Pod Listener Testing: Updated the pod listener to function as a no-op during node testing, ensuring it does not interfere with test scenarios.

August 12, 2024 - Agent v1.13.0

Note: This release contains a critical bug. All agents running version 1.13.0 must upgrade to v1.14.0.

July 30, 2024 - Agent v1.12.0

This release includes several significant updates, enhancements, and bug fixes aimed at improving the functionality and performance of Edge Delta agents. Below are the details of the changes included in this release:

Enhancements

• Splunk Output Update: Splunk output node can now ingest signal type messages.
• Add k8s.container.name to Pod Info Extraction: Added the ability to extract k8s.container.name using CEL from the k8s resource.
• Add Health Check for OTLP Input: Introduced a health check service for OTLP input nodes, necessary for new cloud fleets as load balancers perform health checks at certain intervals.
• Support for service.name in *nix Environments: Added support for the OTEL standard field resource.service.name for sources detected in *nix environments.
• Authentication and Path Filtering for HTTP(S) Input: Added authentication (Bearer and Basic) as well as path filtering options to the HTTP input node.
• The ED_CUSTOM_TAGS environment variable can now be used for attaching attributes statically. The format should be in form: <key 1>:<value 1>|<key 2>:<value 2>
• Unroll JSON from a Field Path: Added functionality to process JSON data from non-body locations using json_field_path.
• The Grok Parsing node has been updated with patterns for AWS VPC Flow as well as MYSQL Slow Query and Error logs.

Bug Fixes

• Leader Election in Non-Processor Agents: Rollups, aggregators, and compactor agents will no longer participate in leader elections. Only processor agents will now have leader election.
• Resource Transform Fields Update: Aligned the Resource Transform node configuration to match log transform nodes, replacing source_field_overrides with transformations and ensuring backward compatibility for older configurations.
• Missing Validation in Resource Transform Node: Added missing validation checks after the deprecation of source field overrides.
• Improve Error Message for k8s Event Tailer: Improved the error message to help diagnose potential issues with Kubernetes event tailers in customer environments.
• Track Errors in Gzip Decompressor: Errors from gzip decompression will now be tracked and logged, instead of being propagated upstream, to reduce noise.
• Error Counting in Gzip Decompressor: Errors are now counted with an errorCount counter and logged at most five times per minute.
• Fix for Splunk Mapping: Ensured messages appear the same during migration from config v2 to config v3 by making Log-transform and Splunk-mapper compatible.
• Prevent Push Strategy Creation Errors from Stopping Agent Execution: Added a No-Op strategy to handle push strategy creation failures, allowing the agent to continue running when some output nodes fail.
• Use Bool Pointer for Log to Metric parameters: Changed skip_empty_intervals and only_report_nonzeros to booleans to differentiate between false values and unset variables.
• Kubernetes Input Labels & Annotations: Switched to opt-in collection for pod labels and annotations, added support for node and namespace labels.

Miscellaneous

• Batching Bug in Compactor: Fixed issues with batching in compactors that were causing unnecessary overhead and inefficiencies.
• Group Anomaly Settings Using Log to Metric Top Anomaly Setting: Improved the handling of anomaly settings where group anomaly settings are missing.

These improvements and bug fixes enhance the robustness, usability, and functionality of the Edge Delta agent.

July 16, 2024 - Agent v1.11.0

This update provides robust enhancements and fixes for better performance, reliability, and functionality across different environments and workflows.

Enhancements

• A new Knowledge Library has been released, providing solutions for common regex, CEL, and GROK patterns.
• Fleets on the Pipelines - Dashboard page can be sorted by telemetry data such as number of deployed agents.
• Support for OTEL Log Format: Improved support for sending logs in OTEL format to the backend, ensuring backwards compatibility and gradual transition while running older and newer schemas concurrently.
• Helm Chart Improvements: Fixed missing annotations for push services in the Helm chart to avoid issues when setting up a load balancer.
• Kubernetes Events Handling: Updated K8s events tailer to include modified events, ensuring all events are ingested even when a Back-off event is patched.
• Log to Pattern Node Enhancement: Added capability to pick a field in the agent data item for clustering instead of the default body field.
• Log Forwarding via OTEL: Improved the flushing logic and interval management for OTEL logs, and enabled log forwarding to pusher when archive ingestion is active.
• Field Name for File Input: Updated source detection to add ed.source.name and replace invalid characters with an underscore.

Bug Fixes

• Debug Output with Linux: Addressed a nil pointer exception in the debug output with Linux pipelines.
• CPU Profile Capture on Windows: Removed the attempt to capture CPU profile on Windows to prevent error messages.
• Metrics Reporting Fix: Corrected an error with log to metric processor to avoid stopping item processing due to variable shadowing.
• Update Archive Payload Schema: Updated schema to align with OTEL schema and fixed a deadlock issue during the stop procedure.
• Fixed host.ip value for agents behind NAT: Changed the ingestion method to use the default gateway interface’s IP instead of backend-imposed IPs, ensuring the correct internal IP address is reported.

Miscellaneous

• HTTP Input Enhancements: Support for all routes and multiple methods (POST, PUT, PATCH, DELETE) for HTTP input, with attributes added for enhanced flexibility.
• Leader Election Improvements: Removed the ED_LEADER_ELECTION_ENABLED environment variable requirement for leader election and made it enabled by default.
• K8s Role Permissions: Added get secret permissions for Kubernetes roles required for specific cleanup operations.
• Log Threshold Monitors: Fixed the issue with log threshold monitors not ingesting empty strings.
• Rehydration Fixes: Corrected issues with rehydration not pulling Avro/Zstd files.
• Grok Node Fixes: Combined pattern and custom pattern fields to prevent validation failures and improve reliability.
• Data from Debug Output nodes is no longer available on the node itself. It is only available on the full Debug Output page.
• RBAC group names are now case insensitive.
• Default k8s_input exclusions have now been trimmed to automatically ingest all K8s control and management plane telemetry data.

July 1, 2024 - Agent v1.10.0

This update introduces several new nodes, enhancements to existing nodes, and bug fixes.

New Features

• Google Cloud Logging Output node is introduced in Beta status.
• S3 input and JSON Unroll is introduced in Beta release.
• Horizontal Pod Autoscaler (HPA) is introduced.

Enhancements

• Resource Transform nodes have relaxed restrictions when applied to fields.
• Debug Output node is now available in all environments.
• Added support for ingress in Helm. More about in Edge Delta Helm.

Bug Fixes

• Various bug fixes but none notable enough to mention.

Miscellaneous

• Agent v1.10.0 is released thus fully embracing SemVer.
• Various performance improvements.

June 17, 2024 - Agent v0.1.105

This update introduces several new features, enhancements, and bug fixes to improve the functionality, integration, and overall performance of the Edge Delta Agent.

New Features

• CloudTrail Format: Added CloudTrail format to Demo Input for better support of CloudTrail use cases.
• Output Transform nodes can now send transformed logs to HTTP output nodes in addition to Datadog, Splunk, Sumo, and Elastic.
• Custom HTTP Headers: Introduced custom HTTP header support for HTTP output. Users can now create any header using key/value pairs for the HTTP output node. This currently supports hardcoded strings. For example, headers like X-Token: abc123456 can now be configured.

Enhancements

• Streamer Component: Corrected success counting in to ensure it only counts when items are created for pushing.
• Error Handling: Unified error counting and logging intervals across processors, ensuring all counters stop correctly at stopping time.
• Helm Chart: Introduced support for defining separate node selectors, tolerations, and priority class names for rollup and compactor agents.
• Kubernetes Integration: Improved K8s file tailer to create a complete set of OTEL semantic conventions attributes.

Bug Fixes

• Helm Upgrade Secrets: Fixed an issue where Helm upgrade didn’t create a new secret with a valid secretApiKey value, ensuring proper handling of secrets.
• Windows Agent Logging Location: Changed the log destination for the Windows agent to C:\Program Files\EdgeDelta\edgedelta.log to align with standard application log locations.
• HTTP output Ordering: Addressed an ordering issue for node specifications in HTTP output to prevent headers from being hidden.

Miscellaneous

• Archiver Code: Deprecated and removed archiver agent.

June 5, 2024 - Agent v0.1.104

This release brings numerous improvements in sample collection, integration capabilities, logging detail, and bug fixes, enhancing the overall performance and reliability of the Edge Delta Agent.

New Features

• GROK Parser Node: Added a new parser node used for parsing log data into attributes using common log patterns.
• Google Cloud Logging Node: Introduced a new output node for sending data to Google Cloud Logging API.

Enhancements

• Custom HTTP Headers: Enabled the creation of custom HTTP headers with key/value pairs for HTTP output nodes. Currently supports hardcoded strings.
• Splunk Integration: Removed legacy support for features from v3 Splunk output.

Bug Fixes

• Agent Exit Code: Changed the agent to explicitly set exit code to 1 on graceful shutdown due to errors, ensuring proper restart by controllers like systemd, Kubernetes, etc.
• Container Stats Memory Issue: Fixed a memory leak issue caused by the ED Container Stats input node in Kubernetes environments.
• Log Metadata: Enhanced agent self logs to include package, file, and line number information for better debugging.
• Compactor Buffer Flush Fix: Resolved an issue with compactor prematurely flushing the internal archive buffer before the S3 pusher could send the data. Disabled compaction except for Avro encoding and uncompressed third-party archive outputs.
• Late Arrival Handling: Removed non-configurable late arrival handling from several input nodes. Late arrival is handled by the line-separator.
• Kubernetes Resources: Fix issue of missing resource values for Kubernetes agents.

Miscellaneous

• Converted datatype-related functions to variables to optimize memory utilization.

Agent v0.1.103

Version 0.1.103 was not released. Please use v0.1.104.

May 20, 2024 - Agent v0.1.102

The following changes been implemented in this agent release:

Product

• Changed the release status of compound nodes from Beta to Released, reflecting their stable release state.

Enhancements

• Loki and NewRelic Output Nodes: Added support for custom items to be processed, enhancing the data modification capabilities before sending to these destinations. Enhanced Loki integration to allow custom labeling, giving users more control over their data attributes.
• Self-Logs Metadata: Added extra metadata (package and file plus line number) to self-logs to ease troubleshooting and investigations.

Bugfixes

• Fixed the extraction of timestamps from messages, ensuring timely logging and improved search accuracy.
• Route Path Processing: Fixed the issue where the “Exit if Matched” field radio button wasn’t correctly set.
• Updated the Event Schema to include event.type and event.domain fields at the top level in the AVRO schema to ensure complete event representation.
• Agent panic when testing: Testing uses the proper schema abstraction while accessing data, preventing agent panics.
• Removed legacy Prometheus Metrics settings such as rule_metrics_prom_stats_enabled, internal_prom_stats_enabled, and enable_reporting_in_prometheus to eliminate conflicts with the Prometheus metrics exporter output node.
• Fixed an issue with the UDP input node handling read timeouts, ensuring it gracefully restarts listening.
• Fixed handling of number values when applying math functions.

Security

• Updated dependencies to mitigate a high severity vulnerability CVE-2022-29583 [OWASP Top Ten 2017 Category A9 - Using Components with Known Vulnerabilities].
• Fixed medium severity vulnerabilities including:
  • CVE-2023-45288 [Uncontrolled Resource Consumption]
  • CVE-2024-24786 [Loop with Unreachable Exit Condition]
  • CVE-2024-24557 [Insufficient Verification of Data Authenticity]
  • CVE-2024-29018 [Incorrect Resource Transfer Between Spheres]

Upgrade Instructions

Note: this version removes the default use of Persistent Volume Claims (PVC) for the Compactor in Kubernetes installations. Therefore, to upgrade to this version a clean install is required. To enable PVC, use the compactorProps.usePVC=true helm value. For kubectl installations, update the manifest with these details.

May 6, 2024 - Agent v0.1.101

The following changes been implemented in this agent release:

New Features

• The Mask node now includes a number of regex patterns out of the box for masking important fields such as email, bitcoin address, different types of credit card numbers etc.
• Metrics Generation for Log Threshold Monitors: Agents now generate metrics required for log threshold monitors. A hidden Log-to-Metric node fetches monitors from the backend and associates metrics with them. This node is also connected to the ed_metrics_output node type.

Improvements

• Health Signal Enhancements in Debug Output: The health node output can now be viewed in the debug output node, allowing users to visualize the health signals it produces. The base strategy for health status now registers a ’not ok’ status if there are more than 10 issues within a 10-minute window.

Bug Fixes

• Elastic Output Warning Suppression: An issue where the lastErr variable in the Elastic output node wasn’t being set properly has been addressed, ensuring that warnings are now logged if entries encounter issues during processing.
• Enhanced Isolation in Pipeline Testing: Changes have been made to pipeline testing to prevent the activation of components. During testing, Output nodes are substituted with Void nodes, ensuring that no real output operations are executed. This approach solidifies the testing environment’s isolation, ensuring that it remains unaffected by any output-side effects.
• Source Detector and Enrichment Tagging Fix: A bug concerning the incorrect tagging of log file names in source detector and enrichment in config v2 has been rectified, improving tagging accuracy in specific use cases.
• Corrected Outgoing Bytes and Lines Reporting: A previous bug that prevented successful reporting of outgoing bytes and lines, even when there were successes, has been fixed. This enhances the accuracy of transmission analytics.
• Panic Prevention During OTEL Patterns Emission: Adjustments have been made to check whether the compactor is running before proceeding with any data upload. This change prevents the Agent from panicking and attempting to write to a closed channel when stopping, thereby ensuring a smoother shutdown process.
• Concurrent Map Read/Write Issue Resolved: An agent panic caused by concurrent map read/write operations, especially relevant during output node and dependencies construction, has been addressed. This fix includes the enforcement of correct logic to determine if a link should copy the object considering both Incoming and Outgoing data configurations.
• Pusher Logging and Health Tracking: Improvements in Pusher logging have been applied, allowing more detailed error logging and better health status accuracy.

April 22, 2024 - Agent v0.1.100

In this agent version, the following improvements were released:

New Features

• The Kafka input and Kafka output nodes were released to beta.
• The Loki output node was released to beta.
• The Debug Output node was released to beta.

Improvements

• The Demo input node validation was improved.
• The kubernetes manifest now includes CPU limits for the agent to align with best practice and prevent scheduling issues
• Node testing traffic to the API was optimized.
• The Parse Json Attributes node’s handling of numbers was improved.

April 8, 2024 - Agent v0.1.99

In this agent version, the following nodes were released in beta:

• Resource Transform to transform the resource parameters.
• New Relic Output to send data to your New Relic destination.

The following improvements were released:

• The TCP Output node and the Local Storage Output node can now accept additional data types: custom, metric, cluster_pattern_and_sample, and signal.
• Improved health data reporting of last_error to improve API stability.
• The Compactor logging level was changed from debug to trace to improve troubleshooting and diagnostics.
• Various improvements are made to the Edge Delta Helm chart improving efficiency and improving use with Continuous Deployment.

The following fixes were released:

• The Demo input node now emits logs that use UTC time.
• Mask node testing where a blank Golang Regex pattern was set no longer results in a null pointer exception that causes an agent panic.
• The Log to Metric processor no longer reports unprocessed logs as errors
• The compactor component gracefully stops when shutting down

The following features were deprecated:

• The Enrichment node is now deprecated. Please use the Log Transform, Output Transform or Mask nodes.

March 28, 2024 - Agent v0.1.98

This agent release

• Fixes missing resources in HTTP Input, TCP Input and UDP Input nodes.
• Fixes a demo node input issue regarding timestamps.
• Updates software dependencies for the Edge Delta agent to versions that address recent CVE-listed security vulnerabilities.

In addition, the following improvements are released:

• Improved health and EDAC ingestion.
• Improved counting of hits and errors in processors.
• Improved agent performance monitoring.
• Improved Enrich and Source Manipulate nodes’ forms.

March 26, 2024 - Agent v0.1.97

This agent release

• introduces the OTLP input node to collect raw logs or metrics in the OTEL format,
• improves the Demo input node by adding additional log types the node can emit,

Note: after upgrading to this version, existing demo nodes need to be removed and replaced with the new demo node.

• improves validation on the Enrich and Log Transform nodes,
• adds support for nested Compound Nodes, and
• improves validation for compound nodes.

Breaking Change

Customers with HTTP Input, TCP Input or UDP Input nodes should not use or update to Agent version v0.1.97.

March 12, 2024 - Agent v0.1.96

This agent release continued to deprecate the custom Edge Delta schema in favor of standardization on the OTEL schema for v3 configurations. Patterns have been enriched with ed.team and ed.source fields. In addition, the default for the TCP Output and FluentD nodes Host name was changed to 127.0.0.1.

This agent release resolved technical debt related to multiple metrics ingestion paths. It contains a breaking change for existing customers with any agents older than v0.1.63. It can be mitigated as follows:

1. Update all agents to v0.1.95.
2. Notify Edge Delta to migrate ingestion settings to use the new metric provider.
3. After 7 days, update to v0.1.96 or above.

February 26, 2024 - Agent v0.1.95

The agent release introduces Compound Nodes - an aggregation of pipeline nodes that you can add to a pipeline as a single object - as a beta feature. In addition, the first_non_empty() CEL macro was updated to handle nested functions within it. It extended the Log to Metrics node to allow CEL fields to be used to define paths. In addition, it improved adoption of the OTEL schema along with other performance and stability improvements.

February 9, 2024 - Agent v0.1.94

In this agent release the Output Transform node was upgraded to support removal of metric attributes to improve pipeline optimization. Among other fixes and optimizations, the Route node user experience has been improved and node self-logs have been optimized.

February 9, 2024 - Agent v0.1.93

This agent release contained a critical bug. Please upgrade to v0.1.94.

January 30, 2024 - Agent v0.1.92

This agent release introduced the beta Prometheus output node to enable the Edge Delta pipeline to send metrics to Prometheus. In addition, the Kubernetes Events input node entered beta. This node enables ingestion of kubernetes events into Edge Delta v3 pipelines.

January 16, 2024 - Agent v0.1.91

This agent release improved back end stability and performance. In addition, it improved the Datadog Mapper node by allowing a custom dd_hostname and it contains CEL improvements such as the introduction of a CEL macro for converting timestamps.

January 3, 2024 - Agent v0.1.90

This agent released added support for case lower/upper control in data streams. In addition, there were updates and bug fixes to the Log Compactor Agent.

December 19, 2023 - Agent v0.1.89

This agent release added name validation to integrations. As with node names, integration names containing a period character (.) will not pass validation. In addition it improved memory usage of the Agent’s Aggregator component.

November 30, 2023 - Agent v0.1.88

This agent release improved configuration validation for duration and CEL macro fields. In addition it improved agent health monitoring.

November 20, 2023 - Agent v0.1.87

This agent release improved parsing of JSON attributes to allow for more precise data extraction from JSON logs. It added support for environment variables in the agent configuration. In addition, a new agent manifest includes the new compactor.

November 6, 2023 - Agent v0.1.86

This agent release improved agent metric reporting including agent health among other stability and performance improvements.

November 2, 2023 - Agent v0.1.85

This agent release improved log to pattern node sampling and the Slack integration payload.

October 20, 2023 - Agent v0.1.84

This agent release fixed a log transform upsert bug. It improved pipeline reliability for metrics collection and it reduced error suppression for agent start logs.

October 17, 2023 - Agent v0.1.83

This agent release improved log transform upsert functionality and validation. It also reduced alert noise caused by the Webhook output node.

October 8, 2023 - Agent v0.1.82

This agent release improved collection of health and I/O metrics from visual pipeline nodes.

October 5, 2023 - Agent v0.1.81

This agent release improved performance and error handling of parsing, transformation and mask type nodes. In addition, Visual Pipelines moved out of beta, while the following nodes were introduced as beta:

• TCP Output
• HTTP Output
• FluentD Output

September 20, 2023 - Agent v0.1.80

This agent release fixed a trace log issue that caused higher than normal trace traffic.

September 18, 2023 - Agent v0.1.79

This agent release defaults to the OTEL schema for Patterns and Metrics when using nodes with the v3 configuration. It also introduces the following nodes:

• JSON Parse
• Log Transform
• Output Transform
• Splunk Mapper
• Datadog Mapper

August 25, 2023 - Agent v0.1.78

This agent release defaults to the OTEL schema for log data types. This aligns data in the pipeline with data discovered using log search. In addition, this release enables Edge Delta agents to detect and capture HTTP2/GRPC connection information using EBPF.

August 9, 2023 - Agent v0.1.77

This agent release reduced memory usage when handling EBPF packets. It introduced the Pipeline IO Stats input node to enable IO stats reporting, and it exposed the item_schema agent setting to enable schema switching.

August 3, 2023 - Agent v0.1.76

This agent release introduced aggregator agent support for the Top-K and Log to Pattern nodes.

July 30, 2023 - Agent v0.1.75

This agent release includes the beta Docker Input Node for Visual Pipelines. It captures log input from Docker containers.

July 26, 2023 - Agent v0.1.74

This agent release improved Prometheus integration behavior in the event of a source change. In addition, the Log Transform Node entered beta.

July 20, 2023 - Agent v0.1.73

This agent release exposes bulk indexer configurations to help debug issues and enable better configuration of elastic clients.

July 17, 2023 - Agent v0.1.72

This agent release improved a number of configuration v3 nodes. The following Visual Pipeline nodes entered beta:

• TCP Output
• HTTP Output
• Microsoft Teams Output
• Webhook Output
• Ratio Processor
• TopK Processor
• OpenTSDB parser

In addition, parent source fields were added to the metrics payload sent to s3 to enable downstream processing, and the FluentD input node and log-to-pattern processor node reliability was improved.

June 21, 2023 - Agent v0.1.71

This agent release supports a new agent configuration format (v3) in addition to version 2. Agents with the version 3 format can be configured using either the normal YAML text editor or the new Visual Pipelines interface. There are a number of v3 nodes that can be configured using Visual Pipelines, for example,

• Kubernetes Input
• File Input
• Log to Pattern
• Log to Metric
• Regex Filter
• Mask
• Datadog Output
• Splunk Output
• S3 Output

See the full list of input nodes, processor nodes, and output nodes.

June 16, 2023 - Agent v0.1.70

This agent release improved back end stability and performance.

May 31, 2023 - Agent v0.1.69

This agent release improved back end stability and performance.

May 26, 2023 - Agent v0.1.68

This agent release improved enrichment logic to deal with aliases and reserved keywords. It also optimized backend performance for metrics and archiving and improved helm chart customization.

May 15, 2023 - Agent v0.1.67

This version improved log search and hosted agent performance as well as backend stability.

May 10, 2023 - Agent v0.1.66

This agent release introduced support for a specifying a worker count for Elastic and OpenSearch data destinations to improve peak traffic performance.

May 8, 2023 - Agent v0.1.65

This agent release improved back end stability and performance.

May 1, 2023 - Agent v0.1.64

This agent release improves Elastic integration performance by adding validation for send as is configurations.

April 27, 2023 - Agent v0.1.63

This agent release improved retry performance for S3 archiving and it adds the option to remove root names for Wavefront.

April 21, 2023 - Agent v0.1.62

This agent release improved agent performance with pre-start agent validation. It also removed the PVC dependency from aggregators and it added a more granular drop metric column option.

April 14, 2023 - Agent v0.1.61

This agent release improved the Helm template and added template validation.

April 12, 2023 - Agent v0.1.60

This agent release improved essential metric reporting and optimized S3 usage by reducing the file count.

April 10, 2023 - Agent v0.1.59

This agent release improved handling of agent health data as well as improved Elastic integration. It added support for creating AVRO formatted archive files and it added a distinct count type regex processor.

March 28, 2023 - Agent v0.1.58

This agent release improved handling of agent health data and it implemented a number of performance fixes.

March 16, 2023 - Agent v0.1.57

This agent release improved regex processor and log to metric performance. It also added options for handling internal certificates.

February 15, 2023 - Agent v0.1.56

This agent release improved back end stability and performance.

February 9, 2023 - Agent v0.1.55

This agent release added an option to disable TLS certificate verification for Elastic Streaming destinations. This may be useful for self-signed certificates.

February 1, 2023 - Agent v0.1.54

This agent release improved agent and back end stability and performance.

January 23, 2023 - Agent v0.1.53

This agent release improved stability and performance. In addition, Edge Delta Anomaly Context (EDAC) logs were improved for Amazon S3.

December 23, 2022 - Agent v0.1.52

This agent release made particular improvements to filter error handling and improved hosted agent environment performance among other backend stability and performance improvements.

December 6, 2022 - Agent v0.1.51

This agent release improved regex processors by making it easier to identify dimension group metrics using a custom suffix. In addition to agent and backend stability and performance improvements, script-based filters were updated to handle errors more effectively.

November 30, 2022 - Agent v0.1.50

This agent release improved agent and back end stability and performance.

November 28, 2022 - Agent v0.1.49

This agent release improved the querying experience for customers who use Elastic destinations.

November 24, 2022 - Agent v0.1.48

This agent release added support for enriching logs using K8s controller information. In addition attribute mode was added to the source detection filter to enable field mapping using keys from the source attributes rather than the log body. A number of stability and performance improvements were also deployed.

November 15, 2022 - Agent v0.1.47

This agent release improved agent functionality and performance particularly for Elastic and Datadog. In addition, support was added for Prometheus to scrape metrics with dimensions as attributes.

November 9, 2022 - Agent v0.1.46

This agent release improved stability and performance. In addition, an Enrichment filter that uses a script function was released.

October 11, 2022 - Agent v0.1.45

This agent release improved Prometheus integration. Rule metrics in Prometheus now work end to end:

You can now configure an agent to expose regex processor rule metrics at the metric endpoint for Prometheus scraping. The agent can now handle late-coming source tags for rule metrics. Rule metrics that are exposed from regex processors in the Prometheus format now support aggregator agents. Regex processor reporting now aligns with Prometheus 1 minute scraping intervals. In addition, pipeline performance has been improved with better handling of Elastic destinations not being available.

October 6, 2022 - On Prem UI v0.1.13

This release enabled the Observability - Metrics and the Data Pipeline - Pipeline Status pages for the self-hosted user interface. In addition, users of the self-hosted user interface can now access Observability, Data Pipeline and Management features without any 3rd party identity provider integrations, for example, for internal demo purposes:

October 3, 2022 - Agent v0.1.44

This agent release improved agent and back end stability and performance.

September 27, 2022 - Agent v0.1.43

This agent release added a new filter that can perform JavaScript enrichment and log transformation. In addition, the log files of newly created Kubernetes pods are now scraped from when they are created, including startup logs. The agent can also resume scraping from the previous position in a pod log file if the pod is restarted.

September 19, 2022 - Agent v0.1.42

This agent release added support in the numeric capture regex processor for multiple dimension groups with a range of metric types. In addition, the agent can now be configured to conform with the Amazon Web Services CloudWatch log quota limits.

September 9, 2022 - Agent v0.1.41

This agent release added Transport Layer Security (TLS) configuration support for S3-compatible archive destinations. The config wizard was updated and TLS support was added for AWS session settings.

September 5, 2022 - Agent v0.1.40

This agent release improved the Ratio Processor which is now out of Beta.

August 29, 2022 - Agent v0.1.39

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

On Demand Log Forwarding Log forwarding can now be triggered via an API call. It is used to temporarily forward specific sets of raw data to streaming destinations for a given time period. Both the duration and the log sources can be granularly defined to meet a number of use cases such as forwarding for 30 minutes following a deployment or when an alert triggers. The API call can be automated with integration into CI/CD tooling or third party alerting systems.

For more information, see On Demand Log Forwarding.

Prometheus Integration The Edge Delta agent can now be configured with a service monitor to expose metrics on an endpoint for Prometheus to scrape. The following metrics are exposed:

Count of incoming lines Sum of incoming bytes Count of outgoing lines Sum of outgoing bytes Count of successful outgoing streams Count of failed outgoing streams Open file status To learn more, see Prometheus Integration.

Updated Azure AppInsight Streaming Output The Azure AppInsight streaming output has been updated to allow you to specify where to funnel data. Previously, you could only funnel data to an event index.

Now, you can use the newly published base_type parameter to specify where to funnel data.

To funnel data into a tracing index, enter MessageData.

To funnel data into an event index, enter EventData.

To learn more, see Azure AppInsight.

New Filter Type You can use the newly created Log Transformer Javascript filter to transform specified log messages. Specifically, this filter uses Goja, a type of script in JavaScript. When a log matches the criteria, it will be transformed, and then passed through the filter.

To learn more, see Log Transformer Javascript.

Updated agent_settings For agent configuration, under agent_settings, you can use the newly created agent_stats_enabled parameter to display agent-related information (such as CPU and memory usage) in the Metrics page in the Edge Delta App.

To learn more, see Agent Settings.

June 6, 2022 - Agent v0.1.25

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updates Agent Installation The logging directory for the Edge Delta agent has been updated.

Specifically, if you install the agent via the installation script, then:

For Windows agents, logs can be found under %AppData%/edgedelta.

For all other operating systems, logs can be found under /var/logs/edgedelta.

Updated Input Enrichment In the Edge Delta App, input enrichment has been updated. Specifically, the field_name parameter under dynamic enrichment can be applied to a template.

      enrichments:
        dynamic:
          field_mappings:
            - field_name: `{{if eq .controllerKind "replicaSet"}}kube_deployment{{else}}kube_{{.controllerKind}}{{end}}`
              value: "{{.controllerName}}"

For more complicated templates that include if / else statements or range statements, you must use bracket as a delimiter.

Updated source_detection Parameter The source_detection parameter for inputs has been updated.

Specifically, source_detection now supports custom as a source_type. When you enter custom , you must configure the field_mappings parameter with a key-value pair.

    - labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Additionally, source_detection now supports regex as a processing_mode.

    - labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

May 10, 2022 - Agent v0.1.24

This agent release contains backend enhancements and bug fixes.

May 6, 2022 - Agent v0.1.23

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Streaming Output In the Edge Delta App, the Splunk streaming output has been updated to support custom tags via the custom_tags parameter.

You can use this parameter to define key-value pairs that are streamed with every request.

- name: my-splunk
      type: splunk
      endpoint: "://:/"
      token: "32-character GUID token"
      custom_tags:
        "app": "test"
        "region": "us-west-2"
        "File Path": "{{.FileGlobPath}}"
        "K8s PodName": "{{.K8sPodName}}"
        "K8s Namespace": "{{.K8sNamespace}}"
        "K8s ControllerKind": "{{.K8sControllerKind}}"
        "K8s ContainerName": "{{.K8sContainerName}}"
        "K8s ContainerImage": "{{.K8sContainerImage}}"
        "K8s ControllerLogicalName": "{{.K8sControllerLogicalName}}"
        "ECSCluster": "{{.ECSCluster}}"
        "ECSContainerName": "{{.ECSContainerName}}"
        "ECSTaskVersion": "{{.ECSTaskVersion}}"
        "ECSTaskFamily": "{{.ECSTaskFamily}}"
        "DockerContainerName": "{{.DockerContainerName}}"
        "ConfigID": "{{.ConfigID}}"
        "Host": "{{.Host}}"
        "Source": "{{.Source}}"
        "SourceType": "{{.SourceType}}"
        "Tag": "{{.Tag}}"

Updated Agent Settings In the Edge Delta App, the Agent Settings section has been updated with new parameters.

  max_file_per_glob_path: 100
  forget_file_after: 1h
  total_seek_capacity: "5 MB"
  max_seek_size: "4 MB"
  source_discovery_interval: 5s
  file_tailer_buffer_size: 1000
  router_per_source_buffer_size: 1000
  archive_flush_interval: 5m
  archive_max_byte_limit: "16MB"

To learn more, see Agent Settings.

May 2, 2022 - Agent v0.1.22

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Enrichment Options In the Edge Delta App, data enrichment options have been updated to address failed or failing sources.

To troubleshoot potential mapping failures, you can configure the failure_behavior parameter.

Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed.

      enrichments:
        failure_behavior: stop_enrichment
        dynamic:
          field_mappings:
            - field_name: "service"
              value: '{{".labels.service"}}'
            - field_name: "source"
              value: '.annotations.kubernetes.io/{{.container_name}}.logs'
              json_path: "[0].source"
              fallback_value: '{{".short_container_image"}}'

To learn more, see Enrich Input Data.

Updated Source Types In the Edge Delta App, streaming outputs have been updated.

Specifically, for the source_type parameter, you can now enter custom.

Previously, this parameter only supported K8s, Docker, ECS, and File.

When you enter custom, you must add field_mappings parameters to indicate the file source.

- labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Updated File Inputs In the Edge Delta App, the file input type has been updated.

Specifically, you can use the newly created exclude parameter to enter a glob path to exclude matched patterns.

files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"

April 27, 2022 - Agent v0.1.21

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Output In the Edge Delta App, the Splunk streaming output has been updated with the ability to send data in a JSON format.

Specifically, to use this option, you must update the endpoint parameter to point to Splunk’s API services/collector/raw, instead of services/collector/event.

    - name: splunk-integration
      type: splunk
      endpoint: ..../services/collector/raw
      token: ....
      features: log,metric,edac,cluster,alert
      index: rehydration

New Filter Type In the Edge Delta App, you can use the newly created JSON Field Extractor filter to extract a field’s value and replace the whole JSON content with the field’s value.

  - name: extract_severity
    type: extract-json-field
    field_path: "severity"
  - name: extract_first_data
    type: extract-json-field
    field_path: "records.[0].data"

To learn more, see JSON Field Extractor Filters.

Updated Enrichments for AWS ECS Inputs The configurations to enrich input data have been updated.

Specifically, you can use the dynamic parameter to enrich input data from AWS ECS.

In the agent configuration, you can

To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            - field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            - field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'
            - field_name: "cluster_name"
              value: '{{".aws-instance.cluster-name"}}'
            - field_name: "ec2launchtemplate_id"
              value: '{{".aws-instance.ec2launchtemplate-id"}}'
            - field_name: "ec2launchtemplate_version"
              value: '{{".aws-instance.ec2launchtemplate-version"}}'
            - field_name: "inspector_enabled"
              value: '{{".aws-instance.inspector-enabled"}}'
            - field_name: "cluster_autoscaler_enabled"
              value: '{{".aws-instance.cluster-autoscaler-enabled"}}'
            - field_name: "autoscaling_groupName"
              value: '{{".aws-instance.autoscaling-groupName"}}'
            - field_name: "nodegroup_name"
              value: '{{".aws-instance.nodegroup-name"}}'
            - field_name: "ec2_fleet_id"
              value: '{{".aws-instance.ec2-fleet-id"}}'

To learn how to enrich input, see Enrich Input Data.

To learn how to retrieve instance metadata, review this document from AWS.

New Filter Type In the Edge Delta App, you can use the newly createdSplit with Delimiter filter to match, then split a single log into multiple logs.

For example, the abc\n\ndef\nxyz\n log would split into 3 separate logs (abc , def , xyz ), based on the configured delimiter, (newline character ( \n )).

 - name: split_logs_using_specified_delimiter
    type: split-with-delimiter
    delimiter: ","

To learn more, see Split Lines Filters.

Updated Archiving Outputs In the Edge Delta App, archiving outputs have been updated.

Specifically, you can use the new use_native_compression option to compress data, but not metadata.

This option can be useful with big data cloud applications, such as AWS Athena and Google BigQuery.

To use this parameter, you must set the encoding parameter to parquet.

    - name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      s3_force_path_style: true
      encoding: parquet
      compression: zstd
      use_native_compression: true

Updated Cluster Processors In the Edge Delta App, cluster processors have been updated with a new configuration.

Specifically, you can use the newly created include_pattern_info_in_samples parameter to include pattern information in a cluster sample, such as patterns, pattern counts, and sentiment scores.

processors:
  cluster:
    name: clustering
    num_of_clusters: 100
    samples_per_cluster: 20
    reporting_frequency: 30s
    retention: 10m
    cpu_friendly: true
    throttle_limit_per_sec: 200
    include_pattern_info_in_samples: true

To learn more, see Cluster Processors.

New Input type In the Edge Delta App, NATS JetStream is now a supported input type.

This input type allows you to specify a NATS stream subscription for Edge Delta to monitor.

  nats:
    - labels: "my-nats-normal"
      input_mode: "normal"
      consumer_mode: "pull"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject: "example-subject-1"
      timeout: 1m
      ack_wait_duration: 10s
    - labels: "my-nats-distributed"
      input_mode: "distributed"
      consumer_mode: "push"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject_prefix: "example-subject"
      total_agent_count: 5
      total_subject_count: 10
      should_split_lines: true
      timeout: 1m
      disable_acks: true

To learn more, see NATS JetStream Inputs.

Updated Datadog Streaming Output In the Edge Delta App, the Datadog Streaming Output has been updated with buffered-related options.

Specifically, you can use the following, newly created parameters to configure the output’s buffering behavior:

Parameter Description buffer_ttl

Enter a length of time to retry failed streaming data.

After this length of time is reached, the failed streaming data will no longer be tried.

This parameter is optional.

buffer_ttl: 2h

buffer_path

Enter a folder path to temporarily store failed streaming data.

The failed streaming data will be retried until the data reaches its destinations or until the Buffer TTL value is reached.

If you enter a path that does not exist, then the agent will create directories, as needed.

This parameter is optional.

buffer_path: /var/log/edgedelta/pushbuffer/

buffer_max_bytesize

Enter the maximum size of failed streaming data that you want to retry.

If the failed streaming data is larger than this size, then the failed streaming data will not be retried.

This parameter is optional.

buffer_max_bytesize: 100MB

April 12, 2022 - Agent v0.1.20

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Log Enrichment In the Edge Delta App, log enrichment features have been updated to now support enrichment from Kubernetes annotations.

Specifically, you can use the from_k8s parameter to enrich streaming data with K8s attributes.

You can enter a pod, namespace, or node attributes.

        from_k8s:
          pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.*
          field_mappings:
            - field_name: instance_id
              pod_attribute: pod
              transformers:
                # replace all "source" matches with "target"
                - source: "-"
                  target: "_"
                  type: "replace"
                # remove all "test" words
                - source: "test*"
                  target: ""
                  type: "regex"
            - field_name: namespace
              pod_attribute: namespace
            # fields from labels should have pod_attribute start with "labels."
            - field_name: service
              pod_attribute: labels.service

To learn more, review the Enrich Input Data section in the Inputs document.

Updated Numeric Capture (Regexes) Processors In the Edge Delta App, the Numeric Capture (Regexes) processor has been updated to support multiplication and division for numeric value captures.

Specifically, you can use the newly created value_adjustment_rules parameter to create a rule per capture group.

The rule must follow the “(*|/)” format where:

An asterisk ( * ) represents multiplication A slash ( / ) represents division

    - name: "flog"
      pattern: " (?P\\d+) (?P\\d+)$"
      value_adjustment_rules:
        responsesize:
          operator: "/"
          operand: 1000.0

To learn more, review theNumeric Capture (Regexes) Processor section in the Processors document.

Updated Edge Delta Agent In an effort to improve security during agent installation, makeself –sha256 option has been enabled on agent deployments.

Specifically, makeself performs md5 and crc checks for content integrity.

Starting with version 0.1.20 of the agent, this upgraded security measure update will be included in all agent deployments.

Updated Sumo Logic Output In the Edge Delta App, the Sumo Logic streaming output has been updated.

Specifically, you can use the newly created send_as_json parameter to send data in a JSON format, which allows the fields to be auto-parsed and extracted in Sumo.

    - name: sumo-us-2
      type: sumologic
      endpoint: '{{ Env "EMPTY" "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/XYZ" }}'
      send_as_json: true

Updated Enriched Data for AWS In the Edge Delta App, you can use the dynamic enrichment feature to obtain data from an AWS EC2 instance.

Specifically, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            # if the field value starts with "aws-instance" then instance metadata is get from aws ec2 instance.
            # for more info ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
            - field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            - field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'

To learn more, review the Enrich Input Data section of the Inputs document.

For additional information, please review the Retrieve instance metadata document from Amazon.

Updated Enriched Data for JSON In the Edge Delta App, the from_logs enrichment feature has been with the json_path parameter.

You can use the json_path parameter to enrich data with fields extracted from JSON logs.

        # from_logs is used to enrich data with fields extracted from logs
        from_logs:
          field_mappings:
            - field_name: component
              # extracting using json_path is also supported
              json_path: fields.[1].component

To learn more, review the Enrich Input Data section of the Inputs document.

Updated Workflows In the Edge Delta App, you can use the newly created enabled_hosts parameter to limit the workflow to specific hosts.

With this parameter, the workflow will only run for specified agent hosts.

  enabled_hosts_workflow:
    description: "runs only specified hosts"
    input_labels:
      - system
      - docker
      - agent
      - infa-processes
    filters:
      - info
    destinations:
      - '{{ Env "TEST_SUMO" }}'
    enabled_hosts:
      - my.host.us1
      - my.host.us2

To learn more, see Workflows.

New Filter / Process Type In the Edge Delta App, you can use the newly created OTLP filter / processor to process OTLP (Open Telemetry) logs.

  - name: opentelemetry_trace_filter
    type: buffered-otlp-trace-processor
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "attributes.result_code"
    failure_value_pattern: "(4|5)xx"
    latency_threshold: 2500.0
    success_sample_rate: 0.1

To learn more, see Filters.

March 21, 2022 - Agent v0.1.19

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - GCP Cloud Monitoring In the Edge Delta App, GCP Cloud Monitoring is now a supported streaming output.

The GCP Cloud Monitoring output will stream custom Google Cloud metrics to a Cloud project.

In the app, you can use the visual editor or YAML file to add GCP Cloud Monitoring to an agent configuration.

GCP Cloud Monitoring was previously known as GCP Stackdriver.

New Flush Mode In the Edge Delta App, you can use the newly created custom_local_per_group flush mode to specify custom groups that should flush together if one of the groups triggers an alert.

In other words, if you set up multiple inputs, and one input triggers an alert, then all (or selected) inputs will flush.

agent_settings:
  tag: sett_test_custom
  log:
    level: debug
  capture_flush_mode: custom_local_per_group
  capture_flush_custom:
    label_grouping:
      group1:
        - file1
        - file2
      group2:
        - file1
        - file3

inputs:
  files:
    - labels: "file1"
      path: "test1.log"
    - labels: "file2"
      path: "test2.log"
    - labels: "file3"
      path: "test3.log"
    - labels: "file4"
      path: "test4.log"

outputs:
  streams:
    - name: sumo
      type: sumologic
      endpoint: "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1ozOeONNQ8LuYTYUj7SaKgr6dt1ueSTOc6mMS2pQz9BM169sb8_UQs5IRaqaRcbpKdI4Tms9S5La9ZFRTL_bf-Ptf_I5ICXcQz2WEQg0fNfA=="
      features: alert

processors:
  regexes:
    - name: "error-regex"
      pattern: "error|ERROR|problem|ERR|Err|POST|hostname|GET"
      interval: 10s
      retention: 1h
      trigger_thresholds:
        upper_limit_per_interval: 3

workflows:
  error-anomaly-workflow:
    input_labels:
      - file1
      - file2
      - file3
      - file4
    processors:
      - error-regex
    destinations:
      - sumo

The following actions will take place:

If an alert is triggered for test1.log, then the file1, file2, and file3 sources will be flushed together. If an alert is triggered for test2.log, then the file1 and file2 sources will be flushed together. If an alert is triggered for test3.log, then the file1 and file3 sources will be flushed together. If an alert is triggered for test4.log, then the file4 source will be flushed. Since file4 was not specified in any group and a fallback_mode was not provided, the agent will use the default fallback_mode local_per_source and only flush file4. When fallback_mode: local_all is added, and alert is triggered for test4.log, then all sources will be flushed. To learn more about filters, see Agent Settings.

New Filter / Processor Type In the Edge Delta App, you can use the newly created buffered-elastic-apm filter / processor to process Elastic APM logs.

  - name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor

To learn more about processors, see Processors.

New Filter Type - base64 decoder In the Edge Delta App, you can use the newly created base64 decoder filter type to decode base64 encoding.

This update helps to support a base64 encoded input on Edge Delta’s hosted collector. In other words, you can attach this filter to a source to display logs that are base64 encoded.

  - name: base64_decoder
    type: base64-decode

To learn more about filters, see Filters.

New Monitor Types You can use the new Pattern Alert and Skyline Alert monitors to trigger an alert for negative patterns.

If an alert is triggered, then the monitor will create a finding.

To learn more, see Patterns.

New feature - Suppress Notifications In the Edge Delta App, you can use the newly created Finding Status option to suppress notifications for a specific finding.

When you suppress a finding, the finding will no longer be displayed in the Insights page. Additionally, any future detection of the finding will not be displayed.

To learn more, see Patterns.

New Filter Type - APM In the Edge Delta App, you can use the newly created APM filter to process Elastic APM logs.

This filter type samples failed and high-latency traces with successful traces and a sampling probability.

  - name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor
    payload_separator: "-----------------"
    enabled_types: "transaction,span,error,metricset"
    optimize_types: "transaction,span"
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "transaction.result"
    failure_value_pattern: "HTTP (4|5)xx"
    transaction_latency_path: "transaction.duration"
    span_latency_path: "span.duration"
    latency_threshold: 35.5
    success_sample_rate: 0.2

To learn more, see Filters.

March 24, 2022 - New Organizations Feature

In the Edge Delta App, you can now create and join different organizations for you and your users.

At a high level, organizations can be considered as different environments, platforms, or sub-accounts that live within your main Edge Delta account.

You can create and belong to multiple organizations, as well as invite users to join various organizations.

Additionally, you can switch between organizations without the need to log off and log back into the app.

To learn more, see Invite Users, Manage Permissions, Access Organizations.

March 21, 2022 - Agent v0.1.18

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - ObserveInc In the Edge Delta App, ObserveInc is now a supported streaming output.

The ObserveInc output will stream analytics and insights to your ObserveInc endpoint.

In the app, you can use the visual editor or YAML file to add ObserveInc to an agent configuration.

New Filter - Custom Attributes In the Edge Delta App, you can use the newCustom Attributes filter to filter for custom attributes.

Specifically, you can use the Attribute Key andAttribute Value parameters to filter for custom attributes.

  # Custom attribute filter do all the log filtering with given key-value of the attribute
  - name: custom_attributes_filter
    type: custom-attributes
    key: service
    value: billing
  - name: negate_custom_attributes_filter
    type: custom-attributes
    key: component
    # A comma separated values to match. If any of them matches the given attribute's value then the log will be pass through
    value: credithandler,debithandler
    # Negate is also supported for attribute filter
    negate: true
  # Filtering custom attributes also support regex matching
  - name: regex_custom_attributes_filter
    type: custom-attributes
    key: level
    pattern: "error|ERROR|problem|ERR|Err"

To learn more, see Filters.

Updated Agent Settings - Log In the Edge Delta App, the Log parameter in the Agent Settings has been updated.

Specifically, you can use theSecure Logging option to hide sensitive data from the specified agent logs, such as API keys, secrets, and authentication information.

To learn more, see Agent Settings.

Updated Integration - Loki In the Edge Delta App, the Loki integration has been updated with a new option.

Specifically, the Send Alert As Loki Log option allows you to send alerts as a log to a Loki endpoint.

New Input Type - Google Pub/Sub In the Edge Delta App, Pub/Sub is now a supported input type.

This input type allows you to specify a Pub/Sub project for Edge Delta to monitor. Specifically, EdgeDelta will consume messages from Pub/Sub subscriptions.

In the app, you can use the visual editor or YAML file to add PubSub to an agent configuration.

To learn more, see Inputs.

Updated Input - File In the Edge Delta App, the File input type has been updated. Specifically, there are 2 new parameters:

Add Ingestion Timestamp

You can use this parameter to ingest a timestamp if the input format is in JSON. Skip Ingestion Timestamp On Failure

You can use this parameter to skip the ingestion of the timestamp when the input is broken or in an invalid format.

files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      # ingest timestamp if input is JSON format.
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true # skip ingestion time when the input is broken or invalid format.

New Filter - Combinations In the Edge Delta App, you can use the newly created combination filter to combine with other, existing filters. Specifically, you can use and or or terms to combine filters to create a more customized filter.

filters:
  - name: combine_two_filters
    type: combination
    operator: or
    filters_list:
      - pattern: "INFO"
      - filter_name: error

To learn more, see Filters.

New Filter Type - Drop Json Fields In the Edge Delta App, you can use the newly created drop-json-fields filter to filter and drop specified JSON fields.

filters:
  - name: drop_some_fields
    type: drop-json-fields
    field_paths: # Each field path is a dot separated path of the field (i.e. "log.source")
      - "level"
      - "details"
      - "log.source"

New Input Type - EDPort In the Edge Delta App, you can use the newly created EDPort Collector Inputs input type to specify a set of ports and protocols for the agent to listen on for incoming traffic.

inputs:
  ed_ports:
    - labels: request
      port: 9000
      protocol: tcp
      read_size: 1
      read_timeout: 30s
      source_detection:
        source_type: "K8s"
        optional: false
        field_mappings:
          k8s_namespace: "kubernetes.namespace"
          k8s_pod_name: "kubernetes.pod.name"
          k8s_container_name: "kubernetes.container.name"
          k8s_container_image: "kubernetes.container.image"
      enrichments:
        from_logs:
          field_mappings:
            - field_name: environment
              json_path: kubernetes.tags.env

To learn more, see Inputs.

February 11, 2022 - Updated Edge Delta App Design

In order to provide a better user experience, the overall look and feel of the Edge Delta App has been updated.

February 1, 2022 - Agent v0.1.16

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Multi-Threshold Support In the Edge Delta App, you can add multi-threshold settings to an agent configuration file.

For example, you can set a threshold to generate an alert when the following conditions are met:

anomaly score is > 90 response_time_ms.avg is > 250 To set this configuration, use the type parameter, specifically set to AND.

  - name: cluster-errors-multi-threshold
    type: and
    interval: 1m
    conditions:
    - metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    - metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

To learn more, see Thresholds.

Graylog Integration In the Edge Delta App, Graylog is now a supported streaming output.

The Graylog output will stream analytics and insights to your Graylog endpoint. In the app, you can use a YAML file to add Graylog to a configuration.

Dynatrace Integration In the Edge Delta App, Dynatrace is now a supported streaming output.

The Dynatrace output will stream analytics and insights to a Dynatrace environment. In the app, you can use the visual editor or YAML file to add Dynatrace to a configuration.

Updates to Datadog Integration The Datadog Integration has been updated with a new parameter called Send Alert As Datadog Log. With this update, you can now send alerts as logs.

Additional Information This agent release contains additional enhancements and bug fixes.

To see the complete list of changes, please visit the Changelog - Agent Releases page.

January 31, 2022 - Notification for Throttled Data

In the Edge Delta App, a notification has been created to let users know when data ingestion is being throttled.

To learn more about these limits, please contact Edge Delta Support.

December 15, 2021 - VictorOps Integration

In the Edge Delta App, VictorOps is now a supported triggering output.

The VictorOps output streams notifications and alerts to a VictorOps endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

VictorOps is also known as Splunk On-Call; however, the app will refer to this output as simply VictorOps.

December 30, 2021 - AppDynamics Integration

In the Edge Delta App, AppDynamics is now a supported streaming output.

The AppDynamics output will stream analytics and insights to an AppDynamics environment. In the app, you can use the visual editor or YAML file to add AppDynamics to a configuration.

December 17, 2021 - OpenMetrics Integration

In the Edge Delta App, OpenMetrics is now a supported streaming output.

The OpenMetrics output will stream analytics and insights to an OpenMetrics endpoint. In the app, you can use a YAML file or visual editor to add OpenMetrics to a configuration.

December 16, 2021 - S3 Integration

In the Edge Delta App, S3 is now a supported streaming output.

The S3 output will stream analytics and insights to an S3 bucket. In the app, you can use a YAML file or visual editor to add S3 to a configuration.

December 15, 2021 - Cribl Integration

In the Edge Delta App, Cribl is now a supported streaming output.

The Cribl output streams analytics and insights to a Cribl endpoint. In the app, you can use the visual editor or YAML file to add Cribl to a configuration.

December 4, 2021 - Moogsoft Integration

In the Edge Delta App, Moogsoft is now a supported triggering output.

The Moogsoft output will stream notifications and alerts to a specified Moogsoft URL. In the app, you can use the visual editor or YAML file to add Moogsoft to a configuration.

December 1, 2021 - Honeycomb Integration

In the Edge Delta App, Honeycomb is now a supported streaming output.

The Honeycomb output will stream analytics and insights to a Honeycomb environment. In the app, you can use the visual editor or YAML file to add Honeycomb to a configuration.

December 1, 2021 - FluentD Integration

In the Edge Delta App, FluentD is now a supported streaming output.

The FluentD output will stream analytics and insights to your FluentD endpoint. In the app, you can use the visual editor or YAML file to add FluentD to a configuration.

December 1, 2021 - AWS CloudWatch Event Logs Input

In the Edge Delta App, Cloudwatch Event Logs is now a supported input.

The Cloudwatch Event Logs input type allows you to specify a set of AWS CloudWatch Log Events for Edge Delta to monitor. With this input, you can monitor multiple regions and log streams.

In the app, you can use the visual editor or YAML file to add Cloudwatch Event Logs to a configuration.

To learn more, see Inputs.

December 1, 2021 - Big Panda Integration

In the Edge Delta App, Big Panda is now a supported triggering output.

The Big Panda output will stream notifications and alerts to a specified BigPanda endpoint. In the app, you can use the visual editor or YAML file to add Big Panda to a configuration.

December 1, 2021 - Loki Integration

In the Edge Delta App, Loki is now a supported streaming output.

The Loki output streams analytics and insights to your Loki endpoint. In the app, you can use the visual editor or YAML file to add Loki to a configuration.

December 1, 2021 - Logz.io Integration

In the Edge Delta App, Logz.io is now a supported streaming output.

The Logz.io output will stream analytics and insights to your Logz.io endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

November 30, 2021 - Child Configurations

In the Edge Delta App, you can add a configuration into another, existing configuration. With this action, the existing configurations will convert into a parent and child configuration.

To learn more, review the Create and Add a Child Configuration section of the Cloud Configuration Backend (CCB) document.

November 30, 2021 - New Compression and Encoding Settings

In the Edge Delta App, you can change compression and encoding settings for Outputs - Archives.

For encoding, Edge Delta now supports Parquet. For compression, Edge Delta now supports zstd and Snappy. Previously, Edge Delta only offered gzip compression and JSON encoding.

To update these settings, you must access the configuration’s YAML file.

archives:
    - name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      # Force archive destination to use {endpoint}/{bucket} format instead of {bucket}.{endpoint}/ when reaching buckets.
      s3_force_path_style: true
      encoding: parquet # supported ones: json, parquet
      compression: zstd # supported ones: gzip, zstd, snappy, uncompressed

November 23, 2021 - EDPort Integration

The EDPort streaming integration has been updated to offer JSON as a schema option for sending data.

November 22, 2021 - Local Storage

In the Edge Delta App, Local Storage is now a supported archiving output.

The Local Storage output will send logs to a file on your local machine.

November 18, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported streaming output.

The Azure Event Hubs output will stream analytics and insights to an Azure Event Hubs endpoint. In the app, you can use a YAML file to add Azure Event Hubs to a configuration.

November 12, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported triggering output.

The Azure Event Hubs output will stream notifications and alerts to a specified Event Hub URL. In the app, you can use the visual editor or YAML file to add Azure Event Hubs to a configuration.

November 9, 2021 - Create a Token

In the Edge Delta App, you can create a token to give your users specific access to the Edge Delta API system.

With tokens, you can specify read / write access for specific backend functionality for your users. In other words, you can create a token to give your users specific read / write access to the Edge Delta API system.