Edge Delta Release Notes

January 5, 2026

New Features

• EDXLookup Pattern Matching: Extended the EDXLookup OTTL function with pattern matching capabilities including regex, contain, prefix, and suffix match modes, case-insensitive matching, and the ability to return multiple matches from lookup tables.
• Excluded Paths for HTTP Input: Added an excluded_paths field to HTTP input nodes, allowing you to filter out specific paths from processing while using wildcard includes.
• Splunk HEC Acknowledgment Endpoint: Implemented the acknowledgment endpoint for Splunk HEC input nodes, completing support for Splunk indexer acknowledgment protocol.
• JSON Unroll for Array Bodies: Extended JSON unroll functionality to work on message bodies containing arrays of JSON objects, enabling individual processing of each array element.

Improvements

• Live Tail Settings Configuration: Moved live tail and sampling settings from environment variables to pipeline configuration, providing easier access while still supporting environment variable overrides when needed.
• Self Log Flush Interval Configuration: Added the self log flush interval to agent settings, allowing configuration through the pipeline interface instead of requiring environment variables.
• Prometheus Input Service Name: Fixed the Prometheus input node to correctly populate the job field as service.name, enabling proper identification of metric sources without requiring additional transformations.
• Probabilistic Sample Processor: Extended the probabilistic sample processor to handle any data type for hash calculation, supporting object types and manipulated body fields beyond strings, integers, and floats.
• OTTL Transform Error Modes: Added configurable error modes to the OTTL transform processor with silent mode for parsing processors and strict mode for others, providing appropriate error handling based on processor type.
• Parse Processor Error Handling: Introduced a configurable error_mode field for parse processors (ParseJSON, Grok, ExtractJSONField, JSONUnroll) with silent mode as default to reduce unnecessary error logs, and strict mode for when validation errors should be reported.
• Kafka Library Upgrade: Updated the Kafka client library to v0.4.35, incorporating upstream fixes for improved reliability in coordinator relay services.
• Filebeat Node Rename: Renamed the Elastic API input node to Filebeat to better reflect its purpose of receiving data from Filebeat agents.
• Kafka Source Validation: Added proper validation to mark endpoint and topic fields as required in the Kafka source node configuration.

Bug Fixes

• GCS Write Request Optimization: Fixed a regression introduced in v2.8.0 that caused excessive GCS write requests.
• S3 Input URL Decoding: Fixed S3 input failing to download objects with special characters in keys by properly URL-decoding object keys from SQS notifications, resolving NoSuchKey errors for files with characters like colons or plus signs.
• HTTP Input NDJSON Parsing: Fixed HTTP input incorrectly treating newline-delimited JSON (NDJSON) as a single message, enabling proper splitting of multiple JSON objects in the request body.
• Local Storage Destination Fixes: Fixed the Local Storage destination node to honor the configured path prefix and respect the compression setting instead of using hardcoded values.
• Compactor Client Event Subscription: Fixed unnecessary warning logs for compactor client status change events when no compactor component is subscribed, now conditionally unsubscribing based on component presence.
• Kubernetes Metrics Nil Pointer: Fixed a nil pointer exception in the Kubernetes metrics input node that could occur when pod information is not found in the agent cache while using node_labels.
• Cloud Fleet Secret Handling: Added secret dereferencing for cloud fleet configurations after retrieval, ensuring secrets are properly resolved when connecting through the backend coordinator.
• Coordinator Connection Reset: Fixed coordinator client to properly reset the connection on ping failures, preventing stale connections and associated error logs.

Security

• CVE-2025-68156: Updated github.com/expr-lang/expr to v1.17.7 to address a high severity vulnerability where deeply nested or cyclic data structures could cause stack overflow panics in builtin functions.

December 16, 2025

New Features

• ClickHouse Destination: Added a new HTTP-based ClickHouse destination node for direct write operations to ClickHouse instances with compressed HTTP support.
• Splunk Load Balanced Destination: Added a new destination node that retrieves all available indexers from the cluster master and load balances across them, with configurable polling rate for indexer list updates.
• HTTP Workflow Pagination: Implemented full pagination execution for HTTP Workflow inputs, including Link header parsing (RFC 5988), JSON path URL extraction, parallel page fetching, header inheritance with security controls, and configurable error handling strategies.
• Google SecOps Destination: Added a new destination node for sending logs to Google SecOps (Chronicle) with support for regional endpoints (US, Europe, Asia), service account authentication or Application Default Credentials, optional customer ID, and gzip compression.
• Archive Destination Schema Selection: Added support for specifying custom file paths for user-defined schemas in archive destinations using the file://<path> format.

Improvements

• Datadog Data Fidelity: Improved data fidelity for Datadog metrics and traces including metric type preservation (gauge, rate, count), interval forwarding for rate/count metrics, device field forwarding for per-device system metrics, source type name forwarding, proper hostname extraction for traces, stub handlers for unprocessed Datadog agent endpoints, and handling of uncompressed requests.
• Lookup Processor Multi-Telemetry Support: Extended the lookup processor to handle metrics and traces in addition to logs, enabling consistent enrichment via lookup tables across all telemetry types.
• Buffer Event Metrics: Added new metrics to track individual event counts and bytes in the persistent buffer (ed.buffer.memory.events, ed.buffer.disk.events, ed.buffer.memory.event_bytes, ed.buffer.disk.event_bytes) for better visibility into actual data volume being buffered.
• Self Log Flush Interval: Added dynamic self-log flush interval that uses a more frequent flush (configurable via ED_SELF_LOG_FLUSH_INTERVAL_POST_CONFIG_CHANGE, default 1 minute) for the first 15 minutes after startup or config changes, then reverts to the normal flush interval.
• Destination Error Reporting: Hardened all destination nodes to report errors to self-telemetry when destinations are unreachable or errors occur during data transmission.
• OTLP Input Hardening: Added configurable WriteTimeout and IdleTimeout, log writer for HTTP server errors, unified error handling, and self-telemetry error tracking for the OTLP input node.
• Kafka Source Rate Limiting: Added events_per_minute configuration option to the Kafka source node to control throughput at the source level and prevent overwhelming downstream destinations.
• Beta Nodes Released: The following nodes are now generally available: Splunk TCP Input, Filebeat Input, SNMP Pull Input, SNMP Trap Input, Syslog Input, Splunk TCP Output, Google BigQuery Output, Securonix Output, Grokstream AIOps Output, and OTTL Context Filter Processor.
• JSON and Gzip Performance: Improved CPU and GC usage for JSON and Gzip operations.
• Client-Side Caching for Redis: Added client-side caching to EDXRedis for improved performance.

Bug Fixes

• Persistent Queue Disk Usage: Fixed an issue where the max_size buffer configuration only limited logical data size but not actual disk usage, which could grow unbounded. The fix adds periodic disk size checks and enforces disk-based write gating. Config Reload: Fixed an issue where items in the buffer were not processed when a configuration reload occurred.
• HTTP Workflow Pagination Persistence: Fixed an issue where pagination configuration was lost when switching between UI and YAML tabs in the pipeline builder. SSRF Protection: Disabled SSRF protection for HTTP Workflow input when running on user infrastructure, allowing agents to reach internal and private network APIs while keeping protection enabled for backend UI test services.
• Syslog Input Timestamp Handling: Added defensive logic to replace zero timestamps with the observed time in syslog input, resolving issues with malformed syslog messages. Error Visibility: Fixed syslog parsing errors not being captured in self-telemetry by creating a custom logger that intercepts and forwards errors to the self-telemetry system.
• Datadog Output Tag Handling: Fixed an issue where Datadog output was incorrectly adding all attributes as tags when using a Datadog input.
• GCL Field Deletion: Fixed keep_overridden_field_name not working correctly in the GCL destination by ensuring field deletion happens before JSON serialization.
• Field Deletion in Destinations: Updated all destination nodes to completely remove keys during delete operations instead of just setting values to nil.
• Compactor Kubernetes API Calls: Reduced excessive Kubernetes API calls from the compactor component by replacing continuous informer-based polling with on-demand resolution.
• Self Telemetry Node Protection: Prevented disabling of self-telemetry nodes.

Known Issues

• GCS Destination Excessive Write Requests: A regression introduced in v2.8.0 causes the GCS destination to issue excessive write requests, potentially increasing costs and API usage. This issue is fixed in v2.11.0. Users experiencing high GCS API usage should upgrade to v2.11.0 or later.

December 4, 2025

New Features

• HTTP Workflow Source: Added support for chained API requests with variable extraction, enabling complex data retrieval workflows where subsequent requests depend on values extracted from previous responses. Includes webhook trigger support with bearer and signature authentication, rate limiting, and Redis caching for step responses with configurable TTL.
• HTTP Destination Compression: Added compression options to HTTP destination node with support for gzip, zstd, and snappy compression algorithms, reducing network bandwidth usage and improving transfer efficiency when sending data to HTTP endpoints.

Enhancements

• OTLP Input Buffer Logging: Added buffer full error logging for OTLP input nodes, providing visibility when buffers reach capacity. The default buffer size is 1,000 entries and can be increased using the ED_OTLP_TAILER_BUFFER_SIZE environment variable.
• Configurable Request Timeouts: Made request timeouts configurable for HTTP Pull nodes, allowing users to adjust timeout values for API endpoints with longer response times and ensuring retry mechanisms use the configured timeout instead of hardcoded defaults.
• Parallel Worker Count Default: Corrected the default value documentation for parallel_worker_count to accurately reflect the actual default of 2 workers, ensuring configuration expectations match runtime behavior.

Bug Fixes

• Gateway Destination Telemetry: Fixed self telemetry reporting for gateway destination nodes to correctly report the node type, enabling proper metrics collection and inventory tracking for gateway outputs.

Known Issues

• GCS Destination Excessive Write Requests: A regression introduced in v2.8.0 causes the GCS destination to issue excessive write requests, potentially increasing costs and API usage. This issue is fixed in v2.11.0. Users experiencing high GCS API usage should upgrade to v2.11.0 or later.

December 2, 2025

New Features

• Pipeline Secrets: Introduced pipeline secrets for storing encrypted credentials in configuration files with placeholder references. Secrets are encrypted at rest using AES-256-GCM and decrypted only at runtime, ensuring sensitive values are never logged or exposed in plaintext. Secrets support both UI-created (default mode) and CLI-created (masterkey mode) encryption methods.
• Persistent Queue Strategies: Added configurable persistent queue strategies to destination nodes (see S3 for example) for output buffering with three modes: error (default) writes to disk only when destinations are unreachable for lowest latency; backpressure writes to disk when in-memory buffers reach 80% capacity for balanced durability; and always writes every event to disk before sending for maximum durability. The strict_ordering option controls whether newer events can bypass older buffered events during backpressure recovery.
• Google SecOps Destination: Added Google SecOps destination node for sending telemetry data directly to Google Security Operations (Chronicle) for security analytics and threat detection.
• Kerberos Authentication for Kafka: Added Kerberos authentication support to Kafka source node, enabling secure data ingestion from Kerberos-protected Kafka clusters in enterprise environments.
• Apache Kudu Security: Enhanced Apache Kudu destination with Kerberos authentication and encryption support for secure enterprise data warehouse integration.
• Datadog Trace Support: Added trace capability to Datadog output node, enabling distributed tracing data to be sent directly to Datadog APM for end-to-end observability.
• Metrics in Splunk HEC Input: Extended Splunk HEC input node to support ingesting metrics in addition to logs, enabling unified telemetry collection from Splunk HEC-compatible sources.
• Dynamic Elastic Index: Added dynamic index support to Elastic destination via expressions, allowing index names to be computed dynamically based on data item attributes for flexible index routing.
• Cron-based HTTP Pull Scheduling: Added cron-based scheduling support to HTTP Pull sources, enabling precise scheduling of data retrieval at specific intervals using standard cron syntax.

Enhancements

• Keep Overridden Fields: Added keep_overridden_index, keep_overridden_token, and similar options to multiple destination nodes including Splunk TCP, Elastic, Splunk HEC, GCS, CloudWatch, and GCL. These options preserve original field values when dynamic expressions override them, enabling dual-write scenarios where both original and computed values are needed.
• Prometheus Remote Write Authentication: Added authentication and custom header support to Prometheus remote write destination, enabling secure metric forwarding to authenticated Prometheus-compatible endpoints.
• New Relic Output Buffer: Added output buffering to New Relic destination, improving reliability and throughput when sending telemetry data to New Relic.
• OTEL Histogram Support: Updated OTLP input and output nodes to support the latest metric histogram structure from OpenTelemetry, ensuring compatibility with current OTEL collector versions.
• User-Provided Parquet Schema: Added support for user-provided schema definitions when processing raw Parquet files in archive operations (see S3 or GCS), enabling precise control over column types and structure.
• Raw Schema for Archive: Added support for Raw schema type in archive operations, providing flexibility in how archived data is structured and stored.
• Buffer Metrics: Added ed.buffer.* metrics for monitoring persistent queue usage and performance, and ed.pipeline.node.throttle metric for tracking node throttling events, improving observability of pipeline backpressure conditions.
• OTEL Semantic Conventions: Added additional OpenTelemetry semantic convention resource attributes and labels to improve trace and metric correlation.
• Splunk Field Retention: Ensured Splunk-specific fields are retained after deotel processing, preserving source, sourcetype, and index metadata through OTEL transformations.
• Trace ID Parsing: Added decimal parsing support for trace and span IDs, improving compatibility with trace systems that use decimal ID formats.
• S3 Test Event Handling: Enhanced S3 input node to properly handle AWS S3 test events, preventing unnecessary processing of test notifications.
• Splunk TCP Noise Reduction: Reduced log noise from expected EOF events during Splunk TCP connection handshakes, improving log clarity.
• GCL Self Telemetry: Fixed self telemetry reporting in GCL destination when using custom item types.

Bug Fixes

• OTTL Context Filter Deadlock: Fixed a deadlock issue in OTTL context filter processor that could cause agent crashloops under certain conditions.

Security

• CVE-2025-47913 Fix: Updated golang.org/x/crypto dependency to address CVE-2025-47913, a security vulnerability in the cryptographic library.

Known Issues

• GCS Destination Excessive Write Requests: A regression in this release causes the GCS destination to issue excessive write requests, potentially increasing costs and API usage. This issue is fixed in v2.11.0. Users experiencing high GCS API usage should upgrade to v2.11.0 or later.

Breaking Changes

• Source Samples Input Removed: The ed_source_samples_input node type has been removed. Pipelines using this node type will fail to start. This node was previously used for internal source detection and has been replaced with improved mechanisms.

November 7, 2025

New Features

• Elastic API Input: Introduced Filebeat endpoint to consume data from Vector, Logstash, and Filebeat, enabling seamless integration with Elastic-compatible data sources.
• EDXLookup Function: Added custom OTTL function for data enrichment by looking up values from lookup tables, supporting dynamic data enrichment with configurable column access.
• Edge Delta AI Event Output Node: Released AI Event Output node for public use, enabling AI-powered event processing with support for non-log data types and enhanced connector metadata.
• Apache Kudu Destination: Introduced secure Apache Kudu destination with Kerberos authentication and encryption support for enterprise data warehouse integration.
• BigQuery Destination: Added Google Cloud BigQuery destination node for sending log data directly to BigQuery via API, enabling native GCP data warehouse integration with nil object handling.
• Secrets and Keys Management: Added secrets and keys sections to pipeline configuration with reference validation, enabling secure credential management and encryption key handling.

Enhancements

• HTTP Input JSON Parsing: Added parse mode configuration with auto-detection, JSON object preservation, and line-delimited modes to HTTP Input source, resolving issues with embedded newlines in JSON payloads and ensuring complete JSON objects are preserved as single log entries.
• HTTP Pull Source Enhancements: Added body and body expression fields to HTTP Pull source with POST method support, and configurable request timeout for flexible HTTP data retrieval.
• OTLP JSON Encoding: Fixed JSON encoding support in OTLP source and destination nodes for HTTP protocol, resolving HTTP 400 errors when exporting logs.
• Splunk TCP Dynamic Indexing: Added dynamic index value support to Splunk TCP destination based on data item attributes, enabling flexible index routing similar to GCS bucket expression capability.
• TCP Input Concurrency: Enabled concurrent TCP connections with proper connection limits in TCP input, allowing multiple simultaneous client connections for improved throughput.
• Live Tail Payload Management: Added configurable truncation for large payloads with environment variables (ED_CAPTURER_ITEM_MAX_BODY_SIZE and ED_CAPTURER_ITEM_TRUNCATION_SIZE) for body size and truncation thresholds, preventing UI freezing with oversized data during live capture.
• Pipeline Node Disabling: Added disabled property support for source and destination nodes, allowing nodes to remain visible in UI while being operationally inactive without requiring removal from pipeline configuration.
• Edge Delta Gateway Destination: Extended Edge Delta Gateway destination availability to all environment types including non-Kubernetes deployments, enabling gateway connectivity via Kubernetes ingress or host ports.
• Splunk HEC Input Docker Support: Enhanced Splunk HEC input to handle Docker Splunk driver time field format and endpoint path variations, ensuring compatibility with Docker logging drivers.
• Splunk HEC Authentication: Added authentication support to Splunk HEC input node with hec_authentication configuration, enabling token validation and access control for incoming HEC requests.
• Splunk HEC Error Codes: Improved Splunk HEC input error code responses with proper feedback mechanisms, including code 1 for disabled tokens, providing better diagnostics for token validation issues.
• Splunk Destination Compression: Added gzip compression support to Splunk destination, enabling data compression before transmission to reduce bandwidth usage and improve transfer efficiency.
• TLS Configuration Separation: Separated TLS specifications into client and server roles with appropriate field restrictions, improving mutual TLS configuration clarity and preventing misconfiguration.
• Trace Type Recognition: Enhanced trace type recognition with expanded support for databases and messaging systems, and relaxed detection logic to prevent overwriting existing trace types set earlier in the pipeline.
• OTLP Input Compression: Added compression support for OTLP input to ensure compatibility with newer OpenTelemetry Collector versions that enable compression by default.

Bug Fixes

• S3/SQS Data Loss Prevention: Fixed critical bug in S3 input where SQS messages were deleted even when S3 object downloads failed. The agent now only deletes SQS messages after all downloads succeed for that message, ensuring failed downloads are automatically retried after the visibility timeout. Messages with partial download failures remain in the queue for retry.
• IRSA Authentication: Fixed authentication on EKS when role_arn is specified in S3/SQS configuration by using AWS SDK default credential chain with AssumeRole, enabling cross-region access and maintaining backward compatibility.
• OTLP Destination Endpoints: Fixed HTTP endpoint construction in OTLP destination, ensuring proper URL formatting for outbound telemetry.
• Output Buffer Encoding: Expanded gob encoding/decoding support to all item types beyond metrics, resolving serialization issues in HTTP output buffering.
• BigQuery Operations: Added nil object handling for BigQuery responses to prevent nil pointer errors when API returns nil without explicit errors.
• Kubernetes Trace IDs: Fixed trace and span ID generation to use hex-encoded strings, resolving failures when sending traces in OTEL format through gateway destination.
• Elastic API Input Configuration: Fixed Elastic API input to properly honor configuration variables including enable_health_check, active_request_limit, custom_api_version_response, and api_version that were previously not being applied in the tailer.
• Splunk HEC Input JSON Handling: Fixed Splunk HEC input to properly handle incoming requests with JSON payload events, resolving parsing issues for JSON-formatted event data.

Configuration Changes

• Sample Collection: Changed live capture sample collection to disabled by default to reduce unnecessary data processing and performance overhead in production environments. Live capture is designed for the pipeline design phase and can be re-enabled by setting the ED_DISABLE_SAMPLE_COLLECTOR=0 environment variable via helm (see helm values) if needed for ongoing pipeline development.
• Pipeline Secret: Coordinator now automatically generates secure pipeline.secret during initial run for encryption and decryption of secrets and configuration keys.

Deployment and Upgrade Notes

Coordinator Agent Upgrade (v2.7.0):

A new volume mount has been introduced for the coordinator agent to support persistent storage of the pipeline.secret.

• Upgrades: If upgrading an existing coordinator agent to this version, you must run helm upgrade to apply the updated deployment configuration.
• Requirements: Ensure that your Kubernetes cluster has a default StorageClass defined, as the new PersistentVolumeClaim depends on it.

Backward Compatibility Notes:

The following new configuration fields require agent version v2.7.0 or higher. Agents running older versions will not honor these fields:

• HTTP Pull Source - request_body and request_body_expression fields (added for POST request support)
• Splunk TCP Destination - index_expression field (added for dynamic index routing)
• Splunk HEC Input - hec_authentication field (added for token validation and access control)

September 22, 2025

New Features

• Apache Kudu Destination: Added an Apache Kudu output node for routing data to Kudu, enabling low-latency analytics on high-volume datasets.
• Splunk TCP Destination Node: Introduced a Splunk TCP (S2S) output node to send data directly to Splunk over TCP, simplifying migrations and hybrid deployments.
• Webhook Output (Metrics): Added support for the Metric datatype (gauge, sum, histogram) in the Webhook output with new templating options; also fixes header propagation so custom headers are forwarded as configured.
• EDXCode Function (Beta): Added an OTTL function to execute inline JavaScript expressions for advanced transformations and rapid prototyping within Telemetry Pipelines.
• Exponential Histogram Support: Added exponential histogram handling across OTLP and Prometheus inputs, updated the OTLP tailer, storage format, and schemas to capture high-dynamic-range metrics end to end.

Improvements

• AWS Region Configuration for S3/SQS Inputs: Added s3_config and sqs_config settings (with validation and fallback) for S3 and CrowdStrike FDR inputs to support cross-region deployments without redirects.
• OTTL Library Upgrade: Upgraded OTTL to v0.135.0 to enable newer functions (e.g., Index) and improvements across transformation logic.
• Self Telemetry Node Controls: Added source metadata and an advanced option to unselect unwanted dimensions to control cardinality and reduce ingestion.
• Input Node UX: Added documentation links to input node messages and improved warnings when binding to ports 0-1024 without root privileges to help avoid ingestion issues.
• Capture Payload Controls: Parameterized capture lower/upper thresholds to prevent oversized payloads and reduce browser memory usage during troubleshooting.
• Live Tail Payload Management: Added configurable truncation for large payloads with environment variables (ED_CAPTURER_ITEM_MAX_BODY_SIZE and ED_CAPTURER_ITEM_TRUNCATION_SIZE) for body size and truncation thresholds, preventing UI freezing with oversized data during live capture.
• SNMP Trap Source (v3): Added SNMPv3 support to the SNMP Trap source with security fields and engine ID for secure trap ingestion.
• Gateway Connections: Added a passthrough name-resolution option for gRPC gateway connections to support environments that require external resolver behavior.

Bug Fixes

• HTTP Pagination (JSONPath): Always processes initial response data even when no pagination URLs are present and handles null pagination URLs with debug logging to prevent data loss on final pages.
• HTTP Dispatcher Stability: Added the missing Start call, introduced early-exit checks for subscribe/unsubscribe when the component is not running, and corrected locking to prevent race conditions and inaccessible fleets.
• Source Metadata: Fixed missing SourceDef/metadata for Syslog, SNMP Trap, and SNMP Pull nodes to ensure consistent connector context and enrichment.
• Event Hubs Input: Added connection string format validation to prevent startup crash loops and provide actionable error messages.

Security

• OTLP gRPC Transport Hardening: Fixed a critical issue where the OTLP gRPC server accepted plaintext; with TLS configured, connections are now encrypted, and when TLS is absent the server logs a clear warning while using existing Edge Delta TLS configuration.

September 4, 2025

New Features

• SNMP Trap Source Node: Added an SNMP Trap source node that listens for SNMP trap events, enabling real-time monitoring and alerting from network devices and infrastructure components.
• SNMP Pull Source Node: Introduced an SNMP Pull source node (previously SNMP) that actively polls SNMP data from devices, providing comprehensive network monitoring capabilities with configurable polling intervals.
• Syslog Source Node: Added a Syslog source node supporting both TCP and UDP protocols for receiving syslog messages, enabling centralized log collection from network devices, applications, and systems.
• Splunk TCP Source Node: Introduced a Splunk TCP source node for receiving data from Splunk Universal Forwarders, facilitating seamless migration and integration with existing Splunk deployments.
• HTTP Dispatcher Component: Added an HTTP dispatcher component that enables multiple HTTP sources to share the same port with request filtering and routing capabilities, eliminating port conflicts and simplifying configuration.
• EDXEncrypt and EDXDecrypt Functions: Introduced OTTL crypto functions EDXEncrypt and EDXDecrypt providing AES-256-CBC/GCM encryption capabilities with local key management for secure data transformation within pipelines.
• EDXRedis Extension: Added Redis integration to OTTL enabling advanced use cases such as stateful processing, caching, and cross-pipeline data sharing.
• EDXDecode Function: Introduced an OTTL function EDXDecode to decode URL-encoded and hex-encoded strings, simplifying data normalization tasks.

Improvements

• HTTP Pull Source Enhancements: Added URL following mode for APIs that return content URLs, implemented RFC 5988 Link header parsing for pagination, enabled parallel fetching with configurable concurrency, added OTTL expression support for dynamic endpoints and headers, implemented header inheritance for paginated requests with security controls, and added support for OAuth2 client credentials flow.
• Splunk Integration Enhancements: Enhanced Splunk HEC source with configurable header extraction to attributes for preserving authentication tokens and metadata. Updated Splunk destination to support dynamic token override via OTTL expressions and allow empty index values for improved flexibility.
• Source Detection Performance: Refactored source detection to use on-demand processing with Redis backing, significantly reducing detection time and improving onboarding speed. Fixed concurrent access issues and excluded Edge Delta internal sources from detection.
• Telemetry Generator Enhancements: Enhanced the Telemetry Generator with support for multiple metric configurations per template and introduced combined log format with separators for improved usability and template management.
• OTLP Support Enhancements: Added JSON encoding support for OTLP-based sources and destinations with HTTP protocol, improving compatibility with various OTLP implementations.
• GCS Destination Improvements: Enhanced the GCS destination with fixed path prefix handling to honor configured bucket paths correctly. Enhanced authentication support for Workload Identity and improved nil data handling for increased stability.
• Connector Metadata Support: Extended connector metadata enrichment to File, Kubernetes, Exec, and port-based sources, providing consistent context across all source types.
• Validation and Error Handling: Added validation for MinIO endpoints to require protocol specification, preventing configuration errors. Centralized node upload failure reporting for consistent error handling across all destination nodes. Enhanced capture payload handling with size thresholds to prevent oversized payloads.
• OTTL Function Improvements: Added EDXEnv function for retrieving environment variables with fallback support, enabling secure credential management in configurations.
• HTTP Source Rate Limiting: Added configurable rate limiting for HTTP-based sources to control data ingestion rates and prevent overwhelming the pipeline.

Bug Fixes

• GCS Path Prefix: Fixed an issue where GCS destination ignored configured path prefixes, causing files to be written to incorrect locations.
• Nil Pointer Fixes: Resolved nil pointer dereferences in GCS destination strategy and Telemetry Generator node that could cause agent crashes.
• Compactor Test Data Race: Fixed a data race condition in Compactor unit tests improving test stability.

Security

• OAuth2 Authentication: Added OAuth2 client credentials flow support to HTTP Pull source and HTTP destination nodes, enhancing security for API integrations.
• Encryption Support: Implemented local key management with AES-256 encryption for sensitive data processing within pipelines.
• Header Security: Added same-origin validation and automatic blocking of hop-by-hop headers in pagination requests to prevent token leakage.

July 28, 2025

New Features

• Deotel Processor: Added the Deotel processor which converts OpenTelemetry formatted data back to simplified formats for non-OTLP destinations. This processor extracts core content from OTLP wrapper metadata, enabling integration with legacy systems, webhooks, and custom applications that don’t support OpenTelemetry format.

Improvements

• Source Detection Enhancements: Added trace-level logging for the source detection node to improve debugging and operational visibility. Included error or message fields in source detection results for improved feedback to the user interface. Improved log detection performance by immediately triggering detection tasks and retrieving Kubernetes logs via API (up to 1MB) for more accurate log rate calculations. Modified permission checks for Linux/Mac source detection to skip sources when not authorized and log skipped cases. Updated permission-checking logic to avoid use of syscall. Enhanced coordinator and gRPC integration to support sample collection for non-deployed inputs. Made error handling more robust and results more explanatory for the frontend. Added support for collecting and displaying live tail data from detected sources before pipeline deployment, enabling faster validation of data visibility in the UI.
• Telemetry Generator Template Improvements: Enhanced template preview to display realistic sample values for variables (e.g., IPs, UUIDs) instead of placeholders. Enabled automatic variable detection from pasted log samples, generating consistent and editable template variables for dynamic fields. Added support for dynamic variables in resource and attributes sections of telemetry generator templates. Integrated tool to generate and validate templates from sample logs, with validation errors reported for further manual editing.
• Telemetry Generator Trace Enhancements: Added backend support for calculating and emitting error ratios when generating traces. Added support for setting parent-to-child span ratios and trace depth in trace generation.
• GCS Destination & Archive Improvements: Allowed use of pod credentials (Workload Identity) for GCS destination by making credentials path optional if HMAC keys are also missing. Made HMAC key fields visible in the GCS destination configuration. Introduced a new GCS archive buffer strategy and improved stability by handling nil data gracefully.
• OTLP Source Log Attribute Support: Added extraction of severity text from OTLP log records when present, improving attribute fidelity.
• OTTL Context Filter Improvements: Added a flush check during shutdown, implemented a default timeout (10s) and maximum capacity limit (10MB), and introduced force flush and interval setting functionality from sequence nodes. Improved test node wait handling for stateful processors and fixed flaky tests.
• Destination Health Metrics: Added health metrics for backpressure, rejected, and timeout states to pipeline destination handling, providing deeper insights into pipeline health.
• Sample Processor Enhancements: Improved error handling for probabilistic sampling by allowing partial hash calculations when some default log fields are missing or malformed. Errors encountered during sampling are now joined and reported collectively.

Bug Fixes

• Sample Collector Shutdown Fix: Resolved an issue where the sample collector failed to stop properly, which could result in agent shutdown delays.
• Telemetry Generator Node Nil Pointer Fix: Addressed a nil pointer dereference when running the Telemetry Generator node.
• OTTL Context Filter Issue: Fixed the flushing logic for consecutive extended windows in the OTTL context filter, ensuring timely log emission. Corrected setting of the flush interval in OTTL context filter nodes when configured from sequence nodes.

July 14, 2025

New Features

• HTTP Pull/Output OAuth2.0 Support: Added support for OAuth2.0 Client Credentials flow to the HTTP Pull source and HTTP Destination nodes, enabling secure authentication with external services and improved connection security.
• Telemetry Generator Source Released: The Telemetry Generator Source node is released and the previous Demo Source and Demo Template Source nodes are deprecated.
• Distinct Count Aggregation in Aggregate Metrics: Added a distinct count aggregation method to the aggregate metric processor, supporting approximate mode (HyperLogLog) and group by key configuration for advanced metric analytics. Added an option to the aggregate metric processor to keep only fields used in group by, enabling privacy and data minimization in aggregations.
• Rate Limiter for Non-Push Sources: Added a rate limiter component to non-push based source nodes to better control data ingestion rates.
• KSM Metric Filtering and New Metric: Added support for filtering Kubernetes State Metrics (KSM) via configuration and included a new kube_pod_status_reason metric, improving Kubernetes observability.

Improvements

• Supported Environments and Fleets for Source Nodes: Updated supported environments, fleets, and subfleet types, improving pipeline compatibility checking and avoiding misconfiguration.
• Refactor for GCS Destination: Updated GCS destination to no longer check agent version before requiring HMAC keys or credentials path, simplifying user configuration and avoiding unnecessary incompatibilities. Updated GCS archive strategy to take credentials definition directly and improved flexibility for various authorization mechanisms.
• Component Tag & Metrics for Tailers: Added missing component keys to certain tailers and made obsReport a no-op for component health nodes, improving telemetry labeling and reducing noise.

Bug Fixes

• Filter Empty Condition: Fixed logic in the filter processor to correctly allow all items if the context condition is empty. Fixed an issue where OTTL expressions would return errors if top-level fields were undefined, allowing valid fallback behavior.
• Remove Unused __logical_src Field: Removed the unused field from pipeline items, reducing data size and eliminating unnecessary fields.

July 3, 2025

Improvements

• Add Warning for External ID with Assume Role: Added a warning when an external ID is provided along with an AWS assume role for cloud fleets.
• OTLP Output Header Support: Added support for custom headers in OTLP output, enabling integration with third-party destinations that require header-based authentication or metadata.
• Nil Check for Stats in Obs Report: Added a nil check for stats before using them in obs report for the Edge Delta pusher to prevent nil pointer exceptions when closing the agent during upload.

Bug Fixes

• CEL Transform Concurrent Map Write: Fixed a race condition in CEL transform that could cause the agent to crash due to concurrent map writes.

July 2, 2025

New Features

• Source-based Metadata Filtering: Added the ability to define and filter metadata fields for eligible source nodes, enabling more granular control over ingested metadata at the configuration level.
• Lookup Processor Matching Modes: Introduced new match modes (contain, prefix, suffix) for the lookup processor.
• TelemetryGen Source Node: Added a new TelemetryGen source node supporting metrics and traces with multiple template configurations, enabling richer and more customizable synthetic telemetry data generation.
• Securonix Output Node: Added support for Securonix SIEM as an output destination, allowing seamless integration with Securonix for security event ingestion.
• Router Node (OTTL-based): Introduced a new Route node supporting OTTL statements, expanding routing capabilities with advanced expression support.
• Expose Parallel Worker Count for Outputs: Exposed configuration for parallel request (worker) count in advanced settings for non-notifier output nodes, allowing for increased throughput customization.
• TLS Config for HTTP Pull Source: Added TLS configuration options to the HTTP Pull input node, enabling secure HTTPS data ingestion.
• Support for EndpointSlices in Helm Chart: Added native support for the EndpointSlices Kubernetes resource in the Helm chart to improve load balancing and scalability in Kubernetes environments.

Improvements

• Telemetry Provider Resource Pre-initialization: Pre-initialized resource and attribute maps within self telemetry, improving performance and reducing latency in metric bucketing.
• Self Telemetry Node Advanced Setting: Added an advanced setting to the self telemetry node to disable intermediate node telemetry, only emitting data from input and output nodes for streamlined reporting.
• Disable Old Telemetry Components: Disabled pipeline I/O stats and node health counting when new self telemetry is enabled, reducing resource usage and metric duplication.
• Deprecated Health Manager: Finalized removal of the health manager in favor of using obsreport and self telemetry for pipeline health metrics, simplifying telemetry collection and infrastructure.
• Parameterize File Tailer Settings: Made tailer buffer size and seek capacity for file tailing configurable via advanced settings and pipeline YAML, enhancing tuning options for high-throughput or resource-constrained environments.
• File Tailer Buffer Default Update: Updated file tailer to be unbuffered by default, reducing memory usage and preventing OOM scenarios, with an option to adjust via advanced settings.
• Support Numeric Types in Probabilistic Sampling: Enabled support for all numeric types in the priority field of the probabilistic sampling processor, improving flexibility in rule definition.
• GCS Destination Native Auth Support: The GCS output node now supports empty authentication fields for use with Google workload identity, allowing seamless integration without static credentials; backward compatibility retained with HMAC keys.
• Common S3 Path Format for Presigned Uploads: Updated presigned S3 upload endpoint to include destination name in the object key, preventing key collisions and improving upload reliability.
• Support Identifier-based pprof Ingestion: Enhanced pprof ingestion to associate data with unique agent identifiers in addition to host names, improving debugging and analytics resolution.

Bug Fixes

• Metadata Field Name Correction for Windows Event: Corrected incorrect field naming for Windows event sources, fixing compatibility and ingestion issues on Windows environments.
• Fix for Compactor and Coordinator Clients: Addressed bugs related to S3 path handling and coordinator client state refresh, improving upload reliability and connection management.
• Fix Winevent Tailer Field Traversal: Stopped sending Edge Delta-specific fields (ed.tag, ed.conf.id, ed.org.id) in logs and events that can be inferred from file upload paths, reducing data overhead and destination pollution.
• Self Log & ObsReport Gaps for Gateway Connection: Fixed gaps in self log and obsreport metrics for gateway connection, ensuring accurate telemetry and reporting.
• Path Extraction Logic in JSON Unroll: Updated JSON Unroll node to use proper OTTL and correct path extraction logic, preventing errors and supporting expected behavior.
• Output Error Log Rate Limiter Adjustments: Increased error log rate limiter interval for output nodes to 5 minutes and added granular error log rate limiters for processors and output nodes to reduce log spam and improve clarity.
• Rate Limiter for Self Log Uploader: Introduced a rate limiter to the self log uploader, preventing excessive log uploads and protecting backend resources.
• Error Log Rate Limiter Updates: Increased error log rate limiter interval to 1 minute for specific output scenarios to further control error log verbosity.

Security

• CVE-2025-30204: Upgraded the echo package to address a high severity vulnerability.

Performance

• Increase SQS Messages Per Request: Increased the maximum number of SQS messages downloadable per request, significantly boosting data throughput for SQS source nodes.
• Archive Buffer Refactor: Refactored archive buffering and strategy construction, allowing maximum buffer size and flush interval to be set at construction time and improving configuration for scalable data archival.

June 16, 2025

New Features

• Node, gateway, and coordinator pipelines: Introduced a new agent architecture that separates responsibilities across three distinct components: node, gateway, and coordinator pipelines. The node pipeline remains lightweight and focused on local data collection and preprocessing at the node or host level. The gateway pipeline acts as a centralized point for data aggregation, transformation, and advanced processing tasks such as trace tail sampling. The coordinator pipeline manages orchestration and configuration distribution across node and gateway agents, ensuring consistent behavior and scalability across large deployments. This modular structure improves performance, simplifies management, and enables more advanced use cases in distributed environments.
• Windows Event Source: Added a new Windows Event source node for collecting logs from Windows events, enabling observability for Windows-based node pipelines.
• OTTL Conditional Functions: Introduced EDXIfElse and EDXCoalesce custom OTTL functions for inline conditional operations and null coalescing, enabling ternary-like operations and fallback value handling.
• Add Trace and Span ID to Log Items: Added trace ID and span ID as top-level fields to log item payloads, enabling improved traceability and correlation across pipelines.
• Tail Sampling: Introduced the tail sample processor to filter and manage traces and spans by applying various sampling policies

Improvements

• Aggregate Metric Processor Passthrough: Introduced a passthrough on no match flag for the aggregate metric processor, allowing unmatched items to be forwarded to subsequent nodes in sequence processors.
• OTLP GRPC Connection State Management: Implemented connection state management, cleanup, and reconnection logic for OTLP gRPC outputs to prevent orphaned connections and address issues with excessive pings after config reloads.
• Data Transformation Hook for Output Strategies: Updated the output node strategy interface to encapsulate data transformation (compression/formatting) before sending to third-party destinations.
• Set Cluster Name Only from Environment Variable: The agent now sets the ed.cluster.name field exclusively from the environment variable to avoid ambiguity between different sources.
• Output Node Self Log and ObsReport Gaps: Ensured all output nodes record error logs and upload errors using both self logs and obsreport metrics.
• Expand Delete Empty Values Validation: Made validation error messages for the Delete Empty Field processor more descriptive and specific to improve user feedback.
• GZIP Node and HTTP Source Improvements: Added decompression support for HTTP sources and increased max item size to enhance GZIP handling, particularly with AWS Firehose data.
• Remove Log from ED Output: Cleaned up unnecessary extra logging in the ED output node for metrics.
• Modify Dimensions of Trace Data in OTLP Input: Adjusted the scope of resources for trace data in OTLP input for accurate dimension labeling.
• Update Validation for JSON Unroll Node: Fixed path extraction logic in the JSON Unroll processor to handle OTTL properly.
• Implemented a number of changes to improve graceful agent shutdown.
• Self Telemetry Memory Usage by cgroups: Migrated self telemetry memory usage reporting from cAdvisor to cgroup-based calculation.
• Memory Working Set Fallback Scenario: Fixed fallback logic for memory working set determination in self telemetry to handle scenarios where cgroup data is unavailable.

Bug Fixes

• Fix Dedup Processor Timestamp: Resolved an issue where deduplicate processor output items had incorrect timestamps due to ordering in processing logic.
• Fix Sequence Node OTTL Data Type Filtering: Corrected the sequence node’s behavior when filtering by data type with OTTL filters to avoid unexpected drops.
• Fix JSON Unroll Path Extraction: Updated the JSON Unroll node to skip CEL expressions when extracting paths, resolving path extraction bugs.
• Do Not Build sysctl on Windows: Excluded sysctl build for Windows to fix Windows image build failures.
• Prevent Crash on Source Detector During Shutdown: Applied context cancellation to stop blocking source detector processes on config reload shutdown, preventing agent crashes.

Deprecations

• Deprecate Component Health Node: Deprecated the ed_component_health_input node in favor of relying on self telemetry for health metrics.

May 29, 2025

Bug Fixes

• Version Requirement Bug: Fixed an issue with the required agent version for the CrowdStrike FDR source node.

May 29, 2025

New Features

• Add CrowdStrike FDR Source: Introduced a new log source for CrowdStrike FDR, enabling ingestion of logs from AWS SQS with specific message formatting and broadening support for security telemetry pipelines.
• Log Parsing Mode for Kubernetes Input: Enabled a parse JSON source option for the Kubernetes Log source node.
• Add Agent Identifier to Self Telemetry: Enriched all self-telemetry data (metrics and logs) with the agent identifier and updated to use node name as the host name for Kubernetes components, improving observability and traceability.
• Add ed.pipeline.node.category & Component to Agent Self Logs: Appended pipeline node category and component attributes to agent self logs, improving the granularity of telemetry data for input nodes.

Improvements

• Log Threshold Monitor Performance: Optimized internal logic to avoid repeated map flattening and unnecessary computation, substantially improving evaluation speed and reducing memory pressure.
• Self-Log Uploader Tag Map Initialization: Fixed occasional failures in the self log uploader by ensuring other tags map is always initialized, improving reliability of self-logs.
• Deduplicate Logs Item-Time Batching: Introduced batching based on item timestamps (instead of system time) for the Deduplicate Logs processor, resulting in more accurate batched results and improved behavior with out-of-order data.
• Aggregation Temporality Standardization: Modified aggregation temporality to be consistently lowercase, ensuring protocol compliance and consistent behavior across integrations.
• Removed src_type: Removed the src_type attribute to all destinations and updated rollup rules to use category-based rules, streamlining data and complying with updated metric structures.
• Name Optional for Aggregate Metric Rules: Made the name field for aggregate metric rules optional, enabling support for rollup mode within the aggregate metric processor and making aggregations more flexible for users.
• Move Trace Attribute Enrichment to Output: Shifted trace attribute enrichment for Edge Delta-specific fields from the input/tailer logic to the output destination logic, centralizing attribute management and simplifying tailer implementations.

Bug Fixes

• Log Item Body Return Result Correction: Fixed incorrect handling of log item body for non-string and non-byte array types, resolving compatibility and processor issues introduced with the previous update to support any type.

May 19, 2025

Improvements

• Source Detection: Enabled asynchronous emission of sources detected events for new source detection payloads to improve responsiveness and system efficiency. Excluded sources with EdgeDelta in their name from source detection. Fixed concurrent map read/write issues for source detection, enhancing data integrity and system stability.
• Validate Cloud Fleet HTTP Source Ports: Added validation to restrict HTTP source ports in Cloud Fleet to only allow ports 80 or 443.
• Logging Behavior Refactoring: Refactored logging behaviors for rollup agents, compactor agents, and log to pattern nodes to suppress frequent error logs and improve user experience.
• Helm Chart Repository URL: Updated Helm chart repository URL to improve accessibility and ensure users are accessing the latest charts.
• Drain on Stop for Pusher in GCL: Introduced a drain on stop mechanism for pusher in GCL to ensure data is not lost during stop operations.
• TelemetryGen Source: The Demo Template Source has been renamed as the TelemetryGen Source (currently beta) with demo templates for logs, plus new templates for metrics and traces.

Bug Fixes

• Live Capture: Fixed the missing live capture status, improving visibility and user experience. Resolved the issue with live capture task status updates in the coordinator relay server to ensure accurate task tracking.
• Sample Collector: Addressed a data race condition in the sample collector that could lead to crashes. This fix improves stability and reliability. Introduced checks to handle nil resource map in sample collector, preventing potential panic scenarios.
• Azure Log Analytics Reporting: Fixed an issue where Azure Logs Analytics was reporting 0 bytes out, ensuring accurate metric reporting.
• Mount Path Field Requirement: Made the mount path field required in the UI, aligning with validation requirements and preventing misconfigurations.

May 5, 2025

Enhancements

• Compactor Improvements:
  • Changed Compactor deployment model from StatefulSet to Deployment.
  • Fixed Horizontal Pod Autoscaler (HPA) behavior for compactors.
  • Added fleet_type and agent_type to heartbeat metrics.
  • Adjusted agent identifier handling and internal constants for self-logging.
• GCL Output Flexibility: The Google Cloud Logging (GCL) output node now supports both raw strings and OTTL expressions in the log_name field, enabling more flexible static bucket routing.
• Cluster Resource Tagging: Added support for the ED_CLUSTER_NAME environment variable. When set, this value is injected into the ed.cluster.name resource field, providing better pipeline grouping across deployments.
• TLS Support for OTLP Output: Introduced TLS configuration support for OTLP output during node creation. This includes improved validation for TCP and UDP port usage across inputs.
• Self Telemetry Node Promotion: The self_telemetry node has been promoted to Released status. Deprecated: ed_pipeline_io_stats_input, ed_node_health_input, and ed_agent_stats_input.

Fixes

• OTEL Alignment: Deprecated legacy formatting and custom tag/label configurations in the OpenMetrics, SumoLogic, and Slack output nodes to streamline with OpenTelemetry (OTEL) semantic conventions and improve configuration consistency.
• Metric & Trace Conversion Fixes: Resolved recent regressions in metric and trace conversions. Histogram metrics now include additional fields for improved fidelity.
• S3 Input Reporting: Fixed incorrect reporting of ed.pipeline.node.read.bytes in the S3 input node.
• Volume Name Mismatch: Fixed volume naming conflict that occurred when the HTTP recorder feature was enabled.
• OTLP Output Proto Size: OTLP output now reports the size of outgoing protobuf messages, providing visibility into traffic volume.
• Port Reuse Restriction: Pipeline validation now prevents the use of the same port by multiple source nodes within a single pipeline, ensuring safer network configurations.
• Output Attribution in Multiprocessors: Packs can now attach and propagate their output names under multi-processor pipelines.
• Agent Metric Cleanup: Removed agent metrics in favor of pprof data:
  • ed.agent.go.routine.value
  • ed.agent.memory.to_be_freed
  • ed.agent.gc.target
  • ed.agent.gc.pause_time
  • ed.agent.gc.count
  • ed.agent.gc.forced_count

Security

• CVE-2025-22872 Fix: Upgraded golang.org/x/net to v0.38.0, resolving a Medium severity CVE related to incorrect tag parsing in the tokenizer.

April 18, 2025

New Features

• Introduced Multiprocessor nodes, enabling multiple processors within a single node for advanced pipeline processing.
• Added Live Capture support to allow real-time data ingestion and inspection within pipelines.
• Added a new Google Cloud Logging output node that uses OTTL expressions for configuration; the previous node is now marked as Beta for existing users.
• Released the demo template source to Beta.

Enhancements

• Updated OTLP conversion logic to group items by resource hash (instead of service name), use hex decoding for IDs, include span events, and improve scope batching to support mixed scopes.
• Made the json_field_path field required in the JSON Unroll node for consistency with node validation requirements.
• Added validation to the HTTP Pull endpoint setting to prevent leading and trailing spaces that could cause failures.
• Added failure path handling for additional processors, improving reliability of multiprocessor node construction.
• Enabled reporting of AWS S3 and Azure Blob storage traffic via self-telemetry, addressing discrepancies in metrics dashboards and editors.
• Improved error visibility by reporting processor and strategy errors with self-telemetry.
• Made name and interval required fields in the aggregate metrics processor for more consistent configuration validation.
• Improved OTTL support for nodes:
  • Updated the Prometheus exporter output node to use OTTL expressions for custom metric labeling, enabling flexible configurations.
  • Updated the Loki output node to use OTTL expressions for custom labels, allowing dynamic configuration.
  • Updated AWS Cloudwatch output node to support OTTL expressions for fetching data attributes, with backward compatibility for CEL.

Bug Fixes

• Fixed missing ED output data in the cost optimization API by ensuring outputs are checked in both output and ED output node lists.
• Prevented duplicate data in sequence processors by avoiding passthrough when termination or condition mismatch occurs.
• Fixed processor data type filtering so that data is not dropped when the incoming type does not match the processor’s supported types.
• Fixed double logging in some cases for compactor receive errors by improving error handling.
• Tightened validation of custom tags in the Sumo Logic Pusher by dropping invalid key-value pairs before sending.
• Fixed config fetch error handling so errors are now properly reported instead of being ignored.
• Fixed processor data type handling so unsupported item types pass through the processor rather than being dropped.

April 3, 2025

Features & Improvements

• OTLP Destination: The OTLP destination node is now in Beta. This enables broader observability pipelines with enhanced compatibility.
• Source Detection: Added a parameter to override the environment type in Source Detection, improving testability and alignment across deployment types.
• Health Metrics Expanded: Enabled health metrics by default. Filled in previously missing metrics: ed.pipeline.node.compactor_client.request.total, ed.pipeline.node.http.request.successful, and ed.pipeline.node.http.request.failed.
• OTTL Dynamic Indexing Support: Upgraded OTTL to v0.122.0. Added dynamic indexing support for maps and slices. Adjusted custom parser logic for full compatibility.
• Log Item Body Handling Improvements: Added support for any type in the body field of log items. Changed log item body representation from byte array to string, simplifying parsing and processing.
• Compactor Logging Clarity: Improved internal logging by adding contextual identifiers to repeated error messages in the compactor, making traceability easier.

Bug Fixes

• OTLP Output Buffer Serialization Fix: Switched OTLP pusher to use protobuf marshal/unmarshal to avoid gob serialization conflicts during buffering.
• Config Reload Stability: Added a safeguard against nil pointer dereference during configuration reloads.
• JSON Subsequence Parsing: Fixed JSON parser to properly handle nested inline objects, improving consistency with complex structures.
• Self Log Enrichment Fixes: Resolved missing self log attributes for inputs, outputs, and processors. This ensures consistent telemetry enrichment across the board.
• Health Item Cleanup: Removed lastError field from health entries to prepare for full deprecation of component_health node.
• Sample Node Field Path Consistency: Removed unnecessary item wrapping in Sample node to match behavior of other OTTL nodes.
• Demo Log Formatting: Prevented unwanted newline characters from being appended to demo log entries.
• System & Container Traffic Stats: Fixed inaccurate system and container stats where traffic was reported as zero.

Security

• [CVE-2025-30204] JWT Library Upgrade: Patched a high-severity vulnerability by upgrading github.com/golang-jwt/jwt to v4.5.1.

March 24, 2025

Breaking Change

• New helm chart variables have been added and the kubernetes manifest has been deprecated. To install to this version and above, please perform a clean install using the Helm command provided in the interface.

Enhancements

• Improved HTTP Pull Input Logging: HTTP Pull input logs now include response status code, response headers, response body size, and request latency. Self-telemetry reporting is also added.
• Validate config with environment type: Added validation for environment and pipeline type, which restricts certain source nodes to specific environments.
• Kubernetes Traffic Node Renamed: Changed display name of k8s traffic node to k8s service map source to make it clear it powers the Service Map page.
• Compactor Telemetry Improvements: Set compactor stats to use self telemetry node detail; Changed compactor metrics to gauges; Use obsReport for compactor stats.
• Added new ed.agent.identifier field for heartbeats.
• Added ed.agent.identifier field for identifying agents instead of host.name.
• Removed ED_HOST_OVERRIDE from Helm chart. This prevents overriding the hostname.
• Added ED_K8S_POD_NAME, ED_K8S_NAMESPACE_NAME, and ED_K8S_NODE_NAME environment variables to enhance agent identification in Kubernetes environments.

Performance Improvements

• Optimized Pod Listener Exports: Use pointers instead of concrete structures for pod listener’s exports.

March 17, 2025

New Features

• HTTP Pull Source Node: Introduced the HTTP Pull source node in Beta. It allows the agent to pull data from an endpoint by sending HTTP requests to it.
• OTTL Type Introspection: Added EDXDataType custom OTTL function for runtime type inspection of fields, enabling conditional processing based on data types.

Enhancements

• Prometheus Exporter Node Refactor: Refactoring of the Prometheus exporter nodes for improved performance and maintainability.
• Self Telemetry Integration for Agent Stats: Integrated self-telemetry provider for ed_agent_stat, ed_container_stat, and ed_system_stat nodes, now utilizing the new metric naming convention. Enabled agent stats collection with self telemetry by default and integrated obs_report for more efficient data reporting.
• PPROF Settings Improvement: Added additional settings for PPROF within agents; agents can now fetch hosts and types from the API to publish PPROFS accordingly.

Fixes

• Counters Management: Fixed counters not stopping when the exporter was stopped, ensuring a smoother operational process.
• Host IP Reporting: Fixed missing host IP cases in metric resources, ensuring completeness of metric reporting data.

March 10, 2025

Features & Enhancements

• Health Metrics for All Nodes & Components: Added health metrics to all nodes and components.
• Support for Compression/Decompression in OTTL: Added EDXCompress, EDXDecompress, and EDXUnescapeJSON custom OTTL functions, supporting GZIP, Snappy, and ZSTD compression, plus handling of multiply-escaped JSON strings.
• Enable pprof from UI with TTL per Fleet: Added the ability to enable pprof profiling from the UI with a TTL per fleet, improving visibility into system performance.
• Add Severity Text to Pattern Item Attributes: Included severity text in pattern item attributes for better categorization.
• Fleet Type & Environment Support in Nodespecs: Added supported pipeline types and environments to nodespecs to ensure proper input-to-pipeline compatibility.

Telemetry & Metrics Improvements

• Self-Telemetry Enhancements
  • Deprecated node_health input node in favor of self_telemetry.
  • Ingested new dimensions (ed.source.name, service.name) for outgoing metrics.
  • Ensured obs_report integration tracks inbound/outbound traffic from ed_debug_output, prometheus_exporter_output, and other outputs.
  • Added __metric_category: self_metric/diagnostic to diagnostic telemetry.
  • Removed double ingestion of self-diagnostic metrics by differentiating between classical and granular metrics.
• Updated Size Calculation for outgoing_bytes.sum Metric: Aligned data size calculations with log data size calculations in the archive, using a byte stream approach for consistency.
• Set Timestamp for OTLP Input Log Type: Ensured logs received via OTLP input have a valid timestamp when missing.

Bug Fixes

• Fix obs_report Usage in ed_output: Adjusted telemetry provider initialization to resolve obs_report consumption issues in ed_output.
• Fix Certain Processors Not Being Addable to Packs: Resolved an issue preventing specific processor nodes from being added to packs (formerly called compound nodes).
• Fix Validation for L2M Field Dimensions: Added missing validation checks for L2M processor dimensions.
• Fix Issues with JSON Marshalling of Resource Fields: Changed resource fields to use struct pointers instead of concrete structs, preventing empty object serialization.

Security Fixes

• Fix Multiple CVEs: Addressed security vulnerabilities: CVE-2024-45339, CVE-2025-22868, CVE-2025-22869.

Other Improvements

• Optimize FlattenAnyToString Method: Improved memory allocation and execution time for FlattenAnyToString.
• Replace gopsutil/v3 with gopsutil/v4: Upgraded gopsutil to v4 to resolve system stats collection issues on macOS.
• Update Nodespecs for Elastic & Datadog Destinations:
  • Fixed formatting issues in Elastic destination link.
  • Standardized quotes in Datadog destination nodespecs.
  • Added a Makefile target for creating nodespecs.
• Remove Default GOMEMLIMIT in Helm: Removed the default GOMEMLIMIT value in Helm as auto-memlimit is now implemented.

February 24, 2025

New Features

• Metric Aggregation: Introduced the aggregate metric node to batch and aggregate metrics.
• Extract Metric Node: Introduced the extract metric node, improving data extraction capabilities in pipelines.
• Datadog Source Node: Released for more comprehensive data ingestion from Datadog agents.
• Prometheus Source Node: moved to released status.
• Sampling Node: moved to released status.
• SCC Support in Helm: Added support for pod security context configurations in Helm to simplify deployment in OpenShift environments.
• DNS Overrides: Added support for DNS service configuration environment variable overrides.

Enhancements

• Self Telemetry Optimization: Optimized self telemetry to reduce CPU and memory consumption. This includes improvements in how telemetry maps are handled and passed.
• Logging Level Control: Updated self-telemetry settings to use internal logging levels to prevent verbose and unnecessary logs when pipeline log level is set to “debug”.
• Template Reuse in Logs: Improved support for Demo node format templates to allow reuse of variables within logs and added enhancements for timestamp and number formatting.
• Node Status and Environment Compatibility: Defined and added support for specific environments in source node configurations, enhancing pipeline configuration validation and operational restrictions.
• Pattern Clustering with Log Levels: Updated the pattern clustering algorithm to incorporate log levels, enabling more accurate clustering of logs with similar severity.
• Self-Telemetry Integration: Integrated telemetry reporting across Prometheus input and tailers for enhanced monitoring of inbound and outbound data flows.
• Kubernetes Metrics Collection: Refactored the include/exclude logic of k8s metrics source node to support regex patterns and improve component-specific metric filtering. The exclude block takes precedence, and if both blocks are empty, only kube_state_metrics will be scraped.

Bug fixes

• Fix for OTLP Collector Issue: Resolved metric conversion issues between OTLP Collector and OTLP source, ensuring consistent metric handling.
• Pattern Source Fix: Corrected an issue where the source type was not properly set for generated patterns.
• Configuration Adjustments: Streamlined configurations within Helm charts to use a unified repository field for container management, reducing redundancy.
• Metric Discrepancy Resolution: Addressed metric discrepancies by fixing type conversions, ensuring alignment between different metric input streams.
• OTTL in Route Node: Changed Route node to use standard OTTL syntax when in OTTL mode.

February 10, 2025

New Features

• Demo Templates: Introduced new demo node templates for comprehensive data generation using existing log types such as Apache common and VPC flow.
• Advanced Log Pattern Configuration: Added configuration settings for log to pattern nodes to enhance pattern similarity management.

Improvements

• Stateful Metric Processing: Enhanced Prometheus input node with stateful metric processing.
• Otel Log Item Integration: Updated the self-logger to use the OpenTelemetry (OTel) Log Item.
• Item Data Type Refinements: Converted integer fields to int64, aligning with OTTL requirements.
• Component Lifecycle Logging: Refactored lifecycle logging for better clarity and information.
• Cloud Fleet Node Type Restrictions: Implemented restrictions on node types for cloud fleets.
• Kube State Metrics Leadership Management: Applied leader election to manage Kube State metrics to prevent duplicate metric ingestion.

Bug Fixes

• DataDog Logger Upgrade: Added lifecycle logs for the DataDog mapper to increase transparency in processor events.
• JSON Demo Template Fixes: Corrected JSON formatting for Demo logs to adhere to standards.
• Self-Telemetry Shutdown Optimization: Fixed the shutdown behavior in the Self Telemetry Provider to prevent redundancy.
• CA Certificate Error Resolution: Addressed CA certificate errors in Target Allocator, ensuring operational reliability.
• Performance Improvements with Pointers: Utilized pointers for rollup items instead of concrete structures to enhance performance.
• Kubelet CI Metric Adjustments: Discontinued unnecessary scraping of rest_client_request_duration_seconds from Kubelet.
• Extended eBPF Byte Passing: Increased bytes passed from the kernel to user space in eBPF to resolve URL truncation issues.
• K8s Event Emission Optimization: Formatted Kubernetes events as ED events to improve efficiency and reduce memory usage.
• Debug Pusher Fix: Resolved issue of debug pusher going no-op when leader election is disabled

Miscellaneous

• Prometheus Integration Improvements: Integrated health details for the Target Allocator component into Prometheus.
• KSM Metric Enhancements: Added new KSM metrics for container restarts; removed default L2P field values.
• Prometheus Histogram Handling: Fixed handling for min and max values in Prometheus histograms.
• Additional Prometheus Logging: Added extra logs for Prometheus transactions for better traceability.
• Node Type Addition: Added node type to agent self log resources.

January 27, 2025

New Features

• Crowdstrike Falcon LogScale Destination: Added support for sending log data to Crowdstrike Falcon LogScale.
• IBM QRadar Destination: Introduced an IBM QRadar output using the Syslog protocol, supporting multiple formats like Universal LEEF.
• Dynatrace Destination: Added Dynatrace as a streaming destination for logs, metrics, and trace data, enabling seamless data flow to Dynatrace environments.
• OTTL Extension: Introduced a new function edx_delete_empty_values() OTTL extension to delete keys with empty values.
• New Node: Introduced the Suppress Processor node to manage and reduce redundant log data.

Improvements

• Kafka Source: Default values for consumer group IDs are now automatically generated if absent in Kafka Source.
• Logging Enhancements: Refactored logging to include missing logs and standardized error tracking across nodes and components for better diagnostics and observability.

Bug Fixes

• Prometheus Configuration Validation: Removed the restriction for empty configuration validation in Prometheus settings.
• Validation of Required Nodes: Addressed an issue with pack configurations validation to bypass unnecessary checks when outputs are not defined.

Miscellaneous

• Agent Self-Telemetry: Introduced a central telemetry provider and observability report components.
• Load Distribution of Targets to Agents: Enhanced target load distribution to agents, ensuring efficient target allocation.

January 13, 2025

New Features

• Gigamon AMX Demo Source: The demo source node can now emit Gigamon sample data.
• Microsoft DNS Demo Source: The demo source node can now emit Microsoft DNS sample data.
• Minimum Agent Version Support: Configuration validation now supports a minimum agent version for nodes. This enhancement ensures that pipeline components are compatible with the agent version in use.
• OTTL Array Type Handling: Enhanced OTTL to accommodate other array types and improved error identification by including processor names in error messages. Users can now map Edge Delta data to their schema tables more seamlessly.

Improvements

• Azure Blob Storage Refactor: Updated Azure Blob and Local Storage implementations. The refactor also includes the addition of a logger as a parameter for the S3 implementation.
• GOMEMLIMIT Enhancements: Introduced default GOMEMLIMIT values for all components managed under Helm, with added support for automatic GOMEMLIMIT configuration if not predefined.
• GCL Resource Label Handling: Changed GCL destination configuration to manage resource labels via whole field mapping instead of individual entry mapping.
• OpenTSDB Processor Logging Reduction: Reduced verbosity in OpenTSDB logging.
• Kubernetes Traffic Metrics: K8s traffic metrics now include domain information for enhanced data granularity.
• Kubernetes and OpenTelemetry Libraries: Upgraded Kubernetes, OpenTelemetry, and other auxiliary libraries to their latest versions.

Bug Fixes

• Node I/O Stats Discrepancy Resolution: Addressed issues causing disparities in pipeline views related to node I/O statistics.
• Mutating Processors Item Sizes Compliance: Updated processors to better respect item size constraints.
• OTTL Transform Node Casting: Eliminated unnecessary casting to BaseItem in the OTTL Transform node, which resolves a panic issue encountered after the Destination Transform.
• Kubernetes Metrics Node Configuration: By default, kubelet, cadvisor, and node_exporter metrics are now excluded in the k8s_metrics node. Users can re-enable them if needed by updating the exclusion parameters.

December 30, 2024

New Features

• Prometheus Remote Write Destination: Introduced a new destination node for Prometheus remote write.
• Support for Isolated Node Testing with JSON Items: Added an endpoint to enable testing of individual nodes using item data.

Improvements

• Enhanced Port Validation: Updated port validation range for TCP/UDP source node configurations. The new range now reflects permissions for non-root agents, setting it from 1024-65535.
• New Relic Endpoints Configuration: Added support for custom log and metric endpoints within New Relic destinations.
• ED Data Schema Mapping: Introduced a new OTTL function edx_map_keys to facilitate mapping ED data to user-defined schemas.
• GCL Rehydration Enhancements: Improved the GCL implementation by considering API request size limits and performing pre-batching to avoid errors.

Bug Fixes

• Node Health Metrics: Fixed a domain issue in node_health histogram metrics by adding ed.domain to the missing metrics.
• Docker Source Container Restarts: Resolved issues with Docker source by resetting the log reader upon container restarts and introducing a mechanism to prevent excessive restart attempts within a short time frame.
• Corrected convert_timestamp Function: Addressed an incorrect scaling issue in the convert_timestamp CEL macro time.UnixMilli.
• Splunk Destination Optimization: Optimized raw submission to Splunk by using a struct as a key for grouping, improving the efficiency of the process.
• Incoming and Outgoing Throughput Metrics Adjustments: Added two new metrics dimensions, ed.source.type and ed.destination.type, and corrected the domain issues for these throughput metrics.
• Log Threshold Metric Update: Renamed the log_threshold_monitor_metric.histogram to ed.pipeline.l2m.log_threshold to align with new naming conventions.
• Other fixes: Destination buffer configuration for Azure Log Analytics; Fixes for Azure Sentinel concerning destination configuration; Constants for path variables.

Security

• Version Upgrade for golang.org/x/net: Updated the package version to v0.33.0 to address a CVE with high severity.

Maintenance

• Azure Sentinel Configuration Validation: Added validation to the destination buffer configuration for Azure Sentinel destinations and other destination nodes.
• Integration Tests Migration: Continued efforts in updating and migrating integration tests to v3 native standards, improving overall testing frameworks.

December 17, 2024

New Features

• Amazon CloudWatch Destination Node: Introduced a new Amazon CloudWatch destination node. This feature includes dynamic namespace, group, and stream name configurations, enhancing integration flexibility.
• Azure Log Analytics Destination Node: Implemented a new Azure Log Analytics destination node.
• Microsoft Sentinel Destination Node: Added a new destination node for Azure Sentinel.
• EDXEncode OTTL Function: Introduced the EDXEncode OTTL function for string-to-byte array conversions, such as modifying log bodies.

Enhancements

• Service Name Dimension in Metrics: Enhanced the IOMetricCollector to support the service.name dimension in metrics.
• Generalized NaN and Inf Handling: Extended the logic for discarding NaN and Inf values across all metric-related tailers.
• Node Health Metrics Update: Added ed.pipeline.node.type to node_health metrics and updated rollup rules to incorporate this new label.
• Dynamic Resource Type for GCL: Supported dynamic resource type definition for the GCL destination node.
• Logging Directory Customization: Introduced ED_LOGGING_DIR environment variable to allow users to override the default logging directory of the agent.

Bug Fixes

• Prometheus Collector Synchronization: Resolved a stop procedure issue in the Prometheus collector by removing unnecessary synchronization variables.
• Proper Message Splitting for GCL: Improved message splitting for the GCL destination node.

Security

• CVE-2024-45337 Mitigation: Updated golang/x/crypto package to version 0.31 to address a high-severity CVE.

Miscellaneous

• Metric Naming and Schema Updates: Renamed metrics and functions to align with OTEL schema changes, including ed.pipeline.<read/write>_lines to ed.pipeline.<read/write>_items, and updated item-related function names for consistency.
• Internal ED Destination Features Removal: Completed the removal of the features parameter from internal ED outputs, streamlining data handling processes.
• Logging Optimization: Reduced excessive logging in the Log to Pattern node by implementing error rate limiting and adjusting log levels for operational messages.

December 2, 2024

New Features

• OTTL Condition Support in Route Node: Introduced OTTL condition evaluation to the route node, enhancing routing capabilities with conditional logic.
• eBPF Support for Azure and Azure CNI: Added support for eBPF on Azure and Azure CNI, including improvements for the eBPF verifier and error logging for unsupported CNIs.
• This agent release enables the Edge Delta service map and Trace Explorer.

Enhancements

• Kubernetes Traffic Metrics Naming Convention: Applied new metric naming convention to Kubernetes traffic metrics.
• Datadog API Key Handling: Updated Datadog destination configuration to use API key headers instead of URL parameters.

Bug Fixes

• Integration Connectivity Test Fixes: Resolved issues with the Integration Connectivity Test API for Splunk, Kafka, and Webhook integrations.
• OTLP ed.domain Fix: Corrected the ed.domain value for OTLP input.
• NaN and Inf Values in OTLP Metrics: Implemented a condition to discard NaN and Inf values from incoming OTLP metrics.

Miscellaneous

• Removal of Classic Kubernetes Metrics: Deprecated old Kubernetes metrics in favor of granular metrics.
• Retirement of Old License Trial Checker: Removed the old license trial checker job and Influx Throttle references from the agent.
• ED Schema Cleanup: Eliminated remnants of the old ED schema.
• Kubenet Support on Azure AKS: Enhanced support for kubenet on Azure AKS with eBPF tailing, including testing and development for the Azure CNI plugin.

November 19, 2024

New Features

• Consolidated Edge Delta Destination Node: Introduced a new consolidated Edge Delta Destination node that ingests logs, metrics, traces, and other data items for the Edge Delta back end.
• Log Deduplication Processor: A new deduplication node has been implemented to aggregate identical logs over a configured interval and emit a single log with the count of deduplicated logs.
• Google Pub/Sub Input Support: Added Google Pub/Sub input implementation for v3, providing seamless transition from v2 to v3.

Security

• OpenSSL and Golang Crypto/TLS Decryption: Enhanced Kubernetes trace and traffic by adding support for OpenSSL and Golang Crypto/TLS decryption using eBPF.

Enhancements

• Granular Metric Controls: Introduced kill-switch controls for critical pipeline nodes, allowing selective granularity in metric ingestion.
• Event Type Alignment: Aligned event type and subtype for Kubernetes traffic and trace data with OTLP standards.
• Custom Editor Functions: Introduced new custom editor functions (edx_delete_keys, edx_delete_matching_keys, edx_keep_keys, edx_matching_keys) to the OTTL library for enhanced key and pattern handling.
• Source Attribution in Metrics: Added ed.source.name and ed.source.type to enhance agent metric insights.

Bug Fixes

• Log to Metric Avg Option: Added an avg option to the Log To Metric node to support averaging of log metrics.
• Domain Fix for Pipeline Metrics: Corrected the ed.domain value for pipeline metrics to ensure accurate categorization across all entries.
• External Pusher Features Removed: Removed features parameter from all output nodes to streamline data handling.
• Cloud Fleet Validation: Enhanced configuration validation for cloud fleets and backend scenarios to prevent invalid customizations.
• Grok Library Consistency: Standardized grok library usage across OTTL and pipeline for consistent processing.
• Tracer/Connection Monitor Logging: Reduced excessive logging.

Miscellaneous

• Kill-Switch Optimization: Extended agent metrics kill-switch controls for new granular metric naming conventions. Added new Kubernetes metrics with granular naming controlled via kill-switches.
• eBPF Process Inclusion: Incorporated eBPF process support for OpenSSL parsing.

November 5, 2024

New Features

• Domain for K8s Metrics: Introduced a domain attribute for Kubernetes metrics to enhance data categorization.
• Node Status Updates: Promoted several nodes including Splunk HEC, Kubernetes Trace Source, Top-k, OpenTSDB Parser, Ratio, and Generic Transform to Stable status.
• Cisco ASA Log Formatting: Updated Cisco ASA logs in the Demo source for improved compatibility and standardized formatting.
• Multi-Select Option Support: Added UI support for multi-select options, initially applied to the Log to Metric node.
• OTTL Custom Functions: Added EDXExtractPatterns and EDXParseKeyValue OTTL custom functions. EDXExtractPatterns enables dynamic regex patterns from field references. EDXParseKeyValue handles duplicate keys in key-value pair strings by supporting multiple merge strategies such as first, last, append, concat, and indexed.

Security

• Agent running as a service in Linux OS: The agent service can now be run by a non-root user. To upgrade Linux fleets to this agent version or higher, uninstall the existing root-user fleet and install a fresh non-root instance.

Enhancements

• CEL Expression Evaluation Optimization: Reused evaluation context across expressions to minimize CPU and memory usage.
• Data Item Type Casting: Streamlined and optimized item processing to eliminate redundant memory and CPU usage across various item types.
• Lookup Optimization: Optimized memory usage and processing efficiency for lookup tables and fixed handling of tables with duplicate keys. Optimized process by prioritizing rows-first search during table lookup operations, streamlining resource usage.
• Extended User-Agent header to include version, OS, architecture, and Go version for detailed reporting.
• Severity and Timestamp: Enhanced GCL by supporting severity and timestamp extraction.
• Log Sentiment Efficiency: Reduced CPU workload by calculating log sentiment scores only upon discovery of new pattern variations.

Bug Fixes

• eBPF Load and Reload Fixes: Restored functionality in eBPF attaching under reload scenarios without waiting for pod change events.
• Metric Name and Mapping Fixes: Addressed incorrect metric mappings for ed_k8s_metric_container_network_transmit_bytes and removed outdated metric paths.
• Enhanced label validation for Loki, Prometheus and GCL destinations.
• Log Duplication: Fixed duplication of logs to debug destination.

Miscellaneous

• Metric Refactoring: Refined metric ingestion with new naming conventions and deprecated extraneous pathways.
• Namespace Information: Added net peer namespace to Kubernetes Traffic and Kubernetes Trace nodes.

This release focuses on optimizing system performance, enhancing security measures on Linux services, and broadening metric categorization to enhance data clarity.

October 21, 2024

New Features

• Kubernetes Trace source: Introduced a beta feature for tracing in Kubernetes using the ed_k8s_trace node. This feature utilizes eBPF to monitor specified Kubernetes namespaces.
• OTTL Transform Node: Released an ottl_transform node to beta. It can transform data items using OTTL statements.
• Cisco ASA Format Support in Demo source node: Enhanced demo source to support Cisco ASA log formats, addressing common variants to improve compatibility.
• Dynamic Pattern Location for grok and Regex filter nodes: Users can now specify locations for patterns within data items using pattern_field_path, enhancing pattern matching capabilities.

Bug Fixes

• Halt Uploading Unused Diagnostic Health Data: Stopped the upload of redundant diagnostic health data to optimize resource usage.
• Self Logs Upload Context Issue: Fixed a problem where the main context cancellation prevented self log uploads during agent shutdown by switching to a flush context.

Enhancements

• Custom Installation Path Support: Unix-based systems can now set a custom installation path through the ED_INSTALL_PATH environment variable, increasing flexibility in deployments.
• Removal of Preset Features Parameter: Simplified nodes by removing preset features, with default outputs now enabling all features unless otherwise specified.

Metric Improvements

• Kubernetes Metric Enhancements: After a comprehensive inventory review, unused Kubernetes metrics were removed and missing metrics added to ensure a robust cluster summary.
• OTLP Metric Category & Rollup Improvement: Introduced metric categorization in OTLP inputs for direct passthrough in roll-up services, streamlining metric management and discovery.

October 9, 2024

Updates

• MicroK8s Compatibility: MicroK8s clusters now use Calico CNI by default with the vxlan backend. This update introduces eBPF tracing support for Calico CNI-generated virtual interfaces, enhancing MicroK8s integration.

Bug Fixes

• Agent Shutdown Stability: Investigated and resolved an issue causing the agent to not shut down gracefully due to stuck sample collectors.
• Kubernetes Parsing Correction: Addressed a bug with the Parsing Pattern in the Discovery section for Kubernetes input.

Enhancements

• Expanded Log Output Formats: Enhanced the demo input to support additional output formats, including CEF without a syslog header, CEF with a syslog header, Fortigate Traffic logs, and Fortigate UPM DNS logs.
• Security Validation: Added validation for the HMAC access key and secret.

October 1, 2024

Enhancements

• Optimized Kubernetes API Interaction: Redesigned the leader election lease process to minimize interactions with the Kubernetes API.
• Wildcard Field Support: Enhanced the agent log threshold visitor to support wildcard fields.
• Cached Environment Variables: Introduced a cache for the CEL environment function to reduce unnecessary system calls.

New Features

• Map Folding Functions: Added new CEL functions fold_left and fold_right to facilitate in-place map attribute merging.

Bug Fixes

• Diagnostic Upload Timeout: Increased the admin client timeout to resolve issues reported with uploading diagnostics to S3.
• Kubernetes Input Error: Resolved error while editing Kubernetes input node.
• OTLP Input Deadlock: Fixed a deadlock during the stop procedure by ensuring OTLP input channels are properly closed post-stop signal.
• Logging Level Adjustment: Downgraded tracer/connection_monitor logs from ERROR level for non-fatal HTTP2 header decoding mismatches.
• Kafka Source Node Fix: Corrected an issue where Kafka brokers were not correctly passed to the agent.
• Elastic Cloud ID Format Validation: Updated validation messages to correctly reflect expected formats for Elastic Cloud IDs.
• Kubernetes Consistency Adjustments: Adjusted K8s-related inputs to consistently use controller as the service.name field.
• System Stats CPU Metrics on macOS: Fixed CPU metric collection for macOS by ensuring compatibility for non-CGO environments.

September 23, 2024

Enhancements

• New CEL Macros: Introduced two additional CEL macros (iterate and reduce) enhancing the agent’s capabilities.
• HTTP Output Batching: Added batching support for HTTP output to improve data transmission efficiency.
• Custom File Discovery under Kubernetes Input: Introduced support for custom file discovery paths with Kubernetes input, allowing users to collect data from specific glob paths and filter undesired files. Additionally, added regex capabilities to capture pod UID and container names.
• Demo Log: Added ArgoCD and Istio Access log types to the Demo node, along with their respective grok patterns, facilitating easier log analysis in these environments.

Bug Fixes

• New Relic Output: Resolved an issue where the whole resource and attributes map were being removed when pushing to New Relic. Now, attributes and resource fields are appropriately included.

Deprecated Features

• Compactor Incoming Stats: Removed dependency on incoming stats for compactor metrics in favor of heartbeat-based representation.

September 9, 2024

Enhancements

• Enhanced Kubernetes Input Source Detection: The source detection mechanism for Kubernetes inputs has been improved. Previously, it utilized the first parent of a pod, such as a ReplicaSet, to create the k8s.pod.name filter. The update now ensures that the root controller is used for creating the k8s.pod.name filter, providing a more stable and reliable source reference. Additionally, the unused attribute ed.k8s.pods has been removed to streamline the system.

• Debug Output Stability: A bug was fixed in the Debug output that caused a crashloop issue due to undefined streamer and integration names. The fix involved ensuring that both names are appropriately set, thus stabilizing the Debug output functionality and preventing further crashes.

• Refinement in Metrics Collection: The default logic for collecting pod labels and annotations for cAdvisor and Kube-state metrics has been overhauled. The old system, which automatically gathered these by default, has been replaced with a new resource field selection logic. This new logic mirrors the approach used for the Kubernetes Input node, ensuring a more targeted and efficient collection of metrics which improves performance and accuracy.

Fixes

• Google Cloud Logging API Limitations: An issue with the Google Cloud Logging API’s limit of 256KB per log entry has been addressed. Log entries will now be split correctly to adhere to this size limit while maintaining their labels. Furthermore, a new label called edgedelta_GCL_split_id has been introduced to store the UUID of split messages. This ensures proper handling of logs without exceeding API restrictions, which include no more than 64 labels, label keys being at most 512 bytes, and label values being at most 64KB.

September 2, 2024

Enhancements

• Conversion to OTEL Definition: Attributes’ data types have been converted to more closely follow the OpenTelemetry (OTEL) specifications. This ensures better compliance and integration with the OTEL ecosystem.
• CEL Evaluations Optimization: Enhanced ordered evaluations in the Common Expression Language (CEL) to optimize performance, particularly for evaluating multiple data types sequentially (e.g., string, float, integer, map).
• Pod Listener Caching: Introduced a caching mechanism for non-existent pods in the pod listener, leading to a reduction in redundant Kubernetes calls for ephemeral containers.
• CEL Version Update: Updated to the latest CEL version and introduced benchmarks for evaluating CEL performance. Added external libraries from the CEL official repository to support this enhancement.
• Support for Granular Stream Stats: Updated ED Log Output, Loki, Google Cloud Logging (GCL), and New Relic to support granular stream statistics in the OpenTelemetry logs format.

New Features

• Updated Inputs and Services: Enhanced service name collection across multiple inputs including Kafka, S3 SQS, Docker, file input, demo, Exec, OTLP, ports, Kubernetes, Edge Delta-generated inputs, and container stats inputs.
• Null Value Option: Added an option (ignore_if_null: true) to transforms, allowing the system to ignore null values during transformations, thus preventing them from being added to the final output.

Bug Fixes

• Push Processor Error Handling: Improved handling of errors in the push processor during the agent shutdown procedure to ensure smoother termination.
• Deprecated Data Type Dependency in OTLP Input: Removed dependency on the data_type in the OTLP input configuration, streamlining data handling and reducing configuration complexity.

August 26, 2024

New Features

• S3 Input Compression Selection Support: Compression options for the S3 Input node have been expanded. You can now select from “gzip”, “zstd”, “snappy”, and “uncompressed” compression types. This update allows for better support of AWS logs such as ALB and CloudTrail.
• HTTP Protocol Support for OTLP Input Nodes: The OTLP Input node now supports HTTP in addition to gRPC, providing more flexibility in how you can send OTLP data to the Edge Delta agent.

Security Fixes

• Environment Variable Masking: Environment variables loaded by the agent are now masked before being written to logs, ensuring sensitive information is protected.
• gRPC Metadata Security Fix: Addressed a potential PII concern where gRPC metadata, which may include private information, was being logged. The gRPC version has been updated to address this.

Improvements

• Granular Stream Stats for Port-based Outputs: Enhanced metrics for outputs using ports now include more granular stream stats, providing detailed insights into data streams.

Bug Fixes

• JSON Unroll Casting and Validation Fix: Fixed an unexpected behavior in JSON Unroll casting. Also corrected JSON Unroll config validation.
• ED_ENV_VARS Functionality Fix: Resolved an issue where the ED_ENV_VARS functionality was not working for users trying to pass their variables to the Agent on Linux and macOS.
• Kubernetes Short-lived Container Status Fix: Fixed an out-of-bounds error in Source Detection.
• Improved Grok Processor Handling of Redis Logs: Enhanced the Grok node to better handle Redis logs, improving log parsing accuracy.
• Support for ed_logs Node Type: The agent now accepts ed_logs as a valid node type.

August 20, 2024

Critical Fix

• Health Data Upload Fix: Resolved an issue with health data uploads causing throttling. Now, health and diagnostic data are buffered to ensure they are uploaded as a single file, significantly reducing upload frequency. All agents running version 1.13.0 must upgrade to 1.14.0.

New Features

• Granular Stream Stats: Added support for granular stream statistics in S3 and Azure Blob Storage, with similar functionality extended to the ED Archive when metadata is enabled.
• CEL Function: Introduced the to_json CEL macro.
• Health Data Debugging: Allowed ingestion of health data by the Debug output node.
• Datadog and Splunk Mapper Updates: Allowed ingestion of metrics by Datadog and Splunk mappers.
• Cluster-Pattern Item Manipulation: Enabled Datadog and Splunk mappers as well as Output Transform nodes to ingest the cluster-pattern data type.

Enhancements

• Agent handling of large data items: To improve agent performance, the agent will split any incoming message larger than 1Mb into individual messages. In addition, the Edge Delta archive will not ingest telemetry messages larger than 2Mb.
• Improved Kubernetes CEL Function: Added the GetPod function to improve use of the from_k8s CEL macro. Introduced a Kubernetes API fetch step if the pod is not found in the cache.
• OTEL Log Ingestion: Made OpenTelemetry (OTEL) log ingestion the default path and removed the old ingestion path in the v3 codebase.
• Rename ed_archive_output to ed_logs_output: The node type for ed_archive_output will be changed to ed_logs_output. To ensure backward compatibility, both of these node types are supported.
• Data Type Validation in OTLP Input: Introduced stricter string data type validation for OTLP input and changed the input field to a dropdown menu instead of a freeform text field.
• Display Name Consistency: Updated the display name for the unescape JSON node to JSON Unescape.
• The following advanced firewall rules are no longer required:
  • ed-agent-log.s3.us-west-2.amazonaws.com
  • ed-overflow-agent-log.s3.us-west-2.amazonaws.com
  • agent-pprof.s3.us-west-2.amazonaws.com
• Kubernetes Pod Topology Spread Constraints: Introduced pod topology spread constraints to our Helm chart. This feature helps control how Pods are spread across your cluster among failure domains such as regions, zones, nodes, and other user-defined topology domains, improving operability with KaaS and overall K8s scheduling.
• Cache Health Observability: Added health data to the pod listener component, allowing better observation of cache contents over time.
• OTLP Traces: Added support for OTLP traces to the OTLP input node. This enhancement, data_type: trace, improves tracing capabilities.

Fixes

• Docker Library Update: Updated the Docker library from version 24.0.9 to 26.1.5 to address critical CVEs including CVE-2024-41110.
• Remove Config Content ID: Removed all references to the now-unsupported config content ID.
• Stream Stats Calculation: Fixed potential divide-by-zero panics by adding length checks before performing average calculations on metadata.
• Resource Flexibility: Made source attributes more flexible by removing mappings that prevented certain labels from propagating downstream when added by users.
• Large Stack Trace Handling: Increased minimum seek size to handle large stack traces more effectively.
• Nested Compound Nodes: Resolved issues with compound nodes (now called Packs) having the same name as their parent compound node, ensuring correct pipeline imports.
• Node Creation for Rollup: Reduced memory consumption on rollup agents by limiting the creation of unnecessary components.
• Sample Collection Time: Increased the default sample collection time from 1 minute to 15 minutes to ensure coverage for lower volume sources.
• Compactor Service DNS Resolver: Fixed an issue where the compactor service’s DNS resolver watched for changes in all services in a K8s cluster. The DNS resolver now only monitors the compactor service, reducing unnecessary load. Also fixed the deregistration of the pod listener from the health manager.
• K8s Metrics Collection: Corrected an issue where some metrics collectors did not check if metric items were nil, causing errors during K8s metrics collection.
• Health Endpoints in HTTP(S) Input: Removed constraints on health endpoints.
• Transform Node Updates: Fixed on-screen wording and updated examples for transform nodes.
• Ingest Health Data Type: Allowed health data type ingestion by debug output.

Stability and Performance Improvements:

• Several stability and performance improvements have been made, including Zstd encoder thread safety. Additionally, error logging for output nodes has been added, dependencies on deprecated ingest configuration fields removed, transformation nodes set to use a no-op poder, and a feedback channel added for the health manager to ensure proper stop procedures.

Maintenance

• Pod Listener Testing: Updated the pod listener to function as a no-op during node testing, ensuring it does not interfere with test scenarios.

August 12, 2024

Note: This release contains a critical bug. All agents running version 1.13.0 must upgrade to v1.14.0.

July 30, 2024

This release includes several significant updates, enhancements, and bug fixes aimed at improving the functionality and performance of Edge Delta agents. Below are the details of the changes included in this release:

Enhancements

• Splunk Output Update: Splunk output node can now ingest signal type messages.
• Add k8s.container.name to Pod Info Extraction: Added the ability to extract k8s.container.name using CEL from the k8s resource.
• Add Health Check for OTLP Input: Introduced a health check service for OTLP input nodes, necessary for new cloud fleets as load balancers perform health checks at certain intervals.
• Support for service.name in *nix Environments: Added support for the OTEL standard field resource.service.name for sources detected in *nix environments.
• Authentication and Path Filtering for HTTP(S) Input: Added authentication (Bearer and Basic) as well as path filtering options to the HTTP input node.
• The ED_CUSTOM_TAGS environment variable can now be used for attaching attributes statically. The format should be in form: <key 1>:<value 1>|<key 2>:<value 2>
• Unroll JSON from a Field Path: Added functionality to process JSON data from non-body locations using json_field_path.
• The Grok node has been updated with patterns for AWS VPC Flow as well as MYSQL Slow Query and Error logs.

Bug Fixes

• Leader Election in Non-Processor Agents: Rollups, aggregators, and compactor agents will no longer participate in leader elections. Only processor agents will now have leader election.
• Resource Transform Fields Update: Aligned the Resource Transform node configuration to match log transform nodes, replacing source_field_overrides with transformations and ensuring backward compatibility for older configurations.
• Missing Validation in Resource Transform Node: Added missing validation checks after the deprecation of source field overrides.
• Improve Error Message for k8s Event Tailer: Improved the error message to help diagnose potential issues with Kubernetes event tailers in customer environments.
• Track Errors in Gzip Decompressor: Errors from gzip decompression will now be tracked and logged, instead of being propagated upstream, to reduce noise.
• Error Counting in Gzip Decompressor: Errors are now counted with an errorCount counter and logged at most five times per minute.
• Fix for Splunk Mapping: Ensured messages appear the same during migration from config v2 to config v3 by making Log-transform and Splunk-mapper compatible.
• Prevent Push Strategy Creation Errors from Stopping Agent Execution: Added a No-Op strategy to handle push strategy creation failures, allowing the agent to continue running when some output nodes fail.
• Use Bool Pointer for Log to Metric parameters: Changed skip_empty_intervals and only_report_nonzeros to booleans to differentiate between false values and unset variables.
• Kubernetes Input Labels & Annotations: Switched to opt-in collection for pod labels and annotations, added support for node and namespace labels.

Miscellaneous

• Batching Bug in Compactor: Fixed issues with batching in compactors that were causing unnecessary overhead and inefficiencies.
• Group Anomaly Settings Using Log to Metric Top Anomaly Setting: Improved the handling of anomaly settings where group anomaly settings are missing.

These improvements and bug fixes enhance the robustness, usability, and functionality of the Edge Delta agent.

July 16, 2024

This update provides robust enhancements and fixes for better performance, reliability, and functionality across different environments and workflows.

Enhancements

• A new Knowledge Library has been released, providing solutions for common regex, CEL, and GROK patterns.
• Fleets on the Pipelines - Dashboard page can be sorted by telemetry data such as number of deployed agents.
• Support for OTEL Log Format: Improved support for sending logs in OTEL format to the backend, ensuring backwards compatibility and gradual transition while running older and newer schemas concurrently.
• Helm Chart Improvements: Fixed missing annotations for push services in the Helm chart to avoid issues when setting up a load balancer.
• Kubernetes Events Handling: Updated K8s events tailer to include modified events, ensuring all events are ingested even when a Back-off event is patched.
• Log to Pattern Node Enhancement: Added capability to pick a field in the agent data item for clustering instead of the default body field.
• Log Forwarding via OTEL: Improved the flushing logic and interval management for OTEL logs, and enabled log forwarding to pusher when archive ingestion is active.
• Field Name for File Input: Updated source detection to add ed.source.name and replace invalid characters with an underscore.

Bug Fixes

• Debug Output with Linux: Addressed a nil pointer exception in the debug output with Linux pipelines.
• CPU Profile Capture on Windows: Removed the attempt to capture CPU profile on Windows to prevent error messages.
• Metrics Reporting Fix: Corrected an error with log to metric processor to avoid stopping item processing due to variable shadowing.
• Update Archive Payload Schema: Updated schema to align with OTEL schema and fixed a deadlock issue during the stop procedure.
• Fixed host.ip value for agents behind NAT: Changed the ingestion method to use the default gateway interface’s IP instead of backend-imposed IPs, ensuring the correct internal IP address is reported.

Miscellaneous

• HTTP Input Enhancements: Support for all routes and multiple methods (POST, PUT, PATCH, DELETE) for HTTP input, with attributes added for enhanced flexibility.
• Leader Election Improvements: Removed the ED_LEADER_ELECTION_ENABLED environment variable requirement for leader election and made it enabled by default.
• K8s Role Permissions: Added get secret permissions for Kubernetes roles required for specific cleanup operations.
• Log Threshold Monitors: Fixed the issue with log threshold monitors not ingesting empty strings.
• Rehydration Fixes: Corrected issues with rehydration not pulling Avro/Zstd files.
• Grok Fixes: Combined pattern and custom pattern fields to prevent validation failures and improve reliability.
• Data from Debug Output nodes is no longer available on the node itself. It is only available on the full Debug Output page.
• RBAC group names are now case insensitive.
• Default k8s_input exclusions have now been trimmed to automatically ingest all K8s control and management plane telemetry data.

July 1, 2024

This update introduces several new nodes, enhancements to existing nodes, and bug fixes.

New Features

• Google Cloud Logging Output node is introduced in Beta status.
• S3 input and JSON Unroll is introduced in Beta release.
• Horizontal Pod Autoscaler (HPA) is introduced.

Enhancements

• Resource Transform nodes have relaxed restrictions when applied to fields.
• Debug Output node is now available in all environments.
• Added support for ingress in Helm. More about in Edge Delta Helm.

Bug Fixes

• Various bug fixes but none notable enough to mention.

Miscellaneous

• Agent v1.10.0 is released thus fully embracing SemVer.
• Various performance improvements.

June 17, 2024

This update introduces several new features, enhancements, and bug fixes to improve the functionality, integration, and overall performance of the Edge Delta Agent.

New Features

• CloudTrail Format: Added CloudTrail format to Demo Input for better support of CloudTrail use cases.
• Output Transform nodes can now send transformed logs to HTTP output nodes in addition to Datadog, Splunk, Sumo, and Elastic.
• Custom HTTP Headers: Introduced custom HTTP header support for HTTP output. Users can now create any header using key/value pairs for the HTTP output node. This currently supports hardcoded strings. For example, headers like X-Token: abc123456 can now be configured.

Enhancements

• Streamer Component: Corrected success counting in to ensure it only counts when items are created for pushing.
• Error Handling: Unified error counting and logging intervals across processors, ensuring all counters stop correctly at stopping time.
• Helm Chart: Introduced support for defining separate node selectors, tolerations, and priority class names for rollup and compactor agents.
• Kubernetes Integration: Improved K8s file tailer to create a complete set of OTEL semantic conventions attributes.

Bug Fixes

• Helm Upgrade Secrets: Fixed an issue where Helm upgrade didn’t create a new secret with a valid secretApiKey value, ensuring proper handling of secrets.
• Windows Agent Logging Location: Changed the log destination for the Windows agent to C:\Program Files\EdgeDelta\edgedelta.log to align with standard application log locations.
• HTTP output Ordering: Addressed an ordering issue for node specifications in HTTP output to prevent headers from being hidden.

Miscellaneous

• Archiver Code: Deprecated and removed archiver agent.

June 5, 2024

This release brings numerous improvements in sample collection, integration capabilities, logging detail, and bug fixes, enhancing the overall performance and reliability of the Edge Delta Agent.

New Features

• Grok: Added a new parser node used for parsing log data into attributes using common log patterns.
• Google Cloud Logging Node: Introduced a new output node for sending data to Google Cloud Logging API.

Enhancements

• Custom HTTP Headers: Enabled the creation of custom HTTP headers with key/value pairs for HTTP output nodes. Currently supports hardcoded strings.
• Splunk Integration: Removed legacy support for features from v3 Splunk output.

Bug Fixes

• Agent Exit Code: Changed the agent to explicitly set exit code to 1 on graceful shutdown due to errors, ensuring proper restart by controllers like systemd, Kubernetes, etc.
• Container Stats Memory Issue: Fixed a memory leak issue caused by the ED Container Stats input node in Kubernetes environments.
• Log Metadata: Enhanced agent self logs to include package, file, and line number information for better debugging.
• Compactor Buffer Flush Fix: Resolved an issue with compactor prematurely flushing the internal archive buffer before the S3 pusher could send the data. Disabled compaction except for Avro encoding and uncompressed third-party archive outputs.
• Late Arrival Handling: Removed non-configurable late arrival handling from several input nodes. Late arrival is handled by the line-separator.
• Kubernetes Resources: Fix issue of missing resource values for Kubernetes agents.

Miscellaneous

• Converted datatype-related functions to variables to optimize memory utilization.

Agent v0.1.103

Version 0.1.103 was not released. Please use v0.1.104.

May 20, 2024

The following changes been implemented in this agent release:

Product

• Changed the release status of compound nodes (now called Packs) from Beta to Released, reflecting their stable release state.

Enhancements

• Loki and NewRelic Output Nodes: Added support for custom items to be processed, enhancing the data modification capabilities before sending to these destinations. Enhanced Loki integration to allow custom labeling, giving users more control over their data attributes.
• Self-Logs Metadata: Added extra metadata (package and file plus line number) to self-logs to ease troubleshooting and investigations.

Bugfixes

• Fixed the extraction of timestamps from messages, ensuring timely logging and improved search accuracy.
• Route Path Processing: Fixed the issue where the “Exit if Matched” field radio button wasn’t correctly set.
• Updated the Event Schema to include event.type and event.domain fields at the top level in the AVRO schema to ensure complete event representation.
• Agent panic when testing: Testing uses the proper schema abstraction while accessing data, preventing agent panics.
• Removed legacy Prometheus Metrics settings such as rule_metrics_prom_stats_enabled, internal_prom_stats_enabled, and enable_reporting_in_prometheus to eliminate conflicts with the Prometheus metrics exporter output node.
• Fixed an issue with the UDP input node handling read timeouts, ensuring it gracefully restarts listening.
• Fixed handling of number values when applying math functions.

Security

• Updated dependencies to mitigate a high severity vulnerability CVE-2022-29583 [OWASP Top Ten 2017 Category A9 - Using Components with Known Vulnerabilities].
• Fixed medium severity vulnerabilities including:
  • CVE-2023-45288 [Uncontrolled Resource Consumption]
  • CVE-2024-24786 [Loop with Unreachable Exit Condition]
  • CVE-2024-24557 [Insufficient Verification of Data Authenticity]
  • CVE-2024-29018 [Incorrect Resource Transfer Between Spheres]

Upgrade Instructions

Note: this version removes the default use of Persistent Volume Claims (PVC) for the Compactor in Kubernetes installations. Therefore, to upgrade to this version a clean install is required. To enable PVC, use the compactorProps.usePVC=true helm value. For kubectl installations, update the manifest with a persistent volume .

May 6, 2024

The following changes been implemented in this agent release:

New Features

• The Mask node now includes a number of regex patterns out of the box for masking important fields such as email, bitcoin address, different types of credit card numbers etc.
• Metrics Generation for Log Threshold Monitors: Agents now generate metrics required for log threshold monitors. A hidden Log-to-Metric node fetches monitors from the backend and associates metrics with them. This node is also connected to the ed_metrics_output node type.

Improvements

• Health Signal Enhancements in Debug Output: The health node output can now be viewed in the debug output node, allowing users to visualize the health signals it produces. The base strategy for health status now registers a ’not ok’ status if there are more than 10 issues within a 10-minute window.

Bug Fixes

• Elastic Output Warning Suppression: An issue where the lastErr variable in the Elastic output node wasn’t being set properly has been addressed, ensuring that warnings are now logged if entries encounter issues during processing.
• Enhanced Isolation in Pipeline Testing: Changes have been made to pipeline testing to prevent the activation of components. During testing, Output nodes are substituted with Void nodes, ensuring that no real output operations are executed. This approach solidifies the testing environment’s isolation, ensuring that it remains unaffected by any output-side effects.
• Source Detector and Enrichment Tagging Fix: A bug concerning the incorrect tagging of log file names in source detector and enrichment in config v2 has been rectified, improving tagging accuracy in specific use cases.
• Corrected Outgoing Bytes and Lines Reporting: A previous bug that prevented successful reporting of outgoing bytes and lines, even when there were successes, has been fixed. This enhances the accuracy of transmission analytics.
• Panic Prevention During OTEL Patterns Emission: Adjustments have been made to check whether the compactor is running before proceeding with any data upload. This change prevents the Agent from panicking and attempting to write to a closed channel when stopping, thereby ensuring a smoother shutdown process.
• Concurrent Map Read/Write Issue Resolved: An agent panic caused by concurrent map read/write operations, especially relevant during output node and dependencies construction, has been addressed. This fix includes the enforcement of correct logic to determine if a link should copy the object considering both Incoming and Outgoing data configurations.
• Pusher Logging and Health Tracking: Improvements in Pusher logging have been applied, allowing more detailed error logging and better health status accuracy.

April 22, 2024

In this agent version, the following improvements were released:

New Features

• The Kafka input and Kafka output nodes were released to beta.
• The Loki output node was released to beta.
• The Debug Output node was released to beta.

Improvements

• The Demo input node validation was improved.
• The kubernetes manifest now includes CPU limits for the agent to align with best practice and prevent scheduling issues
• Node testing traffic to the API was optimized.
• The Parse Json Attributes node’s handling of numbers was improved.

April 8, 2024

In this agent version, the following nodes were released in beta:

• Resource Transform to transform the resource parameters.
• New Relic Output to send data to your New Relic destination.

The following improvements were released:

• The TCP Output node and the Local Storage Output node can now accept additional data types: custom, metric, cluster_pattern_and_sample, and signal.
• Improved health data reporting of last_error to improve API stability.
• The Compactor logging level was changed from debug to trace to improve troubleshooting and diagnostics.
• Various improvements are made to the Edge Delta Helm chart improving efficiency and improving use with Continuous Deployment.

The following fixes were released:

• The Demo input node now emits logs that use UTC time.
• Mask node testing where a blank Golang Regex pattern was set no longer results in a null pointer exception that causes an agent panic.
• The Log to Metric processor no longer reports unprocessed logs as errors
• The compactor component gracefully stops when shutting down

The following features were deprecated:

• The Enrichment node is now deprecated. Please use the Log Transform, Output Transform or Mask nodes.

March 28, 2024

This agent release

• Fixes missing resources in HTTP Input, TCP Input and UDP Input nodes.
• Fixes a demo node input issue regarding timestamps.
• Updates software dependencies for the Edge Delta agent to versions that address recent CVE-listed security vulnerabilities.

In addition, the following improvements are released:

• Improved health and EDAC ingestion.
• Improved counting of hits and errors in processors.
• Improved agent performance monitoring.
• Improved Enrich and Source Manipulate nodes’ forms.

March 26, 2024

This agent release

• introduces the OTLP input node to collect raw logs or metrics in the OTEL format,
• improves the Demo input node by adding additional log types the node can emit,

Note: after upgrading to this version, existing demo nodes need to be removed and replaced with the new demo node.

• improves validation on the Enrich and Log Transform nodes,
• adds support for nested Compound Nodes (now called Packs), and
• improves validation for packs.

Breaking Change

Customers with HTTP Input, TCP Input or UDP Input nodes should not use or update to Agent version v0.1.97.

March 12, 2024

This agent release continued to deprecate the custom Edge Delta schema in favor of standardization on the OTEL schema for v3 configurations. Patterns have been enriched with ed.team and ed.source fields. In addition, the default for the TCP Output and FluentD nodes Host name was changed to 127.0.0.1.

This agent release resolved technical debt related to multiple metrics ingestion paths. It contains a breaking change for existing customers with any agents older than v0.1.63. It can be mitigated as follows:

1. Update all agents to v0.1.95.
2. Notify Edge Delta to migrate ingestion settings to use the new metric provider.
3. After 7 days, update to v0.1.96 or above.

February 26, 2024

The agent release introduces Compound Nodes (now called Packs) - an aggregation of pipeline nodes that you can add to a pipeline as a single object - as a beta feature. In addition, the first_non_empty() CEL macro was updated to handle nested functions within it. It extended the Log to Metrics node to allow CEL fields to be used to define paths. In addition, it improved adoption of the OTEL schema along with other performance and stability improvements.

February 9, 2024

In this agent release the Output Transform node was upgraded to support removal of metric attributes to improve pipeline optimization. Among other fixes and optimizations, the Route node user experience has been improved and node self-logs have been optimized.

February 9, 2024

This agent release contained a critical bug. Please upgrade to v0.1.94.

January 30, 2024

This agent release introduced the beta Prometheus output node to enable the Edge Delta pipeline to send metrics to Prometheus. In addition, the Kubernetes Events input node entered beta. This node enables ingestion of kubernetes events into Edge Delta v3 pipelines.

January 16, 2024

This agent release improved back end stability and performance. In addition, it improved the Datadog Mapper node by allowing a custom dd_hostname and it contains CEL improvements such as the introduction of a CEL macro for converting timestamps.

January 3, 2024

This agent released added support for case lower/upper control in data streams. In addition, there were updates and bug fixes to the Log Compactor Agent.

December 19, 2023

This agent release added name validation to integrations. As with node names, integration names containing a period character (.) will not pass validation. In addition it improved memory usage of the Agent’s Aggregator component.

November 30, 2023

This agent release improved configuration validation for duration and CEL macro fields. In addition it improved agent health monitoring.

November 20, 2023

This agent release improved parsing of JSON attributes to allow for more precise data extraction from JSON logs. It added support for environment variables in the agent configuration. In addition, a new agent manifest includes the new compactor.

November 6, 2023

This agent release improved agent metric reporting including agent health among other stability and performance improvements.

November 2, 2023

This agent release improved log to pattern node sampling and the Slack integration payload.

October 20, 2023

This agent release fixed a log transform upsert bug. It improved pipeline reliability for metrics collection and it reduced error suppression for agent start logs.

October 17, 2023

This agent release improved log transform upsert functionality and validation. It also reduced alert noise caused by the Webhook output node.

October 8, 2023

This agent release improved collection of health and I/O metrics from visual pipeline nodes.

October 5, 2023

This agent release improved performance and error handling of parsing, transformation and mask type nodes. In addition, Visual Pipelines moved out of beta, while the following nodes were introduced as beta:

• TCP Output
• HTTP Output
• FluentD Output

September 20, 2023

This agent release fixed a trace log issue that caused higher than normal trace traffic.

September 18, 2023

This agent release defaults to the OTEL schema for Patterns and Metrics when using nodes with the v3 configuration. It also introduces the following nodes:

• JSON Parse
• Log Transform
• Output Transform
• Splunk Mapper
• Datadog Mapper

August 25, 2023

This agent release defaults to the OTEL schema for log data types. This aligns data in the pipeline with data discovered using log search. In addition, this release enables Edge Delta agents to detect and capture HTTP2/GRPC connection information using EBPF.

August 9, 2023

This agent release reduced memory usage when handling EBPF packets. It introduced the Pipeline IO Stats input node to enable IO stats reporting, and it exposed the item_schema agent setting to enable schema switching.

August 3, 2023

This agent release introduced aggregator agent support for the Top-K and Log to Pattern nodes.

July 30, 2023

This agent release includes the beta Docker Input Node for Visual Pipelines. It captures log input from Docker containers.

July 26, 2023

This agent release improved Prometheus integration behavior in the event of a source change. In addition, the Log Transform Node entered beta.

July 20, 2023

This agent release exposes bulk indexer configurations to help debug issues and enable better configuration of elastic clients.

July 17, 2023

This agent release improved a number of configuration v3 nodes. The following Visual Pipeline nodes entered beta:

• TCP Output
• HTTP Output
• Microsoft Teams Output
• Webhook Output
• Ratio Processor
• TopK Processor
• OpenTSDB parser

In addition, parent source fields were added to the metrics payload sent to s3 to enable downstream processing, and the FluentD input node and log-to-pattern processor node reliability was improved.

June 21, 2023

This agent release supports a new agent configuration format (v3) in addition to version 2. Agents with the version 3 format can be configured using either the normal YAML text editor or the new Visual Pipelines interface. There are a number of v3 nodes that can be configured using Visual Pipelines, for example,

• Kubernetes Input
• File Input
• Log to Pattern
• Log to Metric
• Regex Filter
• Mask
• Datadog Output
• Splunk Output
• S3 Output

See the full list of input nodes, processor nodes, and output nodes.

June 16, 2023

This agent release improved back end stability and performance.

May 31, 2023

This agent release improved back end stability and performance.

May 26, 2023

This agent release improved enrichment logic to deal with aliases and reserved keywords. It also optimized backend performance for metrics and archiving and improved helm chart customization.

May 15, 2023

This version improved log search and hosted agent performance as well as backend stability.

May 10, 2023

This agent release introduced support for a specifying a worker count for Elastic and OpenSearch data destinations to improve peak traffic performance.

May 8, 2023

This agent release improved back end stability and performance.

May 1, 2023

This agent release improves Elastic integration performance by adding validation for send as is configurations.

April 27, 2023

This agent release improved retry performance for S3 archiving and it adds the option to remove root names for Wavefront.

April 21, 2023

This agent release improved agent performance with pre-start agent validation. It also removed the PVC dependency from aggregators and it added a more granular drop metric column option.

April 14, 2023

This agent release improved the Helm template and added template validation.

April 12, 2023

This agent release improved essential metric reporting and optimized S3 usage by reducing the file count.

April 10, 2023

This agent release improved handling of agent health data as well as improved Elastic integration. It added support for creating AVRO formatted archive files and it added a distinct count type regex processor.

March 28, 2023

This agent release improved handling of agent health data and it implemented a number of performance fixes.

March 16, 2023

This agent release improved regex processor and log to metric performance. It also added options for handling internal certificates.

February 15, 2023

This agent release improved back end stability and performance.

February 9, 2023

This agent release added an option to disable TLS certificate verification for Elastic Streaming destinations. This may be useful for self-signed certificates.

February 1, 2023

This agent release improved agent and back end stability and performance.

January 23, 2023

This agent release improved stability and performance. In addition, Edge Delta Anomaly Context (EDAC) logs were improved for Amazon S3.

December 23, 2022

This agent release made particular improvements to filter error handling and improved hosted agent environment performance among other backend stability and performance improvements.

December 6, 2022

This agent release improved regex processors by making it easier to identify dimension group metrics using a custom suffix. In addition to agent and backend stability and performance improvements, script-based filters were updated to handle errors more effectively.

November 30, 2022

This agent release improved agent and back end stability and performance.

November 28, 2022

This agent release improved the querying experience for customers who use Elastic destinations.

November 24, 2022

This agent release added support for enriching logs using K8s controller information. In addition attribute mode was added to the source detection filter to enable field mapping using keys from the source attributes rather than the log body. A number of stability and performance improvements were also deployed.

November 15, 2022

This agent release improved agent functionality and performance particularly for Elastic and Datadog. In addition, support was added for Prometheus to scrape metrics with dimensions as attributes.

November 9, 2022

This agent release improved stability and performance. In addition, an Enrichment filter that uses a script function was released.

October 11, 2022

This agent release improved Prometheus integration. Rule metrics in Prometheus now work end to end:

You can now configure an agent to expose regex processor rule metrics at the metric endpoint for Prometheus scraping. The agent can now handle late-coming source tags for rule metrics. Rule metrics that are exposed from regex processors in the Prometheus format now support aggregator agents. Regex processor reporting now aligns with Prometheus 1 minute scraping intervals. In addition, pipeline performance has been improved with better handling of Elastic destinations not being available.

October 6, 2022 - On Prem UI v0.1.13

This release enabled the Observability - Metrics and the Data Pipeline - Pipeline Status pages for the self-hosted user interface. In addition, users of the self-hosted user interface can now access Observability, Data Pipeline and Management features without any 3rd party identity provider integrations, for example, for internal demo purposes:

October 3, 2022

This agent release improved agent and back end stability and performance.

September 27, 2022

This agent release added a new filter that can perform JavaScript enrichment and log transformation. In addition, the log files of newly created Kubernetes pods are now scraped from when they are created, including startup logs. The agent can also resume scraping from the previous position in a pod log file if the pod is restarted.

September 19, 2022

This agent release added support in the numeric capture regex processor for multiple dimension groups with a range of metric types. In addition, the agent can now be configured to conform with the Amazon Web Services CloudWatch log quota limits.

September 9, 2022

This agent release added Transport Layer Security (TLS) configuration support for S3-compatible archive destinations. The config wizard was updated and TLS support was added for AWS session settings.

September 5, 2022

This agent release improved the Ratio Processor which is now out of Beta.

August 29, 2022

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

On Demand Log Forwarding Log forwarding can now be triggered via an API call. It is used to temporarily forward specific sets of raw data to streaming destinations for a given time period. Both the duration and the log sources can be granularly defined to meet a number of use cases such as forwarding for 30 minutes following a deployment or when an alert triggers. The API call can be automated with integration into CI/CD tooling or third party alerting systems.

For more information, see On Demand Log Forwarding.

Prometheus Integration The Edge Delta agent can now be configured with a service monitor to expose metrics on an endpoint for Prometheus to scrape. The following metrics are exposed:

Count of incoming lines Sum of incoming bytes Count of outgoing lines Sum of outgoing bytes Count of successful outgoing streams Count of failed outgoing streams Open file status To learn more, see Prometheus Integration.

Updated Azure AppInsight Streaming Output The Azure AppInsight streaming output has been updated to allow you to specify where to funnel data. Previously, you could only funnel data to an event index.

Now, you can use the newly published base_type parameter to specify where to funnel data.

To funnel data into a tracing index, enter MessageData.

To funnel data into an event index, enter EventData.

To learn more, see Azure AppInsight.

New Filter Type You can use the newly created Log Transformer Javascript filter to transform specified log messages. Specifically, this filter uses Goja, a type of script in JavaScript. When a log matches the criteria, it will be transformed, and then passed through the filter.

To learn more, see Log Transformer Javascript.

Updated agent_settings For agent configuration, under agent_settings, you can use the newly created agent_stats_enabled parameter to display agent-related information (such as CPU and memory usage) in the Metrics page in the Edge Delta App.

To learn more, see Agent Settings.

June 6, 2022

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updates Agent Installation The logging directory for the Edge Delta agent has been updated.

Specifically, if you install the agent via the installation script, then:

For Windows agents, logs can be found under %AppData%/edgedelta.

For all other operating systems, logs can be found under /var/logs/edgedelta.

Updated Input Enrichment In the Edge Delta App, input enrichment has been updated. Specifically, the field_name parameter under dynamic enrichment can be applied to a template.

      enrichments:
        dynamic:
          field_mappings:
            * field_name: `{{if eq .controllerKind "replicaSet"}}kube_deployment{{else}}kube_{{.controllerKind}}{{end}}`
              value: "{{.controllerName}}"

For more complicated templates that include if / else statements or range statements, you must use bracket as a delimiter.

Updated source_detection Parameter The source_detection parameter for inputs has been updated.

Specifically, source_detection now supports custom as a source_type. When you enter custom , you must configure the field_mappings parameter with a key-value pair.

    * labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Additionally, source_detection now supports regex as a processing_mode.

    * labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        processing_mode: regex
        field_mappings:
          namespace: namespace (?P<field>\w+)
          serviceName: service (?P<field>\w+)
          roleName: user_role (?P<field>\w+)
          systemType: system (?P<field>\w+)

May 10, 2022

This agent release contains backend enhancements and bug fixes.

May 6, 2022

This agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Streaming Output In the Edge Delta App, the Splunk streaming output has been updated to support custom tags via the custom_tags parameter.

You can use this parameter to define key-value pairs that are streamed with every request.

- name: my-splunk
      type: splunk
      endpoint: "://:/"
      token: "32-character GUID token"
      custom_tags:
        "app": "test"
        "region": "us-west-2"
        "File Path": "{{.FileGlobPath}}"
        "K8s PodName": "{{.K8sPodName}}"
        "K8s Namespace": "{{.K8sNamespace}}"
        "K8s ControllerKind": "{{.K8sControllerKind}}"
        "K8s ContainerName": "{{.K8sContainerName}}"
        "K8s ContainerImage": "{{.K8sContainerImage}}"
        "K8s ControllerLogicalName": "{{.K8sControllerLogicalName}}"
        "ECSCluster": "{{.ECSCluster}}"
        "ECSContainerName": "{{.ECSContainerName}}"
        "ECSTaskVersion": "{{.ECSTaskVersion}}"
        "ECSTaskFamily": "{{.ECSTaskFamily}}"
        "DockerContainerName": "{{.DockerContainerName}}"
        "ConfigID": "{{.ConfigID}}"
        "Host": "{{.Host}}"
        "Source": "{{.Source}}"
        "SourceType": "{{.SourceType}}"
        "Tag": "{{.Tag}}"

Updated Agent Settings In the Edge Delta App, the Agent Settings section has been updated with new parameters.

  max_file_per_glob_path: 100
  forget_file_after: 1h
  total_seek_capacity: "5 MB"
  max_seek_size: "4 MB"
  source_discovery_interval: 5s
  file_tailer_buffer_size: 1000
  router_per_source_buffer_size: 1000
  archive_flush_interval: 5m
  archive_max_byte_limit: "16MB"

To learn more, see Agent Settings.

May 2, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Enrichment Options In the Edge Delta App, data enrichment options have been updated to address failed or failing sources.

To troubleshoot potential mapping failures, you can configure the failure_behavior parameter.

Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed.

      enrichments:
        failure_behavior: stop_enrichment
        dynamic:
          field_mappings:
            * field_name: "service"
              value: '{{".labels.service"}}'
            * field_name: "source"
              value: '.annotations.kubernetes.io/{{.container_name}}.logs'
              json_path: "[0].source"
              fallback_value: '{{".short_container_image"}}'

To learn more, see Enrich Input Data.

Updated Source Types In the Edge Delta App, streaming outputs have been updated.

Specifically, for the source_type parameter, you can now enter custom.

Previously, this parameter only supported K8s, Docker, ECS, and File.

When you enter custom, you must add field_mappings parameters to indicate the file source.

- labels: "my-kafka-events"
      endpoint: "something"
      topic: "topic"
      group_id: "my-group"
      sasl:
        username: kafka_username
        password: p@ssword123
        mechanism: PLAIN
      source_detection:
        source_type: "Custom"
        optional: false
        field_mappings:
          namespace: "kubernetes.namespace"
          serviceName: "service"
          roleName: "user.role"
          systemType: "system"

Updated File Inputs In the Edge Delta App, the file input type has been updated.

Specifically, you can use the newly created exclude parameter to enter a glob path to exclude matched patterns.

files:
    * labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    * labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        * "/etc/systemd/system/billingservice/test.log"
        * "/etc/systemd/system/billingservice/dev.log"

April 27, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Splunk Output In the Edge Delta App, the Splunk streaming output has been updated with the ability to send data in a JSON format.

Specifically, to use this option, you must update the endpoint parameter to point to Splunk’s API services/collector/raw, instead of services/collector/event.

    * name: splunk-integration
      type: splunk
      endpoint: ..../services/collector/raw
      token: ....
      features: log,metric,edac,cluster,alert
      index: rehydration

New Filter Type In the Edge Delta App, you can use the newly created JSON Field Extractor filter to extract a field’s value and replace the whole JSON content with the field’s value.

  * name: extract_severity
    type: extract-json-field
    field_path: "severity"
  * name: extract_first_data
    type: extract-json-field
    field_path: "records.[0].data"

To learn more, see JSON Field Extractor Filters.

Updated Enrichments for AWS ECS Inputs The configurations to enrich input data have been updated.

Specifically, you can use the dynamic parameter to enrich input data from AWS ECS.

In the agent configuration, you can

To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            * field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            * field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'
            * field_name: "cluster_name"
              value: '{{".aws-instance.cluster-name"}}'
            * field_name: "ec2launchtemplate_id"
              value: '{{".aws-instance.ec2launchtemplate-id"}}'
            * field_name: "ec2launchtemplate_version"
              value: '{{".aws-instance.ec2launchtemplate-version"}}'
            * field_name: "inspector_enabled"
              value: '{{".aws-instance.inspector-enabled"}}'
            * field_name: "cluster_autoscaler_enabled"
              value: '{{".aws-instance.cluster-autoscaler-enabled"}}'
            * field_name: "autoscaling_groupName"
              value: '{{".aws-instance.autoscaling-groupName"}}'
            * field_name: "nodegroup_name"
              value: '{{".aws-instance.nodegroup-name"}}'
            * field_name: "ec2_fleet_id"
              value: '{{".aws-instance.ec2-fleet-id"}}'

To learn how to enrich input, see Enrich Input Data.

To learn how to retrieve instance metadata, review this document from AWS.

New Filter Type In the Edge Delta App, you can use the newly createdSplit with Delimiter filter to match, then split a single log into multiple logs.

For example, the abc\n\ndef\nxyz\n log would split into 3 separate logs (abc , def , xyz ), based on the configured delimiter, (newline character ( \n )).

 - name: split_logs_using_specified_delimiter
    type: split-with-delimiter
    delimiter: ","

To learn more, see Split Lines Filters.

Updated Archiving Outputs In the Edge Delta App, archiving outputs have been updated.

Specifically, you can use the new use_native_compression option to compress data, but not metadata.

This option can be useful with big data cloud applications, such as AWS Athena and Google BigQuery.

To use this parameter, you must set the encoding parameter to parquet.

    * name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      s3_force_path_style: true
      encoding: parquet
      compression: zstd
      use_native_compression: true

Updated Cluster Processors In the Edge Delta App, cluster processors have been updated with a new configuration.

Specifically, you can use the newly created include_pattern_info_in_samples parameter to include pattern information in a cluster sample, such as patterns, pattern counts, and sentiment scores.

processors:
  cluster:
    name: clustering
    num_of_clusters: 100
    samples_per_cluster: 20
    reporting_frequency: 30s
    retention: 10m
    cpu_friendly: true
    throttle_limit_per_sec: 200
    include_pattern_info_in_samples: true

To learn more, see Cluster Processors.

New Input type In the Edge Delta App, NATS JetStream is now a supported input type.

This input type allows you to specify a NATS stream subscription for Edge Delta to monitor.

  nats:
    * labels: "my-nats-normal"
      input_mode: "normal"
      consumer_mode: "pull"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject: "example-subject-1"
      timeout: 1m
      ack_wait_duration: 10s
    * labels: "my-nats-distributed"
      input_mode: "distributed"
      consumer_mode: "push"
      cluster_url: "nats://localhost:4222"
      stream_name: "example-stream"
      subject_prefix: "example-subject"
      total_agent_count: 5
      total_subject_count: 10
      should_split_lines: true
      timeout: 1m
      disable_acks: true

To learn more, see NATS JetStream Inputs.

Updated Datadog Streaming Output In the Edge Delta App, the Datadog Streaming Output has been updated with buffered-related options.

Specifically, you can use the following, newly created parameters to configure the output’s buffering behavior:

Parameter Description buffer_ttl

Enter a length of time to retry failed streaming data.

After this length of time is reached, the failed streaming data will no longer be tried.

This parameter is optional.

buffer_ttl: 2h

buffer_path

Enter a folder path to temporarily store failed streaming data.

The failed streaming data will be retried until the data reaches its destinations or until the Buffer TTL value is reached.

If you enter a path that does not exist, then the agent will create directories, as needed.

This parameter is optional.

buffer_path: /var/log/edgedelta/pushbuffer/

buffer_max_bytesize

Enter the maximum size of failed streaming data that you want to retry.

If the failed streaming data is larger than this size, then the failed streaming data will not be retried.

This parameter is optional.

buffer_max_bytesize: 100MB

April 12, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Updated Log Enrichment In the Edge Delta App, log enrichment features have been updated to now support enrichment from Kubernetes annotations.

Specifically, you can use the from_k8s parameter to enrich streaming data with K8s attributes.

You can enter a pod, namespace, or node attributes.

        from_k8s:
          pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.*
          field_mappings:
            * field_name: instance_id
              pod_attribute: pod
              transformers:
                # replace all "source" matches with "target"
                * source: "-"
                  target: "_"
                  type: "replace"
                # remove all "test" words
                * source: "test*"
                  target: ""
                  type: "regex"
            * field_name: namespace
              pod_attribute: namespace
            # fields from labels should have pod_attribute start with "labels."
            * field_name: service
              pod_attribute: labels.service

To learn more, review the Enrich Input Data section in the Inputs document.

Updated Numeric Capture (Regexes) Processors In the Edge Delta App, the Numeric Capture (Regexes) processor has been updated to support multiplication and division for numeric value captures.

Specifically, you can use the newly created value_adjustment_rules parameter to create a rule per capture group.

The rule must follow the “(*|/)” format where:

An asterisk ( * ) represents multiplication A slash ( / ) represents division

    * name: "flog"
      pattern: " (?P\\d+) (?P\\d+)$"
      value_adjustment_rules:
        responsesize:
          operator: "/"
          operand: 1000.0

To learn more, review theNumeric Capture (Regexes) Processor section in the Processors document.

Updated Edge Delta Agent In an effort to improve security during agent installation, makeself –sha256 option has been enabled on agent deployments.

Specifically, makeself performs md5 and crc checks for content integrity.

Starting with version 0.1.20 of the agent, this upgraded security measure update will be included in all agent deployments.

Updated Sumo Logic Output In the Edge Delta App, the Sumo Logic streaming output has been updated.

Specifically, you can use the newly created send_as_json parameter to send data in a JSON format, which allows the fields to be auto-parsed and extracted in Sumo.

    * name: sumo-us-2
      type: sumologic
      endpoint: '{{ Env "EMPTY" "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/XYZ" }}'
      send_as_json: true

Updated Enriched Data for AWS In the Edge Delta App, you can use the dynamic enrichment feature to obtain data from an AWS EC2 instance.

Specifically, in the value parameter, you must enter aws-instance.

      enrichments:
        dynamic:
          field_mappings:
            # if the field value starts with "aws-instance" then instance metadata is get from aws ec2 instance.
            # for more info ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
            * field_name: "instance_id"
              value: '{{".aws-instance.instance-id"}}'
            * field_name: "instance_type"
              value: '{{".aws-instance.instance-type"}}'

To learn more, review the Enrich Input Data section of the Inputs document.

For additional information, please review the Retrieve instance metadata document from Amazon.

Updated Enriched Data for JSON In the Edge Delta App, the from_logs enrichment feature has been with the json_path parameter.

You can use the json_path parameter to enrich data with fields extracted from JSON logs.

        # from_logs is used to enrich data with fields extracted from logs
        from_logs:
          field_mappings:
            * field_name: component
              # extracting using json_path is also supported
              json_path: fields.[1].component

To learn more, review the Enrich Input Data section of the Inputs document.

Updated Workflows In the Edge Delta App, you can use the newly created enabled_hosts parameter to limit the workflow to specific hosts.

With this parameter, the workflow will only run for specified agent hosts.

  enabled_hosts_workflow:
    description: "runs only specified hosts"
    input_labels:
      * system
      * docker
      * agent
      * infa-processes
    filters:
      * info
    destinations:
      * '{{ Env "TEST_SUMO" }}'
    enabled_hosts:
      * my.host.us1
      * my.host.us2

To learn more, see Workflows.

New Filter / Process Type In the Edge Delta App, you can use the newly created OTLP filter / processor to process OTLP (Open Telemetry) logs.

  * name: opentelemetry_trace_filter
    type: buffered-otlp-trace-processor
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "attributes.result_code"
    failure_value_pattern: "(4|5)xx"
    latency_threshold: 2500.0
    success_sample_rate: 0.1

To learn more, see Filters.

March 21, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - GCP Cloud Monitoring In the Edge Delta App, GCP Cloud Monitoring is now a supported streaming output.

The GCP Cloud Monitoring output will stream custom Google Cloud metrics to a Cloud project.

In the app, you can use the visual editor or YAML file to add GCP Cloud Monitoring to an agent configuration.

GCP Cloud Monitoring was previously known as GCP Stackdriver.

New Flush Mode In the Edge Delta App, you can use the newly created custom_local_per_group flush mode to specify custom groups that should flush together if one of the groups triggers an alert.

In other words, if you set up multiple inputs, and one input triggers an alert, then all (or selected) inputs will flush.

agent_settings:
  tag: sett_test_custom
  log:
    level: debug
  capture_flush_mode: custom_local_per_group
  capture_flush_custom:
    label_grouping:
      group1:
        * file1
        * file2
      group2:
        * file1
        * file3

inputs:
  files:
    * labels: "file1"
      path: "test1.log"
    * labels: "file2"
      path: "test2.log"
    * labels: "file3"
      path: "test3.log"
    * labels: "file4"
      path: "test4.log"
outputs:
  streams:
    * name: sumo
      type: sumologic
      endpoint: "https://endpoint4.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1ozOeONNQ8LuYTYUj7SaKgr6dt1ueSTOc6mMS2pQz9BM169sb8_UQs5IRaqaRcbpKdI4Tms9S5La9ZFRTL_bf-Ptf_I5ICXcQz2WEQg0fNfA=="
      features: alert
processors:
  regexes:
    * name: "error-regex"
      pattern: "error|ERROR|problem|ERR|Err|POST|hostname|GET"
      interval: 10s
      retention: 1h
      trigger_thresholds:
        upper_limit_per_interval: 3
workflows:
  error-anomaly-workflow:
    input_labels:
      * file1
      * file2
      * file3
      * file4
    processors:
      * error-regex
    destinations:
      * sumo

The following actions will take place:

If an alert is triggered for test1.log, then the file1, file2, and file3 sources will be flushed together. If an alert is triggered for test2.log, then the file1 and file2 sources will be flushed together. If an alert is triggered for test3.log, then the file1 and file3 sources will be flushed together. If an alert is triggered for test4.log, then the file4 source will be flushed. Since file4 was not specified in any group and a fallback_mode was not provided, the agent will use the default fallback_mode local_per_source and only flush file4. When fallback_mode: local_all is added, and alert is triggered for test4.log, then all sources will be flushed. To learn more about filters, see Agent Settings.

New Filter / Processor Type In the Edge Delta App, you can use the newly created buffered-elastic-apm filter / processor to process Elastic APM logs.

  * name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor

To learn more about processors, see Processors.

New Filter Type - base64 decoder In the Edge Delta App, you can use the newly created base64 decoder filter type to decode base64 encoding.

This update helps to support a base64 encoded input on Edge Delta’s hosted collector. In other words, you can attach this filter to a source to display logs that are base64 encoded.

  * name: base64_decoder
    type: base64-decode

To learn more about filters, see Filters.

New Monitor Types You can use the new Pattern Alert and Skyline Alert monitors to trigger an alert for negative patterns.

If an alert is triggered, then the monitor will create a finding.

To learn more, see Patterns.

New feature - Suppress Notifications In the Edge Delta App, you can use the newly created Finding Status option to suppress notifications for a specific finding.

When you suppress a finding, the finding will no longer be displayed in the Insights page. Additionally, any future detection of the finding will not be displayed.

To learn more, see Patterns.

New Filter Type - APM In the Edge Delta App, you can use the newly created APM filter to process Elastic APM logs.

This filter type samples failed and high-latency traces with successful traces and a sampling probability.

  * name: elastic_apm_trace_filter
    type: buffered-elastic-apm-processor
    payload_separator: "-----------------"
    enabled_types: "transaction,span,error,metricset"
    optimize_types: "transaction,span"
    trace_deadline: 1m
    should_filter_traces: true
    failure_path: "transaction.result"
    failure_value_pattern: "HTTP (4|5)xx"
    transaction_latency_path: "transaction.duration"
    span_latency_path: "span.duration"
    latency_threshold: 35.5
    success_sample_rate: 0.2

To learn more, see Filters.

March 24, 2022 - New Organizations Feature

In the Edge Delta App, you can now create and join different organizations for you and your users.

At a high level, organizations can be considered as different environments, platforms, or sub-accounts that live within your main Edge Delta account.

You can create and belong to multiple organizations, as well as invite users to join various organizations.

Additionally, you can switch between organizations without the need to log off and log back into the app.

To learn more, see Invite Users, Manage Permissions, Access Organizations.

March 21, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

New Streaming Destination - ObserveInc In the Edge Delta App, ObserveInc is now a supported streaming output.

The ObserveInc output will stream analytics and insights to your ObserveInc endpoint.

In the app, you can use the visual editor or YAML file to add ObserveInc to an agent configuration.

New Filter - Custom Attributes In the Edge Delta App, you can use the newCustom Attributes filter to filter for custom attributes.

Specifically, you can use the Attribute Key andAttribute Value parameters to filter for custom attributes.

  # Custom attribute filter do all the log filtering with given key-value of the attribute
  * name: custom_attributes_filter
    type: custom-attributes
    key: service
    value: billing
  * name: negate_custom_attributes_filter
    type: custom-attributes
    key: component
    # A comma separated values to match. If any of them matches the given attribute's value then the log will be pass through
    value: credithandler,debithandler
    # Negate is also supported for attribute filter
    negate: true
  # Filtering custom attributes also support regex matching
  * name: regex_custom_attributes_filter
    type: custom-attributes
    key: level
    pattern: "error|ERROR|problem|ERR|Err"

To learn more, see Filters.

Updated Agent Settings - Log In the Edge Delta App, the Log parameter in the Agent Settings has been updated.

Specifically, you can use theSecure Logging option to hide sensitive data from the specified agent logs, such as API keys, secrets, and authentication information.

To learn more, see Agent Settings.

Updated Integration - Loki In the Edge Delta App, the Loki integration has been updated with a new option.

Specifically, the Send Alert As Loki Log option allows you to send alerts as a log to a Loki endpoint.

New Input Type - Google Pub/Sub In the Edge Delta App, Pub/Sub is now a supported input type.

This input type allows you to specify a Pub/Sub project for Edge Delta to monitor. Specifically, EdgeDelta will consume messages from Pub/Sub subscriptions.

In the app, you can use the visual editor or YAML file to add PubSub to an agent configuration.

To learn more, see Inputs.

Updated Input - File In the Edge Delta App, the File input type has been updated. Specifically, there are 2 new parameters:

Add Ingestion Timestamp

You can use this parameter to ingest a timestamp if the input format is in JSON. Skip Ingestion Timestamp On Failure

You can use this parameter to skip the ingestion of the timestamp when the input is broken or in an invalid format.

files:
    * labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      # ingest timestamp if input is JSON format.
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true # skip ingestion time when the input is broken or invalid format.

New Filter - Combinations In the Edge Delta App, you can use the newly created combination filter to combine with other, existing filters. Specifically, you can use and or or terms to combine filters to create a more customized filter.

filters:
  * name: combine_two_filters
    type: combination
    operator: or
    filters_list:
      * pattern: "INFO"
      * filter_name: error

To learn more, see Filters.

New Filter Type - Drop Json Fields In the Edge Delta App, you can use the newly created drop-json-fields filter to filter and drop specified JSON fields.

filters:
  * name: drop_some_fields
    type: drop-json-fields
    field_paths: # Each field path is a dot separated path of the field (i.e. "log.source")
      * "level"
      * "details"
      * "log.source"

New Input Type - EDPort In the Edge Delta App, you can use the newly created EDPort Collector Inputs input type to specify a set of ports and protocols for the agent to listen on for incoming traffic.

inputs:
  ed_ports:
    * labels: request
      port: 9000
      protocol: tcp
      read_size: 1
      read_timeout: 30s
      source_detection:
        source_type: "K8s"
        optional: false
        field_mappings:
          k8s_namespace: "kubernetes.namespace"
          k8s_pod_name: "kubernetes.pod.name"
          k8s_container_name: "kubernetes.container.name"
          k8s_container_image: "kubernetes.container.image"
      enrichments:
        from_logs:
          field_mappings:
            * field_name: environment
              json_path: kubernetes.tags.env

To learn more, see Inputs.

February 11, 2022 - Updated Edge Delta App Design

In order to provide a better user experience, the overall look and feel of the Edge Delta App has been updated.

February 1, 2022

While this agent release contains many enhancements and bug fixes, review the following noteworthy updates:

Multi-Threshold Support In the Edge Delta App, you can add multi-threshold settings to an agent configuration file.

For example, you can set a threshold to generate an alert when the following conditions are met:

anomaly score is > 90 response_time_ms.avg is > 250 To set this configuration, use the type parameter, specifically set to AND.

  * name: cluster-errors-multi-threshold
    type: and
    interval: 1m
    conditions:
    * metric_name: http_request_method_updateconfig_latency.avg
      operator: ">="
      value: 100
    * metric_name: http_request_method_deleteconfig_latency.max
      operator: ">"
      value: 125
      consecutive: 5

To learn more, see Thresholds.

Graylog Integration In the Edge Delta App, Graylog is now a supported streaming output.

The Graylog output will stream analytics and insights to your Graylog endpoint. In the app, you can use a YAML file to add Graylog to a configuration.

Dynatrace Integration In the Edge Delta App, Dynatrace is now a supported streaming output.

The Dynatrace output will stream analytics and insights to a Dynatrace environment. In the app, you can use the visual editor or YAML file to add Dynatrace to a configuration.

Updates to Datadog Integration The Datadog Integration has been updated with a new parameter called Send Alert As Datadog Log. With this update, you can now send alerts as logs.

Additional Information This agent release contains additional enhancements and bug fixes.

To see the complete list of changes, please visit the Changelog - Agent Releases page.

January 31, 2022 - Notification for Throttled Data

In the Edge Delta App, a notification has been created to let users know when data ingestion is being throttled.

To learn more about these limits, please contact Edge Delta Support.

December 15, 2021 - VictorOps Integration

In the Edge Delta App, VictorOps is now a supported triggering output.

The VictorOps output streams notifications and alerts to a VictorOps endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

VictorOps is also known as Splunk On-Call; however, the app will refer to this output as simply VictorOps.

December 30, 2021 - AppDynamics Integration

In the Edge Delta App, AppDynamics is now a supported streaming output.

The AppDynamics output will stream analytics and insights to an AppDynamics environment. In the app, you can use the visual editor or YAML file to add AppDynamics to a configuration.

December 17, 2021 - OpenMetrics Integration

In the Edge Delta App, OpenMetrics is now a supported streaming output.

The OpenMetrics output will stream analytics and insights to an OpenMetrics endpoint. In the app, you can use a YAML file or visual editor to add OpenMetrics to a configuration.

December 16, 2021 - S3 Integration

In the Edge Delta App, S3 is now a supported streaming output.

The S3 output will stream analytics and insights to an S3 bucket. In the app, you can use a YAML file or visual editor to add S3 to a configuration.

December 15, 2021 - Cribl Integration

In the Edge Delta App, Cribl is now a supported streaming output.

The Cribl output streams analytics and insights to a Cribl endpoint. In the app, you can use the visual editor or YAML file to add Cribl to a configuration.

December 4, 2021 - Moogsoft Integration

In the Edge Delta App, Moogsoft is now a supported triggering output.

The Moogsoft output will stream notifications and alerts to a specified Moogsoft URL. In the app, you can use the visual editor or YAML file to add Moogsoft to a configuration.

December 1, 2021 - Honeycomb Integration

In the Edge Delta App, Honeycomb is now a supported streaming output.

The Honeycomb output will stream analytics and insights to a Honeycomb environment. In the app, you can use the visual editor or YAML file to add Honeycomb to a configuration.

December 1, 2021 - FluentD Integration

In the Edge Delta App, FluentD is now a supported streaming output.

The FluentD output will stream analytics and insights to your FluentD endpoint. In the app, you can use the visual editor or YAML file to add FluentD to a configuration.

December 1, 2021 - AWS CloudWatch Event Logs Input

In the Edge Delta App, Cloudwatch Event Logs is now a supported input.

The Cloudwatch Event Logs input type allows you to specify a set of AWS CloudWatch Log Events for Edge Delta to monitor. With this input, you can monitor multiple regions and log streams.

In the app, you can use the visual editor or YAML file to add Cloudwatch Event Logs to a configuration.

To learn more, see Inputs.

December 1, 2021 - Big Panda Integration

In the Edge Delta App, Big Panda is now a supported triggering output.

The Big Panda output will stream notifications and alerts to a specified BigPanda endpoint. In the app, you can use the visual editor or YAML file to add Big Panda to a configuration.

December 1, 2021 - Loki Integration

In the Edge Delta App, Loki is now a supported streaming output.

The Loki output streams analytics and insights to your Loki endpoint. In the app, you can use the visual editor or YAML file to add Loki to a configuration.

December 1, 2021 - Logz.io Integration

In the Edge Delta App, Logz.io is now a supported streaming output.

The Logz.io output will stream analytics and insights to your Logz.io endpoint. In the app, you can use the visual editor or YAML file to add VictorOps to a configuration.

November 30, 2021 - Child Configurations

In the Edge Delta App, you can add a configuration into another, existing configuration. With this action, the existing configurations will convert into a parent and child configuration.

To learn more, review the Create and Add a Child Configuration section of the Cloud Configuration Backend (CCB) document.

November 30, 2021 - New Compression and Encoding Settings

In the Edge Delta App, you can change compression and encoding settings for Outputs - Archives.

For encoding, Edge Delta now supports Parquet. For compression, Edge Delta now supports zstd and Snappy. Previously, Edge Delta only offered gzip compression and JSON encoding.

To update these settings, you must access the configuration’s YAML file.

archives:
    * name: my-minio
      type: minio
      access_key: my_access_key_123
      secret_key: my_secret_key_123
      endpoint: play.minio.com:9000
      bucket: ed-test-bucket-minio
      disable_ssl: true
      # Force archive destination to use {endpoint}/{bucket} format instead of {bucket}.{endpoint}/ when reaching buckets.
      s3_force_path_style: true
      encoding: parquet # supported ones: json, parquet
      compression: zstd # supported ones: gzip, zstd, snappy, uncompressed

November 23, 2021 - EDPort Integration

The EDPort streaming integration has been updated to offer JSON as a schema option for sending data.

November 22, 2021 - Local Storage

In the Edge Delta App, Local Storage is now a supported archiving output.

The Local Storage output will send logs to a file on your local machine.

November 18, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported streaming output.

The Azure Event Hubs output will stream analytics and insights to an Azure Event Hubs endpoint. In the app, you can use a YAML file to add Azure Event Hubs to a configuration.

November 12, 2021 - Azure Event Hubs

In the Edge Delta App, Azure Event Hubs is now a supported triggering output.

The Azure Event Hubs output will stream notifications and alerts to a specified Event Hub URL. In the app, you can use the visual editor or YAML file to add Azure Event Hubs to a configuration.

November 9, 2021 - Create a Token

In the Edge Delta App, you can create a token to give your users specific access to the Edge Delta API system.

With tokens, you can specify read / write access for specific backend functionality for your users. In other words, you can create a token to give your users specific read / write access to the Edge Delta API system.