Agent v2.10.0

ClickHouse destination, Splunk Load Balanced destination, HTTP Workflow pagination, Lookup processor multi-telemetry support, and buffer monitoring improvements.

  4 minute read  

December 16, 2025

New Features

  • ClickHouse Destination: Added a new HTTP-based ClickHouse destination node for direct write operations to ClickHouse instances with compressed HTTP support.
  • Splunk Load Balanced Destination: Added a new destination node that retrieves all available indexers from the cluster master and load balances across them, with configurable polling rate for indexer list updates.
  • HTTP Workflow Pagination: Implemented full pagination execution for HTTP Workflow inputs, including Link header parsing (RFC 5988), JSON path URL extraction, parallel page fetching, header inheritance with security controls, and configurable error handling strategies.
  • Google SecOps Destination: Added a new destination node for sending logs to Google SecOps (Chronicle) with support for regional endpoints (US, Europe, Asia), service account authentication or Application Default Credentials, optional customer ID, and gzip compression.
  • Archive Destination Schema Selection: Added support for specifying custom file paths for user-defined schemas in archive destinations using the file://<path> format.

Improvements

  • Datadog Data Fidelity: Improved data fidelity for Datadog metrics and traces including metric type preservation (gauge, rate, count), interval forwarding for rate/count metrics, device field forwarding for per-device system metrics, source type name forwarding, proper hostname extraction for traces, stub handlers for unprocessed Datadog agent endpoints, and handling of uncompressed requests.
  • Lookup Processor Multi-Telemetry Support: Extended the lookup processor to handle metrics and traces in addition to logs, enabling consistent enrichment via lookup tables across all telemetry types.
  • Buffer Event Metrics: Added new metrics to track individual event counts and bytes in the persistent buffer (ed.buffer.memory.events, ed.buffer.disk.events, ed.buffer.memory.event_bytes, ed.buffer.disk.event_bytes) for better visibility into actual data volume being buffered.
  • Self Log Flush Interval: Added dynamic self-log flush interval that uses a more frequent flush (configurable via ED_SELF_LOG_FLUSH_INTERVAL_POST_CONFIG_CHANGE, default 1 minute) for the first 15 minutes after startup or config changes, then reverts to the normal flush interval.
  • Destination Error Reporting: Hardened all destination nodes to report errors to self-telemetry when destinations are unreachable or errors occur during data transmission.
  • OTLP Input Hardening: Added configurable WriteTimeout and IdleTimeout, log writer for HTTP server errors, unified error handling, and self-telemetry error tracking for the OTLP input node.
  • Kafka Source Rate Limiting: Added events_per_minute configuration option to the Kafka source node to control throughput at the source level and prevent overwhelming downstream destinations.
  • Beta Nodes Released: The following nodes are now generally available: Splunk TCP Input, Filebeat Input, SNMP Pull Input, SNMP Trap Input, Syslog Input, Splunk TCP Output, Google BigQuery Output, Securonix Output, Grokstream AIOps Output, and OTTL Context Filter Processor.
  • JSON and Gzip Performance: Improved CPU and GC usage for JSON and Gzip operations.
  • Client-Side Caching for Redis: Added client-side caching to EDXRedis for improved performance.

Bug Fixes

  • Persistent Queue Disk Usage: Fixed an issue where the max_size buffer configuration only limited logical data size but not actual disk usage, which could grow unbounded. The fix adds periodic disk size checks and enforces disk-based write gating. Config Reload: Fixed an issue where items in the buffer were not processed when a configuration reload occurred.
  • HTTP Workflow Pagination Persistence: Fixed an issue where pagination configuration was lost when switching between UI and YAML tabs in the pipeline builder. SSRF Protection: Disabled SSRF protection for HTTP Workflow input when running on user infrastructure, allowing agents to reach internal and private network APIs while keeping protection enabled for backend UI test services.
  • Syslog Input Timestamp Handling: Added defensive logic to replace zero timestamps with the observed time in syslog input, resolving issues with malformed syslog messages. Error Visibility: Fixed syslog parsing errors not being captured in self-telemetry by creating a custom logger that intercepts and forwards errors to the self-telemetry system.
  • Datadog Output Tag Handling: Fixed an issue where Datadog output was incorrectly adding all attributes as tags when using a Datadog input.
  • GCL Field Deletion: Fixed keep_overridden_field_name not working correctly in the GCL destination by ensuring field deletion happens before JSON serialization.
  • Field Deletion in Destinations: Updated all destination nodes to completely remove keys during delete operations instead of just setting values to nil.
  • Compactor Kubernetes API Calls: Reduced excessive Kubernetes API calls from the compactor component by replacing continuous informer-based polling with on-demand resolution.
  • Self Telemetry Node Protection: Prevented disabling of self-telemetry nodes.

Known Issues

  • GCS Destination Excessive Write Requests: A regression introduced in v2.8.0 causes the GCS destination to issue excessive write requests, potentially increasing costs and API usage. This issue is fixed in v2.11.0. Users experiencing high GCS API usage should upgrade to v2.11.0 or later.