Ingest Data from Splunk Forwarders

Overview

Edge Delta can receive data directly from Splunk Universal Forwarders (UF) and Heavy Forwarders (HF), enabling seamless migration from Splunk infrastructure or hybrid deployment scenarios. This integration is ideal for organizations looking to:

• Gradually migrate from Splunk without reconfiguring forwarder agents across infrastructure
• Run hybrid deployments where Edge Delta processes data alongside existing Splunk infrastructure
• Optimize costs by processing and filtering data before selective forwarding to Splunk
• Multi-destination routing to send forwarder data to multiple analytics platforms
• Transform Splunk data using Edge Delta processors before forwarding elsewhere

S2S protocol support: Starting with agent v2.24.0, the Splunk TCP source speaks the S2S v4 protocol that Splunk 9.x and 10.x forwarders use by default, including indexer acknowledgments (useACK). The legacy v3 protocol is still available for forwarders configured to use it.

Edge Delta supports two methods for receiving data from Splunk forwarders:

Choosing the Right Source Integration

Splunk TCP Source (S2S Protocol)

The Splunk TCP source implements the native Splunk-to-Splunk (S2S) protocol, making it the ideal choice when:

• Migrating from Splunk without modifying existing Universal Forwarder configurations
• Working with on-premises Splunk infrastructure
• Certificate-based authentication is preferred or required
• You need protocol-level compatibility with Splunk forwarders

Key advantages:

• Drop-in replacement for Splunk indexers
• Minimal forwarder reconfiguration required
• Native S2S protocol support with acknowledgments
• Preserves Splunk metadata (source, sourcetype, host, index)

Splunk HEC Source

The Splunk HEC source receives data using the HTTP Event Collector protocol, best suited for:

• Modern, cloud-native architectures
• Token-based authentication requirements
• Integration with systems that already send to Splunk HEC
• HTTP-friendly network environments

Key advantages:

• Simple token-based authentication
• HTTP/HTTPS protocol for firewall-friendly communication
• Easy load balancing with standard HTTP load balancers
• Flexible routing options

Method 1: Splunk TCP Source (S2S Protocol)

Version requirement: This page describes agent v2.24.0 and later. Earlier agents support only the v3 protocol, do not send indexer acknowledgments, and require enableOldS2SProtocol = true, negotiateProtocolLevel = 0, and useACK = false on the forwarder. With useACK = true an older agent never acknowledges, the forwarder’s wait queue fills, and it stops sending.

Choose a Protocol Version

Splunk 10.x forwarders ship with enableOldS2SProtocol = false and cannot fall back to the legacy protocol. A v3 source receiving from one of these forwarders logs message size ... exceeds maximum errors and ingests nothing. Switch the source to v4 or enable the legacy protocol on the forwarder.

Both protocol versions support indexer acknowledgments on agent v2.24.0 and later. A forwarder with useACK = true resends frames that were not acknowledged before a connection loss. Events the agent acknowledged and then dropped, for example by a rate limiter, are not resent.

Configure Edge Delta Splunk TCP Source

Add a Splunk TCP source node to your Edge Delta pipeline.

With v4 protocol (recommended):

nodes:
- name: splunk_tcp_receiver
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997
  listen: "0.0.0.0"
  max_connections: 200

With legacy v3 protocol:

nodes:
- name: splunk_tcp_receiver
  type: splunk_tcp_input
  protocol_version: v3
  port: 9997
  listen: "0.0.0.0"
  max_connections: 200

With TLS encryption (applicable to both protocol versions):

nodes:
- name: splunk_tcp_secure
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997
  listen: "0.0.0.0"
  tls:
    enabled: true
    cert_file: /path/to/server.crt
    key_file: /path/to/server.key
    ca_file: /path/to/ca.crt

Complete pipeline example receiving with both protocol versions and forwarding to Edge Delta:

version: v3

links:
  - from: splunk_tcp_v4
    to: splunk_tcp_v4_multiprocessor
  - from: splunk_tcp_v4_multiprocessor
    to: edgedelta_multiprocessor
  - from: splunk_tcp_v3
    to: splunk_tcp_v3_multiprocessor
  - from: splunk_tcp_v3_multiprocessor
    to: edgedelta_multiprocessor
  - from: edgedelta_multiprocessor
    to: edgedelta

nodes:
  - name: splunk_tcp_v4
    type: splunk_tcp_input
    listen: 0.0.0.0
    port: 9997
    read_timeout: 1m
    protocol_version: v4

  - name: splunk_tcp_v3
    type: splunk_tcp_input
    listen: 0.0.0.0
    port: 9998
    read_timeout: 1m
    protocol_version: v3

  - name: splunk_tcp_v4_multiprocessor
    type: sequence
    processors: []

  - name: splunk_tcp_v3_multiprocessor
    type: sequence
    processors: []

  - name: edgedelta_multiprocessor
    type: sequence
    processors: []

  - name: edgedelta
    type: ed_output

Configure Splunk Universal Forwarder

The forwarder configuration depends on which protocol version you configured in Edge Delta. In both cases leave compression off; the Splunk TCP source does not support compressed = true.

For v4 protocol:

# $SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = edgedelta_agent
indexAndForward = false
forwardedindex.filter.disable = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist =

[tcpout:edgedelta_agent]
server = <edge-delta-host>:9997
sendCookedData = true
# negotiateProtocolLevel: leave unset so the forwarder uses its native
# protocol level, which Edge Delta advertises as well.
compressed = false
# Requires agent v2.24.0 or later; set to false on earlier agents.
useACK = true

Keep the forwardedindex filter in place. It stops the forwarder from sending its own internal indexes (_internal, _audit, _introspection, and so on) to Edge Delta; without it, forwarder housekeeping data can dominate the ingested volume.

For legacy v3 protocol:

# $SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = edgedelta_agent
indexAndForward = false
forwardedindex.filter.disable = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist =
# REQUIRED for v3: allow the forwarder to use the legacy S2S protocol.
enableOldS2SProtocol = true

[tcpout:edgedelta_agent]
server = <edge-delta-host>:9997
sendCookedData = true
compressed = false
# Requires agent v2.24.0 or later; set to false on earlier agents.
useACK = true

Important: With protocol_version: v3, enableOldS2SProtocol = true is required on the forwarder. Without it, Splunk 10.x forwarders switch to the v4 protocol during the handshake and the source cannot decode the stream. Agent v2.24.0 and later no longer need negotiateProtocolLevel = 0; earlier agents still require it.

After configuration, restart the Splunk Universal Forwarder:

$SPLUNK_HOME/bin/splunk restart

Event Handling

Forwarders send file data in chunks, not events. With protocol_version: v4, the source splits each chunk into one event per line and rejoins lines cut between two chunks, so events match the lines in the source file. With protocol_version: v3, each chunk becomes a single event, which can span several lines.

The source drops forwarder bookkeeping messages such as ForwarderInfo heartbeats.

Metadata Preservation

The Splunk TCP source preserves Splunk metadata under the _splunk_tcp_event attribute:

Use these attributes in processors and filters, or map them back onto outgoing Splunk payloads with a Splunk Mapper when forwarding to a Splunk TCP destination:

- name: splunk_mapper
  type: splunk_mapper
  splunk_time: item["timestamp"]
  splunk_host: item["attributes"]["_splunk_tcp_event"]["host"]
  splunk_source: item["attributes"]["_splunk_tcp_event"]["source"]
  splunk_source_type: item["attributes"]["_splunk_tcp_event"]["sourcetype"]
  splunk_index: item["attributes"]["_splunk_tcp_event"]["index"]
  splunk_event: item["body"]

Method 2: Splunk HEC Source

Configure Splunk Forwarder for HEC

If using the HEC protocol, configure your Splunk forwarder or application to send data to Edge Delta’s HEC endpoint:

curl -k -H "Authorization: Splunk <your-token>" \
  https://<edge-delta-host>:8088/services/collector \
  -d '{"event": "your log message", "sourcetype": "custom:log"}'

Configure Edge Delta Splunk HEC Source

Add a Splunk HEC source node to your Edge Delta pipeline:

nodes:
- name: splunk_hec_receiver
  type: splunk_hec_input
  port: 8088
  token: your-secure-token-here

Advanced Configurations

The examples below use the v4 protocol and useACK = true, which need agent v2.24.0 or later. For a v3 source, add enableOldS2SProtocol = true to the [tcpout] stanza. Keep the forwardedindex filter from the samples above in every configuration.

Load Balancing Across Multiple Edge Delta Nodes

Distribute forwarder load across multiple Edge Delta agents.

# outputs.conf
[tcpout]
defaultGroup = edge_delta_lb

[tcpout:edge_delta_lb]
server = <edge-delta-1>:9997,<edge-delta-2>:9997,<edge-delta-3>:9997
sendCookedData = true
compressed = false
useACK = true
autoLB = true
autoLBFrequency = 30

Dual Destination: Edge Delta + Splunk

Send data to both Edge Delta and traditional Splunk indexers simultaneously during migration.

# outputs.conf
[tcpout]
defaultGroup = edge_delta,splunk_indexers

[tcpout:edge_delta]
server = <edge-delta-host>:9997
sendCookedData = true
compressed = false
useACK = true

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Selective Routing by Input Type

Route different log sources to different destinations.

# inputs.conf
[monitor:///var/log/application/*.log]
_TCP_ROUTING = edge_delta

[monitor:///var/log/system/*.log]
_TCP_ROUTING = splunk_indexers

# outputs.conf
[tcpout:edge_delta]
server = <edge-delta-host>:9997
sendCookedData = true
compressed = false
useACK = true

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Migration Strategy

Phased Migration Approach

1. Parallel Processing Phase

   • Configure forwarders to send data to both Edge Delta and Splunk
   • Validate data completeness and accuracy in both systems
   • Monitor Edge Delta pipelines for any processing issues

2. Validation Phase

   • Compare data quality between systems
   • Verify all expected fields are extracted correctly
   • Use Edge Delta’s Live Capture to spot-check data

3. Gradual Cutover

   • Migrate forwarders in groups, starting with non-critical systems
   • Monitor each group for several days before proceeding
   • Maintain rollback capability throughout

4. Full Migration

   • Complete migration of all forwarders to Edge Delta
   • Decommission Splunk indexers (if applicable)
   • Maintain backup configurations for recovery

Linux Permissions Considerations

On some Linux distributions (particularly Ubuntu 24.04 and newer), Edge Delta may need additional permissions to bind to network ports:

# Find the Edge Delta agent binary path
which edgedelta

# Grant network binding capability
sudo setcap 'cap_net_bind_service=+ep' /path/to/edgedelta

# Restart the agent
sudo systemctl restart edgedelta

This capability allows the agent to bind to ports without running as root, providing better security.

Troubleshooting Common Issues

Forwarders Cannot Connect

Symptoms: “Connection refused” errors in forwarder logs

Solutions:

1. Verify Edge Delta is listening: sudo netstat -tulpn | grep 9997
2. Check firewall rules allow incoming connections
3. Verify Edge Delta has permission to bind to the port
4. Review Edge Delta logs for startup errors

Data Not Appearing

Symptoms: Connections succeed but no data flows

Solutions:

1. For v4 protocol: Verify Edge Delta is configured with protocol_version: v4
2. For v3 protocol: Verify enableOldS2SProtocol = true in the forwarder’s outputs.conf
3. Verify compressed = false on the forwarder
4. Check forwarder is actually sending data (review forwarder logs)
5. Use Edge Delta Live Capture to verify data reception

Protocol Mismatch

Symptoms: Edge Delta logs message size ... exceeds maximum ... the forwarder may be sending s2s v4 frames, or the forwarder reconnects every 30 seconds without delivering data.

A forwarder is sending the v4 protocol to a source configured for v3.

Solution 1: Switch the source to v4 (recommended)

nodes:
- name: splunk_receiver
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997

Solution 2: Keep the legacy protocol on the forwarder

[tcpout]
enableOldS2SProtocol = true

Forwarder Stops Sending After a Few Seconds

Symptoms: The forwarder connects, sends a burst of data, then goes quiet; its splunkd.log shows the acknowledgment wait queue filling.

The forwarder has useACK = true but the agent is older than v2.24.0 and never sends acknowledgments. Upgrade the agent or set useACK = false.

Unknown Frame or Entry Marker Errors

Symptoms: Edge Delta logs unknown v4 frame type or unknown entry marker errors and the forwarder reconnects.

The source closes the connection rather than guess at data it does not recognize. Check that the forwarder has compressed = false, then share the error details with Edge Delta support.

Best Practices

Security

• Always use TLS/SSL for production deployments
• Implement proper certificate management and rotation
• Use firewall rules to restrict source IPs
• Regularly audit authentication configurations

Performance

• Configure max_connections based on forwarder count
• Monitor CPU and memory usage on Edge Delta nodes
• Scale horizontally with load balancing for high volumes
• Implement rate limiting for non-critical data sources

Reliability

• On agent v2.24.0 and later, enable useACK = true so forwarders resend frames that were not acknowledged before a connection loss
• Use dual-destination configuration during migrations
• Implement health checks and monitoring
• Maintain backup configurations
• Test rollback procedures before full migration

Related Documentation

• Splunk TCP Source Node
• Splunk HEC Source Node
• Splunk TCP Destination
• Splunk HEC Destination
• Splunk Integration Troubleshooting