Ingest Data from Splunk Forwarders

Overview

Edge Delta can receive data directly from Splunk Universal Forwarders (UF) and Heavy Forwarders (HF), enabling seamless migration from Splunk infrastructure or hybrid deployment scenarios. This integration is ideal for organizations looking to:

• Gradually migrate from Splunk without reconfiguring forwarder agents across infrastructure
• Run hybrid deployments where Edge Delta processes data alongside existing Splunk infrastructure
• Optimize costs by processing and filtering data before selective forwarding to Splunk
• Multi-destination routing to send forwarder data to multiple analytics platforms
• Transform Splunk data using Edge Delta processors before forwarding elsewhere

S2S Protocol v4 Support: Agent version v2.11.0 or higher supports the S2S v4 protocol, enabling compatibility with modern Splunk forwarders without requiring legacy protocol settings on the forwarder side.

Edge Delta supports two methods for receiving data from Splunk forwarders:

Choosing the Right Source Integration

Splunk TCP Source (S2S Protocol)

The Splunk TCP source implements the native Splunk-to-Splunk (S2S) protocol, making it the ideal choice when:

• Migrating from Splunk without modifying existing Universal Forwarder configurations
• Working with on-premises Splunk infrastructure
• Certificate-based authentication is preferred or required
• You need protocol-level compatibility with Splunk forwarders

Key advantages:

• Drop-in replacement for Splunk indexers
• Minimal forwarder reconfiguration required
• Native S2S protocol support
• Preserves Splunk metadata (source, sourcetype, host, index)

Splunk HEC Source

The Splunk HEC source receives data using the HTTP Event Collector protocol, best suited for:

• Modern, cloud-native architectures
• Token-based authentication requirements
• Integration with systems that already send to Splunk HEC
• HTTP-friendly network environments

Key advantages:

• Simple token-based authentication
• HTTP/HTTPS protocol for firewall-friendly communication
• Easy load balancing with standard HTTP load balancers
• Flexible routing options

Method 1: Splunk TCP Source (S2S Protocol)

Configure Edge Delta Splunk TCP Source

Add a Splunk TCP source node to your Edge Delta pipeline.

With v4 protocol (Recommended for v2.11.0+):

nodes:
- name: splunk_tcp_receiver
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997
  listen: "0.0.0.0"
  max_connections: 200

With legacy v3 protocol:

nodes:
- name: splunk_tcp_receiver
  type: splunk_tcp_input
  port: 9997
  listen: "0.0.0.0"
  max_connections: 200

With TLS encryption (applicable to both protocol versions):

nodes:
- name: splunk_tcp_secure
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997
  listen: "0.0.0.0"
  tls:
    enabled: true
    cert_file: /path/to/server.crt
    key_file: /path/to/server.key
    ca_file: /path/to/ca.crt

Configure Splunk Universal Forwarder

The forwarder configuration depends on which protocol version you configured in Edge Delta.

For v4 protocol (v2.11.0+):

# $SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = edge_delta
disabled = false

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true

This simplified configuration works with Edge Delta’s v4 protocol without requiring legacy settings.

For legacy v3 protocol:

# $SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = edge_delta
disabled = false
# REQUIRED for v3: Enable legacy S2S protocol
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false

Important: When using the legacy v3 protocol, the enableOldS2SProtocol = true setting is required. Edge Delta’s v3 protocol uses S2S protocol level 0, which Splunk forwarders reject by default unless this flag is enabled.

After configuration, restart the Splunk Universal Forwarder:

$SPLUNK_HOME/bin/splunk restart

Metadata Preservation

The Splunk TCP source preserves important Splunk metadata as attributes:

These attributes can be used in Edge Delta processors and filters to maintain data organization schemes from Splunk.

Method 2: Splunk HEC Source

Configure Splunk Forwarder for HEC

If using the HEC protocol, configure your Splunk forwarder or application to send data to Edge Delta’s HEC endpoint:

curl -k -H "Authorization: Splunk <your-token>" \
  https://<edge-delta-host>:8088/services/collector \
  -d '{"event": "your log message", "sourcetype": "custom:log"}'

Configure Edge Delta Splunk HEC Source

Add a Splunk HEC source node to your Edge Delta pipeline:

nodes:
- name: splunk_hec_receiver
  type: splunk_hec_input
  port: 8088
  token: your-secure-token-here

Advanced Configurations

Load Balancing Across Multiple Edge Delta Nodes

Distribute forwarder load across multiple Edge Delta agents.

With v4 protocol (v2.11.0+):

# outputs.conf
[tcpout]
defaultGroup = edge_delta_lb
disabled = false

[tcpout:edge_delta_lb]
server = <edge-delta-1>:9997,<edge-delta-2>:9997,<edge-delta-3>:9997
useACK = false
sendCookedData = true
autoLB = true
autoLBFrequency = 30

With legacy v3 protocol:

# outputs.conf
[tcpout]
defaultGroup = edge_delta_lb
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta_lb]
server = <edge-delta-1>:9997,<edge-delta-2>:9997,<edge-delta-3>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false
autoLB = true
autoLBFrequency = 30

Dual Destination: Edge Delta + Splunk

Send data to both Edge Delta and traditional Splunk indexers simultaneously during migration.

With v4 protocol (v2.11.0+):

# outputs.conf
[tcpout]
defaultGroup = edge_delta,splunk_indexers
disabled = false

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

With legacy v3 protocol:

# outputs.conf
[tcpout]
defaultGroup = edge_delta,splunk_indexers
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Selective Routing by Input Type

Route different log sources to different destinations.

With v4 protocol (v2.11.0+):

# inputs.conf
[monitor:///var/log/application/*.log]
_TCP_ROUTING = edge_delta

[monitor:///var/log/system/*.log]
_TCP_ROUTING = splunk_indexers

# outputs.conf
[tcpout]
disabled = false

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

With legacy v3 protocol:

# inputs.conf
[monitor:///var/log/application/*.log]
_TCP_ROUTING = edge_delta

[monitor:///var/log/system/*.log]
_TCP_ROUTING = splunk_indexers

# outputs.conf
[tcpout]
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Migration Strategy

Phased Migration Approach

1. Parallel Processing Phase

   • Configure forwarders to send data to both Edge Delta and Splunk
   • Validate data completeness and accuracy in both systems
   • Monitor Edge Delta pipelines for any processing issues

2. Validation Phase

   • Compare data quality between systems
   • Verify all expected fields are extracted correctly
   • Use Edge Delta’s Live Capture to spot-check data

3. Gradual Cutover

   • Migrate forwarders in groups, starting with non-critical systems
   • Monitor each group for several days before proceeding
   • Maintain rollback capability throughout

4. Full Migration

   • Complete migration of all forwarders to Edge Delta
   • Decommission Splunk indexers (if applicable)
   • Maintain backup configurations for recovery

Linux Permissions Considerations

On some Linux distributions (particularly Ubuntu 24.04 and newer), Edge Delta may need additional permissions to bind to network ports:

# Find the Edge Delta agent binary path
which edgedelta

# Grant network binding capability
sudo setcap 'cap_net_bind_service=+ep' /path/to/edgedelta

# Restart the agent
sudo systemctl restart edgedelta

This capability allows the agent to bind to ports without running as root, providing better security.

Troubleshooting Common Issues

Forwarders Cannot Connect

Symptoms: “Connection refused” errors in forwarder logs

Solutions:

1. Verify Edge Delta is listening: sudo netstat -tulpn | grep 9997
2. Check firewall rules allow incoming connections
3. Verify Edge Delta has permission to bind to the port
4. Review Edge Delta logs for startup errors

Data Not Appearing

Symptoms: Connections succeed but no data flows

Solutions:

1. For v4 protocol: Verify Edge Delta is configured with protocol_version: v4
2. For v3 protocol: Verify enableOldS2SProtocol = true and negotiateProtocolLevel = 0 in forwarder config
3. Check forwarder is actually sending data (review forwarder logs)
4. Use Edge Delta Live Capture to verify data reception

Protocol Compatibility Errors

Symptoms: “unexpected EOF” or “failed to read signature” errors

Solution 1: Upgrade to v4 protocol (Recommended for v2.11.0+)

Configure Edge Delta to use the v4 protocol:

nodes:
- name: splunk_receiver
  type: splunk_tcp_input
  protocol_version: v4
  port: 9997

With v4 protocol, Splunk forwarders work with default settings.

Solution 2: Configure forwarders for legacy v3 protocol

If using the default v3 protocol, ensure all required settings are present in the forwarder configuration:

[tcpout]
enableOldS2SProtocol = true

[tcpout:edge_delta]
sendCookedData = true
negotiateProtocolLevel = 0

Best Practices

Security

• Always use TLS/SSL for production deployments
• Implement proper certificate management and rotation
• Use firewall rules to restrict source IPs
• Regularly audit authentication configurations

Performance

• Configure max_connections based on forwarder count
• Monitor CPU and memory usage on Edge Delta nodes
• Scale horizontally with load balancing for high volumes
• Implement rate limiting for non-critical data sources

Reliability

• Use dual-destination configuration during migrations
• Implement health checks and monitoring
• Maintain backup configurations
• Test rollback procedures before full migration

Related Documentation

• Splunk TCP Source Node
• Splunk HEC Source Node
• Splunk TCP Destination
• Splunk HEC Destination
• Splunk Integration Troubleshooting