Ingest Data from Splunk Forwarders

Overview

Edge Delta can receive data directly from Splunk Universal Forwarders (UF) and Heavy Forwarders (HF), enabling seamless migration from Splunk infrastructure or hybrid deployment scenarios. This integration is ideal for organizations looking to:

• Gradually migrate from Splunk without reconfiguring forwarder agents across infrastructure
• Run hybrid deployments where Edge Delta processes data alongside existing Splunk infrastructure
• Optimize costs by processing and filtering data before selective forwarding to Splunk
• Multi-destination routing to send forwarder data to multiple analytics platforms
• Transform Splunk data using Edge Delta processors before forwarding elsewhere

Edge Delta supports two methods for receiving data from Splunk forwarders:

Choosing the Right Source Integration

Splunk TCP Source (S2S Protocol)

The Splunk TCP source implements the native Splunk-to-Splunk (S2S) protocol, making it the ideal choice when:

• Migrating from Splunk without modifying existing Universal Forwarder configurations
• Working with on-premises Splunk infrastructure
• Certificate-based authentication is preferred or required
• You need protocol-level compatibility with Splunk forwarders

Key advantages:

• Drop-in replacement for Splunk indexers
• Minimal forwarder reconfiguration required
• Native S2S protocol support
• Preserves Splunk metadata (source, sourcetype, host, index)

Splunk HEC Source

The Splunk HEC source receives data using the HTTP Event Collector protocol, best suited for:

• Modern, cloud-native architectures
• Token-based authentication requirements
• Integration with systems that already send to Splunk HEC
• HTTP-friendly network environments

Key advantages:

• Simple token-based authentication
• HTTP/HTTPS protocol for firewall-friendly communication
• Easy load balancing with standard HTTP load balancers
• Flexible routing options

Method 1: Splunk TCP Source (S2S Protocol)

Configure Splunk Universal Forwarder

To send data from Splunk Universal Forwarders to Edge Delta, modify the forwarder’s outputs.conf file:

# $SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = edge_delta
disabled = false
# REQUIRED: Enable legacy S2S protocol for Edge Delta compatibility
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
# Disable indexer acknowledgment
useACK = false
# Send cooked data (parsed events)
sendCookedData = true
# Use protocol level 0 for compatibility
negotiateProtocolLevel = 0
# Disable compression
compressed = false

Important: The enableOldS2SProtocol = true setting is required for Splunk Universal Forwarders to communicate with Edge Delta. Edge Delta’s Splunk TCP source currently uses S2S protocol level 0, which Splunk forwarders reject by default unless this flag is explicitly enabled.

After configuration, restart the Splunk Universal Forwarder:

$SPLUNK_HOME/bin/splunk restart

Configure Edge Delta Splunk TCP Source

Add a Splunk TCP source node to your Edge Delta pipeline:

nodes:
- name: splunk_tcp_receiver
  type: splunk_tcp_input
  port: 9997
  listen: "0.0.0.0"
  max_connections: 200

With TLS encryption:

nodes:
- name: splunk_tcp_secure
  type: splunk_tcp_input
  port: 9997
  listen: "0.0.0.0"
  tls:
    enabled: true
    cert_file: /path/to/server.crt
    key_file: /path/to/server.key
    ca_file: /path/to/ca.crt

Metadata Preservation

The Splunk TCP source preserves important Splunk metadata as attributes:

These attributes can be used in Edge Delta processors and filters to maintain data organization schemes from Splunk.

Method 2: Splunk HEC Source

Configure Splunk Forwarder for HEC

If using the HEC protocol, configure your Splunk forwarder or application to send data to Edge Delta’s HEC endpoint:

curl -k -H "Authorization: Splunk <your-token>" \
  https://<edge-delta-host>:8088/services/collector \
  -d '{"event": "your log message", "sourcetype": "custom:log"}'

Configure Edge Delta Splunk HEC Source

Add a Splunk HEC source node to your Edge Delta pipeline:

nodes:
- name: splunk_hec_receiver
  type: splunk_hec_input
  port: 8088
  token: your-secure-token-here

Advanced Configurations

Load Balancing Across Multiple Edge Delta Nodes

Distribute forwarder load across multiple Edge Delta agents:

# outputs.conf
[tcpout]
defaultGroup = edge_delta_lb
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta_lb]
server = <edge-delta-1>:9997,<edge-delta-2>:9997,<edge-delta-3>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false
autoLB = true
autoLBFrequency = 30

Dual Destination: Edge Delta + Splunk

Send data to both Edge Delta and traditional Splunk indexers simultaneously during migration:

# outputs.conf
[tcpout]
defaultGroup = edge_delta,splunk_indexers
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Selective Routing by Input Type

Route different log sources to different destinations:

# inputs.conf
[monitor:///var/log/application/*.log]
_TCP_ROUTING = edge_delta

[monitor:///var/log/system/*.log]
_TCP_ROUTING = splunk_indexers

# outputs.conf
[tcpout]
disabled = false
enableOldS2SProtocol = true

[tcpout:edge_delta]
server = <edge-delta-host>:9997
useACK = false
sendCookedData = true
negotiateProtocolLevel = 0
compressed = false

[tcpout:splunk_indexers]
server = <splunk-indexer>:9997
useACK = true

Migration Strategy

Phased Migration Approach

1. Parallel Processing Phase

   • Configure forwarders to send data to both Edge Delta and Splunk
   • Validate data completeness and accuracy in both systems
   • Monitor Edge Delta pipelines for any processing issues

2. Validation Phase

   • Compare data quality between systems
   • Verify all expected fields are extracted correctly
   • Use Edge Delta’s Live Capture to spot-check data

3. Gradual Cutover

   • Migrate forwarders in groups, starting with non-critical systems
   • Monitor each group for several days before proceeding
   • Maintain rollback capability throughout

4. Full Migration

   • Complete migration of all forwarders to Edge Delta
   • Decommission Splunk indexers (if applicable)
   • Maintain backup configurations for recovery

Linux Permissions Considerations

On some Linux distributions (particularly Ubuntu 24.04 and newer), Edge Delta may need additional permissions to bind to network ports:

# Find the Edge Delta agent binary path
which edgedelta

# Grant network binding capability
sudo setcap 'cap_net_bind_service=+ep' /path/to/edgedelta

# Restart the agent
sudo systemctl restart edgedelta

This capability allows the agent to bind to ports without running as root, providing better security.

Troubleshooting Common Issues

Forwarders Cannot Connect

Symptoms: “Connection refused” errors in forwarder logs

Solutions:

1. Verify Edge Delta is listening: sudo netstat -tulpn | grep 9997
2. Check firewall rules allow incoming connections
3. Verify Edge Delta has permission to bind to the port
4. Review Edge Delta logs for startup errors

Data Not Appearing

Symptoms: Connections succeed but no data flows

Solutions:

1. Verify enableOldS2SProtocol = true in forwarder config
2. Ensure negotiateProtocolLevel = 0 is set
3. Check forwarder is actually sending data (review forwarder logs)
4. Use Edge Delta Live Capture to verify data reception

Protocol Compatibility Errors

Symptoms: “unexpected EOF” or “failed to read signature” errors

Solutions: Ensure all required settings are present in the forwarder configuration:

[tcpout]
enableOldS2SProtocol = true

[tcpout:edge_delta]
sendCookedData = true
negotiateProtocolLevel = 0

Best Practices

Security

• Always use TLS/SSL for production deployments
• Implement proper certificate management and rotation
• Use firewall rules to restrict source IPs
• Regularly audit authentication configurations

Performance

• Configure max_connections based on forwarder count
• Monitor CPU and memory usage on Edge Delta nodes
• Scale horizontally with load balancing for high volumes
• Implement rate limiting for non-critical data sources

Reliability

• Use dual-destination configuration during migrations
• Implement health checks and monitoring
• Maintain backup configurations
• Test rollback procedures before full migration

Related Documentation

• Splunk TCP Source Node
• Splunk HEC Source Node
• Splunk TCP Destination
• Splunk HEC Destination
• Splunk Integration Troubleshooting