Performance Tuning

Edge Delta’s syslog implementation includes several performance optimization features that enable it to handle enterprise-scale log volumes while maintaining low resource utilization.

TCP Performance Optimization

Baseline TCP Configuration

Start with a conservative TCP configuration and scale based on observed performance:

nodes:
  - name: baseline_tcp_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514
    max_log_size: 1MiB

This provides a solid baseline for TCP syslog processing.

Optimized TCP Configuration

For high-volume TCP syslog, consider these performance optimizations:

nodes:
  - name: optimized_tcp_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514
    max_log_size: 5MiB
    add_attributes: false

Performance Considerations for add_attributes:

• Minimal CPU overhead
• Slight increase in data size (network attributes per message)
• Disable if downstream systems don’t use OpenTelemetry conventions

UDP Performance Optimization

Baseline UDP Configuration

Standard UDP configuration without async processing:

nodes:
  - name: baseline_udp_syslog
    type: syslog_input
    protocol: rfc3164
    transport_protocol: udp
    port: 514

This single-threaded configuration is suitable for low to moderate volume environments.

High-Performance Async UDP Configuration

Enable async processing for dramatic performance improvements:

nodes:
  - name: high_perf_udp_syslog
    type: syslog_input
    protocol: rfc3164
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 4
      processors_count: 4
      max_queue_length: 2000

Scaling Characteristics by Configuration:

Tuning Async Parameters

Reader Count: Number of goroutines reading from the UDP socket

• Start with CPU core count / 2
• Increase if receiving dropped packets (check netstat -su)
• Maximum benefit typically at 8-16 readers

Processor Count: Number of goroutines processing messages

• Match reader count for balanced pipeline
• Can be lower if parsing is lightweight
• Can be higher if heavy downstream processing

Queue Length: Buffer between readers and processors

• Start with 500-1000 for normal traffic
• Increase to 2000-5000 for bursty traffic patterns
• Monitor for queue full conditions
• Each queued message consumes ~1-2KB memory

Example for Ultra High-Volume Environments:

nodes:
  - name: ultra_high_volume_syslog
    type: syslog_input
    protocol: rfc3164
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 16
      processors_count: 16
      max_queue_length: 10000

System Requirements:

• 16+ CPU cores
• 8+ GB RAM
• 10 Gbps network interface
• OS socket buffer tuning (see below)

Operating System Tuning

For very high volume scenarios, tune OS-level parameters:

Linux Socket Buffer Tuning

# Increase UDP receive buffer size
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.rmem_default=134217728

# For persistence across reboots
echo "net.core.rmem_max=134217728" | sudo tee -a /etc/sysctl.conf
echo "net.core.rmem_default=134217728" | sudo tee -a /etc/sysctl.conf

Check for Dropped Packets

# Monitor UDP socket statistics
watch -n 1 'netstat -su | grep -i "packet receive errors\|buffer errors"'

# If seeing drops, increase queue length or add more readers

Resource Sizing Guidelines

Small Deployment (Low Volume)

# CPU: 1-2 cores
# Memory: 512MB - 1GB
# Network: 1 Gbps
nodes:
  - name: small_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514

Medium Deployment (Moderate Volume)

# CPU: 2-4 cores
# Memory: 2-4GB
# Network: 1-10 Gbps
nodes:
  - name: medium_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 4
      processors_count: 4
      max_queue_length: 1000

Large Deployment (High Volume)

# CPU: 8-16 cores
# Memory: 8-16GB
# Network: 10 Gbps
nodes:
  - name: large_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 8
      processors_count: 8
      max_queue_length: 5000

Enterprise Deployment (Very High Volume)

# CPU: 16-32 cores
# Memory: 16-32GB
# Network: 10-40 Gbps
# Consider horizontal scaling with multiple Edge Delta agents
nodes:
  - name: enterprise_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 16
      processors_count: 16
      max_queue_length: 10000

Monitoring and Metrics

Track these key metrics to ensure optimal performance:

CPU Utilization:

• Target: 60-70% under normal load
• If consistently > 80%: Scale horizontally or increase resources
• If < 30%: Reduce async readers/processors to free resources

Memory Usage:

• Baseline: ~100-200MB
• With async queues: +1-2MB per 1000 queue length
• Monitor for steady growth (indicates memory leak)

Message Rate:

• Track messages/second received
• Compare to expected volume
• Alert on sudden drops (indicates upstream issues)

Dropped Packets (UDP only):

• Check netstat -su for “packet receive errors”
• Should be 0 or near-zero
• If increasing: Tune queue length or OS buffers

Troubleshooting Performance Issues

High CPU Usage:

1. Reduce async processors if CPU-bound
2. Disable add_attributes if not needed
3. Consider splitting load across multiple agents

Dropped UDP Packets:

1. Increase max_queue_length
2. Increase async readers
3. Tune OS socket buffers (see above)
4. Distribute load across multiple agents/ports

High Memory Usage:

1. Reduce max_queue_length
2. Reduce max_log_size to prevent large message accumulation
3. Check for message buildup (downstream processing slow)

Best Practices

Port Selection

When configuring the Syslog source node, choose ports above 1024 to avoid requiring root privileges for the Edge Delta agent. The standard syslog port 514 requires elevated permissions, which may not be available in containerized environments. Common alternatives include port 5514 for UDP traffic and port 6514 for TCP connections, both of which can be bound by non-privileged processes while maintaining compatibility with most syslog clients.

Protocol Selection

For new deployments, RFC 5424 is the recommended syslog format due to its enhanced structure, support for structured data elements, and precise ISO 8601 timestamp formatting with timezone information. This modern format provides better metadata handling and allows for key-value pairs within the message structure. Only use RFC 3164 when dealing with legacy systems that cannot be upgraded, as this older format lacks the structured data capabilities and has limited timestamp precision that can lead to parsing ambiguities.

Transport Selection

The choice between UDP and TCP transport protocols depends on your specific requirements for reliability versus performance. UDP is ideal for high-volume, non-critical log streams where some message loss is acceptable in exchange for lower overhead and better throughput. This connectionless protocol minimizes resource usage and handles traffic spikes more gracefully. TCP should be used for critical logs that require guaranteed delivery, such as security events or compliance data, though it comes with higher resource consumption and connection management overhead. Consider your network’s bandwidth limitations and reliability characteristics when making this decision.

Security Considerations

Traditional syslog transmits data in plain text without encryption, making it vulnerable to interception and tampering. Implement strict firewall rules to restrict which IP addresses can send syslog messages to your Edge Delta agents, reducing the risk of log injection attacks. Regular monitoring for suspicious patterns or potential message spoofing attempts is essential, particularly when receiving logs from internet-facing sources. For highly sensitive environments, consider using dedicated network segments or VPNs for syslog traffic.

Performance Optimization

To optimize performance in high-volume environments, start with UDP transport to minimize processing overhead while accepting that some messages may be lost during traffic spikes. Adjust receive buffer sizes based on your message volume and burst patterns to prevent drops during peak periods. For environments generating substantial log volumes, distribute the load across multiple Edge Delta agents using DNS round-robin or load balancer configurations. Continuously monitor CPU and memory usage of your Edge Delta agents, adjusting resource allocations and scaling horizontally when utilization consistently exceeds 70%. Regular analysis of message patterns can help identify opportunities to filter unnecessary logs at the source, reducing overall processing requirements.