Troubleshooting

Diagnose and resolve common syslog configuration and performance issues.

  4 minute read  

This page provides solutions for common issues when configuring and operating the Edge Delta Syslog source node.

Messages Not Received

Symptoms:

  • Syslog messages not appearing in Edge Delta pipelines
  • Source indicates successful sends but nothing received

Solutions:

  1. Network Connectivity:

    # Test basic connectivity
telnet <edge-delta-host> 514

# Test UDP port (requires netcat)
nc -u -z <edge-delta-host> 514

  2. Firewall Rules:

    # Check if port is listening
sudo netstat -tulpn | grep 514

# Verify firewall allows traffic
sudo iptables -L -n | grep 514

  3. Port Permissions:

    • Ports below 1024 require root/elevated privileges
    • Use ports 5514, 6514, or similar for non-root agents
    • Check Edge Delta startup logs for “permission denied” errors

  4. Protocol Mismatch:

    • Verify source sends UDP and Edge Delta listens on UDP (or TCP/TCP)
    • Check RFC version match (rfc3164 vs rfc5424)

FortiGate Logs Truncated or Corrupted

Symptoms:

  • FortiGate messages incomplete
  • Parsing errors in Edge Delta logs
  • Messages cut off mid-stream

Root Cause: Missing enable_octet_counting: true parameter

Solution:

nodes:
  - name: fortigate_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514
    enable_octet_counting: true  # REQUIRED for FortiGate reliable mode
    max_log_size: 10MiB

Verification:

# FortiGate must use mode reliable
config log syslogd setting
    get
end

# Look for: mode: reliable

High CPU Usage

Symptoms:

  • CPU utilization consistently > 80%
  • Messages processing slowly
  • System becoming unresponsive

Diagnostic:

# Check CPU usage
top -p $(pgrep edge-delta)

# Monitor message rate
# Check Edge Delta metrics

Solutions:

1. Disable Performance Features if Not Needed:

add_attributes: false  # Saves 2-3% CPU

2. Reduce Async Processor Count:

udp_async:
  readers: 4
  processors_count: 2  # Reduce if CPU-bound

3. Distribute Load:

  • Deploy multiple Edge Delta agents
  • Use DNS round-robin or load balancer
  • Split by device type or facility

4. Filter at Source:

  • Configure syslog sources to send only necessary logs
  • Use facility/severity filtering
  • Reduce debug-level logs

Dropped UDP Packets

Symptoms:

  • netstat -su shows increasing “packet receive errors”
  • Message loss during traffic bursts
  • Inconsistent log delivery

Diagnostic:

# Check current drops
netstat -su | grep -i "receive errors"

# Monitor in real-time
watch -n 1 'netstat -su | grep "packet receive errors"'

# Check socket buffer size
sysctl net.core.rmem_default
sysctl net.core.rmem_max

Solutions:

1. Enable Async UDP Processing:

udp_async:
  readers: 4      # Start here
  processors_count: 4
  max_queue_length: 1000

2. Increase Queue Length:

udp_async:
  readers: 4
  processors_count: 4
  max_queue_length: 5000  # Increase for bursts

3. Tune OS Socket Buffers:

# Temporarily increase
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.rmem_default=67108864

# Make permanent
echo "net.core.rmem_max=134217728" | sudo tee -a /etc/sysctl.conf
echo "net.core.rmem_default=67108864" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

4. Scale Horizontally:

  • Add more Edge Delta agents
  • Use different ports per agent
  • Load balance with DNS or hardware LB

Memory Issues

Symptoms:

  • Edge Delta agent consuming excessive memory
  • Out of memory errors
  • Agent crashes under load

Diagnostic:

# Check memory usage
ps aux | grep edge-delta

# Monitor memory over time
watch -n 5 'ps aux | grep edge-delta'

Solutions:

1. Reduce Max Log Size:

max_log_size: 1MiB  # Reduce if seeing large messages

2. Reduce Queue Length:

udp_async:
  max_queue_length: 500  # Each queued message uses ~1-2KB

3. Check for Memory Leaks:

  • Review Edge Delta agent version
  • Update to latest version
  • Check for known issues in release notes

4. Monitor Downstream Processing:

  • Slow downstream causes message accumulation
  • Check pipeline processors for bottlenecks
  • Verify output destinations are accepting data

Parsing Errors

Symptoms:

  • Messages marked as unparseable in logs
  • Missing or incorrect field extraction
  • Unexpected severity_text or priority values

Solutions:

1. Verify Protocol Setting:

# RFC 3164 (BSD syslog) - https://www.rfc-editor.org/rfc/rfc3164.html
protocol: rfc3164  # Format: <PRI>MMM DD HH:MM:SS hostname tag: message

# RFC 5424 (Structured syslog) - https://www.rfc-editor.org/rfc/rfc5424.html
protocol: rfc5424  # Format: <PRI>VERSION TIMESTAMP HOSTNAME APP PID MSGID [SD] MESSAGE

2. Check Encoding:

encoding: utf-8  # Match source encoding
# Other options: ascii, utf-16le, utf-16be, big5, latin-1

3. Test with Known-Good Message:

# RFC 3164 test - https://www.rfc-editor.org/rfc/rfc3164.html
echo '<134>Oct 6 12:00:00 test syslog: Test message' | nc <host> 514

# RFC 5424 test - https://www.rfc-editor.org/rfc/rfc5424.html
echo '<134>1 2025-10-06T12:00:00Z test app - - - Test message' | nc <host> 514

Structured Data Not Parsing

Symptoms:

  • structured_data field empty or missing
  • RFC 5424 messages not showing structured elements

Solutions:

1. Verify RFC 5424 Protocol:

protocol: rfc5424  # Required for structured data - https://www.rfc-editor.org/rfc/rfc5424.html

2. Check Message Format:

# Correct format with structured data
echo '<34>1 2025-10-06T12:00:00Z host app 123 ID [sd@123 key="value"] Message' | nc <host> 514

# Structured data must be in [SD-ID@PEN PARAM="VALUE"] format

3. Verify No PRI Header Skip:

allow_skip_pri_header: false  # Must be false for proper parsing

Timestamp Issues

Symptoms:

  • Incorrect timestamps in parsed logs
  • Timezone offsets wrong
  • Timestamps in future or past

Solutions:

1. Set Location for Timezone-less Logs:

location: America/New_York  # IANA timezone name

2. Verify NTP Synchronization:

# Check time sync on source systems
timedatectl status

# Ensure NTP is active

3. RFC 5424 Includes Timezone:

protocol: rfc5424  # Includes timezone in ISO 8601 format - https://www.rfc-editor.org/rfc/rfc5424.html

Performance Issues

See the Performance Tuning page for comprehensive performance troubleshooting.

Quick checklist:

  • Enable async UDP for high volume
  • Tune OS socket buffers for >50K msgs/sec
  • Scale horizontally for >100K msgs/sec
  • Monitor CPU, memory, and network saturation