Edge Delta Syslog Source

Overview

The Syslog source node enables Edge Delta to receive and process syslog messages sent from various sources including network devices, servers, applications, and security appliances. It supports both RFC 3164 (traditional BSD syslog format) and RFC 5424 (structured syslog format with enhanced metadata) over UDP or TCP transport protocols, allowing you to centralize log collection from your entire infrastructure.

This node requires Edge Delta agent version v2.5.0 or higher.

AI Team: Configure this source using the Syslog connector for streamlined setup in AI Team.

• outgoing_data_types: log

The Syslog source node is ideal for:

• Network Device Monitoring: Collect logs from routers, switches, firewalls, and load balancers
• Application Logging: Receive logs from applications configured to send syslog
• Security Event Collection: Gather security events from IDS/IPS systems and firewalls
• System Log Aggregation: Centralize Linux/Unix system logs
• Legacy System Integration: Collect logs from older systems that only support syslog

Field Mapping

The Syslog source node extracts and maps the following fields from incoming messages:

Severity Mapping

Syslog priorities are automatically mapped to severity levels:

RFC 5424 Structured Data

When using RFC 5424 protocol, the Syslog source node automatically parses structured data elements into nested JSON objects. Structured data provides a standardized way to include key-value metadata within syslog messages, commonly used by enterprise devices like FortiGate firewalls, Cisco devices, and F5 BIG-IP appliances.

Example RFC 5424 Message with Structured Data:

<34>1 2025-10-06T12:00:00Z firewall01 fortios 5678 ID47 [fortigate@32473 vd="root" logid="0000000013" type="traffic" subtype="forward" level="notice"] Traffic allowed from 10.0.0.5 to 192.168.1.10

Extracted Structured Data:

{
  "structured_data": {
    "fortigate@32473": {
      "vd": "root",
      "logid": "0000000013",
      "type": "traffic",
      "subtype": "forward",
      "level": "notice"
    }
  }
}

This automatic parsing allows you to use these structured fields in downstream processors for filtering, routing, and enrichment without additional parsing configuration.

RFC 6587 Framing Support

For reliable TCP syslog transmission, Edge Delta supports RFC 6587 framing methods, which are required by enterprise devices like FortiGate firewalls operating in “reliable” mode. RFC 6587 provides two framing mechanisms to handle multi-line messages and prevent message boundary ambiguity in TCP streams.

Octet Counting

Octet Counting prefixes each message with its byte length, providing reliable message framing even when messages contain embedded newlines or span multiple lines. This is the most robust framing method for TCP syslog.

Configuration:

nodes:
  - name: fortigate_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514
    enable_octet_counting: true
    max_octets: 8192

Message Format:

123 <34>1 2025-10-06T12:00:00Z host app - - - Message content here

The number 123 indicates the message is 123 bytes long, allowing the parser to read exactly that many bytes regardless of content.

Non-Transparent Framing

Non-Transparent Framing uses a delimiter (typically LF newline or NUL character) to mark message boundaries. This method is simpler but less robust than octet counting for messages containing the delimiter character.

Configuration:

nodes:
  - name: legacy_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: tcp
    port: 514
    non_transparent_framing_trailer: LF

When to Use RFC 6587:

• FortiGate firewalls configured with mode reliable
• Any TCP syslog source requiring guaranteed message boundaries
• Logs containing multi-line content (stack traces, formatted output)
• High-reliability environments where message integrity is critical

Example Configurations

Basic RFC 5424 UDP Configuration

This configuration receives RFC 5424 formatted syslog messages over UDP on port 5514. Using a non-standard port helps avoid permission requirements for privileged ports below 1024.

nodes:
- name: syslog_rfc5424
  type: syslog_input
  port: 5514
  protocol: rfc5424
  transport_protocol: udp

RFC 3164 TCP Configuration

This configuration receives traditional RFC 3164 syslog messages over TCP for reliable delivery. TCP ensures messages are not lost during transmission.

nodes:
- name: syslog_traditional
  type: syslog_input
  port: 6514
  protocol: rfc3164
  transport_protocol: tcp
  listen: "0.0.0.0"

TLS-Secured TCP Configuration

This configuration demonstrates secure syslog over TCP using TLS encryption. TLS protects syslog messages from interception and tampering during transmission.

nodes:
- name: secure_syslog
  type: syslog_input
  port: 6514
  protocol: rfc5424
  transport_protocol: tcp
  otel_tls:
    cert_file: /var/run/secrets/syslog-tls/tls.crt
    key_file: /var/run/secrets/syslog-tls/tls.key
    ca_file: /var/run/secrets/syslog-tls/ca-bundle.crt
    client_ca_file: /var/run/secrets/syslog-tls/client-ca.crt

Advanced Configuration with Encoding

This configuration demonstrates advanced options including custom encoding and listening address specification for multi-homed systems.

nodes:
- name: syslog_advanced
  type: syslog_input
  port: 514
  protocol: rfc5424
  transport_protocol: udp
  listen: "0.0.0.0"
  encoding: utf-8
  location: America/New_York

High-Performance Async UDP Processing

Edge Delta provides async UDP processing capabilities that significantly outperform traditional synchronous syslog implementations. This feature enables concurrent message reading and processing, providing substantial throughput improvements for high-volume environments.

Async UDP Configuration

Enable async processing with multiple concurrent readers and processors:

nodes:
  - name: high_volume_syslog
    type: syslog_input
    protocol: rfc3164
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 4
      processors_count: 4
      max_queue_length: 1000

Performance Characteristics

Async processing improves throughput for high-volume syslog environments. Performance scales with the number of readers and processors configured:

• Synchronous (default): Single-threaded processing, lower throughput but minimal resource usage
• Async (2 readers, 2 processors): Moderate throughput improvement with 2-core utilization
• Async (4 readers, 4 processors): Significant throughput improvement with 4-core utilization
• Async (8+ readers, 8+ processors): Maximum throughput for high-volume scenarios with proportional CPU usage

Tuning Async Processing

Start Conservative: Begin with 2 readers and 2 processors, then scale up based on load:

udp_async:
  readers: 2
  processors_count: 2
  max_queue_length: 500

Scale Based on CPU: Increase readers/processors if CPU utilization remains below 70% under load.

Monitor Queue Depth: If messages are being dropped, increase max_queue_length:

udp_async:
  readers: 4
  processors_count: 4
  max_queue_length: 5000  # Increased for burst handling

Balance Memory vs Performance: Higher queue lengths consume more memory but handle traffic bursts better.

When to Use Async UDP

High-Volume Scenarios:

• Data centers with hundreds of syslog sources
• Network devices generating high message rates
• Environments with multiple concurrent senders
• Burst traffic patterns requiring buffering

Resource Availability:

• Multi-core CPU with available capacity
• Sufficient memory for queue buffering (typically 10-50 MB)
• Network interface capable of high packet rates

Not Recommended:

• Low-volume deployments
• Single-core or resource-constrained environments
• When CPU is already at high utilization

Enterprise Deployment Example

High-performance configuration for large-scale deployments:

nodes:
  - name: datacenter_syslog
    type: syslog_input
    protocol: rfc5424
    transport_protocol: udp
    port: 514
    udp_async:
      readers: 8
      processors_count: 8
      max_queue_length: 5000
    add_attributes: true
    location: UTC

This configuration provides high throughput for enterprise deployments while adding OpenTelemetry semantic conventions for standardized downstream processing.

Example Output

When syslog messages are received, they are parsed and converted into structured log items. Below are examples showing the raw syslog message sent and the resulting processed data item.

Info Level Message

Raw syslog message sent:

<14>1 2025-09-04T15:00:00Z webserver nginx - - - User accessed /api/v1/status

Processed data item in pipeline:

{
  "_type": "log",
  "timestamp": 1756998000000,
  "body": "<14>1 2025-09-04T15:00:00Z webserver nginx - - - User accessed /api/v1/status",
  "resource": {
    "host.ip": "172.19.0.2",
    "host.name": "",
    "service.name": ""
  },
  "attributes": {
    "appname": "nginx",
    "facility": 1,
    "hostname": "webserver",
    "message": "User accessed /api/v1/status",
    "priority": 14,
    "version": 1
  },
  "observed_timestamp": 1756955155244,
  "severity_text": "info"
}

Critical Alert Message

Raw syslog message sent:

<10>1 2025-09-04T12:00:00Z database postgres - - - CRITICAL: Database connection lost

Processed data item in pipeline:

{
  "_type": "log",
  "timestamp": 1756955306000,
  "body": "<10>1 2025-09-04T12:00:00Z database postgres - - - CRITICAL: Database connection lost",
  "resource": {
    "host.ip": "172.19.0.2",
    "host.name": "",
    "service.name": ""
  },
  "attributes": {
    "appname": "postgres",
    "facility": 1,
    "hostname": "database",
    "message": "CRITICAL: Database connection lost",
    "priority": 10,
    "version": 1
  },
  "observed_timestamp": 1756955306171,
  "severity_text": "crit"
}

Related Pages