FluentD Connector

Configure the FluentD connector to receive logs and events from Fluentd agents using the forward protocol for unified logging aggregation.

  7 minute read  

Overview

The FluentD connector receives log data and events from Fluentd agents using the native Fluentd forward protocol. It enables Edge Delta to act as a Fluentd aggregation server, accepting data forwarded from Fluentd instances running on application servers, containers, and infrastructure. Content streams into Edge Delta Pipelines for analysis by AI teammates through the Edge Delta MCP connector.

When you add this streaming connector, it appears as a FluentD source in your selected pipeline. AI teammates access this data by querying the Edge Delta backend with the Edge Delta MCP connector.

Add the FluentD Connector

To add the FluentD connector, you configure Edge Delta to listen for incoming Fluentd connections on a specified port with shared key authentication.

Prerequisites

Before configuring the connector, ensure you have:

  • Edge Delta agent deployed with network access to receive incoming connections
  • Fluentd or td-agent installed on source systems forwarding logs
  • Firewall rules allowing inbound traffic on the FluentD port
  • Identified shared key for authentication between Fluentd instances and Edge Delta

Configuration Steps

  1. Navigate to AI Team > Connectors in the Edge Delta application
  2. Find the FluentD connector in Streaming Connectors
  3. Click the connector card
  4. Configure the Port number to listen on (default: 3421)
  5. Set the Shared Key for authentication
  6. Configure Listen address (default: 0.0.0.0)
  7. Set Read Timeout for incoming connections
  8. Optionally configure Advanced Settings
  9. Select a target environment
  10. Click Save

The connector is now listening for Fluentd connections and streaming content.

FluentD connector configuration showing port, shared key, and advanced settings

Configuration Options

Connector Name

Name to identify this FluentD connector instance.

Port

Port number to listen on for incoming Fluentd forward protocol connections.

Format: Port number (1-65535)

Examples:

  • 3421 - Default Edge Delta port
  • 24224 - Standard Fluentd forward protocol port
  • 9880 - Non-privileged port alternative

Shared Key

Shared secret key for authentication between Fluentd nodes and Edge Delta. Must match the shared key configured in your Fluentd instances’ forward output plugin.

Format: String value (alphanumeric and special characters)

Examples:

  • my-secure-shared-key-2024
  • a8f5e2b9c1d4f7a3e6b8c9d2f5a7e3b6

Security Note: Use strong, unique shared keys in production. Avoid simple values like “shared-key” or “secret”.

Listen

IP address to which the FluentD listener will bind. Use 0.0.0.0 to accept connections on all network interfaces, or specify a particular IP to restrict access.

Format: Valid IPv4 address

Examples:

  • 0.0.0.0 - All interfaces (default)
  • 192.168.1.100 - Specific IP address
  • 127.0.0.1 - Localhost only (testing)

Read Timeout

Maximum time to wait for incoming data on established connections before considering the connection idle.

Format: Duration with unit (s, m, h)

Default: 1 minute

Examples:

  • 30s - 30 seconds for low-latency networks
  • 1m - 1 minute (default)
  • 2m - 2 minutes for high-latency networks

Advanced Settings

Add Ingestion Timestamp

When enabled, adds a server-side timestamp to logs when ingested if the log entry doesn’t contain a timestamp field. When disabled, only uses timestamps from the source Fluentd instance.

Default: Disabled

Skip Ingestion Timestamp on Failure

When enabled, continues processing log data even if timestamp parsing fails. When disabled, rejects logs with invalid timestamps.

Default: Disabled

TLS

Optional TLS encryption for Fluentd forward protocol connections. When enabled, all log data transmitted between Fluentd instances and Edge Delta is encrypted.

Configuration Options:

  • Ignore Certificate Check: Disables SSL/TLS certificate verification. Use with caution in testing environments only.
  • CA File: Absolute file path to the CA certificate for SSL/TLS connections.
  • CA Path: Absolute path where CA certificate files are located for SSL/TLS.
  • CRT File: Absolute path to the SSL/TLS certificate file for secure communication.
  • Key File: Absolute path to the private key file used in SSL/TLS connections.
  • Key Password: Optional password for the key file.
  • Client Auth Type: Client authentication type. Default is noclientcert.
  • Minimum Version: Minimum TLS version. Default is TLSv1_2.
  • Maximum Version: Maximum TLS version allowed for connections.

Recommendation: Enable TLS when forwarding logs over untrusted networks or when logs contain sensitive information.

Metadata Level

This option is used to define which detected resources and attributes to add to each data item as it is ingested by Edge Delta. You can select:

  • Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
  • Default: This option includes the required resources and attributes plus those selected by Edge Delta
  • High: This option includes the required resources and attributes along with a larger selection of common optional fields.
  • Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.

FluentD-specific metadata included:

  • ed.source.type - Source type (fluentd_input)

Rate Limit

Rate limit configuration to control maximum data ingestion volume from Fluentd instances and manage processing capacity.

Target Environments

Select the Edge Delta pipeline (environment) where you want to deploy this connector.

How to Use the FluentD Connector

The FluentD connector integrates seamlessly with AI Team, enabling analysis of log data from Fluentd-instrumented infrastructure. AI teammates automatically leverage the ingested data based on the queries they receive and the context of the conversation.

Use Case: Application Log Aggregation from Distributed Servers

Centralize application logs from multiple servers that use Fluentd for local log collection. AI teammates can analyze application errors, performance issues, and patterns across your server fleet. When combined with PagerDuty alerts, teammates automatically query recent application logs during incident investigation to identify which servers are experiencing problems and correlate error patterns across hosts.

Configuration: Configure each Fluentd instance to forward to Edge Delta using the forward output plugin with matching shared key. AI teammates can then query aggregated logs from all servers through the Edge Delta MCP connector.

Use Case: Kubernetes Container Log Collection

Deploy Fluentd as a DaemonSet in Kubernetes to collect container logs and forward them to Edge Delta for centralized analysis. AI teammates can identify containers producing errors, track pod restarts, and correlate issues across microservices. This is valuable when investigating performance degradation—teammates can analyze log patterns from affected pods and identify the service causing bottlenecks.

Configuration: Deploy Fluentd DaemonSet with forward output pointing to Edge Delta FluentD connector. Enable TLS for secure transmission across cluster networks.

Use Case: Syslog Aggregation via FluentD

Use Fluentd as an intermediate aggregation layer that receives syslog messages from network devices and servers, then forwards them to Edge Delta. AI teammates can analyze authentication failures, configuration changes, and security events across your infrastructure. When integrated with Jira, teammates can automatically document security incidents by querying relevant syslog entries and creating comprehensive issue reports.

Configuration: Configure Fluentd to receive syslog on port 5140, then forward to Edge Delta using the forward protocol with shared key authentication.

Troubleshooting

Connection refused errors: Verify Edge Delta is listening on configured port (netstat -tuln | grep 3421). Check firewall rules allow inbound traffic from Fluentd instance IP addresses.

Shared key mismatch: Ensure shared key in Fluentd forward output exactly matches Edge Delta connector configuration (case-sensitive, no leading/trailing spaces).

Logs sent but not appearing: Verify correct target environment selected. Check if “Skip Ingestion Timestamp on Failure” needs enabling for logs with invalid timestamps.

TLS handshake failures: Verify certificates are valid and not expired. Ensure certificate paths in Fluentd configuration are correct and Fluentd has read permissions.

High buffer usage in Fluentd: Check if rate limits are configured and log volume exceeds limits. Verify Edge Delta is reachable and has sufficient capacity. Consider reducing flush interval in Fluentd.

Slow forwarding performance: Investigate network bandwidth and latency. Enable compression in Fluentd forward output to reduce network consumption. Monitor resource usage on Fluentd host (CPU, memory, disk I/O).

Next Steps

For additional help, visit AI Team Support.