Live Capture and In-Stream Debugging

Overview

Effective pipeline design turns on your knowledge of your data. Live capture makes your actual data visible and enables you to preview the expected output of pipeline processors as you build them.

Important: As of agent v2.7.0, live capture sample collection is disabled by default to optimize resource usage in production environments. To enable live capture for pipeline design and testing, set ED_DISABLE_SAMPLE_COLLECTOR=0. See Enabling and Disabling Live Capture below.

Note: In a cluster with multiple nodes, if you do not have a coordinator pipeline deployed, live capture only shows data from the leader node. With a coordinator pipeline deployed, live capture shows data from all nodes.

Live Capture is designed with enterprise security in mind. The live capture data:

• Temporary Storage: Data is stored in secure Redis-based cache with automatic expiration:
  • Payload data (captured items): 10 minutes
  • Task status information: 1 hour
• Secure Transmission: All data is transmitted over TLS-encrypted HTTPS connections between the agent and Edge Delta backend.
• Transient by Design: Data is not persisted to permanent storage and is automatically removed when the capture window expires.
• Data Flow: The agent polls for capture tasks every 5 seconds and uploads captured data every 1 second via encrypted HTTPS POST requests to the Edge Delta backend API.

Resource Impact

Live Capture has a measurable resource cost that scales with data volume:

Resource Consumption:

• Memory: 15-20% overhead in high-volume environments due to in-memory caching of captured items
• CPU: Additional processing for JSON marshaling and serialization
• Network: Periodic uploads to Edge Delta backend (every 1 second during active capture)

When Resource Impact Matters:

• High-volume production environments (>50GB/day per agent)
• Resource-constrained deployments
• Agents already experiencing memory pressure

For detailed optimization strategies, see Reducing Agent Resource Consumption.

Enabling and Disabling Live Capture

Default Behavior (v2.7.0+): Live capture sample collection is disabled by default to optimize production resource usage.

Enabling Live Capture

To enable live capture for pipeline design and testing, set the ED_DISABLE_SAMPLE_COLLECTOR environment variable to 0:

Configuration via Helm:

helm upgrade edgedelta edgedelta/edgedelta \
  --set 'envs[0].name=ED_DISABLE_SAMPLE_COLLECTOR' \
  --set 'envs[0].value=0' \
  -n edgedelta

Linux/macOS:

export ED_ENV_VARS="ED_DISABLE_SAMPLE_COLLECTOR=0"

Docker:

-e "ED_DISABLE_SAMPLE_COLLECTOR=0"

Disabling Live Capture

If you need to explicitly disable live capture (for agents older than v2.7.0 or to ensure it’s disabled), you can use either:

Option 1 - Sample Collector (Recommended for v2.7.0+):

ED_DISABLE_SAMPLE_COLLECTOR=1

Option 2 - Live Capture:

ED_DISABLE_LIVE_CAPTURE=1

Configuration via Helm:

helm upgrade edgedelta edgedelta/edgedelta \
  --set 'envs[0].name=ED_DISABLE_SAMPLE_COLLECTOR' \
  --set 'envs[0].value=1' \
  -n edgedelta

When disabled, the agent will not collect or transmit any live capture data to the Edge Delta backend. This is useful in environments where:

• Resource Optimization: High-volume production environments where 15-20% resource savings are significant
• Security/Compliance: Regulatory requirements prohibit real-time data sampling
• Network Restrictions: Network policies restrict outbound data transmission
• Data Control: Security policies mandate strict control over data leaving the environment

Note: Disabling live capture will prevent you from using in-stream debugging features and viewing live data samples in the Edge Delta UI. Consider keeping it enabled in development and staging environments where pipeline building and troubleshooting are active.

For complete documentation on these environment variables, see Environment Variables.

Managing Large Payloads

When viewing large payloads in live capture, the Edge Delta UI may experience performance issues or freezing. To prevent this, you can configure payload truncation using environment variables (available in v2.7.0+):

ED_CAPTURER_ITEM_MAX_BODY_SIZE: Controls the maximum size of the body field in captured items before truncation is applied. Specify the size with units (e.g., 500KB, 1MB).

ED_CAPTURER_ITEM_TRUNCATION_SIZE: Defines the size threshold that triggers item truncation. When a captured item exceeds this size, it will be truncated to prevent UI performance issues. Specify the size with units (e.g., 100KB, 200KB).

Configuration via Helm:

helm upgrade edgedelta edgedelta/edgedelta \
  --set 'envs[0].name=ED_CAPTURER_ITEM_MAX_BODY_SIZE' \
  --set 'envs[0].value=500KB' \
  --set 'envs[1].name=ED_CAPTURER_ITEM_TRUNCATION_SIZE' \
  --set 'envs[1].value=100KB' \
  -n edgedelta

Linux/macOS:

export ED_ENV_VARS="ED_CAPTURER_ITEM_MAX_BODY_SIZE=500KB ED_CAPTURER_ITEM_TRUNCATION_SIZE=100KB"

Docker:

-e "ED_CAPTURER_ITEM_MAX_BODY_SIZE=500KB" \
-e "ED_CAPTURER_ITEM_TRUNCATION_SIZE=100KB"

These settings help maintain UI responsiveness when working with pipelines that process large log entries or payloads during the design and testing phase.

For complete documentation on these environment variables, see Environment Variables.

Live Tail Samples

To view live tail samples, open a processor node in edit mode:

The left pane shows the last 100 data items that were detected passing through the node. You configure your processor in the centre pane, and preview the output in the pane on the right.

You can select a specific data item in the left pane to view its details. The diff for the data item is shown in the right pane. When you select a data item the live capture is paused and the last 100 data items are listed. You click Play to resume capturing the latest data items.

AI-Powered Recommendations

While working in Live Capture, Edge Delta’s Recommendations feature can suggest optimal processors based on your actual data patterns. The multiprocessor view displays recommended processors in the center pane, helping you quickly identify opportunities to filter, sample, or transform your data for better pipeline efficiency. These contextual recommendations are generated by analyzing your live data stream and identifying patterns that could benefit from optimization.

Optimization Rate

A key metric to take note of is the percentage increase of decrease in data size as a result of logic in the centre pane. This is calculated against the 100 items in the left pane.

Affected Indicator

Data items that are modified by a processor are marked with a blue marker in the input and output live capture panes. This is likely to happen when a condition is set in the processor:

Pretty View

You can switch between a code view and a “pretty” view of the data items:

With pretty view enabled you can select a field to view shortcuts for creating processors using that field:

Filters

You can filter each live capture pane to view one specific data type. For example you might want to see logs in the left and resulting metrics in the right.

Note: this changes the view only, it does not apply a filter to the processor.

You can also search the input logs for a keyword. This further filters the input and output panes. For example, of the 100 samples, 23 contain the word error: