Live Data Pipeline Design in Edge Delta

Overview

Effective pipeline design turns on your knowledge of your data. Live capture makes your actual data visible and enables you to preview the expected output of pipeline processors as you build them.

Live Tail Samples

To view live tail samples, open a processor node in edit mode:

The left pane shows the last 100 data items that were detected passing through the node. You configure your processor in the centre pane, and preview the output in the pane on the right.

You can select a specific data item in the left pane to view its details. The diff for the data item is shown in the right pane. When you select a data item the live capture is paused and the last 100 data items are listed. You click Play to resume capturing the latest data items.

Optimization Rate

A key metric to take note of is the percentage increase of decrease in data size as a result of logic in the centre pane. This is calculated against the 100 items in the left pane.

Affected Indicator

Data items that are modified by a processor are marked with a blue marker in the input and output live capture panes. This is likely to happen when a condition is set in the processor:

Pretty View

You can switch between a code view and a “pretty” view of the data items:

With pretty view enabled you can select a field to view shortcuts for creating processors using that field:

Filters

You can filter each live capture pane to view one specific data type. For example you might want to see logs in the left and resulting metrics in the right.

Note: this changes the view only, it does not apply a filter to the processor.

You can also search the input logs for a keyword. This further filters the input and output panes. For example, of the 100 samples, 23 contain the word error: