Create a Pipeline

  3 minute read  

To create a pipeline you must install a set of agents. See Pipeline Installation for information on installation options. Generally, the following high level instructions apply:

  1. Click Pipelines.
  2. Click New Pipeline.
  3. Select Edge.
  4. Select the appropriate environment template. If you select Kubernetes, specify a Deployment Type and a Cluster Name.
  5. Specify a name to identify the pipeline.
  6. Click Generate Deployment Command.
  7. Execute the installation commands, they include the unique ID for the pipeline.
  8. Click View Pipeline.
  9. Click View Sources Found and select the sources you want to monitor.
  10. Click Add Sources.

Alternatively, edit an existing pipeline:

  1. In the Edge Delta App, click Pipelines.
  2. Select the pipeline you want to view.
  3. The pipeline opens in the Visual Builder tab.

Create a Node

To create a node:

  1. Open Visual Builder.
  2. Click Edit Mode.

When you add a source or a destination, a multiprocessor node is attached to it.

  1. Click Add Node and select the type of node, for example Source - Collect - Demo Source.
  1. Enter the configuration details and click Save Changes, Select Destination, or Select Source, depending on the node type.
  1. You can select one or more existing nodes to automatically connect your new node to. Alternatively you can choose to create a new node, or connect it to no nodes for now.

The multiprocessor for the new node opens and you can configure it using live capture.

Connect Nodes

To connect nodes, click either the input or the output interface of the node and drag it to the input or output of another node. Some nodes differentiate between default and failure outputs so you can send different results of the node to different destinations.

Links between incompatible nodes can’t be configured - indicated by a red interface. As you release the link, the nodes will rearrange themselves to optimize readability of the data pipeline. To delete a link, select the link and click the Bin icon.

Alternatively, you can add a processor node directly onto an existing link - hover over the link, click the Add (+) icon, and select the node type. This creates the new processor node in the position where the add icon was, and also connects it to the upstream and downstream nodes on the link.

To delete a node, select it and hover over it, then select the Bin icon.

Test Node Configuration

To test a node, use the live capture feature to see actual data and the potential impact of the nodes configuration. See Designing Pipelines and Testing Nodes.

Applying Changes

While in Edit Mode, after making changes, click Review Changes then click Save Changes:

After making changes to a Pipeline configuration using the Edge Delta Web UI, there is no need to restart or reinstall the Edge Delta agents. The configuration settings will be automatically applied after a few minutes.

Delete a Pipeline Configuration:

  1. Click Pipelines.
  2. Select the pipeline you want to delete.
  3. Expand the kebab menu and click Delete Pipeline.

Deleting a Pipeline will remove the Pipeline ID and the configuration. Any active agents configured with this ID will no longer be able to communicate with the Edge Delta backend. You still need to remove those agents from their hosts.