Manage Pipelines with the API

The Edge Delta API enables programmatic management of pipeline configurations, making it ideal for:

• Infrastructure as Code: Store pipeline configurations in version control and deploy through CI/CD pipelines
• Automation: Script bulk updates across multiple pipelines or environments
• GitOps workflows: Manage configurations alongside application code
• Backup and restore: Export configurations for disaster recovery or migration

AI Team integration: AI Teammates can also manage pipeline configurations through natural language using the Edge Delta MCP connector. This enables conversational pipeline management where you describe what you need and the AI teammate implements it.

Workflow Overview

The API uses a versioned deployment model with four key steps:

1. Validate - Check configuration syntax before saving
2. Save - Create a new version of the configuration
3. Get history - Retrieve the version ID for deployment
4. Deploy - Mark the version as active (agents poll for changes and reload automatically)

This separation ensures configurations are validated before deployment and provides an audit trail of all changes.

Prerequisites

You need:

• Your Organization ID
• An API Token with appropriate permissions
• A Pipeline ID (visible in the URL when viewing a pipeline in the UI)

Get a Pipeline Configuration

Retrieve the current configuration for a pipeline:

curl -X 'GET' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/confs/<PIPELINE_ID>' \
  -H 'accept: application/json' \
  -H 'X-ED-API-Token: <API_TOKEN>'

The response includes the pipeline metadata and YAML configuration in the content field:

{
  "id": "<PIPELINE_ID>",
  "orgID": "<ORG_ID>",
  "tag": "my-pipeline",
  "content": "version: v3\n\nsettings:\n  tag: my-pipeline\n...",
  "description": "Production pipeline",
  "creator": "user@example.com",
  "created": "2024-11-25T23:59:06Z",
  "updater": "user@example.com",
  "updated": "2024-12-05T03:32:15Z",
  "version": "v3",
  "environment": "Kubernetes"
}

Save the YAML content to a file for editing:

curl -s -X 'GET' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/confs/<PIPELINE_ID>' \
  -H 'accept: application/json' \
  -H 'X-ED-API-Token: <API_TOKEN>' \
  | jq -r '.content' > pipeline.yaml

Update the Configuration

Edit the pipeline.yaml file to make your changes. For example, you might:

• Add a new processor node
• Modify destination settings
• Update filtering rules
• Add new links between nodes

Note: If you change the tag value in the settings section, the pipeline name displayed in the UI will update to match.

Recommended Workflow

Rather than building node configurations from scratch, use the GUI to establish the basic structure:

1. Create nodes in the GUI first: When using a node type for the first time, create it in the pipeline editor. This generates the correct YAML structure with all required fields.

2. Export and modify: Use the API to retrieve the configuration, then modify the specific values you need. This ensures you start with a valid structure.

3. Reference the documentation: See Sources, Processors, and Destinations for node-specific configuration options and examples.

This approach reduces validation errors and ensures compatibility with the platform’s expected node structure.

YAML Editing Considerations

When editing pipeline YAML files, keep these considerations in mind to avoid validation errors:

Indentation: YAML is whitespace-sensitive. Always use spaces (typically 2) for indentation, never tabs. Inconsistent indentation causes parsing failures.

Special characters in strings: Certain characters require quoting:

• Colons in values: description: "Error: connection failed"
• Hash symbols (otherwise interpreted as comments): pattern: "user#123"
• Leading special characters: value: "@mention" or value: "*wildcard"

Multiline strings: Use YAML block scalars for multi-line content:

# Literal block (|) preserves newlines exactly
description: |
  This is line one.
  This is line two.  

# Folded block (>) joins lines with spaces
description: >
  This is a long description
  that spans multiple lines.  

Regex patterns: Backslashes in regex patterns require escaping. In YAML, use double backslashes:

# To match digits (\d+), escape the backslash
pattern: "\\d+"

# To match a literal backslash followed by 'n'
pattern: "\\\\n"

Trailing whitespace: Some editors add trailing spaces or newlines. These can cause unexpected validation failures or create spurious diffs when comparing versions. Configure your editor to trim trailing whitespace.

OTTL statement comments: Within OTTL statements blocks, use double slashes (//) for comments, not hash symbols (#). Hash symbols cause validation errors in OTTL context:

statements: |-
  // This is a valid OTTL comment
  set(attributes["field"], "value")
  set(attributes["other"], "data")  // End-of-line comment also works  

Processor type names: Use the correct type identifiers. Common mistakes include:

• Use ottl_transform, not ottl
• Use parse_json, not json_parser
• Use statements field, not expressions

Validate the Configuration

Before saving, validate the configuration to catch errors:

curl -X 'POST' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/confs/validate' \
  -H 'X-ED-API-Token: <API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d "{\"content\": $(jq -Rs '.' < pipeline.yaml)}"

A successful validation returns:

{
  "valid": true
}

If validation fails, the response includes error details:

{
  "valid": false,
  "errors": [
    "node 'my_processor': unknown type 'invalid_type'"
  ]
}

Fix any errors before proceeding.

Save the Configuration

Save the validated configuration to create a new version:

curl -X 'POST' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/pipelines/<PIPELINE_ID>/save' \
  -H 'X-ED-API-Token: <API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d "{\"content\": $(jq -Rs '.' < pipeline.yaml)}"

The response confirms the save:

{
  "valid": true,
  "lastUpdated": "2024-12-05T10:30:00Z"
}

Get Version History

After saving, retrieve the version history to get the version ID needed for deployment:

curl -X 'GET' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/pipelines/<PIPELINE_ID>/history' \
  -H 'X-ED-API-Token: <API_TOKEN>'

The response is an array of versions in reverse chronological order (newest first):

[
  {
    "timestamp": "1733394600000",
    "author": "user@example.com",
    "status": "saved"
  },
  {
    "timestamp": "1733391000000",
    "author": "user@example.com",
    "status": "deployed"
  }
]

The timestamp field is the version ID you need for deployment.

Important: Always use the version ID from the history response, not from the save response. The save endpoint does not return the version ID directly.

Deploy the Configuration

Deploy using the version ID from the history:

curl -X 'POST' \
  'https://api.edgedelta.com/v1/orgs/<ORG_ID>/pipelines/<PIPELINE_ID>/deploy/<VERSION_ID>' \
  -H 'X-ED-API-Token: <API_TOKEN>' \
  -H 'Content-Type: application/json'

A successful deployment returns:

{
  "rollout_task_id": "abc123-def456"
}

Complete Workflow Script

After updating your pipeline.yaml file, run this script to validate, save, and deploy the changes. The script performs all the steps described above in sequence:

#!/bin/bash
set -euo pipefail

# Configuration
ORG_ID="your-org-id"
PIPELINE_ID="your-pipeline-id"
API_TOKEN="your-api-token"
API_BASE="https://api.edgedelta.com"
CONFIG_FILE="pipeline.yaml"

echo "Step 1: Validating configuration..."
validation=$(curl -s -X POST \
  "${API_BASE}/v1/orgs/${ORG_ID}/confs/validate" \
  -H "X-ED-API-Token: ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"content\": $(jq -Rs '.' < "$CONFIG_FILE")}")

if [[ $(echo "$validation" | jq -r '.valid') != "true" ]]; then
  echo "Validation failed:"
  echo "$validation" | jq '.errors'
  exit 1
fi
echo "✓ Configuration valid"

echo "Step 2: Saving configuration..."
save_response=$(curl -s -X POST \
  "${API_BASE}/v1/orgs/${ORG_ID}/pipelines/${PIPELINE_ID}/save" \
  -H "X-ED-API-Token: ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"content\": $(jq -Rs '.' < "$CONFIG_FILE")}")
echo "✓ Configuration saved"

echo "Step 3: Getting version history..."
history=$(curl -s -X GET \
  "${API_BASE}/v1/orgs/${ORG_ID}/pipelines/${PIPELINE_ID}/history" \
  -H "X-ED-API-Token: ${API_TOKEN}")

# Get the latest version (first in array)
VERSION_ID=$(echo "$history" | jq -r '.[0].timestamp')
echo "✓ Latest version: $VERSION_ID"

echo "Step 4: Deploying configuration..."
deploy_response=$(curl -s -X POST \
  "${API_BASE}/v1/orgs/${ORG_ID}/pipelines/${PIPELINE_ID}/deploy/${VERSION_ID}" \
  -H "X-ED-API-Token: ${API_TOKEN}" \
  -H "Content-Type: application/json")

if echo "$deploy_response" | jq -e '.rollout_task_id' > /dev/null 2>&1; then
  echo "✓ Deployment initiated"
  echo "  Rollout Task ID: $(echo "$deploy_response" | jq -r '.rollout_task_id')"
else
  echo "Deployment response: $deploy_response"
fi

echo ""
echo "Agents will reload the configuration within 1-2 minutes."

API Endpoints Summary

See the Swagger documentation for all available API endpoints.