Custom API Examples

Example configurations for various custom and third-party APIs demonstrating different authentication methods, pagination styles, and data formats.

6 minute read

Overview

This page provides example configurations for integrating with custom APIs and various third-party services, organized by authentication methods, time window patterns, pagination strategies, and advanced scenarios.

Authentication Methods

Common authentication patterns for HTTP Pull sources.

API Key Authentication

API key passed in request headers:

nodes:
- name: custom_rest_api
  type: http_pull_input
  endpoint: https://api.custom.com/v1/logs
  method: GET

  header_expressions:
    X-API-Key: EDXEnv("CUSTOM_API_KEY", "")

  parameters:
    - name: format
      value: json
    - name: limit
      value: "100"

  pull_interval: 5m

Query Parameter Authentication

API key passed as URL parameter:

nodes:
- name: legacy_api
  type: http_pull_input
  endpoint: https://api.legacy-service.com/data
  method: GET

  parameter_expressions:
    api_key: EDXEnv("LEGACY_API_KEY", "")
    from: String(UnixSeconds(Now() - Duration("1h")))
    to: String(UnixSeconds(Now()))
    limit: "500"

  pull_interval: 10m

Basic Authentication

HTTP Basic Auth with username/password:

nodes:
- name: basic_auth_api
  type: http_pull_input
  endpoint: https://api.internal.com/metrics
  method: GET

  header_expressions:
    # Base64 encode username:password
    Authorization: Concat(["Basic ", EDXEnv("BASIC_AUTH_CREDENTIALS", "")], "")

  pull_interval: 1m

Set the environment variable with base64 encoded credentials:

export BASIC_AUTH_CREDENTIALS=$(echo -n "username:password" | base64)

Bearer Token Authentication

Bearer token in Authorization header:

nodes:
- name: bearer_auth_api
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET

  header_expressions:
    Authorization: Concat(["Bearer ", EDXEnv("API_TOKEN", "")], "")

  pull_interval: 5m

Time Window Patterns

Different approaches for time-based data retrieval.

Unix Timestamp Parameters

Many APIs require Unix timestamps for time-based queries:

nodes:
- name: api_with_unix_time
  type: http_pull_input
  endpoint: https://api.example.com/logs
  method: GET

  parameter_expressions:
    # Unix timestamp in seconds - last 24 hours
    start_time: String(UnixSeconds(Now() - Duration("24h")))
    # Unix timestamp in milliseconds - current time
    end_time: String(UnixMilli(Now()))
    # Alternative: specific date in Unix format
    since_date: String(UnixSeconds(Time("2024-01-01")))

  pull_interval: 1h

This creates a rolling 24-hour window that advances with each hourly pull, ensuring continuous coverage without gaps.

ISO 8601 Time Windows

Formatted timestamps for APIs requiring standard date strings:

nodes:
- name: iso_timestamp_api
  type: http_pull_input
  endpoint: https://api.example.com/events
  method: GET

  parameter_expressions:
    # ISO 8601 format with timezone
    start: FormatTime(Now() - Duration("1h"), "%Y-%m-%dT%H:%M:%SZ")
    end: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
    # Alternative formats
    date_only: FormatTime(Now(), "%Y-%m-%d")
    with_millis: FormatTime(Now(), "%Y-%m-%dT%H:%M:%S.000Z")

  pull_interval: 5m

Dynamic Time Windows

Flexible time ranges based on environment variables:

nodes:
- name: dynamic_window_api
  type: http_pull_input
  endpoint: https://api.example.com/metrics
  method: GET

  parameter_expressions:
    # Window size from environment (default 1h)
    start: FormatTime(Now() - Duration(EDXEnv("WINDOW_SIZE", "1h")), "%Y-%m-%dT%H:%M:%SZ")
    end: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
    # Dynamic granularity
    interval: EDXEnv("QUERY_INTERVAL", "5m")

  pull_interval: 10m

Pagination Strategies

Different approaches for handling multi-page responses.

Offset-based Pagination

Traditional page-based pagination:

nodes:
- name: offset_pagination_api
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET

  header_expressions:
    API-Key: EDXEnv("SERVICE_API_KEY", "")

  parameters:
    - name: limit
      value: "100"
    - name: offset
      value: "0"  # Would need manual management

  # Note: Offset pagination typically requires manual tracking
  pull_interval: 10m

Cursor-based Pagination

Efficient pagination using cursor tokens:

nodes:
- name: cursor_pagination_api
  type: http_pull_input
  endpoint: https://api.modern.com/feed
  method: GET

  header_expressions:
    Authorization: Concat(["Bearer ", EDXEnv("MODERN_API_TOKEN", "")], "")

  parameters:
    - name: limit
      value: "100"

  pagination:
    url_json_path: "$.meta.next_cursor_url"
    max_parallel_requests: 3

  pull_interval: 5m

Nested JSON Pagination

Pagination URLs nested in response metadata:

nodes:
- name: nested_pagination_api
  type: http_pull_input
  endpoint: https://api.complex.com/v2/records
  method: GET

  header_expressions:
    X-API-Key: EDXEnv("COMPLEX_API_KEY", "")

  pagination:
    url_json_path: "$.metadata.pagination.links.next"
    max_parallel_requests: 5

  pull_interval: 15m

Link Header Pagination

RFC 5988 standard Link headers:

nodes:
- name: link_header_api
  type: http_pull_input
  endpoint: https://api.github.com/repos
  method: GET

  header_expressions:
    Authorization: Concat(["Bearer ", EDXEnv("GITHUB_TOKEN", "")], "")

  pagination:
    link_relation: "next"
    max_parallel_requests: 3

  pull_interval: 5m

Advanced Scenarios

Complex integration patterns for specific use cases.

Dynamic Endpoint with Environment Variables

Fully dynamic configuration adapting to different environments:

nodes:
- name: api_monitor
  type: http_pull_input
  # Dynamic endpoint based on environment
  endpoint_expression: Concat(["https://", EDXEnv("API_HOST", "api.example.com"), "/v1/metrics"], "")
  method: GET

  # Dynamic headers
  header_expressions:
    X-API-Key: EDXEnv("API_KEY", "")
    X-Request-ID: Concat(["req-", String(UnixMilli(Now()))], "")
    X-Environment: EDXEnv("ENVIRONMENT", "development")
    X-API-Version: EDXEnv("API_VERSION", "v1")

  pull_interval: 30s

The endpoint_expression dynamically constructs the full URL, allowing the same configuration to work across development, staging, and production environments. Environment variables like API_HOST, ENVIRONMENT, and API_VERSION enable seamless deployment across different stages.

Multi-Tenant API

Configure for multi-tenant scenarios:

nodes:
- name: multitenant_api
  type: http_pull_input
  endpoint_expression: Concat(["https://", EDXEnv("TENANT_DOMAIN", "default"), ".api.service.com/v1/data"], "")
  method: GET

  header_expressions:
    X-Tenant-ID: EDXEnv("TENANT_ID", "")
    Authorization: Concat(["Bearer ", EDXEnv("TENANT_TOKEN", "")], "")

  parameter_expressions:
    from: FormatTime(Now() - Duration("15m"), "%Y-%m-%dT%H:%M:%SZ")
    to: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

  pull_interval: 5m

Rate-Limited API

Handle strict rate limits with conservative settings:

nodes:
- name: rate_limited_api
  type: http_pull_input
  endpoint: https://api.ratelimited.com/data
  method: GET

  header_expressions:
    API-Key: EDXEnv("RATE_LIMITED_KEY", "")

  parameters:
    - name: per_page
      value: "10"  # Small page size

  retry_http_code:
    - 429  # Too Many Requests

  pagination:
    link_relation: "next"
    max_parallel_requests: 1  # Sequential pagination only

  pull_interval: 30s  # Respect rate limits

Common Use Cases

Complete examples for typical integration scenarios.

Webhook/Event API

Pull recent events with timestamp filtering:

nodes:
- name: webhook_events
  type: http_pull_input
  endpoint: https://webhooks.service.com/api/events
  method: GET

  header_expressions:
    Authorization: Concat(["Token ", EDXEnv("WEBHOOK_TOKEN", "")], "")

  parameter_expressions:
    since: String(UnixMilli(Now() - Duration("5m")))
    until: String(UnixMilli(Now()))
    event_types: "order.created,order.updated,order.cancelled"

  pull_interval: 5m

Metrics Aggregation API

Pull aggregated metrics with time windows:

nodes:
- name: metrics_aggregator
  type: http_pull_input
  endpoint: https://metrics.platform.com/api/v2/query
  method: GET

  header_expressions:
    X-API-Token: EDXEnv("METRICS_TOKEN", "")

  parameter_expressions:
    query: "avg(cpu.usage)"
    start: String(UnixSeconds(Now() - Duration("15m")))
    end: String(UnixSeconds(Now()))
    step: "60"  # 1-minute resolution

  pull_interval: 5m

Log Management Platform

Retrieve filtered logs:

nodes:
- name: log_management
  type: http_pull_input
  endpoint: https://logs.platform.com/api/v1/search
  method: GET

  header_expressions:
    Authorization: Concat(["Bearer ", EDXEnv("LOG_PLATFORM_TOKEN", "")], "")

  parameter_expressions:
    from: FormatTime(Now() - Duration("10m"), "%Y-%m-%dT%H:%M:%SZ")
    to: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
    query: "level:ERROR OR level:CRITICAL"
    limit: "500"

  pull_interval: 2m

IoT Telemetry

Retrieve device metrics:

nodes:
- name: iot_telemetry
  type: http_pull_input
  endpoint: https://iot.platform.com/api/telemetry
  method: GET

  header_expressions:
    X-API-Key: EDXEnv("IOT_API_KEY", "")
    X-Device-Group: EDXEnv("DEVICE_GROUP_ID", "default")

  parameter_expressions:
    start_time: FormatTime(Now() - Duration("30m"), "%Y-%m-%d %H:%M:%S")
    end_time: FormatTime(Now(), "%Y-%m-%d %H:%M:%S")
    devices: EDXEnv("DEVICE_IDS", "*")
    metrics: "temperature,humidity,pressure"

  pull_interval: 5m

CI/CD Pipeline Monitoring

Track build status:

nodes:
- name: cicd_pipeline
  type: http_pull_input
  endpoint: https://ci.platform.com/api/v4/pipelines
  method: GET

  header_expressions:
    Private-Token: EDXEnv("CICD_TOKEN", "")

  parameters:
    - name: per_page
      value: "50"
    - name: scope
      value: "finished"

  parameter_expressions:
    updated_after: FormatTime(Now() - Duration("30m"), "%Y-%m-%dT%H:%M:%SZ")

  pagination:
    link_relation: "next"

### Ticketing System API

Pull ticket updates from a support system:

```yaml
nodes:
- name: support_tickets
  type: http_pull_input
  endpoint: https://support.company.com/api/v2/tickets
  method: GET

  header_expressions:
    Authorization: Concat(["Token token=", EDXEnv("SUPPORT_API_TOKEN", "")], "")

  parameter_expressions:
    updated_since: FormatTime(Now() - Duration("1h"), "%Y-%m-%dT%H:%M:%S")
    status: "open,pending,resolved"
    include: "requester,assignee"

  pagination:
    url_json_path: "$.pagination.next"
    max_parallel_requests: 2

  pull_interval: 5m

Common Patterns

Quick reference snippets for frequently used patterns in HTTP Pull configurations.

Time Window Queries

parameter_expressions:
  # ISO 8601 format
  start: FormatTime(Now() - Duration("1h"), "%Y-%m-%dT%H:%M:%SZ")
  end: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

  # Unix timestamps (seconds)
  from: String(UnixSeconds(Now() - Duration("1h")))
  to: String(UnixSeconds(Now()))

  # Unix milliseconds
  since_ms: String(UnixMilli(Now() - Duration("1h")))
  until_ms: String(UnixMilli(Now()))

  # Custom date format
  date_from: FormatTime(Now() - Duration("24h"), "%Y-%m-%d")
  date_to: FormatTime(Now(), "%Y-%m-%d")

Dynamic Filtering

parameter_expressions:
  # Conditional filters based on environment
  severity: EDXEnv("LOG_SEVERITY", "ERROR,CRITICAL")
  sources: EDXEnv("LOG_SOURCES", "app1,app2,app3")

  # Complex query building
  query: Concat(["status:active AND created>", FormatTime(Now() - Duration("24h"), "%Y-%m-%d")], "")

  # Dynamic limit based on environment
  limit: EDXEnv("QUERY_LIMIT", "100")

Authentication Patterns

header_expressions:
  # Bearer token
  Authorization: Concat(["Bearer ", EDXEnv("TOKEN", "")], "")

  # API key header
  X-API-Key: EDXEnv("API_KEY", "")

  # Basic authentication (base64 encoded separately)
  Authorization: Concat(["Basic ", EDXEnv("BASIC_CREDENTIALS", "")], "")

  # Custom authentication scheme
  Authentication: Concat(["CustomScheme ", EDXEnv("CUSTOM_TOKEN", "")], "")

  # Multiple auth headers
  X-Auth-Token: EDXEnv("AUTH_TOKEN", "")
  X-Client-ID: EDXEnv("CLIENT_ID", "")
  X-Client-Secret: EDXEnv("CLIENT_SECRET", "")

Request Metadata

header_expressions:
  # Request tracking
  X-Request-ID: Concat(["req-", String(UnixMilli(Now()))], "")
  X-Trace-ID: Concat([EDXEnv("SERVICE_NAME", "edge-delta"), "-", String(UnixMilli(Now()))], "")

  # Client identification
  User-Agent: "EdgeDelta/1.0"
  X-Client-Version: EDXEnv("CLIENT_VERSION", "1.0.0")
  X-Environment: EDXEnv("ENVIRONMENT", "production")