Custom API Integration Examples

Configure HTTP Pull to retrieve data from various custom and third-party APIs demonstrating different authentication methods, pagination styles, and data formats.

7 minute read

Overview

This guide provides comprehensive examples for integrating with custom APIs and third-party services. Each configuration demonstrates best practices for authentication, time-based filtering, pagination handling, and error management across various API architectures.

The examples cover common authentication patterns, pagination strategies, and real-world API integration scenarios.

Examples

Category	Examples	Use Case
Authentication Methods	API Key, Bearer Token, Query Auth, Basic Auth	Various auth patterns
Time-Based Queries	ISO 8601, Unix timestamps, Time windows	Historical data retrieval
Pagination Strategies	JSON path, Cursor, Offset	Large dataset handling
Dynamic Configuration	Multi-tenant, Environment-based	Flexible deployments
Error Handling	Retry codes, Rate limiting	Resilient operations

**Version Requirements:** HTTP Pull requires Edge Delta Agent v2.6.0 or higher.

**OTTL Expressions:** All OTTL expressions MUST be written as single-line strings. Multi-line expressions will fail validation.

Environment Variables

Set these environment variables for your custom API integrations:

# API Authentication
export CUSTOM_API_KEY="your_api_key_here"
export API_TOKEN="your_bearer_token_here"
export API_USERNAME="api_username"
export API_PASSWORD="api_password"

# API Endpoints
export API_BASE_URL="https://api.example.com"
export API_VERSION="v1"

# Configuration
export API_PAGE_SIZE="100"
export API_TIMEOUT="30"
export API_RETRY_LIMIT="3"

Authentication Methods

Common authentication patterns for HTTP Pull sources with complete working examples.

API Key in Headers

Pass API keys through request headers:

nodes:
- name: custom_api_key_auth
  type: http_pull_input
  endpoint: https://api.example.com/v1/logs
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("CUSTOM_API_KEY", "")
    - header: "Accept"
      value_expression: "application/json"

  parameters:
    - name: format
      value: json
    - name: limit
      value: "100"

  parameter_expressions:
    - name: "since"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")

Bearer Token Authentication

OAuth2 bearer tokens and JWT authentication:

nodes:
- name: bearer_token_auth
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("API_TOKEN", "")], "")
    - header: "Accept"
      value_expression: "application/json"

  parameter_expressions:
    - name: "from"
      value_expression: String(UnixSeconds(Now() - Duration("5m10s")))
    - name: "to"
      value_expression: String(UnixSeconds(Now()))

Query Parameter Authentication

Legacy APIs using URL parameter authentication:

nodes:
- name: query_param_auth
  type: http_pull_input
  endpoint: https://api.legacy-service.com/data
  method: GET
  pull_interval: 10m

  parameter_expressions:
    - name: "api_key"
      value_expression: EDXEnv("LEGACY_API_KEY", "")
    - name: "from"
      value_expression: String(UnixSeconds(Now() - Duration("10m10s")))
    - name: "to"
      value_expression: String(UnixSeconds(Now()))
    - name: "limit"
      value_expression: "500"

Basic Authentication

HTTP Basic Auth with username and password:

nodes:
- name: basic_auth_api
  type: http_pull_input
  endpoint: https://api.internal.com/metrics
  method: GET
  pull_interval: 1m

  # Base64 encode credentials externally
  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Basic ", EDXEnv("BASIC_AUTH_CREDENTIALS", "")], "")
    - header: "Accept"
      value_expression: "application/json"

  parameters:
    - name: limit
      value: "1000"

Environment Variables

Set these environment variables with your API credentials:

# API Key Authentication
export CUSTOM_API_KEY="your_api_key_here"

# Bearer Token
export API_TOKEN="your_bearer_token_here"

# Query Parameter Auth
export LEGACY_API_KEY="your_legacy_key_here"

# Basic Auth (base64 encoded username:password)
export BASIC_AUTH_CREDENTIALS=$(echo -n "username:password" | base64)

API Endpoints Reference

Authentication Type	Header/Parameter	Format	Description
API Key (Header)	`X-API-Key`	Plain text	API key in custom header
Bearer Token	`Authorization`	`Bearer {token}`	OAuth2 or JWT token
Query Parameter	URL parameter	`?api_key={key}`	Legacy URL authentication
Basic Auth	`Authorization`	`Basic {base64}`	Base64 encoded credentials

Time Windows and Pull Intervals

Each endpoint uses optimized time windows to prevent data gaps and respect rate limits:

Pattern	Time Window	Pull Interval	Rationale
High-frequency	5m10s	5 minutes	Real-time monitoring, needs overlap
Standard	10m10s	10 minutes	Regular data collection
Low-frequency	1h10m	1 hour	Historical data, batch processing
Rate-limited	30m10s	30 minutes	APIs with strict limits

Time Window Patterns

Different approaches for time-based data retrieval with working examples.

Unix Timestamp Parameters

Many APIs require Unix timestamps for time queries:

nodes:
- name: unix_timestamp_api
  type: http_pull_input
  endpoint: https://api.example.com/logs
  method: GET
  pull_interval: 1h

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")

  parameter_expressions:
    # Unix seconds
    - name: "start_time"
      value_expression: String(UnixSeconds(Now() - Duration("1h10m")))
    - name: "end_time"
      value_expression: String(UnixSeconds(Now()))

    # Unix milliseconds
    - name: "start_ms"
      value_expression: String(UnixMilli(Now() - Duration("1h10m")))
    - name: "end_ms"
      value_expression: String(UnixMilli(Now()))

ISO 8601 Timestamps

Standard date format for modern APIs:

nodes:
- name: iso_timestamp_api
  type: http_pull_input
  endpoint: https://api.example.com/events
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("TOKEN", "")], "")

  parameter_expressions:
    # ISO 8601 with timezone
    - name: "start"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")
    - name: "end"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

    # Date only format
    - name: "date"
      value_expression: FormatTime(Now(), "%Y-%m-%d")

Dynamic Time Windows

Environment-controlled time ranges:

nodes:
- name: dynamic_window_api
  type: http_pull_input
  endpoint: https://api.example.com/metrics
  method: GET
  pull_interval: 10m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")

  parameter_expressions:
    # Configurable window size
    - name: "from"
      value_expression: FormatTime(Now() - Duration(EDXEnv("WINDOW_SIZE", "10m10s")), "%Y-%m-%dT%H:%M:%SZ")
    - name: "to"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
    - name: "interval"
      value_expression: EDXEnv("QUERY_INTERVAL", "5m")

Important: Time Window Overlaps

Always add a small overlap (10 seconds to 1 minute) to your time windows to account for clock drift and processing delays. This ensures no data gaps between pull intervals.

Time Window Expression Breakdown

Here’s how time window expressions work in detail:

# Educational breakdown - write as single line in actual config
parameter_expressions:
  - name: "start_time"
    value_expression: >
      String(                                    # Convert to string
        UnixSeconds(                            # Get Unix timestamp in seconds
          Now() - Duration("5m10s")              # Current time minus 5 minutes 10 seconds
        )
      )

  - name: "formatted_time"
    value_expression: >
      FormatTime(                                # Format time to string
        Now() - Duration("1h"),                 # One hour ago
        "%Y-%m-%dT%H:%M:%SZ"                    # ISO 8601 format
      )

Common Time Formats:

Unix seconds: UnixSeconds(Now())
Unix milliseconds: UnixMilli(Now())
ISO 8601: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
Custom format: FormatTime(Now(), "%Y/%m/%d %H:%M:%S")

Pagination Strategies

Different approaches for handling multi-page API responses.

Offset-based Pagination

Traditional page number or offset pagination:

nodes:
- name: offset_pagination_api
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET
  pull_interval: 10m

  header_expressions:
    - header: "API-Key"
      value_expression: EDXEnv("SERVICE_API_KEY", "")

  parameters:
    - name: limit
      value: "100"
    - name: offset
      value: "0"

  # Note: Manual offset management typically required
  # Consider using cursor-based pagination instead

Cursor-based Pagination

Efficient pagination using continuation tokens:

nodes:
- name: cursor_pagination_api
  type: http_pull_input
  endpoint: https://api.modern.com/feed
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("MODERN_TOKEN", "")], "")

  parameters:
    - name: limit
      value: "100"

  pagination:
    url_json_path: "meta.next_cursor_url"
    max_parallel: 3

Nested JSON Pagination

Complex pagination with nested response structures:

nodes:
- name: nested_pagination_api
  type: http_pull_input
  endpoint: https://api.complex.com/v2/records
  method: GET
  pull_interval: 15m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("COMPLEX_API_KEY", "")

  parameters:
    - name: page_size
      value: "50"

  pagination:
    url_json_path: "metadata.pagination.links.next"
    max_parallel: 5

Link Header Pagination

RFC 5988 standard Link headers (GitHub-style):

nodes:
- name: link_header_api
  type: http_pull_input
  endpoint: https://api.github-style.com/repos
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("GITHUB_TOKEN", "")], "")

  parameters:
    - name: per_page
      value: "100"

  pagination:
    link_relation: "next"
    max_parallel: 3

Advanced Scenarios

Complex integration patterns for specific use cases.

Dynamic Endpoint Construction

Build endpoint URLs dynamically from environment:

nodes:
- name: dynamic_endpoint_api
  type: http_pull_input
  # Dynamic endpoint based on environment
  endpoint_expression: Concat(["https://", EDXEnv("API_HOST", "api.example.com"), "/v1/metrics"], "")
  method: GET
  pull_interval: 30s

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")
    - header: "X-Request-ID"
      value_expression: Concat(["req-", String(UnixMilli(Now()))], "")
    - header: "X-Environment"
      value_expression: EDXEnv("ENVIRONMENT", "development")
    - header: "X-API-Version"
      value_expression: EDXEnv("API_VERSION", "v1")

  parameter_expressions:
    - name: "since"
      value_expression: String(UnixSeconds(Now() - Duration("35s")))

Multi-Tenant Configuration

Support multiple tenants with environment variables:

nodes:
- name: multitenant_api
  type: http_pull_input
  endpoint_expression: Concat(["https://", EDXEnv("TENANT_DOMAIN", "default"), ".api.service.com/v1/data"], "")
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "X-Tenant-ID"
      value_expression: EDXEnv("TENANT_ID", "")
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("TENANT_TOKEN", "")], "")

  parameter_expressions:
    - name: "from"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")
    - name: "to"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

Rate-Limited APIs

Conservative configuration for strict rate limits:

nodes:
- name: rate_limited_api
  type: http_pull_input
  endpoint: https://api.ratelimited.com/data
  method: GET
  pull_interval: 30s

  header_expressions:
    - header: "API-Key"
      value_expression: EDXEnv("RATE_LIMITED_KEY", "")

  parameters:
    - name: per_page
      value: "10"  # Small page size

  retry_http_code:
    - 429  # Too Many Requests
    - 503  # Service Unavailable

  pagination:
    link_relation: "next"
    max_parallel: 1  # Sequential only

API Documentation

For implementing custom API integrations, refer to:

Edge Delta HTTP Pull Documentation - Complete HTTP Pull reference
OTTL Function Reference - Available OTTL functions
Environment Variable Usage - EDXEnv() function guide

Rate Limiting

Different APIs implement various rate limiting strategies:

Request-based: Fixed number of requests per time window
Token bucket: Burst capacity with refill rate
Sliding window: Rolling time window limits
Concurrent limits: Maximum parallel requests

Configure pull intervals and pagination settings to respect these limits.

Troubleshooting

401 Unauthorized:

Verify API key or token in environment variables
Check token expiration for OAuth2/JWT
Ensure correct authentication method (header vs parameter)

400 Bad Request:

Verify time parameter formats match API requirements
Check required parameters are present
Validate URL encoding for special characters

403 Forbidden:

Check API permissions and scopes
Verify IP allowlisting if required
Confirm subscription tier has required access

429 Too Many Requests:

Increase pull_interval to reduce request frequency
Reduce max_parallel for pagination
Implement exponential backoff using retry_http_code

Empty Results:

Verify time windows overlap properly
Check if API has data for the requested period
Ensure filters aren’t too restrictive

Security Best Practices

Credential Management: Never hardcode credentials - always use environment variables
Token Rotation: Regularly rotate API keys and tokens
Least Privilege: Request minimum required permissions
Network Security: Use HTTPS exclusively
Audit Logging: Monitor API usage for anomalies
Error Handling: Don’t expose sensitive data in logs
Rate Limit Compliance: Respect API provider limits

Sample Response Data

Standard JSON Response

{
  "data": [
    {
      "id": "log_12345",
      "timestamp": "2024-01-01T10:00:00Z",
      "level": "INFO",
      "message": "Application started",
      "metadata": {
        "service": "api-gateway",
        "region": "us-west-2"
      }
    }
  ],
  "pagination": {
    "next_cursor": "eyJpZCI6MTIzNDV9",
    "has_more": true,
    "total": 1000
  }
}

Nested Pagination Response

{
  "results": [...],
  "meta": {
    "pagination": {
      "page": 1,
      "per_page": 100,
      "total_pages": 10,
      "links": {
        "next": "https://api.example.com/data?page=2"
      }
    }
  }
}

Custom API Integration Examples

Overview

Examples

Environment Variables

Authentication Methods

API Key in Headers

Bearer Token Authentication

Query Parameter Authentication

Basic Authentication

Environment Variables

API Endpoints Reference

Time Windows and Pull Intervals

Time Window Patterns

Unix Timestamp Parameters

ISO 8601 Timestamps

Dynamic Time Windows

Time Window Expression Breakdown

Pagination Strategies

Offset-based Pagination

Cursor-based Pagination

Nested JSON Pagination

Link Header Pagination

Advanced Scenarios

Dynamic Endpoint Construction

Multi-Tenant Configuration

Rate-Limited APIs

API Documentation

Rate Limiting

Troubleshooting

Security Best Practices

Sample Response Data

Standard JSON Response

Nested Pagination Response

Edge Delta AI Assistant

Conversations

Hi! I'm your Edge Delta AI Assistant

Current Context