Custom API Integration Examples

Configure HTTP Pull to retrieve data from various custom and third-party APIs demonstrating different authentication methods, pagination styles, and data formats.

  7 minute read  

Overview

This guide provides comprehensive examples for integrating with custom APIs and third-party services. Each configuration demonstrates best practices for authentication, time-based filtering, pagination handling, and error management across various API architectures.

The examples cover common authentication patterns, pagination strategies, and real-world API integration scenarios.

Examples

CategoryExamplesUse Case
Authentication MethodsAPI Key, Bearer Token, Query Auth, Basic AuthVarious auth patterns
Time-Based QueriesISO 8601, Unix timestamps, Time windowsHistorical data retrieval
Pagination StrategiesJSON path, Cursor, OffsetLarge dataset handling
Dynamic ConfigurationMulti-tenant, Environment-basedFlexible deployments
Error HandlingRetry codes, Rate limitingResilient operations

Environment Variables

Set these environment variables for your custom API integrations:

# API Authentication
export CUSTOM_API_KEY="your_api_key_here"
export API_TOKEN="your_bearer_token_here"
export API_USERNAME="api_username"
export API_PASSWORD="api_password"

# API Endpoints
export API_BASE_URL="https://api.example.com"
export API_VERSION="v1"

# Configuration
export API_PAGE_SIZE="100"
export API_TIMEOUT="30"
export API_RETRY_LIMIT="3"

Authentication Methods

Common authentication patterns for HTTP Pull sources with complete working examples.

API Key in Headers

Pass API keys through request headers:

nodes:
- name: custom_api_key_auth
  type: http_pull_input
  endpoint: https://api.example.com/v1/logs
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("CUSTOM_API_KEY", "")
    - header: "Accept"
      value_expression: "application/json"

  parameters:
    - name: format
      value: json
    - name: limit
      value: "100"

  parameter_expressions:
    - name: "since"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")

Bearer Token Authentication

OAuth2 bearer tokens and JWT authentication:

nodes:
- name: bearer_token_auth
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("API_TOKEN", "")], "")
    - header: "Accept"
      value_expression: "application/json"

  parameter_expressions:
    - name: "from"
      value_expression: String(UnixSeconds(Now() - Duration("5m10s")))
    - name: "to"
      value_expression: String(UnixSeconds(Now()))

Query Parameter Authentication

Legacy APIs using URL parameter authentication:

nodes:
- name: query_param_auth
  type: http_pull_input
  endpoint: https://api.legacy-service.com/data
  method: GET
  pull_interval: 10m

  parameter_expressions:
    - name: "api_key"
      value_expression: EDXEnv("LEGACY_API_KEY", "")
    - name: "from"
      value_expression: String(UnixSeconds(Now() - Duration("10m10s")))
    - name: "to"
      value_expression: String(UnixSeconds(Now()))
    - name: "limit"
      value_expression: "500"

Basic Authentication

HTTP Basic Auth with username and password:

nodes:
- name: basic_auth_api
  type: http_pull_input
  endpoint: https://api.internal.com/metrics
  method: GET
  pull_interval: 1m

  # Base64 encode credentials externally
  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Basic ", EDXEnv("BASIC_AUTH_CREDENTIALS", "")], "")
    - header: "Accept"
      value_expression: "application/json"

  parameters:
    - name: limit
      value: "1000"

Environment Variables

Set these environment variables with your API credentials:

# API Key Authentication
export CUSTOM_API_KEY="your_api_key_here"

# Bearer Token
export API_TOKEN="your_bearer_token_here"

# Query Parameter Auth
export LEGACY_API_KEY="your_legacy_key_here"

# Basic Auth (base64 encoded username:password)
export BASIC_AUTH_CREDENTIALS=$(echo -n "username:password" | base64)

API Endpoints Reference

Authentication TypeHeader/ParameterFormatDescription
API Key (Header)X-API-KeyPlain textAPI key in custom header
Bearer TokenAuthorizationBearer {token}OAuth2 or JWT token
Query ParameterURL parameter?api_key={key}Legacy URL authentication
Basic AuthAuthorizationBasic {base64}Base64 encoded credentials

Time Windows and Pull Intervals

Each endpoint uses optimized time windows to prevent data gaps and respect rate limits:

PatternTime WindowPull IntervalRationale
High-frequency5m10s5 minutesReal-time monitoring, needs overlap
Standard10m10s10 minutesRegular data collection
Low-frequency1h10m1 hourHistorical data, batch processing
Rate-limited30m10s30 minutesAPIs with strict limits

Time Window Patterns

Different approaches for time-based data retrieval with working examples.

Unix Timestamp Parameters

Many APIs require Unix timestamps for time queries:

nodes:
- name: unix_timestamp_api
  type: http_pull_input
  endpoint: https://api.example.com/logs
  method: GET
  pull_interval: 1h

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")

  parameter_expressions:
    # Unix seconds
    - name: "start_time"
      value_expression: String(UnixSeconds(Now() - Duration("1h10m")))
    - name: "end_time"
      value_expression: String(UnixSeconds(Now()))

    # Unix milliseconds
    - name: "start_ms"
      value_expression: String(UnixMilli(Now() - Duration("1h10m")))
    - name: "end_ms"
      value_expression: String(UnixMilli(Now()))

ISO 8601 Timestamps

Standard date format for modern APIs:

nodes:
- name: iso_timestamp_api
  type: http_pull_input
  endpoint: https://api.example.com/events
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("TOKEN", "")], "")

  parameter_expressions:
    # ISO 8601 with timezone
    - name: "start"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")
    - name: "end"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

    # Date only format
    - name: "date"
      value_expression: FormatTime(Now(), "%Y-%m-%d")

Dynamic Time Windows

Environment-controlled time ranges:

nodes:
- name: dynamic_window_api
  type: http_pull_input
  endpoint: https://api.example.com/metrics
  method: GET
  pull_interval: 10m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")

  parameter_expressions:
    # Configurable window size
    - name: "from"
      value_expression: FormatTime(Now() - Duration(EDXEnv("WINDOW_SIZE", "10m10s")), "%Y-%m-%dT%H:%M:%SZ")
    - name: "to"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
    - name: "interval"
      value_expression: EDXEnv("QUERY_INTERVAL", "5m")

Time Window Expression Breakdown

Here’s how time window expressions work in detail:

# Educational breakdown - write as single line in actual config
parameter_expressions:
  - name: "start_time"
    value_expression: >
      String(                                    # Convert to string
        UnixSeconds(                            # Get Unix timestamp in seconds
          Now() - Duration("5m10s")              # Current time minus 5 minutes 10 seconds
        )
      )      

  - name: "formatted_time"
    value_expression: >
      FormatTime(                                # Format time to string
        Now() - Duration("1h"),                 # One hour ago
        "%Y-%m-%dT%H:%M:%SZ"                    # ISO 8601 format
      )

Common Time Formats:

  • Unix seconds: UnixSeconds(Now())
  • Unix milliseconds: UnixMilli(Now())
  • ISO 8601: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")
  • Custom format: FormatTime(Now(), "%Y/%m/%d %H:%M:%S")

Pagination Strategies

Different approaches for handling multi-page API responses.

Offset-based Pagination

Traditional page number or offset pagination:

nodes:
- name: offset_pagination_api
  type: http_pull_input
  endpoint: https://api.service.com/data
  method: GET
  pull_interval: 10m

  header_expressions:
    - header: "API-Key"
      value_expression: EDXEnv("SERVICE_API_KEY", "")

  parameters:
    - name: limit
      value: "100"
    - name: offset
      value: "0"

  # Note: Manual offset management typically required
  # Consider using cursor-based pagination instead

Cursor-based Pagination

Efficient pagination using continuation tokens:

nodes:
- name: cursor_pagination_api
  type: http_pull_input
  endpoint: https://api.modern.com/feed
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("MODERN_TOKEN", "")], "")

  parameters:
    - name: limit
      value: "100"

  pagination:
    url_json_path: "meta.next_cursor_url"
    max_parallel: 3

Nested JSON Pagination

Complex pagination with nested response structures:

nodes:
- name: nested_pagination_api
  type: http_pull_input
  endpoint: https://api.complex.com/v2/records
  method: GET
  pull_interval: 15m

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("COMPLEX_API_KEY", "")

  parameters:
    - name: page_size
      value: "50"

  pagination:
    url_json_path: "metadata.pagination.links.next"
    max_parallel: 5

RFC 5988 standard Link headers (GitHub-style):

nodes:
- name: link_header_api
  type: http_pull_input
  endpoint: https://api.github-style.com/repos
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("GITHUB_TOKEN", "")], "")

  parameters:
    - name: per_page
      value: "100"

  pagination:
    link_relation: "next"
    max_parallel: 3

Advanced Scenarios

Complex integration patterns for specific use cases.

Dynamic Endpoint Construction

Build endpoint URLs dynamically from environment:

nodes:
- name: dynamic_endpoint_api
  type: http_pull_input
  # Dynamic endpoint based on environment
  endpoint_expression: Concat(["https://", EDXEnv("API_HOST", "api.example.com"), "/v1/metrics"], "")
  method: GET
  pull_interval: 30s

  header_expressions:
    - header: "X-API-Key"
      value_expression: EDXEnv("API_KEY", "")
    - header: "X-Request-ID"
      value_expression: Concat(["req-", String(UnixMilli(Now()))], "")
    - header: "X-Environment"
      value_expression: EDXEnv("ENVIRONMENT", "development")
    - header: "X-API-Version"
      value_expression: EDXEnv("API_VERSION", "v1")

  parameter_expressions:
    - name: "since"
      value_expression: String(UnixSeconds(Now() - Duration("35s")))

Multi-Tenant Configuration

Support multiple tenants with environment variables:

nodes:
- name: multitenant_api
  type: http_pull_input
  endpoint_expression: Concat(["https://", EDXEnv("TENANT_DOMAIN", "default"), ".api.service.com/v1/data"], "")
  method: GET
  pull_interval: 5m

  header_expressions:
    - header: "X-Tenant-ID"
      value_expression: EDXEnv("TENANT_ID", "")
    - header: "Authorization"
      value_expression: Concat(["Bearer ", EDXEnv("TENANT_TOKEN", "")], "")

  parameter_expressions:
    - name: "from"
      value_expression: FormatTime(Now() - Duration("5m10s"), "%Y-%m-%dT%H:%M:%SZ")
    - name: "to"
      value_expression: FormatTime(Now(), "%Y-%m-%dT%H:%M:%SZ")

Rate-Limited APIs

Conservative configuration for strict rate limits:

nodes:
- name: rate_limited_api
  type: http_pull_input
  endpoint: https://api.ratelimited.com/data
  method: GET
  pull_interval: 30s

  header_expressions:
    - header: "API-Key"
      value_expression: EDXEnv("RATE_LIMITED_KEY", "")

  parameters:
    - name: per_page
      value: "10"  # Small page size

  retry_http_code:
    - 429  # Too Many Requests
    - 503  # Service Unavailable

  pagination:
    link_relation: "next"
    max_parallel: 1  # Sequential only

API Documentation

For implementing custom API integrations, refer to:

Rate Limiting

Different APIs implement various rate limiting strategies:

  • Request-based: Fixed number of requests per time window
  • Token bucket: Burst capacity with refill rate
  • Sliding window: Rolling time window limits
  • Concurrent limits: Maximum parallel requests

Configure pull intervals and pagination settings to respect these limits.

Troubleshooting

401 Unauthorized:

  • Verify API key or token in environment variables
  • Check token expiration for OAuth2/JWT
  • Ensure correct authentication method (header vs parameter)

400 Bad Request:

  • Verify time parameter formats match API requirements
  • Check required parameters are present
  • Validate URL encoding for special characters

403 Forbidden:

  • Check API permissions and scopes
  • Verify IP allowlisting if required
  • Confirm subscription tier has required access

429 Too Many Requests:

  • Increase pull_interval to reduce request frequency
  • Reduce max_parallel for pagination
  • Implement exponential backoff using retry_http_code

Empty Results:

  • Verify time windows overlap properly
  • Check if API has data for the requested period
  • Ensure filters aren’t too restrictive

Security Best Practices

  1. Credential Management: Never hardcode credentials - always use environment variables
  2. Token Rotation: Regularly rotate API keys and tokens
  3. Least Privilege: Request minimum required permissions
  4. Network Security: Use HTTPS exclusively
  5. Audit Logging: Monitor API usage for anomalies
  6. Error Handling: Don’t expose sensitive data in logs
  7. Rate Limit Compliance: Respect API provider limits

Sample Response Data

Standard JSON Response

{
  "data": [
    {
      "id": "log_12345",
      "timestamp": "2024-01-01T10:00:00Z",
      "level": "INFO",
      "message": "Application started",
      "metadata": {
        "service": "api-gateway",
        "region": "us-west-2"
      }
    }
  ],
  "pagination": {
    "next_cursor": "eyJpZCI6MTIzNDV9",
    "has_more": true,
    "total": 1000
  }
}

Nested Pagination Response

{
  "results": [...],
  "meta": {
    "pagination": {
      "page": 1,
      "per_page": 100,
      "total_pages": 10,
      "links": {
        "next": "https://api.example.com/data?page=2"
      }
    }
  }
}