Edge Delta HTTP Pull Source

Fetch HTTP log traffic from an endpoint.

  3 minute read  

Overview

The HTTP Pull node allows the Edge Delta agent to periodically send HTTP requests to an endpoint in order to pull data. This type of data ingestion is useful when you need to retrieve logs or data from an HTTP-based API.

  • outgoing_data_types: log

Example Configuration

nodes:
- name: my_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET
  headers:
    - header: Accept
      value: application/json
  parameters:
    - name: tag
      value: source_id
  pull_interval: 1m
  retry_http_code:
    - 409
    - 429

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: http_pull_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

endpoint

The endpoint specifies the URL to which the HTTP requests are sent. It is a required parameter and must be specified as a valid URL.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET

endpoint

The method parameter defines the HTTP method used for requests. Supported values are GET and POST. This is a required parameter.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET

Optional Parameters

headers

The headers parameter allows adding HTTP headers to the requests. It is specified as a list of key-value pairs. If the endpoint requires authorization, you can include the necessary authentication information in the headers parameter. For example, by adding a Bearer token, which is commonly used for authenticated HTTP requests.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET
  headers:
    - header: Accept
      value: application/json

parameters

The parameters field enables you to add query parameters to the requests, if your endpoint supports them. It is specified as a list of key-value pairs.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET
  parameters:
    - name: tag
      value: source_id

pull_interval

The pull_interval is the frequency at which HTTP requests are sent to the endpoint. The default is 1m (1 minute) and it is specified as a duration.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET
  pull_interval: 1m

retry_http_code

The retry_http_code parameter specifies additional HTTP status codes that will trigger a retry of the request. It is specified as a list of integers and is optional.

nodes:
- name: ed_api_http_pull
  type: http_pull_input
  endpoint: https://api.yourapp.com
  method: GET
  retry_http_code:
    - 409
    - 429

Testing an Endpoint

You can test your endpoint with an HTTP request using curl, a command-line tool for transferring data with URLs. The command contents depend on how your endpoint is configured, such as the HTTP method, headers, query parameters, and whether authentication is required.

Syntax:

curl -X <HTTP_METHOD> "<ENDPOINT_URL>?<QUERY_PARAMETERS>" -H "<HEADER>: <HEADER_VALUE>"
  • HTTP_METHOD: This could be GET or POST, depending on the method supported by the endpoint.
  • ENDPOINT_URL: The URL of the endpoint to which the request is being sent.
  • QUERY_PARAMETERS: Optional, used if the endpoint requires query parameters in the URL.
  • HEADER_VALUE: Specified using -H flag, which adds custom header fields to the request. Headers are often needed for specifying content types or for authentication.

If using POST, include -d to specify data to send in the request body:

-d '{"key1":"value1", "key2":"value2"}'

Example Command:

curl -X GET "https://api.your-company.com/data?tag=value" -H "Accept: application/json" -H "Authorization: Bearer XYZ123"