Edge Delta GCS Source

The Edge Delta GCS Source node reads and processes log objects from a Google Cloud Storage bucket, discovered through Pub/Sub object-change notifications.

11 minute read

Overview

The GCS source node allows the Edge Delta agent to read log objects from a Google Cloud Storage bucket. New objects are discovered through a Pub/Sub subscription that receives GCS object-change notifications. The agent pulls each notification, downloads the referenced object, and processes it within the Edge Delta ecosystem.

This node was introduced in v2.21.0.

See Prepare for a GCS Source for information on setting up your environment.

outgoing_data_types: log

Example Edge Delta Pipeline configuration

Inline service account key

This configuration authenticates using a GCP service account JSON key provided inline. Use a YAML block scalar (|) to avoid quoting issues with the JSON content.

nodes:
- name: my_gcs_input
  type: gcs_input
  project_id: "my-gcp-project"
  sub_id: "edgedelta-gcs-sub"
  key: |
    {"type":"service_account","project_id":"my-gcp-project","private_key_id":"abc123...","private_key":"-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n","client_email":"edgedelta-gcs@my-gcp-project.iam.gserviceaccount.com","client_id":"123456789","token_uri":"https://oauth2.googleapis.com/token"}

Service account key file

This configuration authenticates using a file path to a GCP service account JSON key stored on the agent filesystem.

nodes:
- name: my_gcs_input
  type: gcs_input
  project_id: "my-gcp-project"
  sub_id: "edgedelta-gcs-sub"
  key_path: "/etc/edgedelta/gcp-service-account.json"

Advanced version

nodes:
- name: my_gcs_input
  type: gcs_input
  project_id: "my-gcp-project"
  sub_id: "edgedelta-gcs-sub"
  key_path: "/etc/edgedelta/gcp-service-account.json"
  poll_interval: 10s
  immediate_poll_threshold: 5
  compression: gzip
  file_name_filter: ".*\\.log$"

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: gcs_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

project_id

The project_id parameter specifies the Project ID of the Pub/Sub subscription in Google Cloud. It is specified as a string and is required.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>

sub_id

The sub_id parameter indicates the Pub/Sub subscription from which GCS object-change notifications are pulled. It is specified as a string and is required.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>

Optional Parameters

disabled

The disabled parameter disables the node in the pipeline. When set to true, the agent ignores the node at startup. You can toggle nodes on or off in the pipeline builder. It is specified as a boolean, defaults to false, and is optional.

nodes:
  - name: <node name>
    type: <node type>
    disabled: true

key

The key parameter accepts the complete contents of a GCP service account JSON key file. The service account must be able to pull from the Pub/Sub subscription and read objects from the bucket. Use this parameter or key_path, not both.

Use a YAML block scalar (key: |) to pass the JSON content without escaping issues. Do not wrap the JSON in extra quotes or backticks.

This field supports secret references for secure credential management. Instead of hardcoding sensitive values, you can reference a secret configured in your pipeline.

To use a secret in the GUI:

Create a secret in your pipeline’s Settings > Secrets section (see Secrets)
In this field, select the secret name from the dropdown list that appears

To use a secret in YAML: Reference it using the syntax: '{{ SECRET secret-name }}'

Example:

field_name: '{{ SECRET my-credential }}'

Note: The secret reference must be enclosed in single quotes when using YAML. Secret values are encrypted at rest and resolved at runtime, ensuring no plaintext credentials appear in logs or API responses.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  key: |
    {"type":"service_account","project_id":"...","private_key":"-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n","client_email":"...@....iam.gserviceaccount.com"}

key_path

The key_path parameter specifies the absolute file path to a GCP service account JSON key file on the agent filesystem. Use this parameter or key, not both.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  key_path: "/etc/edgedelta/gcp-service-account.json"

poll_interval

The poll_interval parameter sets the interval between Pub/Sub pulls when not using immediate polling. It is specified as a duration and is optional. The default is 5s.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  poll_interval: 10s

immediate_poll_threshold

The immediate_poll_threshold parameter triggers immediate re-polling (without waiting for poll_interval) when the number of messages received meets or exceeds this threshold. The value must be between 1 and 10. Set to 0 to disable immediate polling. It is specified as an integer and is optional. The default is 0 (disabled).

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  poll_interval: 5s
  immediate_poll_threshold: 5

compression

The compression parameter defines the compression format of the objects being ingested. You can specify gzip, zstd, snappy, or uncompressed. It is specified as a string. It is optional and the default is uncompressed.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  compression: gzip

file_name_filter

The file_name_filter parameter specifies a regex pattern for filtering object names to download. Only objects whose full path matches the pattern are processed. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  file_name_filter: ".*\\.log$"

parallel_worker_count

The parallel_worker_count parameter sets the number of workers that download and process objects in parallel. It is specified as an integer and is optional.

nodes:
- name: <node name>
  type: gcs_input
  project_id: <your project ID>
  sub_id: <your subscription ID>
  parallel_worker_count: 2

rate_limit

The rate_limit parameter enables you to control data ingestion based on system resource usage. This advanced setting helps prevent source nodes from overwhelming the agent by automatically throttling or stopping data collection when CPU or memory thresholds are exceeded.

Use rate limiting to prevent runaway log collection from overwhelming the agent in high-volume sources, protect agent stability in resource-constrained environments with limited CPU/memory, automatically throttle during bursty traffic patterns, and ensure fair resource allocation across source nodes in multi-tenant deployments.

When rate limiting triggers, pull-based sources (File, S3, HTTP Pull) stop fetching new data, push-based sources (HTTP, TCP, UDP, OTLP) reject incoming data, and stream-based sources (Kafka, Pub/Sub) pause consumption. Rate limiting operates at the source node level, where each source with rate limiting enabled independently monitors and enforces its own thresholds.

Rate limiting vs backpressure: Rate limiting and backpressure are different mechanisms. Rate limiting controls data ingestion at the source based on CPU or memory thresholds. Backpressure controls data output at the destination when in-memory queues fill up. Backpressure logs such as backpressure: dataCh at 92% capacity do not appear when rate limiting triggers, and rate limiting logs do not appear during backpressure events.

Observability: When rate limiting is active in enforce mode, the agent does not produce INFO, WARN, or ERROR level logs. To confirm that rate limiting is triggering, check the source node’s Bytes I/O metrics in the Edge Delta app for gaps or drops that coincide with expected throttling periods.

Configuration Steps:

Click Add New in the Rate Limit section
Click Add New for Evaluation Policy
Select Policy Type:

CPU Usage: Monitors CPU consumption and rate limits when usage exceeds defined thresholds. Use for CPU-intensive sources like file parsing or complex transformations.
Memory Usage: Monitors memory consumption and rate limits when usage exceeds defined thresholds. Use for memory-intensive sources like large message buffers or caching.
AND (composite): Combines multiple sub-policies with AND logic. All sub-policies must be true simultaneously to trigger rate limiting. Use when you want conservative rate limiting (both CPU and memory must be high).
OR (composite): Combines multiple sub-policies with OR logic. Any sub-policy can trigger rate limiting. Use when you want aggressive rate limiting (either CPU or memory being high triggers).

Select Evaluation Mode. Choose how the policy behaves when thresholds are exceeded:

Enforce (default): Actively applies rate limiting when thresholds are met. Pull-based sources (File, S3, HTTP Pull) stop fetching new data, push-based sources (HTTP, TCP, UDP, OTLP) reject incoming data, and stream-based sources (Kafka, Pub/Sub) pause consumption. Use in production to protect agent resources.
Monitor: Logs when rate limiting would occur without actually limiting data flow. Use for testing thresholds before enforcing them in production.
Passthrough: Disables rate limiting entirely while keeping the configuration in place. Use to temporarily disable rate limiting without removing configuration.

Set Absolute Limits and Relative Limits (for CPU Usage and Memory Usage policies)

Note: If you specify both absolute and relative limits, the system evaluates both conditions and rate limiting triggers when either condition is met (OR logic). For example, if you set absolute limit to 1.0 CPU cores and relative limit to 50%, rate limiting triggers when the source uses either 1 full core OR 50% of available CPU, whichever happens first.

For CPU Absolute Limits: Enter value in full core units:
- 0.1 = one-tenth of a CPU core
- 0.5 = half a CPU core
- 1.0 = one full CPU core
- 2.0 = two full CPU cores
For CPU Relative Limits: Enter percentage of total available CPU (0-100):
- 50 = 50% of available CPU
- 75 = 75% of available CPU
- 85 = 85% of available CPU
For Memory Absolute Limits: Enter value in bytes
- 104857600 = 100Mi (100 × 1024 × 1024)
- 536870912 = 512Mi (512 × 1024 × 1024)
- 1073741824 = 1Gi (1 × 1024 × 1024 × 1024)
For Memory Relative Limits: Enter percentage of total available memory (0-100)
- 60 = 60% of available memory
- 75 = 75% of available memory
- 80 = 80% of available memory

Set Refresh Interval (for CPU Usage and Memory Usage policies). Specify how frequently the system checks resource usage:

Recommended Values:
- 10s to 30s for most use cases
- 5s to 10s for high-volume sources requiring quick response
- 1m or higher for stable, low-volume sources

The system fetches current CPU/memory usage at the specified refresh interval and uses that value for evaluation until the next refresh. Shorter intervals provide more responsive rate limiting but incur slightly higher overhead, while longer intervals are more efficient but slower to react to sudden resource spikes.

The GUI generates YAML as follows:

# Simple CPU-based rate limiting
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: cpu_usage
        evaluation_mode: enforce
        absolute_limit: 0.5  # Limit to half a CPU core
        refresh_interval: 10s

# Simple memory-based rate limiting
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: memory_usage
        evaluation_mode: enforce
        absolute_limit: 536870912  # 512Mi in bytes
        refresh_interval: 30s

Composite Policies (AND / OR)

When using AND or OR policy types, you define sub-policies instead of limits. Sub-policies must be siblings (at the same level)—do not nest sub-policies within other sub-policies. Each sub-policy is independently evaluated, and the parent policy’s evaluation mode applies to the composite result.

AND Logic: All sub-policies must evaluate to true at the same time to trigger rate limiting. Use when you want conservative rate limiting (limit only when CPU AND memory are both high).
OR Logic: Any sub-policy evaluating to true triggers rate limiting. Use when you want aggressive protection (limit when either CPU OR memory is high).