Pub/Sub Connector

Configure the Pub/Sub connector to collect messages from Google Cloud Pub/Sub subscriptions for AI-powered analysis of application logs, events, and real-time data.

6 minute read

Overview

The Pub/Sub connector collects messages from Google Cloud Pub/Sub subscriptions. Pub/Sub is a fully-managed messaging service providing reliable, asynchronous communication through a publish-subscribe pattern. Content streams into Edge Delta Pipelines for analysis by AI teammates through the Edge Delta MCP connector.

The connector uses pull subscription model to consume messages with automatic acknowledgment and service account authentication. It integrates natively with Google Cloud services (Cloud Run, Cloud Functions, GKE) for centralized log and event collection.

When you add this streaming connector, it appears as a Pub/Sub source in your selected pipeline. AI teammates access this data by querying the Edge Delta backend with the Edge Delta MCP connector.

Add the Pub/Sub Connector

To add the Pub/Sub connector, you configure a GCP service account with subscriber permissions and specify which subscription to pull messages from.

Prerequisites

Before configuring the connector, ensure you have:

Google Cloud Platform project with Pub/Sub enabled
Pub/Sub subscription created for the topic to monitor
Service account with roles/pubsub.subscriber role
Service account JSON key file downloaded
Identified which subscription to consume messages from

Create service account:

gcloud iam service-accounts create edgedelta-pubsub \
  --display-name "Edge Delta Pub/Sub Reader"

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:edgedelta-pubsub@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"

gcloud iam service-accounts keys create edgedelta-key.json \
  --iam-account=edgedelta-pubsub@PROJECT_ID.iam.gserviceaccount.com

Create subscription:

gcloud pubsub subscriptions create edgedelta-sub \
  --topic=application-logs \
  --ack-deadline=60 \
  --message-retention-duration=7d

Configuration Steps

Navigate to AI Team > Connectors in the Edge Delta application
Find the Pub/Sub connector in Streaming Connectors
Click the connector card
Enter Project ID (GCP project identifier)
Enter Sub ID (subscription name only, not full resource path)
Paste complete service account JSON key into Key field
Optionally configure Advanced Settings for metadata or rate limiting
Select a target environment
Click Save

The connector deploys and begins pulling messages from your subscription at the configured interval.

Pub/Sub connector configuration showing authentication and subscription settings

Configuration Options

Connector Name

Name to identify this Pub/Sub connector instance.

Project ID

GCP project identifier where the Pub/Sub subscription exists. This is the project ID (not project name or number) shown in GCP Console project selector.

Format: Valid GCP project ID (lowercase letters, numbers, hyphens)

Examples:

production-telemetry-12345
my-gcp-project

Finding Project ID: GCP Console → Project selector dropdown → Copy Project ID column

Sub ID

Pub/Sub subscription name to pull messages from. Provide only subscription name, not full resource path.

Format: Valid subscription name (alphanumeric, hyphens, underscores, periods)

Examples:

edgedelta-logs
cloud-run-logs-sub
app-logs-subscription

Verify subscription exists:

gcloud pubsub subscriptions describe SUBSCRIPTION_ID --project=PROJECT_ID

Key

Complete JSON key file content for GCP service account with Pub/Sub subscriber permissions. This field or Key Path is required.

Format: Complete valid JSON key file with required fields

Example:

{
  "type": "service_account",
  "project_id": "my-gcp-project",
  "private_key_id": "abc123...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...",
  "client_email": "edgedelta-pubsub@my-gcp-project.iam.gserviceaccount.com",
  "client_id": "123456789...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token"
}

Security: Never commit keys to version control. Use secret management systems.

Key Path

Absolute file path to service account JSON key file on agent filesystem. This field or Key is required.

Format: Absolute file path

Example: /etc/edgedelta/gcp-key.json

When to Use:

Key field: Secrets management systems inject credentials
Key Path: File-based credential distribution

Advanced Settings

Metadata Level (Resource Attributes)

This option is used to define which detected resources and attributes to add to each data item as it is ingested by Edge Delta. You can select:

Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
Default: This option includes the required resources and attributes plus those selected by Edge Delta
High: This option includes the required resources and attributes along with a larger selection of common optional fields.
Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.

Pub/Sub-specific metadata included:

Message ID - Unique message identifier
Publish timestamp - When message was published
Message attributes - Custom key-value pairs
Subscription name - Source subscription
Topic name - Originating topic
Ordering key - For ordered delivery (if enabled)

Metadata Level (Attributes)

Additional attribute-level metadata fields to include.

Default: ed.env.id

Rate Limit

Rate limit configuration to control message consumption rate and prevent overwhelming downstream systems.

How to Use the Pub/Sub Connector

The Pub/Sub connector integrates seamlessly with AI Team, enabling AI-powered analysis of Google Cloud Platform logs and events. AI teammates automatically leverage streaming message data to troubleshoot issues, monitor service health, and analyze event flows.

Use Case: Cloud Run Application Error Analysis

Analyze errors across Cloud Run microservices by streaming Cloud Run logs from Pub/Sub. AI teammates identify error patterns, determine critical issues, and provide recommendations. When combined with PagerDuty alerts, teammates automatically investigate error spikes by querying recent Cloud Run logs and identifying root causes.

Configuration: Create Cloud Logging sink routing Cloud Run logs to Pub/Sub topic, configure connector with Project ID: production-gcp, Sub ID: edgedelta-cloudrun, service account key

Use Case: GKE Cluster Monitoring

Monitor Kubernetes workloads by streaming GKE container logs through Pub/Sub. AI teammates provide namespace-level error analysis, identify problematic pods, and detect failures before production impact. This is valuable when managing multi-namespace clusters—teammates categorize errors, detect patterns, and prioritize remediation.

Configuration: Create Cloud Logging sink for GKE cluster, configure connector with Project ID: production-gcp, Sub ID: edgedelta-gke

Use Case: Cloud Functions Timeout Investigation

Identify Cloud Functions experiencing timeouts by streaming function execution logs. AI teammates analyze patterns, suggest configuration improvements, and track resource utilization. When integrated with Jira, teammates automatically document timeout causes by querying function logs and creating tickets with diagnostic details.

Configuration: Create Cloud Logging sink for Cloud Functions, configure connector with Project ID: serverless-project, Sub ID: edgedelta-functions

Troubleshooting

No messages appearing: Verify service account has roles/pubsub.subscriber role (gcloud projects get-iam-policy PROJECT_ID). Check subscription exists (gcloud pubsub subscriptions describe SUB_ID). Verify messages being published to topic (check GCP Console metrics). Review Edge Delta agent logs for authentication errors.

Permission denied errors: Ensure service account key is valid JSON with required fields. Verify Project ID matches project where service account was created. Check IAM bindings at project and subscription levels. Test credentials: gcloud auth activate-service-account --key-file=key.json.

Subscription not found: Provide only subscription name in Sub ID field, not full resource path projects/*/subscriptions/*. Verify subscription in correct project. Check for trailing spaces or special characters in subscription name.

Growing subscription backlog: Increase acknowledgment deadline in subscription (gcloud pubsub subscriptions update SUB_ID --ack-deadline=120). Check Edge Delta pipeline processing time doesn’t exceed acknowledgment deadline. Review pipeline logs for processing errors preventing acknowledgment. Scale horizontally by deploying to multiple environments.

Duplicate messages: Pub/Sub provides at-least-once delivery, duplicates are normal. Implement deduplication using message ID in Edge Delta pipeline. Ensure acknowledgment deadline sufficient for processing. Consider Pub/Sub exactly-once delivery (preview) for strict deduplication.

Messages garbled: Pub/Sub messages are Base64-encoded. Edge Delta should decode automatically—if not, check pipeline configuration. For JSON messages, add JSON parser processor. For Cloud Logging format, use appropriate parsers.

Configuration changes not taking effect: Verify clicked “Save” after changes. Confirm correct target environment selected. Check agent logs for configuration updates. Restart agents if needed.

After key rotation connector stopped: Update connector configuration with new service account key JSON. Save and redeploy. Revoke old key in GCP after confirming new key works.

Next Steps

Learn about Edge Delta MCP connector for querying Pub/Sub data
Explore Kubernetes Logs connector for GKE pod logs
Learn about creating custom teammates that can use Pub/Sub data

For additional help, visit AI Team Support.