Zero-Code Instrumentation of Go using OpenTelemetry

Overview

Zero-code instrumentation in Go with OpenTelemetry uses eBPF (extended Berkeley Packet Filter) technology to automatically capture telemetry data without requiring any changes to your application code. Unlike Java and Python agents that use bytecode injection, the Go implementation uses eBPF to hook into function calls at the kernel level, making it possible to instrument compiled Go binaries without modification. This approach is particularly useful for capturing telemetry at the boundaries of an application, such as incoming HTTP requests, database calls, gRPC communications, and Kafka message handling.

How It Works

The OpenTelemetry Go automatic instrumentation works by:

1. eBPF Probes: Attaching eBPF probes to specific functions in supported Go libraries
2. Runtime Inspection: Monitoring function calls and extracting telemetry data at runtime
3. Auto SDK Integration: If your application uses the OpenTelemetry Go API (even without explicit initialization), the instrumentation automatically integrates with manual spans through the Auto SDK
4. Data Export: Collected telemetry is exported via OTLP to Edge Delta

The instrumentation runs as a separate process with elevated privileges and attaches to your running Go application.

Prerequisites

Ensure that you have:

• Linux Kernel: Version 4.19 or newer (4.4+ for basic support)
• Go Application: Compiled with Go 1.19 or later
• Architecture: x86_64 (amd64) or ARM64
• Privileges: Root or CAP_SYS_ADMIN capability to run eBPF programs
• Edge Delta: An account and pipeline with an OTLP input node

Supported Libraries

The automatic instrumentation currently supports the following Go libraries:

The instrumentation also captures telemetry sent to the OpenTelemetry global tracer provider (versions v0.14.0 - v1.38.0).

Installation

Option 1: Download Pre-built Binary

Download the latest release from the OpenTelemetry Go Instrumentation releases page:

# Download the latest release
wget https://github.com/open-telemetry/opentelemetry-go-instrumentation/releases/latest/download/otel-go-instrumentation-linux-amd64

# Make it executable
chmod +x otel-go-instrumentation-linux-amd64

# Optionally move to system path
sudo mv otel-go-instrumentation-linux-amd64 /usr/local/bin/otel-go-instrumentation

For ARM64 architecture, download otel-go-instrumentation-linux-arm64 instead.

Option 2: Build from Source

If you need to build from source:

# Clone the repository
git clone https://github.com/open-telemetry/opentelemetry-go-instrumentation.git
cd opentelemetry-go-instrumentation

# Build the instrumentation
make build

# The binary will be at ./otel-go-instrumentation

Option 3: Use Docker Image

For containerized deployments, use the official Docker image:

ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest

Configuration

The OpenTelemetry Go instrumentation is configured entirely through environment variables. There are no command-line configuration files to manage.

Required Environment Variables

These variables must be set for the instrumentation to work:

Optional Configuration Variables

Endpoint Configuration

The endpoint configuration depends on your deployment environment:

Kubernetes Environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://ed-data-supply-svc:4318"

Non-Kubernetes (same host as Edge Delta agent):

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

Protocol Selection:

• For HTTP (port 4318): The instrumentation automatically appends the correct paths (/v1/traces, /v1/metrics)
• For gRPC (port 4317): Set OTEL_EXPORTER_OTLP_PROTOCOL=grpc and use port 4317

See Ingest Data from an OTLP Source for detailed Edge Delta configuration.

Deployment Scenarios

Scenario 1: Same-Host Deployment (Development/VM)

This is the simplest deployment for development or virtual machine environments where the instrumentation runs on the same host as your application.

Step 1: Start Your Go Application

First, start your Go application normally:

./myapp

Step 2: Configure Environment Variables

Set the required environment variables:

export OTEL_GO_AUTO_TARGET_EXE="/path/to/myapp"
export OTEL_SERVICE_NAME="my-go-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

Step 3: Run the Instrumentation

Launch the instrumentation with root privileges:

sudo -E otel-go-instrumentation

The -E flag preserves your environment variables when running with sudo.

Complete Example:

# Start your application in the background
./myapp &

# Run instrumentation with all configuration
sudo OTEL_GO_AUTO_TARGET_EXE="/path/to/myapp" \
     OTEL_SERVICE_NAME="my-go-service" \
     OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" \
     OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
     OTEL_LOG_LEVEL="info" \
     otel-go-instrumentation

Scenario 2: Docker Compose Deployment

For applications running in Docker Compose, add the instrumentation as a sidecar container.

Example docker-compose.yml:

version: '3.8'

services:
  # Your Go application
  myapp:
    image: myapp:latest
    # Your app configuration

  # OpenTelemetry instrumentation sidecar
  otel-instrumentation:
    image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest
    privileged: true
    pid: "service:myapp"  # Share PID namespace with the app
    environment:
      - OTEL_GO_AUTO_TARGET_EXE=/app/myapp
      - OTEL_SERVICE_NAME=my-go-service
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://ed-data-supply-svc:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_LOG_LEVEL=info
    volumes:
      - /proc:/host/proc:ro  # Mount host /proc for process inspection
    depends_on:
      - myapp

Key Configuration Points:

1. privileged: true: Required for eBPF operations
2. pid: “service:myapp”: Shares process namespace with your application
3. /proc mount: Allows the instrumentation to inspect running processes
4. depends_on: Ensures your app starts before instrumentation

Launch the stack:

docker compose up

Scenario 3: Kubernetes Deployment

For Kubernetes deployments, add the instrumentation as a sidecar container in your pod specification.

Example Kubernetes Manifest:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  shareProcessNamespace: true  # Critical: allows sidecar to see app process

  containers:
  # Your main application container
  - name: myapp
    image: myapp:latest
    ports:
    - containerPort: 8080
    # Your app configuration

  # OpenTelemetry instrumentation sidecar
  - name: otel-instrumentation
    image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest
    securityContext:
      runAsUser: 0         # Must run as root
      privileged: true     # Required for eBPF
    env:
    - name: OTEL_GO_AUTO_TARGET_EXE
      value: "/app/myapp"
    - name: OTEL_SERVICE_NAME
      value: "my-go-service"
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://ed-data-supply-svc:4318"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "http/protobuf"
    - name: OTEL_LOG_LEVEL
      value: "info"
    volumeMounts:
    - name: proc
      mountPath: /host/proc
      readOnly: true

  volumes:
  - name: proc
    hostPath:
      path: /proc

Using with Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      shareProcessNamespace: true
      containers:
      - name: myapp
        image: myapp:latest
        # ... app configuration

      - name: otel-instrumentation
        image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest
        securityContext:
          runAsUser: 0
          privileged: true
        env:
        - name: OTEL_GO_AUTO_TARGET_EXE
          value: "/app/myapp"
        - name: OTEL_SERVICE_NAME
          value: "my-go-service"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://ed-data-supply-svc.default.svc.cluster.local:4318"
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true

      volumes:
      - name: proc
        hostPath:
          path: /proc

Sampling Configuration

To control the volume of trace data, configure sampling strategies using environment variables:

Ratio-Based Sampling

Sample a percentage of traces:

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"  # Sample 10% of traces

Parent-Based Sampling (Recommended for Distributed Tracing)

Ensures child spans follow their parent’s sampling decision:

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"  # Sample 10% of root traces

This ensures:

• New traces (requests with no parent) are sampled at the specified rate
• Child spans inherit their parent’s sampling decision
• Distributed traces remain complete across service boundaries

Always Sample (Development)

For development and testing, sample all traces:

export OTEL_TRACES_SAMPLER="always_on"

Never Sample (Disable Tracing)

To disable trace collection:

export OTEL_TRACES_SAMPLER="always_off"

Integrating with Manual Instrumentation (Auto SDK)

If your Go application uses the OpenTelemetry Go API for manual instrumentation, the zero-code instrumentation automatically integrates with your manual spans through the Auto SDK. This allows you to combine automatic boundary instrumentation with custom application logic instrumentation.

How It Works

The Auto SDK is automatically enabled when:

1. Your application imports go.opentelemetry.io/otel (even transitively)
2. You do NOT manually set a global TracerProvider
3. The eBPF instrumentation detects the Auto SDK dependency

Adding Manual Spans

Simply use the OpenTelemetry API as you normally would:

package main

import (
    "log"
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

func main() {
    http.HandleFunc("/api/users", handleUsers)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleUsers(w http.ResponseWriter, r *http.Request) {
    // Get tracer (no initialization needed!)
    tracer := otel.Tracer("my-service")

    // Create manual span - automatically linked to eBPF-generated spans
    ctx, span := tracer.Start(r.Context(), "handleUsers")
    defer span.End()

    // Add custom attributes
    span.SetAttributes(
        attribute.String("handler", "users"),
        attribute.String("method", r.Method),
    )

    // Your business logic
    users := fetchUsers(ctx)

    span.AddEvent("users fetched")
    w.WriteHeader(http.StatusOK)
}

func fetchUsers(ctx context.Context) []User {
    tracer := otel.Tracer("my-service")

    // Nested manual span
    ctx, span := tracer.Start(ctx, "fetchUsers")
    defer span.End()

    // Your database call
    return queryDatabase(ctx)
}

Important: Do NOT call otel.SetTracerProvider() in your code. The Auto SDK registers its own TracerProvider that enables integration with eBPF instrumentation.

Verifying Auto SDK Integration

Check your application’s go.mod file:

grep "go.opentelemetry.io/auto/sdk" go.mod

If present, the Auto SDK is available and will be automatically used by the eBPF instrumentation.

Verifying Instrumentation

1. Check Instrumentation Logs

The instrumentation outputs logs indicating successful attachment:

# If running with sudo
sudo tail -f /var/log/otel-go-instrumentation.log

# Or check stdout if running in foreground

Look for messages like:

Successfully attached to process <PID>
Instrumented net/http
Instrumented database/sql
Exporting traces to http://localhost:4318

2. Verify in Edge Delta

1. Log into your Edge Delta account
2. Navigate to your pipeline configuration
3. Check the OTLP input node for incoming data
4. Look for traces with your service name
5. Verify spans from instrumented libraries (HTTP, database, gRPC, etc.)

3. Generate Test Traffic

Generate traffic to your application to produce telemetry:

# For HTTP applications
curl http://localhost:8080/api/test

# Or use a load testing tool
ab -n 100 -c 10 http://localhost:8080/api/test

4. Check for Errors

Common issues and their solutions:

No telemetry appearing:

• Verify the target executable path is correct
• Ensure the application is running when instrumentation starts
• Check network connectivity to Edge Delta endpoint
• Verify firewall rules allow outbound connections on port 4318/4317

Permission denied errors:

• Ensure instrumentation is running with root privileges or CAP_SYS_ADMIN
• In Kubernetes, verify privileged: true and runAsUser: 0

Library not instrumented:

• Verify the library version is in the supported range
• Check the COMPATIBILITY.md file for known limitations
• Consider using manual instrumentation for unsupported libraries

Auto SDK not working:

• Ensure you’re NOT calling otel.SetTracerProvider() in your code
• Verify go.opentelemetry.io/auto/sdk is in your go.mod
• Use Go 1.24+ and OpenTelemetry SDK v1.33.0+

Best Practices

1. Use in Pre-Production First: Test the instrumentation in development or staging before production deployment

2. Configure Sampling: In high-traffic production environments, use ratio-based sampling to control data volume and costs

3. Monitor Resource Usage: The eBPF instrumentation has minimal overhead, but monitor CPU and memory in production

4. Combine with Manual Instrumentation: Use automatic instrumentation for boundaries and manual spans for business logic

5. Set Resource Attributes: Use OTEL_RESOURCE_ATTRIBUTES to add environment-specific metadata:

   export OTEL_RESOURCE_ATTRIBUTES="environment=production,region=us-west-2,version=1.2.3"
   

6. Use Parent-Based Sampling: For distributed tracing, use parentbased_traceidratio to maintain trace completeness

7. Enable Debug Logging During Setup: Use OTEL_LOG_LEVEL=debug when first setting up, then switch to info or warn in production

8. Container Security: In Kubernetes, consider using Pod Security Policies or Pod Security Standards to limit which pods can run privileged containers

Limitations and Considerations

1. Linux Only: eBPF is a Linux kernel feature; this instrumentation does not work on Windows or macOS

2. Kernel Version: Requires Linux kernel 4.19+ for full functionality (4.4+ for basic support)

3. Privileged Access: Requires root or CAP_SYS_ADMIN capability, which may not be allowed in all environments

4. Limited Library Support: Currently only supports a subset of Go libraries (net/http, database/sql, gRPC, Kafka)

5. Binary Symbols: The Go binary must retain symbol information (not stripped with go build -ldflags="-s -w")

6. Go Version Compatibility: Requires Go 1.19+ for most instrumentation

7. Metrics and Logs: Currently focused on distributed tracing; metrics and logs support is limited

8. Performance: While minimal, there is some overhead from eBPF probes, especially under high load

Troubleshooting

Instrumentation Won’t Start

Error: “Failed to attach eBPF probes”

• Solution: Ensure you’re running as root or with CAP_SYS_ADMIN
• Verify kernel version: uname -r should show 4.19+

Error: “Target executable not found”

• Solution: Check OTEL_GO_AUTO_TARGET_EXE path is absolute and correct
• Verify the binary exists: ls -la /path/to/binary

No Telemetry Data

Problem: No spans appearing in Edge Delta

• Check endpoint configuration is correct
• Verify network connectivity: curl http://localhost:4318/v1/traces
• Enable debug logging: OTEL_LOG_LEVEL=debug
• Check Edge Delta agent is running and accepting OTLP data

Problem: Only seeing automatic spans, no manual spans

• Verify you’re NOT calling otel.SetTracerProvider() in code
• Check go.mod includes go.opentelemetry.io/auto/sdk
• Ensure Go version is 1.24+ for best Auto SDK support

Performance Issues

Problem: High CPU usage

• Reduce sampling rate: OTEL_TRACES_SAMPLER_ARG="0.01" (1%)
• Check for excessive logging: Set OTEL_LOG_LEVEL=warn
• Monitor specific instrumentation overhead

Problem: High memory usage

• This is unusual; check for memory leaks in your application
• Verify you’re using the latest instrumentation version
• Consider reporting an issue to the OpenTelemetry project

Next Steps

• Review supported libraries for your dependencies
• Learn about code-based instrumentation for custom business logic
• Set up Edge Delta processing for your telemetry data
• Explore sampling strategies for production deployments
• Configure resource attributes for richer metadata and filtering

For more information on OpenTelemetry Go automatic instrumentation, visit the official GitHub repository.