Code-based Instrumentation of Go using OpenTelemetry

Learn how to instrument Go applications using OpenTelemetry SDK for comprehensive telemetry data generation and integration with Edge Delta.

9 minute read

Overview

Code-based instrumentation in Go with OpenTelemetry involves manually writing code to generate telemetry data using the OpenTelemetry SDK - a comprehensive set of libraries provided by OpenTelemetry. The SDK provides functions and packages you can use to manually instrument your code to capture telemetry data including traces, metrics, and logs. This typically involves creating “spans” to track the execution timing of operations, using “meters” to capture metrics like request counts and durations, and using loggers to capture structured log data.

Prerequisites

Ensure that you have:

Go 1.23 or later
OpenTelemetry SDK dependencies added to your project
An Edge Delta account and a pipeline with an OTLP input node

Setup OpenTelemetry

1. Install Required Dependencies

First, install the necessary OpenTelemetry packages:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp \
  go.opentelemetry.io/otel/sdk/trace \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/sdk/log \
  go.opentelemetry.io/otel/sdk/resource \
  go.opentelemetry.io/otel/propagation

These packages include:

Core OpenTelemetry API
OTLP exporters for traces, metrics, and logs
SDK implementations for managing telemetry pipelines
Resource detection for metadata enrichment

2. Import Required Packages

Import the necessary OpenTelemetry packages in your Go application:

import (
    "context"
    "errors"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
    "go.opentelemetry.io/otel/log/global"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
)

3. Set Up Resource Attributes

Define resource attributes such as service name and version to associate telemetry data with your application:

func newResource(ctx context.Context) (*resource.Resource, error) {
    return resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("my-service"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )
}

The resource provides standard metadata that will be attached to all telemetry data. The semconv package provides semantic convention attributes that help ensure consistency across observability tools.

4. Configure Traces

Set up a TracerProvider to enable distributed tracing:

func newTracerProvider(ctx context.Context, res *resource.Resource) (*trace.TracerProvider, error) {
    // Create OTLP HTTP trace exporter
    traceExporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithURLPath("/v1/traces"),
        otlptracehttp.WithInsecure(), // Remove in production with proper TLS
    )
    if err != nil {
        return nil, err
    }

    // Create tracer provider with batch span processor
    tracerProvider := trace.NewTracerProvider(
        trace.WithBatcher(traceExporter,
            trace.WithBatchTimeout(time.Second)),
        trace.WithResource(res),
    )

    return tracerProvider, nil
}

The TracerProvider is the SDK component responsible for creating and managing traces. Key configuration options include:

Exporter: The OTLP exporter sends trace data to Edge Delta
Batch Processor: Buffers spans before sending to reduce network overhead
Resource: Attaches metadata to all spans

The endpoint configuration depends on your environment. For example, in a Kubernetes Environment, you need a service open, for example ed-data-supply-svc, for the port that the exporters will send logs, metrics, and traces to. For non-kubernetes environments with exporters running in the same environment as the Edge Delta agent, such as on a Linux VM, the exporters will be able to communicate directly with the Edge Delta agent on localhost. For HTTP (4318), include the path /v1/traces, /v1/metrics, or /v1/logs. For gRPC (4317), do not include the path. See Ingest Data from an OTLP Source.

5. Configure Metrics

Set up a MeterProvider to track application metrics:

func newMeterProvider(ctx context.Context, res *resource.Resource) (*metric.MeterProvider, error) {
    // Create OTLP HTTP metric exporter
    metricExporter, err := otlpmetrichttp.New(ctx,
        otlpmetrichttp.WithEndpoint("localhost:4318"),
        otlpmetrichttp.WithURLPath("/v1/metrics"),
        otlpmetrichttp.WithInsecure(), // Remove in production with proper TLS
    )
    if err != nil {
        return nil, err
    }

    // Create meter provider with periodic reader
    meterProvider := metric.NewMeterProvider(
        metric.WithReader(metric.NewPeriodicReader(metricExporter,
            metric.WithInterval(3*time.Second))),
        metric.WithResource(res),
    )

    return meterProvider, nil
}

The MeterProvider manages metric instruments and their aggregation. Key components include:

Metric Exporter: Sends metric data to Edge Delta
Periodic Reader: Collects and exports metrics at regular intervals
Resource: Attaches metadata to all metrics

6. Configure Logs

Set up a LoggerProvider to capture structured log data:

func newLoggerProvider(ctx context.Context, res *resource.Resource) (*log.LoggerProvider, error) {
    // Create OTLP HTTP log exporter
    logExporter, err := otlploghttp.New(ctx,
        otlploghttp.WithEndpoint("localhost:4318"),
        otlploghttp.WithURLPath("/v1/logs"),
        otlploghttp.WithInsecure(), // Remove in production with proper TLS
    )
    if err != nil {
        return nil, err
    }

    // Create logger provider with batch processor
    loggerProvider := log.NewLoggerProvider(
        log.WithProcessor(log.NewBatchProcessor(logExporter)),
        log.WithResource(res),
    )

    return loggerProvider, nil
}

The LoggerProvider manages log record processing and export. It uses a batch processor to efficiently send logs to Edge Delta while attaching resource metadata.

7. Initialize the OpenTelemetry SDK

Create a setup function that initializes all providers and configures context propagation:

func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
    var shutdownFuncs []func(context.Context) error

    // shutdown calls cleanup functions registered via shutdownFuncs
    shutdown = func(ctx context.Context) error {
        var err error
        for _, fn := range shutdownFuncs {
            err = errors.Join(err, fn(ctx))
        }
        shutdownFuncs = nil
        return err
    }

    // handleErr calls shutdown for cleanup and makes sure that all errors are returned
    handleErr := func(inErr error) {
        err = errors.Join(inErr, shutdown(ctx))
    }

    // Set up resource
    res, err := newResource(ctx)
    if err != nil {
        handleErr(err)
        return
    }

    // Set up propagator for context propagation
    prop := propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    )
    otel.SetTextMapPropagator(prop)

    // Set up trace provider
    tracerProvider, err := newTracerProvider(ctx, res)
    if err != nil {
        handleErr(err)
        return
    }
    shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
    otel.SetTracerProvider(tracerProvider)

    // Set up meter provider
    meterProvider, err := newMeterProvider(ctx, res)
    if err != nil {
        handleErr(err)
        return
    }
    shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
    otel.SetMeterProvider(meterProvider)

    // Set up logger provider
    loggerProvider, err := newLoggerProvider(ctx, res)
    if err != nil {
        handleErr(err)
        return
    }
    shutdownFuncs = append(shutdownFuncs, loggerProvider.Shutdown)
    global.SetLoggerProvider(loggerProvider)

    return shutdown, err
}

This setup function:

Creates and configures all three providers (traces, metrics, logs)
Sets up context propagation for distributed tracing
Returns a shutdown function for proper cleanup
Handles errors and ensures cleanup even on failure

8. Use the SDK in Your Application

In your main function or application initialization, set up and use the SDK:

func main() {
    ctx := context.Background()

    // Set up OpenTelemetry
    otelShutdown, err := setupOTelSDK(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Handle shutdown properly to ensure all telemetry is flushed
    defer func() {
        if err := otelShutdown(context.Background()); err != nil {
            log.Fatal(err)
        }
    }()

    // Your application code here
    // The SDK is now configured and ready to use
}

Using the SDK for Instrumentation

Creating Traces

Use the tracer to create spans that track operations:

import (
    "context"
    "go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("my-service")

func myFunction(ctx context.Context) {
    // Start a new span
    ctx, span := tracer.Start(ctx, "myFunction")
    defer span.End()

    // Your business logic here
    // Pass ctx to nested functions to propagate the trace context
}

Adding Attributes to Spans:

import (
    "go.opentelemetry.io/otel/attribute"
)

func processRequest(ctx context.Context, userID string) {
    ctx, span := tracer.Start(ctx, "processRequest")
    defer span.End()

    // Add attributes to provide context
    span.SetAttributes(
        attribute.String("user.id", userID),
        attribute.Int("request.size", 1024),
    )

    // Business logic
}

Recording Events and Errors:

func handleOperation(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "handleOperation")
    defer span.End()

    // Record an event
    span.AddEvent("operation started")

    // If an error occurs
    err := doSomething()
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "operation failed")
        return err
    }

    span.AddEvent("operation completed successfully")
    return nil
}

Recording Metrics

Create and use metric instruments to track application metrics:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
)

var (
    meter = otel.Meter("my-service")
    requestCounter metric.Int64Counter
    requestDuration metric.Float64Histogram
)

func init() {
    var err error

    // Create a counter for request count
    requestCounter, err = meter.Int64Counter(
        "requests.count",
        metric.WithDescription("Number of requests processed"),
        metric.WithUnit("{request}"),
    )
    if err != nil {
        panic(err)
    }

    // Create a histogram for request duration
    requestDuration, err = meter.Float64Histogram(
        "requests.duration",
        metric.WithDescription("Duration of requests"),
        metric.WithUnit("s"),
    )
    if err != nil {
        panic(err)
    }
}

func handleRequest(ctx context.Context) {
    start := time.Now()

    // Increment request counter
    requestCounter.Add(ctx, 1,
        metric.WithAttributes(attribute.String("method", "GET")))

    // Process request
    processRequest(ctx)

    // Record duration
    duration := time.Since(start).Seconds()
    requestDuration.Record(ctx, duration,
        metric.WithAttributes(attribute.String("method", "GET")))
}

Using Different Metric Instruments:

// UpDownCounter for values that can increase or decrease
activeConnections, _ := meter.Int64UpDownCounter(
    "connections.active",
    metric.WithDescription("Number of active connections"),
)

// Observable Gauge for sampled values
_, _ = meter.Int64ObservableGauge(
    "memory.heap",
    metric.WithDescription("Current heap memory usage"),
    metric.WithInt64Callback(func(ctx context.Context, o metric.Int64Observer) error {
        var m runtime.MemStats
        runtime.ReadMemStats(&m)
        o.Observe(int64(m.HeapAlloc))
        return nil
    }),
)

Capturing Logs

Use log bridges to integrate existing logging with OpenTelemetry:

import (
    "log/slog"
    "go.opentelemetry.io/contrib/bridges/otelslog"
)

var logger *slog.Logger

func init() {
    // Create an OpenTelemetry slog handler
    handler := otelslog.NewHandler("my-service")
    logger = slog.New(handler)
}

func processData(ctx context.Context, data string) {
    // Logs will be automatically correlated with traces
    logger.InfoContext(ctx, "processing data",
        "data_length", len(data),
        "operation", "process")

    // Business logic
}

Instrumenting HTTP Services

Here’s a complete example of instrumenting an HTTP server:

import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    ctx := context.Background()

    // Set up OpenTelemetry SDK
    otelShutdown, err := setupOTelSDK(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer func() { _ = otelShutdown(context.Background()) }()

    // Create HTTP handler with automatic instrumentation
    mux := http.NewServeMux()

    // Wrap handler to add automatic tracing
    handleFunc := func(pattern string, handlerFunc func(http.ResponseWriter, *http.Request)) {
        handler := otelhttp.WithRouteTag(pattern, http.HandlerFunc(handlerFunc))
        mux.Handle(pattern, handler)
    }

    handleFunc("/api/users", handleUsers)
    handleFunc("/api/products", handleProducts)

    // Wrap entire server with instrumentation
    handler := otelhttp.NewHandler(mux, "/")

    // Start server
    log.Fatal(http.ListenAndServe(":8080", handler))
}

func handleUsers(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    // Create custom span for business logic
    ctx, span := tracer.Start(ctx, "handleUsers")
    defer span.End()

    // Add custom attributes
    span.SetAttributes(attribute.String("handler", "users"))

    // Record metric
    requestCounter.Add(ctx, 1,
        metric.WithAttributes(attribute.String("endpoint", "/api/users")))

    // Your business logic
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("Users response"))
}

Best Practices

Always pass context: Pass the context.Context through your call chain to maintain trace context and enable correlation between telemetry signals.
Use semantic conventions: Leverage the semconv package for standardized attribute names to ensure consistency with OpenTelemetry standards.
Instrument at boundaries: Focus instrumentation on system boundaries (HTTP handlers, database calls, external service calls) for maximum visibility with minimal overhead.
Handle shutdown gracefully: Always defer the shutdown function to ensure all telemetry data is flushed before the application exits.
Use appropriate metric types: Choose the right metric instrument for your use case:
- Counter for monotonically increasing values
- UpDownCounter for values that can increase or decrease
- Histogram for distributions
- Gauge for sampled values
Add meaningful attributes: Include contextual information in spans and metrics to enable effective filtering and aggregation in Edge Delta.
Batch telemetry data: Use batch processors for traces and logs to reduce network overhead and improve performance.

Troubleshooting

Telemetry not appearing in Edge Delta:

Verify the endpoint configuration matches your Edge Delta OTLP input node
Check that the Edge Delta agent is running and accepting connections
Ensure network connectivity between your application and Edge Delta
Review application logs for SDK initialization errors

High memory usage:

Adjust batch processor settings to export more frequently
Review the volume of telemetry being generated
Consider sampling strategies for high-volume traces

Missing trace context:

Ensure context is passed through all function calls
Verify propagation is configured correctly
Check that instrumentation libraries are properly initialized

Next Steps

Learn about zero-code instrumentation for automatic telemetry
Set up Edge Delta processing for your telemetry data
Explore sampling strategies to control telemetry volume in production
Review context propagation for distributed tracing across services
Configure resource detection for automatic metadata enrichment

For more information on OpenTelemetry in Go, visit the official OpenTelemetry Go documentation.