Code-based Instrumentation of Python using OpenTelemetry

Instrument Python using Code-based OpenTelemetry to emit useful telemetry.

  7 minute read  

Overview

Instrumentation with OpenTelemetry involves integrating and configuring your application to generate telemetry data, such as traces and metrics, which provide insights into the application’s performance and behavior. This process involves setting up the OpenTelemetry API and SDK, configuring exporters to transmit data to your chosen telemetry backend, and using instrumentation techniques to capture the desired information from your application

Prerequisites

Before you begin manually instrumenting your Python application with OpenTelemetry, ensure that Python 3.6 or later is installed on your system. In addition, install the required OpenTelemetry packages:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp

These packages include essential components such as the OTLP exporter needed to transmit data to Edge Delta.

Setup OpenTelemetry

1. Import Required Modules

To collect telemetry data from your Python application, you need to set up and initialize the OpenTelemetry SDK. This involves configuring the providers and setting up exporters to direct the telemetry data to Edge Delta. The OpenTelemetry Protocol (OTLP) exporters are used in this example to maintain the OTEL schema as telemetry data is passed to Edge Delta:

from opentelemetry import trace, logs 
from opentelemetry.metrics import get_meter 

from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.logs import LoggerProvider
from opentelemetry.sdk.logs.export import BatchLogRecordProcessor

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.logger_exporter import OTLPLogExporter

In this section:

  • Core telemetry modules are imported: log, trace and get_meter.
  • A resource to enrich metadata is imported.
  • SDK providers that manage logs, metrics and traces are imported.
  • Exporters that send trace, metric, and log data using OTLP are imported.

If you want to use HTTP for logs instead of gRPC:

from opentelemetry.exporter.otlp.proto.http.logger_exporter import OTLPLogExporter

2. Set Up Resource Attributes

Define a Resource to associate common attributes such as service.name with all telemetry data:

resource = Resource(attributes={"service.name": "your-service-name"})

3. Configure Traces

Configure OpenTelemetry tracing by setting up a TracerProvider, adding a Span Processor, and configuring an OTLP Exporter.

trace_provider = TracerProvider(resource=resource)
span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="your-traces-endpoint"))
trace_provider.add_span_processor(span_processor)
trace.set_tracer_provider(trace_provider)
  • The resource parameter attaches metadata such as a service name or instance ID to traces.
  • The BatchSpanProcessor buffers and batches spans before exporting them to reduce network load. Instead of sending each span immediately, it groups multiple spans and sends them at once.
  • The OTLPSpanExporter sends traces to Edge Delta using the OpenTelemetry Protocol (OTLP).
  • The endpoint="your-traces-endpoint" specifies the endpoint and port (typically 4317 for gPRC) to which you want to send trace data.

The endpoint configuration depends on your environment. For example, in a Kubernetes Environment, you need a service open, for example ed-data-supply-svc, for the port that the exporters will send logs, metrics, and traces to. For non-kubernetes environments with exporters running in the same environment as the Edge Delta agent, such as on a Linux VM, the exporters will be able to communicate directly with the Edge Delta agent on localhost. For HTTP (4318), include the path /v1/traces, /v1/metrics, or /v1/logs. For gRPC (4317), do not include the path. See Ingest Data from an OTLP Source.

4. Configure Metrics

Configure OpenTelemetry metrics by setting up a MeterProvider, adding a Metric Reader, and configuring an OTLP Metric Exporter.

metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="your-metrics-endpoint"))
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
  • MeterProvider is the main provider for managing and generating metrics. The resource parameter attaches metadata (e.g., service name) to the metrics, while metric_readers=[metric_reader] tells the MeterProvider how to collect and export metrics.
  • PeriodicExportingMetricReader collects and exports metrics periodically instead of immediately. It uses a predefined interval (default is 60 seconds) to batch and send metrics efficiently.
  • The OTLPMetricExporter sends metrics to Edge Delta using OTLP. The endpoint includes a port (4317 for gRPC by default). As with the Traces configuration, the endpoint configuration depends on your environment.
  • metrics.set_meter_provider(meter_provider) registers this MeterProvider as the global provider, so all calls to metrics.get_meter() will use this configuration.

5. Configure Logs

Set up logs by initializing a LoggerProvider and configuring an exporter:

logger_provider = LoggerProvider(resource=resource)
log_exporter = OTLPLogExporter(endpoint="your-logs-endpoint")
log_processor = BatchLogRecordProcessor(log_exporter)
logger_provider.add_log_record_processor(log_processor)
logs.set_logger_provider(logger_provider)

This section

  • initializes logger provider with resource
  • sets up the OTLP exporter for logs
  • attaches the processor to the logger provider
  • sets the global logger provider. As with the Traces configuration, the endpoint configuration depends on your environment.

6. Debugging and Console Export (Optional):

For development purposes, you can set up console exporters to view logs, metrics and traces data locally in the console:

from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter
from opentelemetry.sdk.logs.export import ConsoleLogExporter, BatchLogRecordProcessor
span_processor_console = BatchSpanProcessor(ConsoleSpanExporter())
trace_provider.add_span_processor(span_processor_console)
metric_reader_console = PeriodicExportingMetricReader(ConsoleMetricExporter())
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader_console]) 
log_exporter_console = ConsoleLogExporter()
log_processor_console = BatchLogRecordProcessor(log_exporter_console)
logger_provider.add_log_record_processor(log_processor_console)

Instrumentation

1. Acquire a Tracer and Start a Span

Obtain a tracer instance and create spans to delineate specific operations. Spans represent operations or events in your application. They are used for performance monitoring, debugging, and distributed tracing.

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my_custom_span"):
    # Your application logic
  • trace.get_tracer(__name__) gets a tracer instance that allows you to create spans. __name__ is typically used as the name of the module where the tracer is created. The tracer is linked to the global TracerProvider that was set earlier using trace.set_tracer_provider(trace_provider).
  • with tracer.start_as_current_span("my_custom_span"): Creates a new span named "my_custom_span". The with statement ensures span lifecycle management: The span starts when entering the block and it automatically ends when exiting the block, even if an exception occurs.

2. Measure Metrics

Acquire a meter, create a counter and record a metric value.

from opentelemetry.metrics import get_meter

meter = get_meter(__name__)
counter = meter.create_counter("my_custom_counter")

counter.add(1, {"environment": "production"})
  • get_meter(__name__) retrieves a meter instance used for creating metrics. The __name__ module name helps categorize metrics when exported. This meter is linked to the global MeterProvider, which was set earlier using metrics.set_meter_provider(meter_provider).
  • create_counter("my_custom_counter") defines a Counter metric named “my_custom_counter”. Counters are monotonic, meaning they only increase (e.g., request count, processed items). The metric will be exported periodically using PeriodicExportingMetricReader.
  • counter.add(1, {"environment": "production"}) increments the counter by 1. It includes an attribute {"environment": "production"} to provide context. The OpenTelemetry OTLP exporter will collect this metric with the specified attributes.

3. Capture Logs

Use the logging API to capture and export log messages:

logger = logs.get_logger(__name__)
logger.info("This is an informational message", {"additional": "context"})
  • logs.get_logger(__name__) retrieves a logger instance to capture log messages. The __name__ module name helps categorize logs when exported. The logger is linked to the global LoggerProvider, which was set earlier using logs.set_logger_provider(logger_provider).
  • logger.info("message", {...}) logs a message at the INFO level (other levels: debug, warning, error, critical). The second argument adds structured attributes (metadata) to the log.

This approach is different to standard logging module as it allows structured attributes, enables correlation with traces, is exported using OTLP, and allows batch processing with BatchLogRecordProcessor. To link logs with traces, you attach trace context (trace ID, span ID) to log messages:

logger = logs.get_logger(__name__)
span = trace.get_current_span()
trace_id = span.get_span_context().trace_id
span_id = span.get_span_context().span_id
logger.info("User login event", {"trace_id": trace_id, "span_id": span_id, "user_id": 42})

This enables seamless debugging and observability across traces and logs.

Testing and Validation

After setting up OpenTelemetry and instrumenting your Python application, it’s crucial to test and validate that telemetry data is correctly generated, captured, and exported to Edge Delta. This process ensures that the instrumentation is functional and provides accurate insights into your application’s performance.

  1. Start your application in your development or testing environment to initiate data collection:
python my_application.py
  1. Simulate typical user activities or load to generate traces, metrics, and logs within your application. This could involve interacting with your application’s UI or using scripts/tools to simulate API requests.
  2. If you have set up console exporters, monitor the console to see telemetry data such as spans, metrics, and logs being printed as expected:
opentelemetry-instrument flask run
  1. Log into Edge Delta and verify that data is being received. Ensure that the resource attributes, such as service.name, are correctly attached to the telemetry data.
  2. Review logs and traces for any errors or anomalies that could indicate issues with instrumentation or application performance. Use the telemetry data to assess application latency and resource usage.