Home › Observability (OpenTelemetry) › OpenTelemetry instrumentation AGENTS.md
OpenTelemetry instrumentation AGENTS.md
AGENTS.md for adding OpenTelemetry traces, metrics and logs: OTLP to a Collector, env-var config, Node, Python and Go setup, semantic conventions, and no PII.
AGENTS.mdObservability (OpenTelemetry)
How to use
Save this as AGENTS.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.
Note: Review and test before use. Provided as is, without warranty. See Terms.
AGENTS.mdDownload # AGENTS.md: OpenTelemetry instrumentation
## How telemetry flows
App → OTLP → OpenTelemetry Collector → backend(s). Application code only speaks OTLP. Vendor exporters, API keys, sampling at the edge and routing belong in the Collector config (`otel-collector.yaml`), never in app code.
## Configure with environment variables
- `OTEL_SERVICE_NAME`: required, stable, lowercase-kebab (`checkout-api`).
- `OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=staging`
- `OTEL_EXPORTER_OTLP_ENDPOINT`: `http://localhost:4318` for HTTP/protobuf or `http://localhost:4317` for gRPC; set `OTEL_EXPORTER_OTLP_PROTOCOL` to match.
- Sampling: `OTEL_TRACES_SAMPLER=parentbased_traceidratio` with `OTEL_TRACES_SAMPLER_ARG=0.1` (or what the team agreed). Always keep the parent-based sampler so traces aren't cut in half.
- Auth headers come from secrets via `OTEL_EXPORTER_OTLP_HEADERS`. Never hard-code endpoints, tokens or headers.
## Bootstrapping per language
- **Node.js**: install `@opentelemetry/api` and `@opentelemetry/auto-instrumentations-node`, and load them before any app code: `node --require @opentelemetry/auto-instrumentations-node/register app.js` (or via `NODE_OPTIONS`). For a custom setup, keep `NodeSDK` in one `instrumentation` file that is loaded first; never import it from inside the app.
- **Next.js**: an `instrumentation.ts` at the project root that calls `registerOTel({ serviceName })` from `@vercel/otel`.
- **Python**: `pip install opentelemetry-distro opentelemetry-exporter-otlp`, then `opentelemetry-bootstrap -a install`, then run under `opentelemetry-instrument` (e.g. `opentelemetry-instrument uvicorn app.main:app`).
- **Go**: build the `TracerProvider` (OTLP exporter + resource) in `main`, `defer tp.Shutdown(ctx)`, wrap servers with `otelhttp.NewHandler` and clients with `otelhttp.NewTransport`, and pass `ctx` everywhere.
## Spans
- Let auto-instrumentation cover HTTP, gRPC, databases and queues. Add manual spans only around business operations (`orders.place`, `invoice.render`), not every function.
- Get a tracer once per module, named after the module (`trace.getTracer('billing')`).
- Span names must be low-cardinality: `GET /users/{id}`, `orders.charge`. Put ids in attributes, never in names.
- Always end spans (`startActiveSpan` callback, `try/finally`, `defer span.End()`). On failure, record the exception, set status to ERROR, and set `error.type`.
- Use semantic-convention attribute names where one exists (`http.request.method`, `http.route`, `http.response.status_code`, `server.address`, `db.system.name`, `db.operation.name`). Prefix custom attributes with the company or app (`acme.order.tier`).
- Propagate context with W3C `traceparent` (the default). For queues, inject context into message headers on publish and extract it in the consumer, so workers continue the trace instead of starting new roots.
## Metrics and logs
- Create metric instruments once at startup. Durations are histograms in seconds (unit `s`); counts use counters with a unit like `{order}`. Metric attributes stay low-cardinality: no user ids, emails or raw URLs.
- Keep the existing logger and add the OpenTelemetry log bridge or appender so log records carry `trace_id` and `span_id`. Don't build a second logging path.
## Privacy
Never put passwords, tokens, cookies, auth headers, full request or response bodies, emails or other PII into span names, attributes, events, metric attributes or logs. Turn off db statement capture or make sure statements are parameterized.
## Local and tests
- Locally, run a Collector with the `debug` exporter, or Jaeger (`docker run -p 16686:16686 -p 4317:4317 -p 4318:4318 jaegertracing/jaeger`) and open http://localhost:16686.
- Unit tests use the SDK's in-memory span exporter and assert span names, attributes, status and parent/child links, never durations.
## Done when
The service appears under the right name, version and environment; one request produces a single connected trace across services; span names and metric attributes are low-cardinality; and no PII shows up in telemetry.