# OpenTelemetry v2 integration

> For the complete documentation index, see [llms.txt](https://docs.temporal.io/llms.txt).
> Any documentation page is available as raw Markdown by appending `.md` to its URL.

> Configure custom tracing or automatic tracing and metrics for a Temporal Application with the Go SDK OpenTelemetry v2 plugin.

The OpenTelemetry v2 integration connects the Temporal Go SDK to OpenTelemetry through the
[Plugin API](/develop/plugins-guide). Use it to propagate application trace context across Temporal calls. You can also
opt in to Temporal SDK operation spans and SDK metrics.

> **Pre-release**

This guide demonstrates two approaches:

- **Automatic instrumentation:** The Worker plugin creates spans for Temporal SDK operations and reports Temporal SDK
  metrics.
- **Custom instrumentation:** The plugin propagates trace context, but you create the spans that describe your
  application.

This guide assumes that you understand OpenTelemetry fundamentals. For OpenTelemetry concepts and exporter options,
refer to the [OpenTelemetry documentation](https://opentelemetry.io/docs/).

Code snippets in this guide are taken from the
[OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2). Refer to the sample for
the complete, runnable code.

## Prerequisites

- Set up your local development environment by following
  [Set up your local development environment](/develop/go/set-up-your-local-go).
- Leave the Temporal development server running to test the sample locally.
- Install Docker to run Jaeger for the local tracing examples.

## Install the plugin

Install the OpenTelemetry v2 plugin:

```bash
go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest
```

The samples also use the OpenTelemetry OTLP gRPC and Prometheus exporters:

```bash
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@latest
go get go.opentelemetry.io/otel/exporters/prometheus@latest
go get github.com/prometheus/client_golang@latest
```

## Configure tracing

Create and install a replay-safe tracer provider in every process that creates the plugin or calls
`temporalotel.Tracer`. The replay-safe provider gives Workflow spans consistent identities when Workflow code replays.

<!--SNIPSTART samples-go-opentelemetry-v2-tracer-provider-->
[opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go)
```go
func InitializeGlobalTracerProvider(
	ctx context.Context,
	serviceName string,
) (*temporalotel.ReplaySafeTracerProvider, error) {
	exporter, err := otlptracegrpc.New(
		ctx,
		otlptracegrpc.WithEndpoint("127.0.0.1:4317"),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)
	return provider, nil
}
```
<!--SNIPEND-->

The samples use distinct OpenTelemetry service names for each instrumented process. This makes process boundaries clear
when one trace contains spans from both a Client and a Worker.

The OTLP exporter uses an insecure loopback connection for this local example. Configure transport security for your
production telemetry backend.

## Enable automatic instrumentation

Set `AddTemporalSpans` to `true` to have the plugin create spans for Temporal SDK operations performed by the Worker.
Provide `MetricsHandlerOptions` to install the plugin's Temporal SDK metrics handler. A non-nil metrics configuration
enables the complete SDK metric set.

<!--SNIPSTART samples-go-opentelemetry-v2-metrics-plugin-->
[opentelemetry-v2/automatic-instrumentation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/automatic-instrumentation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}
```
<!--SNIPEND-->

`UseMonotonicCounters` represents Temporal SDK counters as OpenTelemetry `Int64Counter` instruments instead of
`Int64UpDownCounter` instruments.

### Export Temporal SDK metrics

Create and install a meter provider backed by the OpenTelemetry Prometheus exporter:

<!--SNIPSTART samples-go-opentelemetry-v2-meter-provider-->
[opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go)
```go
func InitializeGlobalMeterProvider(serviceName string) (*sdkmetric.MeterProvider, error) {
	exporter, err := otelprometheus.New()
	if err != nil {
		return nil, err
	}

	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(exporter),
		sdkmetric.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetMeterProvider(provider)
	return provider, nil
}
```
<!--SNIPEND-->

Start a loopback-only HTTP endpoint that serves the Prometheus handler:

<!--SNIPSTART samples-go-opentelemetry-v2-metrics-server-->
[opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go)
```go
func StartPrometheusEndpoint() (*http.Server, error) {
	mux := http.NewServeMux()
	mux.Handle("/metrics", promhttp.Handler())

	server := &http.Server{
		Addr:              "127.0.0.1:9090",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}
	listener, err := net.Listen("tcp", server.Addr)
	if err != nil {
		return nil, err
	}

	log.Println("Prometheus metrics available at http://127.0.0.1:9090/metrics")
	go func() {
		if err := server.Serve(listener); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Println("Prometheus endpoint failed:", err)
		}
	}()
	return server, nil
}
```
<!--SNIPEND-->

Opening the listener before starting the serving goroutine makes an address conflict fail during Worker startup. The
endpoint at [http://127.0.0.1:9090/metrics](http://127.0.0.1:9090/metrics) returns metrics in Prometheus format for an
external Prometheus server to scrape.

### Run the automatic instrumentation sample

From the root of a `samples-go` checkout, start Jaeger and the Worker:

```bash
docker compose -f opentelemetry-v2/docker-compose.yaml up -d
go run opentelemetry-v2/automatic-instrumentation/worker/main.go
```

In another terminal, start the Workflow:

```bash
temporal workflow execute \
  --task-queue automatic-instrumentation \
  --type Workflow \
  --input '"Temporal"'
```

The trace begins with the Worker's automatic spans, since only the Worker installs the plugin in this sample. These
spans describe Temporal SDK operations that the Worker performs, such as running the Workflow and its Activity.

Open the [Jaeger UI](http://127.0.0.1:16686), select `temporal-otel-v2-automatic-worker`, and inspect the Worker trace. Open
[http://127.0.0.1:9090/metrics](http://127.0.0.1:9090/metrics) to inspect the Temporal SDK metrics.

## Add custom instrumentation

By default, the plugin propagates context: spans your application creates remain connected across Clients, Workflows,
Activities, and other Temporal calls. The custom samples focus on that span propagation.

### Propagate from a Workflow to an Activity

Use `temporalotel.Tracer` to create replay-safe spans in Workflow code. Use an ordinary OpenTelemetry tracer in Activity
code. Passing the span-derived Workflow context to `workflow.ExecuteActivity` connects the Activity span to the Workflow
span.

<!--SNIPSTART samples-go-opentelemetry-v2-application-spans-->
[opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go)
```go
const instrumentationName = "github.com/temporalio/samples-go/opentelemetry-v2/workflow-activity-propagation"

func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}

func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}
```
<!--SNIPEND-->

The Worker creates a default plugin so it can propagate the custom Workflow span to the Activity:

<!--SNIPSTART samples-go-opentelemetry-v2-plugin-client-->
[opentelemetry-v2/workflow-activity-propagation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()
```
<!--SNIPEND-->

From the root of a `samples-go` checkout, start Jaeger and the Worker:

```bash
docker compose -f opentelemetry-v2/docker-compose.yaml up -d
go run opentelemetry-v2/workflow-activity-propagation/worker/main.go
```

In another terminal, start the Workflow:

```bash
temporal workflow execute \
  --task-queue workflow-activity-propagation \
  --type Workflow \
  --input '"Temporal"'
```

Open the [Jaeger UI](http://127.0.0.1:16686), select
`temporal-otel-v2-custom-workflow-activity-propagation-worker`, and find the trace
containing the `workflow-operation` and `activity-operation` spans created by the sample.

### Propagate from a Client to an Update

Install the plugin in both processes when a custom client span must cross a Temporal call. The client-side plugin injects
the span context into the Update headers. The Worker plugin extracts it into the Update handler's Workflow context.

Create the replay-safe tracer provider before the plugin in both processes, and add the default plugin when you create
each Temporal Client. Keep the `send-update` span open until the Update reaches the completed stage and returns its
result:

<!--SNIPSTART samples-go-opentelemetry-v2-client-update-->
[opentelemetry-v2/client-update-propagation/starter/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/client-update-propagation/starter/main.go)
```go
func sendUpdate(
	ctx context.Context,
	c client.Client,
	workflowID string,
	name string,
) (string, error) {
	ctx, span := otel.Tracer(instrumentationName).Start(ctx, "send-update")
	defer span.End()

	handle, err := c.UpdateWorkflow(ctx, client.UpdateWorkflowOptions{
		WorkflowID:   workflowID,
		UpdateName:   clientupdate.UpdateName,
		WaitForStage: client.WorkflowUpdateStageCompleted,
		Args:         []interface{}{name},
	})
	if err != nil {
		return "", fmt.Errorf("unable to send Workflow Update: %w", err)
	}

	var result string
	if err := handle.Get(ctx, &result); err != nil {
		return "", fmt.Errorf("unable to get Workflow Update result: %w", err)
	}
	return result, nil
}
```
<!--SNIPEND-->

The Update validator creates a replay-safe `validate-update` span, and the handler creates a `handle-update` span,
changes the Workflow's completion state, and returns the result:

<!--SNIPSTART samples-go-opentelemetry-v2-update-handler-span-->
[opentelemetry-v2/client-update-propagation/workflow.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/client-update-propagation/workflow.go)
```go
func Workflow(ctx workflow.Context) (string, error) {
	var result string
	updateCompleted := false

	err := workflow.SetUpdateHandlerWithOptions(
		ctx,
		UpdateName,
		func(ctx workflow.Context, name string) (string, error) {
			_, span := temporalotel.Tracer(instrumentationName).Start(ctx, "handle-update")
			defer span.End()

			result = fmt.Sprintf("Hello, %s!", name)
			updateCompleted = true
			return result, nil
		},
		workflow.UpdateHandlerOptions{
			Validator: func(ctx workflow.Context, name string) error {
				_, span := temporalotel.Tracer(instrumentationName).Start(ctx, "validate-update")
				defer span.End()

				if name == "" {
					return fmt.Errorf("name cannot be empty")
				}
				return nil
			},
		},
	)
	if err != nil {
		return "", fmt.Errorf("unable to register Update handler: %w", err)
	}

	if err := workflow.Await(ctx, func() bool { return updateCompleted && workflow.AllHandlersFinished(ctx) }); err != nil {
		return "", err
	}
	return result, nil
}
```
<!--SNIPEND-->

The Workflow waits for the Update to set its completion state and for `workflow.AllHandlersFinished` to report all
Update handlers done before returning.

From the root of a `samples-go` checkout, start the Worker:

```bash
go run opentelemetry-v2/client-update-propagation/worker/main.go
```

In another terminal, run the starter:

```bash
go run opentelemetry-v2/client-update-propagation/starter/main.go
```

In Jaeger, select `temporal-otel-v2-custom-client-update-propagation-client` and open the trace containing `send-update`,
`validate-update`, and `handle-update`. The trace crosses into `temporal-otel-v2-custom-client-update-propagation-worker`,
connected by the context the default plugins propagate.

## Shut down telemetry providers

Install the tracer and meter providers as OpenTelemetry globals before constructing the plugin or a Workflow tracer, so
they're available to instrument every Temporal call the process makes.

When the process exits, close the Temporal Client or stop the Worker first, then shut down the Prometheus-format
endpoint, meter provider, and tracer provider, each with its own fresh, bounded context rather than one that might
already be canceled. Shutting down the tracer provider flushes the batch span processor, so spans created earlier in
the process still reach the exporter before it exits.

## Propagate trace context and baggage

The plugin automatically propagates OpenTelemetry trace context and baggage across Temporal calls, using a composite
W3C Trace Context and Baggage propagator by default instead of the global OpenTelemetry propagator. Use
`PluginOptions.TextMapPropagator` to provide a different propagator.

Temporal headers that carry baggage can be persisted in Workflow Event Histories. Do not put credentials, tokens, or
other sensitive data in OpenTelemetry baggage. Set `PluginOptions.DisableBaggage` to `true` to turn off baggage
propagation.

For other application context, see [Context Propagation](/develop/go/best-practices/context-propagation).

## Resources

- [Automatic instrumentation sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2/automatic-instrumentation)
  — spans the Worker creates for Temporal SDK operations, plus Temporal SDK metrics.
- [Workflow-to-Activity propagation sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2/workflow-activity-propagation)
  — custom spans propagated from a Workflow to an Activity.
- [Client-to-Update propagation sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2/client-update-propagation)
  — custom spans propagated from a Client into a Workflow Update.
- [OpenTelemetry v2 Go package](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry-v2) — the full plugin
  reference.
- [Go SDK observability guide](/develop/go/platform/observability) — where tracing and metrics fit among the SDK's
  other observability tools.