Edit

Enable OpenTelemetry distributed tracing with Durable Task Scheduler

Distributed tracing provides end-to-end visibility into orchestration execution. When you enable OpenTelemetry with Durable Task Scheduler, each orchestration, activity, and sub-orchestration produces linked spans that show timing, ordering, and errors across the entire workflow. You can export these traces to any OpenTelemetry-compatible backend, like Azure Monitor Application Insights, Jaeger, or Zipkin.

Durable Functions and the standalone Durable Task SDKs both support OpenTelemetry distributed tracing when using Durable Task Scheduler as the backend.

How it works

The Durable Task SDKs automatically instrument orchestrations and activities with OpenTelemetry spans. The SDK creates a parent span for each orchestration and child spans for each activity call, sub-orchestration, and timer. Trace context propagates automatically across all these operations, so you get a single, correlated trace for the entire workflow.

The resulting trace tree looks like:

create_orchestration (client)
  └─ orchestration (server)
       ├─ activity:Step1
       ├─ activity:Step2
       └─ activity:Step3

You don't need to add custom instrumentation to your orchestrator or activity code. Register the Microsoft.DurableTask activity source with your OpenTelemetry configuration, and the SDK handles the rest.

Prerequisites

  • An Azure Functions project with the Durable Functions extension version 2.13.0 or later.
  • Durable Task Scheduler configured as the storage back end for your function app.
  • An OpenTelemetry-compatible back end for viewing traces (Application Insights, Jaeger, or another OTLP collector).
  • .NET 8 SDK or later.
  • The Microsoft.DurableTask.Worker.AzureManaged and Microsoft.DurableTask.Client.AzureManaged NuGet packages.
  • The OpenTelemetry, OpenTelemetry.Extensions.Hosting, and OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet packages.
  • An OpenTelemetry-compatible back end for viewing traces, like Application Insights for production or Jaeger for local development.

Enable distributed tracing

To enable distributed tracing in Durable Functions, update your host.json and configure an OpenTelemetry-compatible telemetry backend.

Update host.json

Add the tracing section under durableTask in your host.json file:

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "tracing": {
        "DistributedTracingEnabled": true,
        "Version": "V2"
      }
    }
  }
}

Configure Application Insights

Set the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable in your function app.

For local development, add it to local.settings.json:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "APPLICATIONINSIGHTS_CONNECTION_STRING": "<your-connection-string>"
  }
}

For Azure-hosted apps, add it as an application setting under Configuration in the Azure portal.

Note

If you previously used APPINSIGHTS_INSTRUMENTATIONKEY, switch to APPLICATIONINSIGHTS_CONNECTION_STRING for the latest capabilities.

Reduce telemetry noise

To prevent Application Insights from sampling out trace data, exclude Request from sampling rules in host.json:

{
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  }
}

Register the Microsoft.DurableTask activity source with your OpenTelemetry configuration. The Durable Task SDK automatically creates spans for orchestrations and activities when you register this source.

In your worker's Program.cs, add OpenTelemetry tracing with the Durable Task activity source:

using Microsoft.DurableTask;
using Microsoft.DurableTask.Worker;
using Microsoft.DurableTask.Worker.AzureManaged;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = Host.CreateApplicationBuilder(args);

// Configure OpenTelemetry tracing
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("durable-worker"))
    .WithTracing(tracing =>
    {
        tracing
            .AddSource("Microsoft.DurableTask")
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri(
                    Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
                    ?? "http://localhost:4317");
            });
    });

// Build connection string from environment variables
string endpoint = Environment.GetEnvironmentVariable("ENDPOINT") ?? "http://localhost:8080";
string taskHub = Environment.GetEnvironmentVariable("TASKHUB") ?? "default";
string connectionString = endpoint.Contains("localhost")
    ? $"Endpoint={endpoint};TaskHub={taskHub};Authentication=None"
    : $"Endpoint={endpoint};TaskHub={taskHub};Authentication=DefaultAzure";

// Configure Durable Task worker
builder.Services.AddDurableTaskWorker()
    .AddTasks(tasks =>
    {
        tasks.AddOrchestratorFunc<string, string>(
            "OrderProcessingOrchestration", async (ctx, input) =>
        {
            var validated = await ctx.CallActivityAsync<string>("ValidateOrder", input);
            var payment = await ctx.CallActivityAsync<string>("ProcessPayment", validated);
            var shipment = await ctx.CallActivityAsync<string>("ShipOrder", payment);
            var result = await ctx.CallActivityAsync<string>("SendNotification", shipment);
            return result;
        });

        tasks.AddActivityFunc<string, string>("ValidateOrder", (ctx, input) =>
            Task.FromResult($"Validated({input})"));
        tasks.AddActivityFunc<string, string>("ProcessPayment", (ctx, input) =>
            Task.FromResult($"Paid({input})"));
        tasks.AddActivityFunc<string, string>("ShipOrder", (ctx, input) =>
            Task.FromResult($"Shipped({input})"));
        tasks.AddActivityFunc<string, string>("SendNotification", (ctx, input) =>
            Task.FromResult($"Notified({input})"));
    })
    .UseDurableTaskScheduler(connectionString);

var host = builder.Build();
await host.RunAsync();

The key line is .AddSource("Microsoft.DurableTask"), which tells OpenTelemetry to capture spans that the Durable Task SDK emits.

Configure the OTLP endpoint

The code snippets above reference the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to set the destination for trace data. Set this variable based on your backend:

Backend Endpoint value Protocol
Jaeger (local) http://localhost:4317 gRPC
Jaeger (local, HTTP) http://localhost:4318 HTTP/protobuf
OpenTelemetry Collector http://<collector-host>:4317 gRPC
Azure Monitor (via OTLP) Use the Azure Monitor exporter instead N/A

For local development with Jaeger, the default http://localhost:4317 works when Jaeger is running with OTLP gRPC enabled (port 4317). The JavaScript SDK uses HTTP/protobuf by default, so it targets port 4318 with the /v1/traces path.

View traces locally with Jaeger UI

For local development, use the Durable Task Scheduler emulator with Jaeger to view traces. Use a docker-compose.yml to start both services:

services:
  dts-emulator:
    image: mcr.microsoft.com/dts/dts-emulator:latest
    ports:
      - "8080:8080"  # gRPC
      - "8082:8082"  # Dashboard
  jaeger:
    image: jaegertracing/jaeger:latest
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP

Start the infrastructure:

docker compose up -d

After you run your application, open the Jaeger UI at http://localhost:16686 and search for your service name (for example, durable-worker) to view traces.

For local development with Durable Functions, distributed tracing data is sent to Application Insights by default. To view traces locally without deploying, you can add an OTLP exporter alongside Application Insights in your function app's Program.cs:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource("Microsoft.DurableTask")
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("http://localhost:4317");
            });
    });

Then run Jaeger locally with docker run -d -p 16686:16686 -p 4317:4317 jaegertracing/jaeger:latest and open the Jaeger UI at http://localhost:16686.

View traces in Application Insights

For production workloads, Application Insights is the recommended telemetry backend.

Once DistributedTracingEnabled is set to true with Version set to V2 in host.json, your Durable Functions app emits correlated spans to Application Insights. To view the full orchestration trace in the Azure portal:

  1. Go to your Application Insights resource in the Azure portal.
  2. Open Transaction search and search for your orchestration by name or instance ID.
  3. Select a trace to view the end-to-end transaction with all correlated spans.

The trace shows the orchestration as a parent span with child spans for each activity call, suborchestration, and timer wait. The following patterns produce distinct trace shapes:

Pattern Trace shape
Function chaining Sequential activity spans nested under the orchestrator span.
Fan-out/fan-in Parallel activity spans overlapping in time.
Human interaction An orchestrator span with a long wait for an external event.
Monitor Repeated activity spans with timer waits between iterations.

Configure the OTLP exporter to send traces to Application Insights by using the Azure Monitor OpenTelemetry exporter, or export through OTLP to an OpenTelemetry Collector that forwards to Application Insights.

Install the Azure.Monitor.OpenTelemetry.Exporter NuGet package and replace the OTLP exporter with:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("durable-worker"))
    .WithTracing(tracing =>
    {
        tracing
            .AddSource("Microsoft.DurableTask")
            .AddAzureMonitorTraceExporter(opts =>
            {
                opts.ConnectionString = Environment.GetEnvironmentVariable(
                    "APPLICATIONINSIGHTS_CONNECTION_STRING");
            });
    });

What trace data shows

The trace data produced by the Durable Task SDKs includes:

Span type Description
create_orchestration The client-side span emitted when scheduling a new orchestration
orchestration The server-side orchestration span covering the full execution lifecycle
activity:<name> A span for each activity invocation, showing timing and result
sub_orchestration:<name> A span for each sub-orchestration call
timer A span for durable timer waits

Each span includes attributes like durabletask.type, durabletask.task.name, durabletask.task.instance_id, and durabletask.task.task_id. Failed activities and orchestrations include error details in the span's status and events.

Troubleshooting

Issue Resolution
No traces appear Check that the Microsoft.DurableTask activity source is registered and the exporter endpoint is reachable.
Traces are incomplete Check that the OpenTelemetry SDK is initialized before the Durable Task SDK (especially in JavaScript/TypeScript).
Missing spans in Application Insights Disable or adjust sampling settings to prevent trace data from being dropped.
Traces don't correlate Check that you're using Durable Task Scheduler as the backend. Trace context propagation requires the scheduler.

Sample code