Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Distributed tracing provides end-to-end visibility into orchestration execution. When you enable OpenTelemetry with Durable Task Scheduler, each orchestration, activity, and sub-orchestration produces linked spans that show timing, ordering, and errors across the entire workflow. You can export these traces to any OpenTelemetry-compatible backend, like Azure Monitor Application Insights, Jaeger, or Zipkin.
Durable Functions and the standalone Durable Task SDKs both support OpenTelemetry distributed tracing when using Durable Task Scheduler as the backend.
How it works
The Durable Task SDKs automatically instrument orchestrations and activities with OpenTelemetry spans. The SDK creates a parent span for each orchestration and child spans for each activity call, sub-orchestration, and timer. Trace context propagates automatically across all these operations, so you get a single, correlated trace for the entire workflow.
The resulting trace tree looks like:
create_orchestration (client)
└─ orchestration (server)
├─ activity:Step1
├─ activity:Step2
└─ activity:Step3
You don't need to add custom instrumentation to your orchestrator or activity code. Register the Microsoft.DurableTask activity source with your OpenTelemetry configuration, and the SDK handles the rest.
Prerequisites
- An Azure Functions project with the Durable Functions extension version 2.13.0 or later.
- Durable Task Scheduler configured as the storage back end for your function app.
- An OpenTelemetry-compatible back end for viewing traces (Application Insights, Jaeger, or another OTLP collector).
- .NET 8 SDK or later.
- The
Microsoft.DurableTask.Worker.AzureManagedandMicrosoft.DurableTask.Client.AzureManagedNuGet packages. - The
OpenTelemetry,OpenTelemetry.Extensions.Hosting, andOpenTelemetry.Exporter.OpenTelemetryProtocolNuGet packages.
- An OpenTelemetry-compatible back end for viewing traces, like Application Insights for production or Jaeger for local development.
Enable distributed tracing
To enable distributed tracing in Durable Functions, update your host.json and configure an OpenTelemetry-compatible telemetry backend.
Update host.json
Add the tracing section under durableTask in your host.json file:
{
"version": "2.0",
"extensions": {
"durableTask": {
"tracing": {
"DistributedTracingEnabled": true,
"Version": "V2"
}
}
}
}
Configure Application Insights
Set the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable in your function app.
For local development, add it to local.settings.json:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
"APPLICATIONINSIGHTS_CONNECTION_STRING": "<your-connection-string>"
}
}
For Azure-hosted apps, add it as an application setting under Configuration in the Azure portal.
Note
If you previously used APPINSIGHTS_INSTRUMENTATIONKEY, switch to APPLICATIONINSIGHTS_CONNECTION_STRING for the latest capabilities.
Reduce telemetry noise
To prevent Application Insights from sampling out trace data, exclude Request from sampling rules in host.json:
{
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
}
}
Register the Microsoft.DurableTask activity source with your OpenTelemetry configuration. The Durable Task SDK automatically creates spans for orchestrations and activities when you register this source.
In your worker's Program.cs, add OpenTelemetry tracing with the Durable Task activity source:
using Microsoft.DurableTask;
using Microsoft.DurableTask.Worker;
using Microsoft.DurableTask.Worker.AzureManaged;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = Host.CreateApplicationBuilder(args);
// Configure OpenTelemetry tracing
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService("durable-worker"))
.WithTracing(tracing =>
{
tracing
.AddSource("Microsoft.DurableTask")
.AddOtlpExporter(opts =>
{
opts.Endpoint = new Uri(
Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
?? "http://localhost:4317");
});
});
// Build connection string from environment variables
string endpoint = Environment.GetEnvironmentVariable("ENDPOINT") ?? "http://localhost:8080";
string taskHub = Environment.GetEnvironmentVariable("TASKHUB") ?? "default";
string connectionString = endpoint.Contains("localhost")
? $"Endpoint={endpoint};TaskHub={taskHub};Authentication=None"
: $"Endpoint={endpoint};TaskHub={taskHub};Authentication=DefaultAzure";
// Configure Durable Task worker
builder.Services.AddDurableTaskWorker()
.AddTasks(tasks =>
{
tasks.AddOrchestratorFunc<string, string>(
"OrderProcessingOrchestration", async (ctx, input) =>
{
var validated = await ctx.CallActivityAsync<string>("ValidateOrder", input);
var payment = await ctx.CallActivityAsync<string>("ProcessPayment", validated);
var shipment = await ctx.CallActivityAsync<string>("ShipOrder", payment);
var result = await ctx.CallActivityAsync<string>("SendNotification", shipment);
return result;
});
tasks.AddActivityFunc<string, string>("ValidateOrder", (ctx, input) =>
Task.FromResult($"Validated({input})"));
tasks.AddActivityFunc<string, string>("ProcessPayment", (ctx, input) =>
Task.FromResult($"Paid({input})"));
tasks.AddActivityFunc<string, string>("ShipOrder", (ctx, input) =>
Task.FromResult($"Shipped({input})"));
tasks.AddActivityFunc<string, string>("SendNotification", (ctx, input) =>
Task.FromResult($"Notified({input})"));
})
.UseDurableTaskScheduler(connectionString);
var host = builder.Build();
await host.RunAsync();
The key line is .AddSource("Microsoft.DurableTask"), which tells OpenTelemetry to capture spans that the Durable Task SDK emits.
Configure the OTLP endpoint
The code snippets above reference the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to set the destination for trace data. Set this variable based on your backend:
| Backend | Endpoint value | Protocol |
|---|---|---|
| Jaeger (local) | http://localhost:4317 |
gRPC |
| Jaeger (local, HTTP) | http://localhost:4318 |
HTTP/protobuf |
| OpenTelemetry Collector | http://<collector-host>:4317 |
gRPC |
| Azure Monitor (via OTLP) | Use the Azure Monitor exporter instead | N/A |
For local development with Jaeger, the default http://localhost:4317 works when Jaeger is running with OTLP gRPC enabled (port 4317). The JavaScript SDK uses HTTP/protobuf by default, so it targets port 4318 with the /v1/traces path.
View traces locally with Jaeger UI
For local development, use the Durable Task Scheduler emulator with Jaeger to view traces. Use a docker-compose.yml to start both services:
services:
dts-emulator:
image: mcr.microsoft.com/dts/dts-emulator:latest
ports:
- "8080:8080" # gRPC
- "8082:8082" # Dashboard
jaeger:
image: jaegertracing/jaeger:latest
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
Start the infrastructure:
docker compose up -d
After you run your application, open the Jaeger UI at http://localhost:16686 and search for your service name (for example, durable-worker) to view traces.
For local development with Durable Functions, distributed tracing data is sent to Application Insights by default. To view traces locally without deploying, you can add an OTLP exporter alongside Application Insights in your function app's Program.cs:
builder.Services.AddOpenTelemetry()
.WithTracing(tracing =>
{
tracing
.AddSource("Microsoft.DurableTask")
.AddOtlpExporter(opts =>
{
opts.Endpoint = new Uri("http://localhost:4317");
});
});
Then run Jaeger locally with docker run -d -p 16686:16686 -p 4317:4317 jaegertracing/jaeger:latest and open the Jaeger UI at http://localhost:16686.
View traces in Application Insights
For production workloads, Application Insights is the recommended telemetry backend.
Once DistributedTracingEnabled is set to true with Version set to V2 in host.json, your Durable Functions app emits correlated spans to Application Insights. To view the full orchestration trace in the Azure portal:
- Go to your Application Insights resource in the Azure portal.
- Open Transaction search and search for your orchestration by name or instance ID.
- Select a trace to view the end-to-end transaction with all correlated spans.
The trace shows the orchestration as a parent span with child spans for each activity call, suborchestration, and timer wait. The following patterns produce distinct trace shapes:
| Pattern | Trace shape |
|---|---|
| Function chaining | Sequential activity spans nested under the orchestrator span. |
| Fan-out/fan-in | Parallel activity spans overlapping in time. |
| Human interaction | An orchestrator span with a long wait for an external event. |
| Monitor | Repeated activity spans with timer waits between iterations. |
Configure the OTLP exporter to send traces to Application Insights by using the Azure Monitor OpenTelemetry exporter, or export through OTLP to an OpenTelemetry Collector that forwards to Application Insights.
Install the Azure.Monitor.OpenTelemetry.Exporter NuGet package and replace the OTLP exporter with:
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService("durable-worker"))
.WithTracing(tracing =>
{
tracing
.AddSource("Microsoft.DurableTask")
.AddAzureMonitorTraceExporter(opts =>
{
opts.ConnectionString = Environment.GetEnvironmentVariable(
"APPLICATIONINSIGHTS_CONNECTION_STRING");
});
});
What trace data shows
The trace data produced by the Durable Task SDKs includes:
| Span type | Description |
|---|---|
create_orchestration |
The client-side span emitted when scheduling a new orchestration |
orchestration |
The server-side orchestration span covering the full execution lifecycle |
activity:<name> |
A span for each activity invocation, showing timing and result |
sub_orchestration:<name> |
A span for each sub-orchestration call |
timer |
A span for durable timer waits |
Each span includes attributes like durabletask.type, durabletask.task.name, durabletask.task.instance_id, and durabletask.task.task_id. Failed activities and orchestrations include error details in the span's status and events.
Troubleshooting
| Issue | Resolution |
|---|---|
| No traces appear | Check that the Microsoft.DurableTask activity source is registered and the exporter endpoint is reachable. |
| Traces are incomplete | Check that the OpenTelemetry SDK is initialized before the Durable Task SDK (especially in JavaScript/TypeScript). |
| Missing spans in Application Insights | Disable or adjust sampling settings to prevent trace data from being dropped. |
| Traces don't correlate | Check that you're using Durable Task Scheduler as the backend. Trace context propagation requires the scheduler. |