Rediger

Set autopurge retention policies for Durable Task Scheduler

To prevent reaching the memory limit of a capacity unit (CU), we recommend periodically purging orchestration history data. The Durable Task Scheduler offers a lightweight, configurable autopurge feature that helps you manage orchestration data cleanup without manual intervention.

Setting Default value
Autopurge Enabled
Default retention period 30 days
Maximum retention period 90 days
Applies to All task hubs in the scheduler
Eligible statuses Completed, Failed, Canceled, Terminated

Autopurge operates asynchronously in the background, optimized to minimize system resource usage and prevent interference with other Durable Task operations. Although autopurge doesn't adhere to a strict schedule, its cleanup rate generally aligns with your orchestration scheduling rate.

How it works

Autopurge is enabled by default with a 30 day policy, but it can be customized. You can modify it by defining retention policies that specify how long to keep orchestration data for certain statuses. The autopurge feature removes orchestration data that is in terminal statuses. A terminal status means the orchestration has reached a final state and will no longer schedule tasks, process events, or generate work items. Terminal statuses include:

  • Completed
  • Failed
  • Canceled
  • Terminated

The orchestration instances eligible for autopurge match those targeted by the Durable SDK PurgeInstancesAsync API.

Autopurge ignores orchestration data associated with non-terminal statuses. Non-terminal statuses indicate that the orchestration instance is either actively executing, paused, or in a state where it may resume in the future (waiting for external events or timers). Non-terminal statuses include:

  • Pending
  • Running
  • Suspended
  • Continued_As_New

Note

Orchestrations using ContinueAsNew aren't considered terminal. ContinueAsNew restarts the orchestration with a new execution history while preserving the instance ID, so these instances aren't purged until they reach a true terminal state.

Once enabled, autopurge periodically deletes orchestration data older than the retention period you set. Autopurge only removes data for orchestrations in terminal statuses.

Note

Retention policies you define are applied to all task hubs in a scheduler.

Configure retention policies

Policy value range

Retention values are specified in days and can range from 0 (purge as soon as possible) to 90 days. By default, autopurge retention is set to 30 days.

The retention period starts when the orchestration enters a terminal state, not when it was created. For example, if you set a retention value of 1 day and an orchestration takes 10 days to finish, autopurge deletes it 1 day after it finishes.

Default and specific policy types

When configuring an autopurge retention policy, you can set either a specific or a default policy.

  • Default policy purges orchestration data regardless of orchestrationState. The following policy purges orchestration data of all statuses covered by the feature after 2 days:

    {
  "retentionPeriodInDays": 2
}

  • Specific policy defines purging of orchestration data for specific orchestrationState. The following policy tells Durable Task Scheduler to keep completed orchestration data for 1 day, after which this data is purged.

    {
  "retentionPeriodInDays": 1,
  "orchestrationState": "Completed"
}

Add specific policies to override the default policy applied to orchestrations. In the following example, the second and third policies override the default policy ("retentionPeriodInDays": 1).

  • Data associated with completed orchestrations is deleted as soon as possible.

  • Data associated with failed orchestrations is purged after 60 days.

    [
  {
    "retentionPeriodInDays": 1
  },
  {
    "retentionPeriodInDays": 0,
    "orchestrationState": "Completed"
  },
  {
    "retentionPeriodInDays": 60,
    "orchestrationState": "Failed"
  }
]

Since no specific policy is set for canceled or terminated orchestrations, the default policy still applies to them, purging their data after 1 day.

For more information, see the API reference spec for Durable Task Scheduler retention policies.

Enable autopurge

You can define retention policies using:

  • Durable Task CLI
  • Azure Resource Manager (ARM)
  • Bicep

Make sure you have the latest version of the Durable Task CLI extension.

az extension add --name durabletask
az extension update --name durabletask

Create or update the retention policy by running the following command.

az durabletask retention-policy create --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP --default-days 1 --completed-days 0 --failed-days 60

The following properties specify the retention duration for orchestration data of different statuses.

Property Description
--canceled-days or -x The number of days to retain canceled orchestrations.
--completed-days or -c The number of days to retain completed orchestrations.
--default-days or -d The number of days to retain orchestrations.
--failed-days or -f The number of days to retain failed orchestrations.
--terminated-days or -t The number of days to retain terminated orchestrations.

Example response

If creation is successful, you receive the following response.

{
  "id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAME/retentionPolicies/default",
  "name": "default",
  "properties": {
    "provisioningState": "Succeeded",
    "retentionPolicies": [
      {
        "retentionPeriodInDays": 1
      },
      {
        "orchestrationState": "Completed",
        "retentionPeriodInDays": 0
      },
      {
        "orchestrationState": "Failed",
        "retentionPeriodInDays": 60
      }
    ]
  },
  "resourceGroup": "RESOURCE_GROUP",
  "systemData": {
    "createdAt": "2025-04-23T23:41:17.3165122Z",
    "createdBy": "someone@microsoft.com",
    "createdByType": "User",
    "lastModifiedAt": "2025-04-23T23:41:17.3165122Z",
    "lastModifiedBy": "someone@microsoft.com",
    "lastModifiedByType": "User"
  },
  "type": "microsoft.durabletask/schedulers/retentionpolicies"
}

Tip

Learn more about the retention policy command via the CLI reference.

View current policy

Retrieve the current retention policy for a scheduler:

az durabletask retention-policy show --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP

Disable autopurge

Delete the retention policies using the following command. The Durable Task Scheduler stops cleaning orchestration data within 5 to 10 minutes.

az durabletask retention-policy delete --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP

Related content

  • Last updated on