Reliability in the Azure Health Data Services de-identification service

This article describes reliability support in the Azure Health Data Services de-identification service. For a more detailed overview of reliability principles in Azure, see Azure reliability.

Cross-region disaster recovery

Disaster recovery (DR) refers to practices that organizations use to recover from high-impact events, such as natural disasters or failed deployments that result in downtime and data loss. Regardless of the cause, the best remedy for a disaster is a well-defined and tested DR plan and an application design that actively supports DR. Before you start creating your disaster recovery plan, see Recommendations for designing a disaster recovery strategy.

For DR, Microsoft uses the shared responsibility model. In this model, Microsoft ensures that the baseline infrastructure and platform services are available. However, many Azure services don't automatically replicate data or fall back from a failed region to cross-replicate to another enabled region. For those services, you're responsible for setting up a disaster recovery plan that works for your workload. Most services that run on Azure platform as a service (PaaS) offerings provide features and guidance to support DR. You can use service-specific features to support fast recovery to help develop your DR plan.

Each de-identification service is deployed to a single Azure region. If an entire region isn't available or performance is significantly degraded:

• The control plane functionality of Azure Resource Manager is limited to read-only during the outage. Microsoft always backs up your service metadata (such as resource properties) outside the region. After the outage is over, you can read and write to the control plane.

• All data plane requests, such as de-identification or job API requests, fail during the outage. No customer data is lost, but there's the potential for job progress metadata to be lost. After the outage is over, you can read and write to the data plane.

Disaster recovery tutorial

If an entire Azure region isn't available, you can still assure high availability of your workloads. You can deploy two or more de-identification services in an active-active configuration. An active-active configuration distributes requests across multiple active regions. Use Azure Front Door to route traffic to both regions.

With this example architecture:

• Identical de-identification services are deployed in two separate regions.

• Azure Front Door routes traffic to both regions.

• During a disaster, one region becomes offline, and Azure Front Door routes traffic exclusively to the other region. The recovery time objective is the time Azure Front Door takes to detect that one service is unhealthy.

If you adopt the active-active configuration, you can expect a recovery time objective (RTO) of five minutes. In any configuration, you can expect a recovery point objective (RPO) of zero minutes (no customer data is lost).

Prerequisites

If you don't have an Azure account, create a free account before you begin.

To complete this tutorial:

• Use the Bash environment in Azure Cloud Shell. For more information, see Get started with Azure Cloud Shell.

  

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Authenticate to Azure using Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use and manage extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

Create a resource group

You need two instances of a de-identification service in different Azure regions for this tutorial. The tutorial uses the East US and West US 2 regions, but feel free to choose your own regions.

To make management and clean-up simpler, you use a single resource group for all resources in this tutorial. Consider using separate resource groups for each region or resource to further isolate your resources in a disaster recovery situation.

Run the following command to create your resource group.

az group create --name my-deid --location eastus

Create de-identification services

Follow the steps at Quickstart: Deploy the de-identification service to create two separate services, one in East US and one in West US 2.

Note the service URL of each de-identification service. You need this information when you deploy Azure Front Door in the next step.

Create a deployment

A multiregion deployment can use an active-active or active-passive configuration. An active-active configuration distributes requests across multiple active regions. An active-passive configuration keeps running instances in the secondary region, but doesn't send traffic there unless the primary region fails.

You can enable these configurations in Azure Front Door. For more information on designing apps for high availability and fault tolerance, see Architect Azure applications for resiliency and availability.

Create a profile

You now create a profile in Azure Front Door to route traffic to your services. Run az afd profile create.

az afd profile create --profile-name myfrontdoorprofile --resource-group my-deid --sku Premium_AzureFrontDoor

Add an endpoint

To create an endpoint in your Azure Front Door profile, run az afd endpoint create. This endpoint routes requests to your services. You can create multiple endpoints in your profile after you finish this tutorial.

az afd endpoint create --resource-group my-deid --endpoint-name myendpoint --profile-name myfrontdoorprofile --enabled-state Enabled

Create an origin group

To create an origin group that contains your two de-identification services, run az afd origin-group create.

az afd origin-group create --resource-group my-deid --origin-group-name myorigingroup --profile-name myfrontdoorprofile --probe-request-type GET --probe-protocol Https --probe-interval-in-seconds 60 --probe-path /health --sample-size 1 --successful-samples-required 1 --additional-latency-in-milliseconds 50 --enable-health-probe

Add origins to the origin group

To add an origin to your origin group, run az afd origin create. For the --host-name and --origin-host-header parameters, replace the placeholder value <service-url-east-us> with your East US service URL, leaving out the scheme (https://). You have a value like abcdefghijk.api.eastus.deid.azure.com.

az afd origin create --resource-group my-deid --host-name <service-url-east-us> --profile-name myfrontdoorprofile --origin-group-name myorigingroup --origin-name deid1 --origin-host-header <service-url-east-us> --priority 1 --weight 1000 --enabled-state Enabled --https-port 443

Repeat this step to add your second origin. For the --host-name and --origin-host-header parameters, replace the placeholder value <service-url-west-us-2> with your West US 2 service URL, leaving out the scheme (https://).

az afd origin create --resource-group my-deid --host-name <service-url-west-us-2> --profile-name myfrontdoorprofile --origin-group-name myorigingroup --origin-name deid2 --origin-host-header <service-url-west-us-2> --priority 1 --weight 1000 --enabled-state Enabled --https-port 443

Pay attention to the --priority parameters in both commands. Because both origins are set to priority 1, Azure Front Door treats both origins as active and directs traffic to both regions. If the priority for one origin is set to 2, Azure Front Door treats that origin as secondary. All traffic goes to the other origin unless that origin goes down.

Add a route

To map your endpoint to the origin group, run az afd route create. This route forwards requests from the endpoint to your origin group.

az afd route create --resource-group my-deid --profile-name myfrontdoorprofile --endpoint-name myendpoint --forwarding-protocol MatchRequest --route-name route  --origin-group myorigingroup --supported-protocols Https --link-to-default-domain Enabled 

Allow about 15 minutes for this step to finish. It takes some time for this change to propagate globally. After this period, your Azure Front Door profile is fully functional.

Test the profile

When you create the Azure Front Door profile, it takes a few minutes for the configuration to be deployed globally. After this period, you can access the host that you created.

To get the host name of the Azure Front Door endpoint, run az afd endpoint show. It looks like abddefg.azurefd.net.

az afd endpoint show --resource-group my-deid --profile-name myfrontdoorprofile --endpoint-name myendpoint --query "hostName"

In a browser, go to the endpoint host name that the previous command returned: <endpoint>.azurefd.net/health. Your request automatically gets routed to the primary de-identification service in East US.

To test instant global failover:

1. Open a browser and go to the endpoint host name: <endpoint>.azurefd.net/health.

2. Follow the steps at Configure private access to disable public network access for the de-identification service in East US.

3. Refresh your browser. You see the same information page because traffic is now directed to the de-identification service in West US 2.

   Tip

   You might need to refresh the page a few times for the failover to complete.

4. Now disable public network access for the de-identification service in West US 2.

5. Refresh your browser. This time, you see an error message.

6. Re-enable public network access for one of the de-identification services. Refresh your browser and you see the health status again.

You've now validated that you can access your services through Azure Front Door and that failover functions as intended. Enable public network access on the other service if you're done with failover testing.

Clean up resources

In the preceding steps, you created Azure resources in a resource group. If you don't need these resources in the future, delete the resource group by running the following command:

az group delete --name my-deid

This command might take a few minutes to finish.

Initiate recovery

To check the recovery status of your service, you can send requests to <service-url>/health.

Related content

• Reliability in Azure