Deploy disconnected operations for Azure Local and the management cluster

This article explains how to deploy disconnected operations for Azure Local in your datacenter. This step is key to deploy and operate Azure Local without any outbound network connection. After you deploy the management cluster (control plane), you deploy your first Azure Local instance.

Key considerations

When deploying disconnected operations and creating your management instance, consider the following key points:

• The network configuration and names you enter in the portal should be consistent with your setup and the previously created switches.
• Virtual deployments aren't supported. You must have physical machines.
• If you require VLAN on the control plane appliance you need to make sure to set that up using Set-VMNetworkAdapterIsolation -ManagementOS .
• You need at least three machines to support disconnected operations. You can use up to 16 machines for the management instance.
• The deployment of the Azure Local cluster can take several hours.
• The local control plane can experience periods of downtime during node reboots and updates.
• During the creation of the cluster, the process creates a thinly provisioned 2-TB infrastructure volume for disconnected operations. Don't tamper with or delete the infrastructure volumes created during the deployment process.
• When you create the Azure Local instance, the process moves the disconnected operations VM appliance to cluster storage and converts it to a clustered VM.

Prerequisites

For more information, see Azure Local disconnected operations overview.

For information on known issues with disconnected operations for Azure Local, see Known issues for disconnected operations.

Checklist for deploying disconnected operations

Before you deploy Azure Local with disconnected operations, you need the following:

• Network plan:
  • IP pool, ingress IP, fully qualified domain name (FQDN), domain name system (DNS), and gateway (GW).
• DNS server to resolve IP to FQDN names.
• Local credentials for Azure Local machines.
• Active directory credentials for Azure Local deployment.
• Active directory OU and networking requirements.
• Local credentials and AD credentials to meet minimum password complexity.
• Active directory prepared for Azure Local deployment.
• Certificates to secure ingress endpoints (24 certificates) and the public key (root) used to create these certificates.
• Certificates to secure the management endpoint (two certificates).
• Credentials and parameters to integrate with identity provider:
  • Active Directory Federations Services (ADFS) application, credentials, server details, and certificate chain details for certificates used in identity configuration.
• Disconnected operations deployment files (manifest and appliance).
• The Azure Local composite ISO.
• Firmware/device drivers from OEM.

Deployment sequence

Deploy Azure Local with disconnected operations on non-premier hardware stock keeping units (SKUs). For more information, see Azure Local Catalog.

You deploy and configure Azure Local with disconnected operations in multiple steps. The following image shows the overall journey, including post-deployment.

Here's a brief overview of the tools and processes you use during the deployment. You'll need access to Azure Local nodes (OS or host) for the first phases of the deployment.

1. Use the existing tools and processes to install and configure the OS. You need local admin access on all Azure Local nodes.
2. Run PowerShell and the Operations module on the first machine (sorted by node name like seed node). You must have local admin access.
3. Use the local Azure portal, Azure PowerShell, or Azure CLI. You don't need physical node access, but you do need Azure Role-Based Access Control (RBAC) with the Owner role.
4. Use the local Azure portal, Azure PowerShell, or Azure CLI. You don't need physical node access, but you do need Azure RBAC with the Operator role.

Deploy disconnected operations (control plane)

Disconnected operations must be deployed on the first machine (seed node). Make sure you complete the following steps on every node in your management cluster. For more information, see Prepare Azure Local machines.

To prepare the first machine for the disconnected operations appliance, follow these steps:

1. Modify your path to correct location. If you initialized a data disk or are using a different path than C: modify the $applianceConfigBasePath.

   Here's an example:

   $applianceConfigBasePath = 'C:\AzureLocalDisconnectedOperations'
   

2. Copy the disconnected operations installation files (appliance zip, vhdx, and manifest) to the first machine. Save these files into the base folder you created earlier.

   Copy-Item \\fileserver\share\azurelocalfiles\* $applianceConfigBasePath    
   

3. Verify that you have these files in your base folder using the following command:

   • AzureLocal.DisconnectedOperations.zip
   • AzureLocal.DisconnectedOperations.manifest
   • ArcA_LocalData_A.vhdx
   • ArcA_SharedData_A.vhdx
   • OSAndDocker_A.vhdx

   Get-ChildItem $applianceConfigBasePath  
   

4. Extract the zip file in the same folder:

   Expand-Archive "$($applianceConfigBasePath)\AzureLocal.DisconnectedOperations.zip" -DestinationPath $applianceConfigBasePath  
   

5. Verify that you have these files using the following command:

   • OperationsModule (PowerShell module for installation)
   • AzureLocal.DisconnectedOperations.manifest
   • AzureLocal.DisconnectedOperations.zip
   • manifest.xml
   • IRVM.zip
   • ArcA_LocalData_A.vhdx
   • ArcA_SharedData_A.vhdx
   • ArcA_SharedData_ACSBlob_A.vhdx
   • ArcA_SharedData_ACSTable_A.vhdx
   • OSAndDocker_A.vhdx
   • Storage.json

   Get-ChildItem $applianceConfigBasePath   
   

   Note

   At this point, remove the AzureLocal.DisconnectedOperations.zip file to save some space.

6. Copy the certificates root directory. Save these files into the base folder you created earlier.

   $certsPath = "$($applianceConfigBasePath)\certs"  
   Copy-Item \\fileserver\share\azurelocalcerts $certspath -recurse  
   

7. Verify the certificates, public key, and management endpoint. You should have two folders: ManagementEndpointCerts and IngressEndpointsCerts and at least 24 certificates.

   Get-ChildItem $certsPath 
   Get-ChildItem $certsPath -recurse -filter *.cer  
   

8. Import the Operations module. Run the command as an administrator using PowerShell. Modify the path to match your folder structure.

   Import-Module "$applianceConfigBasePath\OperationsModule\Azure.Local.DisconnectedOperations.psd1" -Force    
   
   $mgmntCertFolderPath = "$certspath\ManagementEndpointCerts"  
   $ingressCertFolderPath = "$certspath\IngressEndpointsCerts"  
   

Initialize the parameters

Populate the required parameters based on your deployment planning. Modify the examples to match your configuration.

1. Populate the management network configuration object.

   $CertPassword = "retracted"|ConvertTo-Securestring -AsPlainText -Force  
   $ManagementIngressIpAddress = "192.168.50.100"  
   $ManagementNICPrefixLength = 24  
   $mgmtNetworkConfigParams = @{  
       ManagementIpAddress = $ManagementIngressIpAddress  
       ManagementIpv4PrefixLength = $ManagementNICPrefixLength  
       TlsCertificatePath = "$mgmntCertFolderPath\ManagementEndpointSsl.pfx"  
       TlsCertificatePassword = $CertPassword  
       ClientCertificatePath = "$mgmntCertFolderPath\ManagementEndpointClientAuth.pfx"  
       ClientCertificatePassword = $CertPassword  
   }  
   $managementNetworkConfiguration = New-ApplianceManagementNetworkConfiguration @mgmtNetworkConfigParams  
   

   Note

   The password for the certificates must be in the secure string format. For certificates pertaining to the management endpoint, see PKI for disconnected operations.

2. Populate the ingress network configuration object.

   $azureLocalDns = "192.168.200.150"  
   $NodeGw = "192.168.200.1"  
   $IngressIpAddress = "192.168.200.115"  
   $NICPrefixLength= 24  
   $ingressNetworkConfigurationParams = @{  
       DnsServer = $azureLocalDns  
       IngressNetworkGateway = $NodeGw  
       IngressIpAddress = $IngressIpAddress  
       IngressNetworkPrefixLength = $NICPrefixLength  
       ExternalDomainSuffix = "autonomous.cloud.private"
   }  
   $ingressNetworkConfiguration = New-ApplianceIngressNetworkConfiguration @ingressNetworkConfigurationParams  
   

   For network configuration details, see Networking for disconnected operations.

3. Populate the identity configuration object.

   $oidcCertChain = Get-CertificateChainFromEndpoint -requestUri 'https://adfs.azurestack.local/adfs'        
   # Omitted this step if you don't have LDAPS.
   
   $ldapsCertChain = Get-CertificateChainFromEndpoint -requestUri 'https://dc01.azurestack.local'
   
   # LDAPS default port (Non secure default = 3268)
   $ldapPort = 3269
   $ldapPassword = 'RETRACTED'|ConvertTo-SecureString -AsPlainText -Force
   
   # Populate params with LDAPS enabled.
   $identityParams = @{  
       Authority = "https://adfs.azurestack.local/adfs"  
       ClientId = "<ClientId>"  
       RootOperatorUserPrincipalName = "operator@azurestack.local"  
       LdapServer = "adfs.azurestack.local"  
       LdapPort = $ldapPort 
       LdapCredential = New-Object PSCredential -ArgumentList @("ldap", $ldapPassword)  
       SyncGroupIdentifier = "<SynGroupIdentifier>"     
       OidcCertChainInfo = $oidcCertChain
       LdapsCertChainInfo = $ldapsCertChain             
   }  
   $identityConfiguration = New-ApplianceExternalIdentityConfiguration @identityParams  
   

   Note

   • LdapsCertChainInfo and OidcCertChain can be omitted completely for debugging or demo purposes. For information on how to get LdapsCertChainInfo and OidcCertChainInfo, see PKI for disconnected operations.

   There's an issue with the Get-CertificateChainFromEndpoint not being exported as intended. Use the steps in Known issues for disconnected operations for Azure Local to mitigate this issue.

   For more information, see Identity for disconnected operations.

4. Populate the external certificates configuration object.

   $ingressCertPassword = "retracted"|ConvertTo-Securestring -AsPlainText -Force  
   $certsParams = @{  
       certificatesFolder = $ingressCertFolderPath  
       certificatePassword = $ingressCertPassword  
   }  
   $CertificatesConfiguration = New-ApplianceCertificatesConfiguration @certsParams  
   

   For more information, see PKI for disconnected operations.

5. Copy the appliance manifest file (Downloaded from Azure) to your configuration folder:

   # Modify your source path accordingly 
   copy-item AzureLocal.DisconnectedOperations.Manifest.json $applianceConfigBasePath\AzureLocal.DisconnectedOperations.manifest.json
   

Install and configure the appliance

To install and configure the appliance on the first machine, use the following command. Point the AzureLocalInstallationFile to a path that contains the IRVM01.zip.

$networkIntentName = 'ManagementComputeStorage' 
$azureLocalInstallationFile = "$($applianceConfigBasePath)"  
$applianceManifestJsonPath = Join-Path $applianceConfigBasePath AzureLocal.DisconnectedOperations.manifest.json

$installAzureLocalParams = @{  
    Path = $azureLocalInstallationFile  
    IngressNetworkConfiguration = $ingressNetworkConfiguration  
    ManagementNetworkConfiguration = $managementNetworkConfiguration  
    IngressSwitchName = "ConvergedSwitch($networkIntentName)"  
    ManagementSwitchName = "ConvergedSwitch($networkIntentName)"  
    ApplianceManifestFile = $applianceManifestJsonPath  
    IdentityConfiguration = $identityConfiguration  
    CertificatesConfiguration = $CertificatesConfiguration      
}  

Install-Appliance @installAzureLocalParams -disconnectMachineDeploy -Verbose  

# Note: If you're deploying the appliance with limited connectivity, you can omit the flag -disconnectMachineDeploy.

Configure observability for diagnostics and support

We recommend that you configure observability to get system-generated logs for your first deployment. This doesn't apply if you're planning to run air-gapped, as system-generated logs and diagnostics require connectivity.

The Azure resources needed:

• A resource group in Azure used for the appliance.
• A Service Principal Name (SPN) with contributor rights on the resource group.

To configure observability, follow these steps:

1. On a computer with Azure CLI (or using the Azure Cloud Shell in Azure portal) create the SPN. Run the following script:

   $resourcegroup = 'azure-disconnectedoperations'
   $appname = 'azlocalobsapp'
   az login
   $g = (az group create -n $resourcegroup -l eastus)|ConvertFrom-Json
   az ad sp create-for-rbac -n $appname --role Owner --scopes $g.id
   
   # Get the Subscription ID
   $j = (az account list | ConvertFrom-Json) | Where-object {$_.IsDefault}
   
   # SubscriptionID:
   $j.id
   

   Here's an example output:

   {
     "appId": "<AppId>",
     "displayName": "azlocalobsapp",
     "password": "<RETRACTED>",
     "tenant": "<RETRACTED>"
   }
   <subscriptionID>
   

2. Set the observability configuration. Modify to match your environment details:

   $observabilityConfiguration = New-ApplianceObservabilityConfiguration -ResourceGroupName "azure-disconnectedoperations" `
     -TenantId "<TenantID>" `
     -Location "<Location>" `
     -SubscriptionId "<SubscriptionId>" `
     -ServicePrincipalId "<AppId>" `
     -ServicePrincipalSecret ("<Password>" | ConvertTo-SecureString -AsPlainText -Force)
   
   Set-ApplianceObservabilityConfiguration -ObservabilityConfiguration $observabilityConfiguration
   

3. Verify that observability is configured:

   Get-ApplianceObservabilityConfiguration
   

Configure Azure PowerShell

On each node, run the following to enable a custom cloud endpoint for Azure PowerShell. You'll use this later when bootstrapping the Azure Local node to the control plane.

$applianceCloudName = "azure.local"
$applianceFQDN = "autonomous.cloud.private"

$AdminManagementEndPointUri = "https://armmanagement.$($applianceFQDN)/"
$DirectoryTenantId = "98b8267d-e97f-426e-8b3f-7956511fd63f"

#Retrieve disconnected operations endpoints

$armMetadataEndpoint = $AdminManagementEndPointUri.ToString().TrimEnd('/') + "/metadata/endpoints?api-version=2015-01-01"

$endpoints = Invoke-RestMethod -Method Get -Uri $armMetadataEndpoint -ErrorAction Stop

$azEnvironment = Add-AzEnvironment `
-Name $applianceCloudName `
-ActiveDirectoryEndpoint ($endpoints.authentication.loginEndpoint.TrimEnd('/') + "/") `
-ActiveDirectoryServiceEndpointResourceId $endpoints.authentication.audiences[0] `
-ResourceManagerEndpoint $AdminManagementEndPointUri.ToString() `
-GalleryEndpoint $endpoints.galleryEndpoint `
-MicrosoftGraphEndpointResourceId $endpoints.graphEndpoint `
-MicrosoftGraphUrl $endpoints.graphEndpoint `
-AdTenant $DirectoryTenantId `
-GraphEndpoint $endpoints.graphEndpoint `
-GraphAudience $endpoints.graphEndpoint `
-EnableAdfsAuthentication:($endpoints.authentication.loginEndpoint.TrimEnd("/").EndsWith("/adfs",[System.StringComparison]::OrdinalIgnoreCase)) 


# Verify that you can connect to the ARM endpoint (example showing device authentication)
Connect-AzAccount -EnvironmentName $applianceCloudName -UseDeviceAuthentication

Subscription placement for the dedicated management cluster

We require deploying a fully dedicated management cluster. The recommended practice is to place your management cluster in the operator subscription. This will help you restrict and isolate the control plane from workloads and you can restrict workloads from being created on the same cluster as other tenants.

Keeping the control plane separated from the workloads provides a clear separation of concerns.

Ensure you limit access to the operator subscription to only required personnel.

Register required resource providers

Make sure you register the required resource providers before deployment.

Here's an example of how to automate the resource providers registration from Azure PowerShell.

$applianceCloudName = "azure.local"
$subscriptionName = "Operator subscription"

# Connect to the ARM endpointusing device authentication
Connect-AzAccount -EnvironmentName $applianceCloudName -UseDeviceAuthentication
Write-Host "Selecting a different subscription than the operator subscription.."
$subscription = Get-AzSubscription -SubscriptionName $subscriptionName

# Set the context to that subscription
Set-AzContext -SubscriptionId $subscription.Id

Register-AzResourceProvider -ProviderNamespace  "Microsoft.EdgeArtifact"
Register-AzResourceProvider -ProviderNamespace "Microsoft.HybridCompute" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.GuestConfiguration" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.HybridConnectivity" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.AzureStackHCI" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.Kubernetes" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.KubernetesConfiguration" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.ExtendedLocation" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.ResourceConnector" 
Register-AzResourceProvider -ProviderNamespace "Microsoft.HybridContainerService"
# Not required on disconnected operations
# Register-AzResourceProvider -ProviderNamespace "Microsoft.Attestation"
# Register-AzResourceProvider -ProviderNamespace "Microsoft.Storage"
# Register-AzResourceProvider -ProviderNamespace "Microsoft.Insights"

Wait until all resource providers are in the state Registered.

To list all resource providers and their statuses, run the following command.

Get-AzResourceProvider | Format-Table

Deploy Azure Local to form the management cluster

You now have a control plane deployed and configured, a subscription and resource group created for your Azure Local deployment, and (optionally) an SPN created to use for deployment automation.

Verify the deployment before creating local Azure resources.

1. Sign in with the root operator you defined during the deployment, or use a subscription owner account.
2. From a client with network access to your Ingress IP, open your browser and go to https://portal.<FQDN>. Replace <FQDN> with your domain name.
   • You're redirected to your identity provider to sign in.
3. Sign in to your identity provider with the credentials you configured during deployment.
   • You should see Azure portal running in your network.
4. Check that a subscription exists for your Azure Local infrastructure (for example, Operator subscription).
5. Check that required resource providers are registered in the subscription.
6. Check that a resource group exists for your Azure Local infrastructure (for example, azurelocal-disconnected-operations).

Initialize each Azure Local node

To initialize each node, run this PowerShell script. Modify the variables necessary to match your environment details:

1. Initialize each Azure Local node.

   $resourcegroup = 'azurelocal-management-cluster' 
   $applianceCloudName = "azure.local"
   $applianceConfigBasePath = "C:\AzureLocalDisconnectedOperations\"
   $applianceFQDN = "autonomous.cloud.private"
   $subscriptionName = "Operator subscription"
   
   Connect-AzAccount -EnvironmentName $applianceCloudName -UseDeviceAuthentication
   Write-Host "Ensuring you are using operator subscription for the management cluster.."
   $subscription = Get-AzSubscription -SubscriptionName $subscriptionName
   
   # Set the context to that subscription
   Set-AzContext -SubscriptionId $subscription.Id
   
   $armTokenResponse = Get-AzAccessToken -ResourceUrl "https://armmanagement.$($applianceFQDN)"
   
   # $ArmAccessToken = $armTokenResponse.Token
   # Convert token to string for use in initialization
   # Workaround needed for Az.Accounts 5.0.4
   $ArmAccessToken = [System.Net.NetworkCredential]::new("", $armTokenResponse.Token).Password
   
   # Bootstrap each node
   Invoke-AzStackHciArcInitialization -SubscriptionID $subscription.Id -TenantID $subscription.TenantId -ResourceGroup $resourceGroup -Cloud $applianceCloudName -Region "Autonomous" -CloudFqdn $applianceFQDN -ArmAccessToken $ArmAccessToken
   

Pre-create Azure Key Vault

Create the Azure Key Vault before you deploy Azure Local to avoid long deployment delays caused by a known issue.

For a code example, see Known issues.

After you create the Key Vault, wait 5 minutes before you continue with the next portal deployment step.

Deploy the management cluster (first Azure Local instance)

With the control plane deployed and configured - you can complete the management cluster by deploying Azure Local using your local control plane.

Follow these steps to create an Azure Local instance (cluster):

1. Access the local portal from a browser of your choice.
2. Navigate to portal.FQDN. For example, https://portal.autonomous.cloud.private.
3. Select your nodes and complete the deployment steps outlined in Deploy Azure Local using the Azure portal.

Tasks after deploying disconnected operations

Perform the following tasks after deploying Azure Local with disconnected operations:

1. Back up the BitLocker keys (do not skip this step). During deployment, the appliance is encrypted. Without the recovery keys for the volumes, you can't recover and restore the appliance. For more information, see Understand security controls with disconnected operations on Azure Local.
2. Assign extra operators. You can assign one or more operators by navigating to Operator subscriptions. Assign the contributor role at the subscription level.
3. Export the host guardian service certificates and back up the folder you export them to on an external share or drive.
4. Register the management cluster
5. Lock down the management cluster. Restrict operators from deploying workloads on the management cluster. Limit operator access to a select few and enforce this restriction using Azure Policy to block workloads on the cluster resource.
6. Clean up disks. If data disks were used during the bootstrap process, ensure they are removed.

Appendix

Lock down management cluster (using Azure Policy)

$operatorSubscriptionId = ''
$resourceGroup = ''
$customLocationId = 'my-cluster'
cd "$applianceConfigBasePath\OperationsModule\AzureLocalOrchestration" 
.\Set-MgmtClusterDenyPolicy.ps1 `
        -SubscriptionId "$operatorSubscriptionId" `
        -MgmtClusterCustomLocationId "/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroup)/providers/Microsoft.ExtendedLocation/customLocations/$($customLocationId)"

Clean up data disks used for bootstrap

# ===============================
# Remove CSV and Return Space to Pool
# ===============================

$CsvName="Cluster Virtual Disk (InfraLocal_1)"
$VirtualDiskName ="InfraLocal_1"

Write-Host "Starting CSV removal process..." -ForegroundColor Cyan

# --- Validate CSV exists ---
$csv = Get-ClusterSharedVolume -Name $CsvName -ErrorAction Stop
Write-Host "Found CSV: $($csv.Name)"

# --- Remove CSV ---
$csv = remove-ClusterSharedVolume -Name $CsvName -ErrorAction Stop
Write-Host "Removed CSV: $($csv.Name)"

# --- Take Cluster Resources offline ---
Write-Host "Taking CSV resource offline..." -ForegroundColor Yellow
Stop-ClusterResource -Name $csv.name -Wait 120

# --- Remove Cluster Resources offline ---
Write-Host "Taking CSV resource offline..." -ForegroundColor Yellow
remove-ClusterResource -Name $csv.name

# --- Remove virtual disk and return space to pool ---
Write-Host "Remoing virtual disk..." -ForegroundColor Yellow
remove-virtualdisk -FriendlyName $VirtualDiskName

Troubleshoot and reconfigure by using management endpoint

To use the management endpoint for troubleshooting and reconfiguration, you need the management IP address used during deployment, along with the client certificate and password used to secure the endpoint.

From a client with network access to the management endpoint, import the OperationsModule and set the context (modify the script to match your configuration):

Import-Module "$applianceConfigBasePath\OperationsModule\Azure.Local.DisconnectedOperations.psd1" -Force

$password = ConvertTo-SecureString 'RETRACTED' -AsPlainText -Force  
$context = Set-DisconnectedOperationsClientContext -ManagementEndpointClientCertificatePath "${env:localappdata}\AzureLocalOpModuleDev\certs\ManagementEndpoint\ManagementEndpointClientAuth.pfx" -ManagementEndpointClientCertificatePassword $password -ManagementEndpointIpAddress "169.254.53.25"  

After setting the context, you can use all the management cmdlets provided by the Operations module, like resetting identity configuration, checking health state, and more.

To get a full list of cmdlets provided, use the PowerShell Get-Command -Module OperationsModule. To get details around each cmdlet, use the Get-Help <cmdletname> command.

Troubleshooting deployments

System not configured

After you install the appliance, you might see this screen for a while. Let the configuration run for 2 to 3 hours. After you wait the 2 to 3 hours, the screen goes away, and you see the regular Azure portal. If the screen doesn't go away and you can't access the local Azure portal, troubleshoot the issue.

• To find logs from the OperationsModule on the first machine, go to C:\ProgramData\Microsoft\AzureLocalDisconnectedOperations\Logs.

• To view the health state of your appliance, use the management endpoint Get-ApplianceHealthState cmdlet. If you see this screen and the cmdlet reports no errors and all services report 100, open a support ticket from the Azure portal.

Install-Appliance fails

Update the appliance configuration and rerun the installation after failure.

For example:

• If the IP address is already in use, modify the configuration object.

  Here's an example to modify the ingress IP address

  # Set a new IP address
  $ingressNetworkConfiguration.IngressIpAddress = '192.168.0.115'
  

• If install-appliance fails during installation, update $installAzureLocalParams and rerun Install-appliance as described in Install and configure the appliance.

• If the appliance deployment succeeded and you're updating network configuration, see Get-Help Set-Appliance for the settings you can update post-deployment.

Next steps

• Register disconnected operations for Azure Local.

This feature is available only in Azure Local 2602 or later.