MSDTC detectó un error (HR=0x80000171) al intentar establecer una conexión segura con el sistema <name server>.

Hello YUSA ,

Thank you for the additional information and screenshots. 

Based on the symptoms you reported and the current MSDTC configuration, the backup failures occur only when VMware/Barracuda runs quiesced (VSS) snapshots. During this process, Windows VSS invokes MSDTC to coordinate application consistency.  Because Network DTC Access is currently disabled, MSDTC cannot establish the required RPC communication and fails with Event ID 4879 (HR=0x80000171). The subsequent VSS errors (12305, 8193, 12293, 12329, 12330) are secondary effects of this initial failure. 

This is a guest OS configuration issue, not a VMware storage or Barracuda appliance problem.  

Recommended configuration (safe for File Server workloads): 

1.Open Component Services (dcomcnfg) 

2.Navigate to Computers > My Computer > Distributed Transaction Coordinator > Local DTC 

Open Properties > Security 

3.Enable Network DTC Access 

4.Under Transaction Manager Communication, select: 

• Allow Inbound 
• Incoming Caller Authentication Required 

Apply the changes and restart the MSDTC service (or reboot the VM) 

If you also encounter VSS/COM permission errors (e.g. Event 8194), add NETWORK SERVICE with Local Access permission under My Computer > Properties > COM Security > Access Permissions (Edit Default), then reboot. 

This configuration is commonly used for file servers that do not rely on distributed transactions (no SQL Linked Servers, COM+, or cross-database transactions) and is recommended by Microsoft/VMware for application-consistent VM backups. 

 After applying the changes, please re-test a quiesced snapshot. We expect the MSDTC and VSS errors to stop and backups to complete successfully. 

Best regards, 
Titus Bui  

Hi YUSA ,

Welcome to the Microsoft Q&A forum.
To better scope this issue, could you please confirm:

1. Does the error occur only during VMware VSS snapshot/backup, while the system works normally otherwise?
2. In Component Services → Local DTC → Security, is Mutual Authentication Required enabled?
3. Is the server domain‑joined (single domain or multi‑domain)? Any recent Windows updates or security hardening applied?
4. Does the application or SQL Server actually use distributed transactions (Linked Servers, COM+, cross‑DB), or is MSDTC otherwise not actively used?

This will help determine whether the issue is MSDTC security/authentication‑related rather than a storage or VMware issue.

Best wishes! 
Titus Bui.

The error code HR=0x80000171 indicates a problem related to the Microsoft Distributed Transaction Coordinator (MSDTC) when trying to establish a secure connection. This can occur in various scenarios, including when creating snapshots in VMware.

To troubleshoot this issue, consider the following steps:

1. Check MSDTC Configuration: Ensure that MSDTC is properly configured on both the source and destination servers. This includes verifying that the MSDTC service is running and that the necessary firewall ports are open.
2. Network Connectivity: Confirm that there is proper network connectivity between the servers involved. Any network issues can lead to connection failures.
3. Security Settings: Review the security settings for MSDTC. Ensure that the security settings allow remote connections and that the appropriate permissions are set.
4. VMware Configuration: Since the error occurs while creating a snapshot, ensure that the VMware environment is correctly configured to support snapshots, especially in the context of inactive disks.
5. Logs and Events: Check the event logs on both the Windows Server and VMware for any additional error messages or warnings that might provide more context about the failure.

If the issue persists after checking these areas, further investigation into the specific VMware and Windows Server configurations may be necessary.

References:

• Error when you manage a VHD file in Windows Server: "A Virtual Disk Provider for the specified file was not found"
• Error retrieving the list of servers when you create a protection group in Azure Site Recovery