Mac RDP client cannot log into new Windows 11 Pro machine – “The credentials used to connect did not work” + certificate warning

Hi Lafayette Williams,

Thank you for giving your question to us.

To my knowledge, this specific error sequence on Windows 11 Pro usually stems from a conflict between the Windows App (formerly Remote Desktop) on macOS and the enhanced security defaults in the latest Windows builds.

Here are the recommended solutions to help you:

1. Disable "Require Windows Hello" for Sign-in

Windows 11 often defaults to allowing only "passwordless" sign-in for Microsoft Accounts. This prevents RDP from accepting a standard password because it expects a PIN or biometric that RDP cannot transmit.

On the host Windows 11 PC, go to Settings > Accounts > Sign-in options.

Under Additional settings, toggle OFF "For improved security, only allow Windows Hello sign-in for Microsoft accounts on this device."

Restart the PC and attempt to connect again.

2. Verify the Username Format

Remote Desktop is pedantic regarding how usernames are passed from macOS.

If using a Local Account: Use .\Username or ComputerName\Username.

If using a Microsoft Account: Use the full email address (e.g., ******@outlook.com).

3. Adjust Network Level Authentication

While NLA is a security best practice, it can occasionally cause the "Credentials did not work" loop if there is a mismatch in security providers.

Go to Settings > System > Remote Desktop.

Click the dropdown arrow next to "Remote Desktop" (Ensure it is On).

Uncheck the box: "Require devices to use Network Level Authentication to connect (recommended)."

Then test the connection again.

4. Group Policy Adjustment

If the above fails, you may need to force the security layer to negotiate correctly.

Press Win + R, type gpedit.msc, and hit Enter.

Navigate to: Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Security.

Find "Require use of specific security layer for remote (RDP) connections".

Set it to Enabled and select RDP from the dropdown menu.

If you find the answer useful, please mark it as the "Accepted Answer" so other members can find the solution easily. Your help in supporting the community is much appreciated!

Best regards,
Scott Nguyen

Use the following checks and fixes on the new Windows 11 Pro machine and the RDP configuration, focusing on authentication and the RDP listener/certificate.

1. Verify the machine state and basic RDP configuration

• Confirm the Windows 11 Pro machine is running and reachable on the network (ping or console access).
• On the Windows 11 Pro machine, open System Properties → Remote and ensure Allow remote connections to this computer is enabled.
• Confirm the account being used is either a local account on that machine or a domain account that has permission, and that it is in the Remote Desktop Users group (or is an administrator).

2. Check that the RDP listener is functioning If basic settings are correct but authentication still fails, verify the RDP-TCP listener:

• On the Windows 11 Pro machine, check the registry key:
  • HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-tcp
• If the listener configuration is suspected to be corrupt, export this key from a working Windows machine of the same version, back up the existing key on the new machine, replace it with the exported one, then restart the Remote Desktop Services (TermService) service and test again.
• After changes, test RDP again from the Mac client.

3. Fix certificate and encryption negotiation issues The certificate warning and repeated credential prompts indicate possible issues with the RDP self-signed certificate or security protocol negotiation.

3.1 Reset the RDP self-signed certificate On the Windows 11 Pro machine:

1. Open mmc.exe.
2. Add the Certificates snap-in for Computer account → Local computer.
3. Navigate to Remote Desktop → Certificates.
4. Delete the existing RDP self-signed certificate.
5. Restart the Remote Desktop Services service.
6. Refresh the Certificates snap-in and confirm a new RDP self-signed certificate is created.

3.2 Verify security protocol-related registry values On the Windows 11 Pro machine, check:

• Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
  • If SSLCertificateSHA1HashValue exists and does not match the Thumbprint of the RDP certificate under Remote Desktop → Certificates, delete SSLCertificateSHA1HashValue, restart the system, and test again. (Do not do this if the server has the Remote Desktop Connection Broker role.)
  • Ensure fAllowSecProtocolNegotiation is set to 1.
  • Ensure MinEncryptionLevel is set to 3.

3.3 Check SSL cipher suite policy If a cipher suite policy is configured, it can break TLS negotiation with some clients:

• In Group Policy, check:
  • Computer Configuration → Administrative Templates → Network → SSL Configuration Settings → SSL Cipher Suite Order.
• Set this policy to Not configured if it is defined.
• Then delete the registry value (if present):
  • Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002 → Functions.
• Reboot and test RDP again.

4. Test with a local account If domain credentials are in use and still fail, test RDP using a local account on the Windows 11 Pro machine. If local accounts work but domain accounts fail intermittently, investigate domain connectivity and possible TCP port exhaustion as described below.
5. Check for intermittent or port-exhaustion-related issues If the error appears intermittently or after the machine has been running for some time:

• On the Windows 11 Pro machine, open Event Viewer → Windows Logs → System and look for events such as 1129 (Group Policy), 4227/4231 (Tcpip), or 5719 (NETLOGON).
• If these events are present, troubleshoot TCP port exhaustion and identify any misbehaving process that may be consuming ports. Stopping that process and then signing in via RDP with domain credentials can restore access.

6. If issues persist If, after resetting the RDP certificate, confirming the RDP listener, and validating encryption and cipher suite settings, the Mac client still cannot authenticate while other clients can, collect logs and consider contacting Microsoft Support for deeper analysis.

References:

• Remote Desktop can't connect to the remote computer
• General Remote Desktop connection troubleshooting
• "An internal error has occurred" when connecting to a remote machine
• "Remote Desktop cannot verify the identity of the remote computer" when connecting to a remote machine