Passkey enrollment errors with the following error mentioned below:

 Invalid data in the WebAuthn enrollment request. attestedRpIdHash in authData did not match SHA256 hash of any trusted hostnames.

• Okta Identity Engine (OIE)
• Passkey/WebAuthn
• Widget / okta-auth-js

Registering Applications might overwrite the Relaying Party (RP) ID for the WebAuthn Authenticator returned by Okta during registration.

To support Passkeys, Okta allows a custom Relaying Party (RP) ID to be defined for the WebAuthn Authenticator.
With a custom RP configured during Passkey registration, Okta returns this value in the activation data.

        "activationData": {
          "rp": {
            "name": "{Org_Name}",
            "id": "{Custom_RP_Domain}"
          },
          "user": {
            ...
          },
          "pubKeyCredParams": [
            ...
          ],
          "challenge": "hQq...",
          "attestation": "direct",
          "authenticatorSelection": {
            ...
          },
          "u2fParams": {
            "...
          },
          "excludeCredentials": [],
          "extensions": {
            ...
          }
        }

Applications might attempt to override the RP ID value with the domain on which the application is hosted. Doing this can lead to the error below when the credential is sent back to Okta to complete the registration:

Invalid data...any trusted hostnames

To resolve this issue, applications should not override the RP ID returned by Okta.

• If the application's root domain is different from the RP root domain, then the /.well-known/webauthn endpoint associated with the RP domain should return the applications domain as an allowed origin.
• If the application shares the same root domain as the RP root domain, then nothing needs to be done.

Related References

• Associated Domains
• Passkey Custom RP ID Setup