An Okta Access Gateway (OAG) High Availability (HA) synchronization fails with a generic error when the worker node cannot copy configuration files from the admin node. This issue occurs when a deleted certificate leaves a dead symbolic link, and administrators must remove the dead link from the admin node to resolve the error.

When configuring high availability or during a sync from admin, Okta generates the following error messages on the worker node:

Failed to synchronize events from admin.

FAILURE RC =2

• Okta Identity Engine (OIE)
• Okta Classic Engine
• Okta Access Gateway (OAG)
• High Availability (HA)
• Secure Sockets Layer (SSL)

After the addition of a node to the cluster during HA configuration, the new worker node pulls all application configuration files, certificates, and keys. If Okta encounters an issue while copying these files, the console logs a generic FAILURE RC=2 error.

In Okta Access Gateway versions prior to 2023.11, Okta does not remove the symbolic link created for multiple Subject Alternative Names (SANs) when an administrator deletes a certificate. As a result, the symbolic link lacks the referred file and becomes inaccessible when the worker node attempts to pull the configuration. This issue also occurs during a sync from admin. 

How do administrators identify the underlying synchronization issue?

Connect to the affected worker node through the shell, switch to the root user, and execute the setup script to trigger the synchronization manually and identify the failing file.

1. Connect to the affected worker node through the shell.
2. Switch to the root user by executing the following command.
   

   sudo su -

3. Execute the following command to trigger the synchronization step manually. Replace <admin-node-hostname> with the hostname of the admin node.
   

   /opt/oag/scripts/oagHAsetup.sh -s -m <admin-node-hostname>

Review the script output to troubleshoot the synchronization failure.

Review the script output to determine which file causes the synchronization issue and remove any dead symbolic links on the admin node.

Review the following example output of a successful synchronization. Administrators can ignore the chgrp error, as it does not impact the synchronization flow.

worker3.support.lab;/root# /opt/oag/scripts/oagHAsetup.sh -s -m oag.support.lab
Starting up connection to admin..
  Generating list of configurations on admin..
  Pulling spgw config from admin: spgw.json
    sleeping for 5 secs to process configuration
  Pulling auth config from admin: auth.82db97a4-170f-47fd-8f73-0f49c78fd4de.json
  Pulling auth config from admin: store.auth.json
  Pulling idp config from admin: idp.16e34e7d-010d-4ecd-8a62-1974b5892aaa.json
  Pulling krb config from admin: krb5.67137634-6556-4448-867d-f4d27b20dd0e.json
  Pulling loglevel config from admin: loglevel.local.json
  Pulling store config from admin: store.spgw.json
<truncated>
  Pulling ebsssoagent config from admin: ebsssoagent.a7a56e33-21a7-4f72-8611-4ddb0a1ca499.json
  Pulling app config from admin: app.2779a889-0ca3-4e86-9c0b-afd863fb9e9b.json
<truncated>
  Pulling app config from admin: app.c2e20d08-b1aa-4d66-9d97-989ba6d20017.json
chgrp: cannot access '/opt/oag/configs/simpleSAMLphp/config/cert/*.key': No such file or directory
Unit /etc/systemd/system/oag-admin.service is masked, ignoring.

If the issue described in the cause section occurs, the script returns the following output. 

worker3.support.lab;/root# /opt/oag/scripts/oagHAsetup.sh -s -m oag.support.lab
Starting up connection to admin..
  Generating list of configurations on admin..
  Pulling spgw config from admin: spgw.json
    sleeping for 5 secs to process configuration
<truncated>
  Pulling ebsssoagent config from admin: ebsssoagent.a7a56e33-21a7-4f72-8611-4ddb0a1ca499.json
  Pulling app config from admin: app.2779a889-0ca3-4e86-9c0b-afd863fb9e9b.json
Unable to get SSL key/cert from host: oag.support.lab
Exiting.

The Unable to get SSL key/cert error indicates that the worker node cannot retrieve the certificate and key files from the admin node.

1. Connect to the admin node through the shell.
2. Navigate to the /opt/oag/nginx/ssl directory.
3. Locate any dead links, which appear as blinking red entries.
4. Execute the following command to remove the dead link safely. Replace <link> with the name of the dead link.

NOTE: Do not delete any files other than the red symbolic link.

sudo rm <link>

Attempt the sync from admin process again from the worker node.

**chgrp error can be ignored. It will be handled in future releases and does not have any impact on the sync flow.

Related References

• Cli Access
• HA Configuration Steps
• Information on how the syncing process works