A renomination failure occurs when the Admin node is on a lower Okta Access Gateway (OAG) version than 2023.12, and the renominated worker is on 2023.12 or later. The following errors can be seen on the console and in the backend logs:

Errors from the oag-mgmt console session: 

<worker_node> ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Perform restore on nominated node [USER="root"] Restoring nominated node using backup file - oagConfigBackup.<timestamp>.tar.bz2

2024-02-15T12:44:55.118-08:00 <worker_node> ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Enable oag-admin service [USER="root"] Enabled oag-admin service successfully on nominated node - <worker_node>
2024-02-15T12:51:56.970-08:00 <worker_node> ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Perform restore on nominated node [USER="root"] Restoring nominated node using backup file - oagConfigBackup.<timestamp>.tar.bz2
2024-02-15T12:52:01.507-08:00 <worker_node> ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Reset admin node HA setup [USER="root"] Reset HA configs of older admin node - <admin_node>

Errors from backend logs (ics_all.log): 

<worker_node> OAG_RESTORE RESTORE DATABASE ERROR - [-] Failed to restore database

<worker_node> OAG_RESTORE RESTORE DATABASE WARN - [-] Failed to restore database

2024-02-15T12:45:59.207-08:00 <worker_node> OAG_RESTORE RESTORE DATABASE ERROR - [-] Failed to restore database
2024-02-15T12:45:59.222-08:00 <worker_node> OAG_RESTORE RESTORE DATABASE WARN - [-] Failed to restore database
.
.
2024-02-15T12:46:26.398-08:00 <worker_node> application_template_service Exception occurred processing event file trusteddomains.create.json. Error : ['Traceback (most recent call last):', '  File "/opt/oag/bin/events_monitor.py", line 299, in <module>', '    td_handler.process_trusteddomains_event(event_info, file_with_path)', '  File "/opt/oag/bin/trusted_domains_event_handler.py", line 105, in process_trusteddomains_event', '    raise ValueError("Unexpected trusteddomains event: %s" % action.upper())', 'ValueError: Unexpected trusteddomains event: CREATE']

• Okta Access Gateway (OAG) 2023.12 and Later
• Okta Classic Engine
• Okta Identity Engine (OIE)

The OAG admin UI database was upgraded on 2023.12. During the upgrade from a lower version, the upgrade process will do a backup and restore of the existing UI database using updated binaries. 

However, during the renomination, a backup request is sent to the admin, and old binaries will be used to export the database. This results in a restore failure on the renominated node because of incompatibility. 

Upgrade the current admin to the same version as the worker before performing renomination. For more information on the admin renomination flow, refer to this documentation.