Okta Access Gateway Nginx Error "[emerg] host not found in upstream"

Okta Access Gateway (OAG) fails to resolve the backend application domain due to outdated Domain Name System (DNS) entries in the Unbound configuration file. Resolve this by verifying the DNS entries in the management console and removing any outdated directives from the configuration file.

Despite successful DNS lookups, Okta Access Gateway cannot resolve the backend application domain. Running an nslookup from the shell resolves successfully when provided with configured DNS servers on the appliance. However, the following error appears in the NGINX status.

2023-08-29T19:18:56.923-07:00 oktatest.com okta-nginx nginx: [emerg] host not found in upstream "okta.testurl.com" in /etc/nginx/conf.d/icsgw_app_<id>.active.conf:121
2023-08-29T19:19:49.000-07:00 localhost ACCESS_GATEWAY WEB_CONSOLE Command:[/opt/oag/bin/checkNginxConf.sh] Exit: 1 Out:HOST_NOTFOUND|
nginx: [emerg] host not found in upstream "okta.testurl.com" in /etc/nginx/conf.d/icsgw_app_<id>.active.conf:121
nginx: configuration file /opt/oag/nginx/conf/nginx.conf test failed

The following output is displayed through nslookup.

[0] oag-local@hostname;/home/oag-local# nslookup okta.testurl.com
Server:  127.0.0.1
Address: 127.0.0.1#53
** server can't find okta.testurl.com: NXDOMAIN

When used with a configured DNS server, nslookup resolves successfully and displays the following output.

[0] oag-local@hostname;/home/oag-local# nslookup okta.testurl.com 10.0.0.44
Server:  10.0.0.44
Address: 10.0.0.44#53
Non-authoritative answer:
Name: okta.testurl.com
Address: 10.0.0.55

• Okta Identity Engine (OIE)
• Okta Classic Engine
• Okta Access Gateway (OAG)

Unbound selects outdated Domain Name System (DNS) entries from /opt/oag/configs/unbound/config/unbound.conf that lack details on the backend resolution. The file contains the old entry at the fourth position, while the DNS list in the management console only displays three. This is an unexpected behavior that cannot be reproduced, making the root cause unknown.

How are the outdated DNS entries removed from the Unbound configuration?

Verify the configured Domain Name System (DNS) entries in the management console, remove any unresolvable or unwanted directives from the Unbound configuration file, and reload the services.

1. Verify that all configured DNS entries shown in the management console resolve the backend host. If any DNS servers fail to resolve, remove the entry or update the DNS entries.
2. Check /opt/oag/configs/unbound/config/unbound.conf on the appliance for more than three forward-addr directives.
3. Remove the unwanted forward-addr entries if discrepancies exist with the entries displayed in the management console.
4. Reload Unbound and NGINX by executing the provided commands.

   sudo systemctl reload okta-unbound
   sudo systemctl reload okta-nginx