Centrally Viewing Okta Logs in Heterogenous Cloud Environment

Today, many enterprises have heterogeneous cloud infrastructure and leverage the unique strengths of different cloud providers. They leverage multiple cloud providers for their unique services and to harvest cost-efficiency, high performance, and cutting-edge features. Also, many of the “Software as service” providers offer their service on a specific cloud provider platform before expanding into other platforms (e.g. Okta Workforce Identity Cloud (WIC) on AWS etc.)

The mainstream cloud providers offer services that include providing infrastructure, platform, and software-as-a-service capabilities for various needs like computing, data storage, analytics, and machine learning. Amazon, Google, Microsoft, and Oracle are some of the predominant Cloud providers.

Many of the large enterprises have standardized and chosen GCP as a platform of choice and leverage it heavily for running their business. They have built monitoring dashboards on GCP that analyze and monitor the health of their IT Systems and can pinpoint bottlenecks impacting the business near real-time. They still leverage “Software as a Service” on other cloud platforms and are effectively able to perform analysis and monitoring of those services in their current ecosystem.  

Below is an example of how Okta Access Gateway logs (which currently can run on GCP) can be funneled into Google cloud logging for monitoring and analysis. Log Explorer & cloud analytics on GCP can be further leveraged to monitor the health of Okta access gateway applications.

The Rocket-fast System for log processing (rsyslog) is a system utility provided in Linux variants which provides support for message logging. It offers high performance and comes with excellent security , and a modular design. This is a very useful utility that can accept input from a wide variety of sources and transform them and store them in multiple and diverse destinations.

The configuration screen shown below can be utilized in Okta Access Gateway for forwarding the logs to rsyslog Server. From Okta Access Gateway the logs namely - audit, access and monitor logs can be forwarded to rsyslog server using TCP/UDP protocol.

A configuration file “rsyslog.conf” is leveraged to ingest, process and forward the logs to defined destinations. In our configuration below rsyslog is configured to receive messages via TCP, write files locally and forward them to remote destinations (ab-usc1-a.dev.companyxyz-gcp.com). Attached below is a sample config file on rsyslog server (rsyslog.conf) which can be tweaked to meet customers desired requirements and outcomes.  

rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module below is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad original # provides access to the systemd journal

#$ModLoad imklog # reads kernel messages (the same are read from journald)

$ModLoad immark # provides --MARK-- message capability

#$ModLoad impstats

module(load="impstats"

       interval="120"

       severity="7"

       log.syslog="off"

       # need to turn log stream logging off!

       log.file="/tmp/stats.log")

# Provides UDP syslog rec

#$ModLoad imudp 

#$UDPServerRun 5514

#$UDPServerRun 5515

#$UDPServerRun 5516

# Provides TCP syslog reception

$ModLoad imtcp

$SystemLogRateLimitInterval 0

$IMUXSockRateLimitInterval 0

$IMJournalRatelimitInterval 0

#$MaxMessageSize 64k

#$InputTCPServerRun 5514

#$InputTCPServerRun 5515

#$InputTCPServerRun 5516

$DebugFile /var/log/rsyslog.debug

$DebugLevel 2

#$InputTCPFlowControl off

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;

# local messages are retrieved through imjournal now.

$OmitLocalLogging on

# File to store the position in the journal

$IMJournalStateFile imjournal.state

$template CustomTemplate1,"%msg%\n"

$template CustomTemplate2,"%msg%\n"

$template CustomTemplate3,"%msg%\n"

#$template gelf, "%$now% %HOSTNAME% %app-guid%\n"

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                        /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

#*.* /var/log/custom_message;CustomTemplate

*.info;mail.none;authpriv.none;cron.none        /var/log/messages

# The authpriv file has restricted access.

authpriv.*                       /var/log/secure

# Log all the mail messages in one place.

mail.*                         -/var/log/maillog

# Log cron stuff

crown.*                         /var/log/cron

# Everybody gets emergency messages

*.emerg                        :omusrmsg:*

# Save news errors of level crit and higher in a special file.

uucp,news.crit                     /var/log/spooler

# Save boot messages also to boot.log

local.*                        /var/log/boot.log

# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

#

#

$InputTCPServerSupportOctetCountedFraming off

$RuleSet remote5514

*.* /var/log/oagaudit

$RuleSet remote5515

*.* /var/log/oagmonitor

$RuleSet remote5516

*.* /var/log/oagaccess

$InputTCPServerBindRuleset remote5514

$InputTCPServerRun 5514

$InputTCPServerBindRuleset remote5515

$InputTCPServerRun 5515

$InputTCPServerBindRuleset remote5516

$InputTCPServerRun 5516

#

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

$ActionQueueFileName fwdRule1 # unique name prefix for spool files

$ActionQueueType LinkedList

#$ActionQueueFileName /var/log/fwdRule1 # unique name prefix for spool files

$ActionQueueSize 8000000

#$ActionQueueLightDelayMark 0

$ActionQueueHighWatermark 4800000

$ActionQueueLowWatermark 650000

#$ActiondeQueueBatchSize 20000

$ActionQueueDiscardMark 6400000

$ActionQueueDiscardSeverity 7

#$ActionQueueFileName /var/log/fwdRule1

$ActionResumeRetryCount -1

#$ActionQueueTimeoutEnqueue 60000

$ActionQueueTimeoutEnqueue 0

#$ActionQueueDequeueSlowdown 10000

#$ActionQueueMaxDiskSpace 50g # 1gb space limit (use as much as possible)

$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList # run asynchronously

#$ActionResumeRetryCount -1  # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

#*.* @@remote-host:514

$ActionQueueWorkerThreads 4

*.* @@oag-admin102-lab-usc1-a.dev.companyxyz-gcp.com:5514;CustomTemplate1

In our case the OAG logs are ingested, processed and forwarded by rsyslog server to the remote rsyslog/syslog server farm (setup for enterprise log harvesting) and then eventually to Cloud Logging on GCP. Log Explorer and Log Analytic Tools can be leveraged to view and analyze the logs. 

Similarly the Okta system logs can be sent to Okta Log collector (log fetched periodically based on Okta Log API ) on Grafana and Grafana Plugin for cloud logging can be utilized to view the cloud logging logs centrally on Grafana.  

Please refer to link below for Okta Log collector and integration details https://grafana.com/blog/2024/08/07/how-to-integrate-okta-logs-with-grafana-loki-for-enhanced-siem-capabilities/ 

Solutions like Grafana can be leveraged to centralize monitoring and visualization of logs in heterogenous cloud environments. Please check https://grafana.com/ for more information.

Grafana is an open source interactive data-visualization platform, developed by Grafana Labs, which allows users to see their data via charts and graphs that are unified into one dashboard (or multiple dashboards!) for easier interpretation and understanding. You can also query and set alerts on your information and metrics from wherever that information is stored, whether that’s traditional server environments, Kubernetes clusters , or various cloud services. It’ll be easier to analyze the data, identify trends and inconsistencies, and ultimately make your processes more efficient.

About the Author: Bala Panatpur, CISSP, is a Certified Technical Architect at Okta and has over 30 years of experience in the security industry. His deep technical expertise spans IAM, Directories, Governance, Security Integrations and Observability technologies.