Root Cause Analysis - Feature Disruption - 09/13/2017
Published: Sep 18, 2017
Updated: Jun 22, 2018
Root Cause Analysis: Feature Disruption September 13, 2017
Problem Description & Impact
On Tuesday evening, September 12th, Okta deployed the weekly release 2017.36. Following the release, customers with Active Directory mastered end-users experienced an unexpected increase in users who became locked-out of Okta. In cases where the end-user entered a locked state within Okta, the locked state persisted even after the end-user’s account was unlocked in Active Directory and subsequent login to Okta was attempted. This required a change in behavior on the part of the Okta Administrator to manually unlock the user in Okta. Alternately, the end-user could self-service unlock themselves when that option was configured/enabled by the Okta Administrator. The unanticipated increase in end-user lockouts continued until Sept 13th, 2017 @ 10:40am PDT when the issue was resolved by Okta.
Okta determined that the root cause of the increased number of end-user lock-outs was attributed to the deployment of a new password-policy soft lock feature (for customers with the Universal Directory SKU) as part of the production release of 2017.36. The new password-policy soft-lock is a security feature which enables access to Okta to be restricted when an Active Directory-mastered users' corresponding Windows accounts are locked, or when users exceed a specified threshold for failed login attempts to Okta.
This new password-policy soft-lock was thoroughly tested and subsequently deployed to the Okta Preview environment on May 10th as part of release 2017.19. Following the Okta Preview release, Okta monitored for errors or customer reported issues. The feature was subsequently cleared for release to production as part of the scheduled weekly release 2017.36 and documented in the release notes. While the expectation was that user impact would be marginal, the production release notes contained instructions for unlocking users if they encountered any soft-locks.
The issue largely impacted customers with AD mastered users who are locked in Active Directory, as the new soft-lock functionality also places the end-users into a locked state within Okta and required a change in Okta Administrator action to explicitly unlock end-users in Okta after they were unlocked within Active Directory.
Mitigating Steps & Corrective Actions
Following the release of 2017.36 to production, customer feedback revealed that the feature led to greater than anticipated end user impact and/or required admin action. In response, Okta reverted the feature for customers who reported the increase in user lock-outs then reverted the feature at 10:40am PDT for all customers who received it with the weekly release deployment.
Okta is working with several impacted customers to capture all contributing environmental factors to enhance our ability to anticipate future customer impact in similar feature releases.
Okta has reviewed the production release and communications processes related to this event and is implementing internal procedural changes regarding feature enablement and customer communication to minimize risk for features that may change end users’ login flows.
Help Article Feedback
We’re sorry this article didn’t meet your needs. What specifically about the article was not helpful?