This article will discuss the root cause and potential workaround when Okta encounters a Slack Provisioning failure due to the following error:
Rate Limit Exceeded
- Slack
- Provisioning
- Okta Integration Network (OIN)
- System for Cross-domain Identity Management (SCIM)
Based on the findings from Okta Engineering's investigation, Okta will only encounter this Slack Provisioning error when there are too many Profile Sync jobs being triggered and sent to the Slack SCIM API remote server, which causes the server to hit the Slack API rate limit - Too Many Requests error (with 429 error code).
This normally occurs due to large push profile updates or large group assignments triggering numerous push user creation/profile update jobs to the Slack API server. According to previous customers who reported the issue, this Slack API rate limit exceeded error will be observed when over 100+ push user creations or push profile update requests are sent to the Slack SCIM API server. However, this could vary for each customer depending on their Slack Rate Limit quota.
The Slack Remote API server throws this error as it reaches Slack's Rate Limit.
The Slack provisioning error occurs not because of an Okta Rate Limit but because of the Service Provider Slack's API rate limit in their remote SCIM API server. Due to this, not much can be done from an Okta configuration standpoint.
It is worth mentioning that when Okta hits the Slack Rate Limit, which causes the Slack SCIM API call, during the Okta/Slack provisioning job, Slack will respond back to Okta with a Retry-After header in the Slack SCIM API response header. In which OKTA will automatically retry the remaining failed app user profile sync jobs based on the number returned by Slack's Retry-After header.
In other words, the Okta Admin does not need to manually click to retry a failed task from the Okta Admin Console > Dashboard > Tasks page, as a retry job has been automatically scheduled to run in the Okta backend based on the Slack Retry-After header returned by the Slack SCIM API response for each failed Slack provisioning job due to the Slack Rate Limit Exceeded error.
Please note that, depending on the total number of Slack profile sync jobs that hit Slack's Rate Limit Exceeded error in the Okta org, it could take a couple of job retries until the job eventually stops hitting Slack's API rate limit/429 Too many requests error on the Slack Remote API server. It could take a couple of hours to fully clear out all previously failed Slack provisioning jobs with the Rate Limit Exceeded error.
Monitor the Okta Admin > Dashboard > Tasks page and locate the failed Slack app assignment/app provisioning task count. The failed task count should reduce automatically over time until it eventually reaches 0 for the Slack Rate Limit Exceeded error.
NOTE: As a best practice, when making large Slack group app assignments (over 100+ Slack app assignments by group assignment) or any Slack attribute mapping (Okta > Slack) updates, which could trigger over 100+ Push User Profile updates when the mappings are saved, the recommendation is to perform these changes after business hours to minimize the impact.
