p8eko (p8eko) asked a question.
'Delayed' events from 'System Log' API
I setup two pipelines ingesting data from 'GET /api/v1/logs'.
One syncs every minute, the other syncs every 2 hours.
Comparing the records, based on Okta-returned UUID, every minute pipeline was missing >2% events that showed up in the every 2 hours pipeline.
Documentation says:
Not all events for the specified time range may be present — events may be delayed. Such delays are rare but possible.
But 2-3% is NOT RARE.
What sync frequency is recommended to get complete records from the API endpoint?
Parameters send to API: '{"since":"max(published) formatted 'YYYY-MM-DDTHH:MI:SS.FF3Z'","until":"current_timestamp() 'formatted YYYY-MM-DDTHH:MI:SS.FF3Z ","sortOrder": "ASCENDING","limit":1000}'
"events may be delayed" I take that statement to mean that it is unknow when delays will occur and logs get added at a later time. The longer you wait to pull logs the more likely you are to get most of the events.