TCFA.18640 (Customer) asked a question.
I am trying to write a script that gathers 1 minute worth of log data, filtered by API endpoint, to populate an on-prem dashboard.
- ## Okta Data Gathering ## $now=Get-Date -format s $time=(Get-Date).AddSeconds(-60) $since='{0:s}' -f $time $filter=$null $token="mytoken" $baseurl="https://mysite.okta.com/api/v1/logs" $headers=@{"Accept"="application/json"; "Content-Type"="application/json"; "Authorization"="SSWS ${token}"} $filter="debugContext.debugData.requestUri%20eq%20`"/api/v1/authn`"" $uri="$baseurl"+"?filter="+"$filter"+"&since="+"$since" $output=@() $data=$null $i=1 do{ $data=Invoke-WebRequest -Headers $headers -Method GET -Uri $uri $output+=$data $check=$data.Headers.Link -match "next" if($check -eq "True") { $uri=$null $nextrun=$data.Headers.Link $uri=$nextrun -replace ".*,<" -replace ">.*" } else { $i++ } } until($i -gt 1)
To sort-of translate, the script generates an ISO8601 timestamp a minute in the past for the since= part of the uri. The $filter is where I define the filter= portion of the URI, the apikey is blanked here, and the baseurl is there as well. All of that is pieced together into a full URI for the Invoke-WebRequest. The headers and URI are passed in the Invoke-WebRequest, and boom I get data back. But I only get 100 lines.
In the API documentation, it indicates that a value of "rel="next" will be passed in the Header data when it is received. Along with a URI for the next chunk of data.
So, I check the headers information for the "Link" and if it matches the word "next", it cuts everything out except the "next" uri link, and sets that as the $uri variable for the next run.
If it doesn't match the word "next", it adds 1 to $i making it greater than 1, and should stop the loop.
The problem is, the loop never stops, because I keep getting "next" passed back. In the API documentation, it said that it will only pass "next" if there is more pages of data for the query.
When I query the same query over the same time span in the Okta Admin panel, I get maybe 300 lines.
But the script will run till my api/v1/logs limit is reached, then 429 me. Last time I ran, i stopped it on the 8th loop, and it indicated I had 800 lines of data. The same time frame with the same filters in the UI showed me around 280.
Is the rel="next" always in the Header? Even if there is no next page? Or is there a better way to get this information?
-Jason LutzTCFBank
- $DateTime.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffK")
@Elliott - You are correct, it was sending the correct data.
I haven't really looked at the outputs completely to verify this, but I think what is happening is that the returned data is more lines of data than the Okta Admin panel displays for the same query. Not that there is more data, but what is considered a "line" is different between what the Admin panel displays as a log item, and what the API rate-limit considers a line.
This morning, I set the filter in my script to just gather rate-limit violations and warnings, and the returned data was within the expected parameters. I think for the filter of /api/v1/authn, the amount of data lines returned for that query is higher than what is displayed in the Admin Panel, so much so, that we hit our 60/min api rate-limit for log data api calls. This is due to the max 100 line pagination of data breaking a single query into multiple API calls to the /logs/ api URI.
I will be at Oktane17, so I plan to discuss with some of the folks in person about this script, and using the logs API to monitor things like authentications per minute and user api calls per minute. The purpose of the script was to monitor those api usages to ensure we aren't closing in on our rate limits to avoid outages proactively.
Thanks everyone for replies!