Getting Log data via Powershell Invoke-WebRequest Skip to main content
https://support.okta.com/help/answers?id=9062a000000xaheqa0&refurl=http%3a%2f%2fsupport.okta.com%2fhelp%2fanswers
How satisfied are you with the Okta Help Center?
Thank you for your feedback!
How satisfied are you with the Okta Help Center?
1
2
3
4
5
Very Dissatisfied
Very satisfied
Enter content less than 200 characters.
Ask Search:
Jason LutzJason Lutz 

Getting Log data via Powershell Invoke-WebRequest

I am trying to write a script that gathers 1 minute worth of log data, filtered by API endpoint, to populate an on-prem dashboard.

## Okta Data Gathering ##

$now=Get-Date -format s
$time=(Get-Date).AddSeconds(-60)
$since='{0:s}' -f $time
$filter=$null
$token="mytoken"
$baseurl="https://mysite.okta.com/api/v1/logs"
$headers=@{"Accept"="application/json"; "Content-Type"="application/json"; "Authorization"="SSWS ${token}"}

$filter="debugContext.debugData.requestUri%20eq%20`"/api/v1/authn`""
$uri="$baseurl"+"?filter="+"$filter"+"&since="+"$since"

$output=@()
$data=$null

$i=1
    do{
    $data=Invoke-WebRequest -Headers $headers -Method GET -Uri $uri
    $output+=$data
    $check=$data.Headers.Link -match "next"
        if($check -eq "True")
        {
        $uri=$null
        $nextrun=$data.Headers.Link
        $uri=$nextrun -replace ".*,<" -replace ">.*"
        }
        else
        {
        $i++
        }
    }
    until($i -gt 1)

To sort-of translate, the script generates an ISO8601 timestamp a minute in the past for the since= part of the uri.  The $filter is where I define the filter= portion of the URI, the apikey is blanked here, and the baseurl is there as well.  All of that is pieced together into a full URI for the Invoke-WebRequest.  The headers and URI are passed in the Invoke-WebRequest, and boom I get data back.  But I only get 100 lines.

In the API documentation, it indicates that a value of "rel="next" will be passed in the Header data when it is received.  Along with a URI for the next chunk of data.

So, I check the headers information for the "Link" and if it matches the word "next", it cuts everything out except the "next" uri link, and sets that as the $uri variable for the next run.

If it doesn't match the word "next", it adds 1 to $i making it greater than 1, and should stop the loop.

The problem is, the loop never stops, because I keep getting "next" passed back.  In the API documentation, it said that it will only pass "next" if there is more pages of data for the query.  

When I query the same query over the same time span in the Okta Admin panel, I get maybe 300 lines.  

But the script will run till my api/v1/logs limit is reached, then 429 me.  Last time I ran, i stopped it on the 8th loop, and it indicated I had 800 lines of data.  The same time frame with the same filters in the UI showed me around 280.  

Is the rel="next" always in the Header?  Even if there is no next page?  Or is there a better way to get this information?

-Jason Lutz

TCFBank

Elliott BradleyElliott Bradley
The next link does work the way the documents describe, it will page through the specified set of records requested by the initial request.

I would suggest looking at the contents of the next link to confirm that it matches your query. I had issues when I was writing my Powershell module with correctly formatting my query, specifically around the date, and wouldn't get the right results back. I would suggest using a different datetime format for the datestring. This is the one that I use and it works great:
$DateTime.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffK")
The timestamp needs to be in UTC and Okta's datetime parser is very particular.
Andrei AldeaAndrei Aldea (Okta, Inc.)
Hi Jason, my name is Andrei with Okta Support and I'll try to address your question.

Typically Support does not provide scripts or assistance regarding custom scripts, so unfortunately we can do very little to assist with this directly. However, our Professional Services department can help you create custom scripts and assist you in finding the best way to work around any rate limitations.

You could also reach out to developers@okta.com in regards to this question, as it does involve the Okta API, though I don't know if providing scripts is something they can do.

Andrei Aldea
Technical Support Engineer
Okta Global Customer Care
Jason LutzJason Lutz

@Elliott - You are correct, it was sending the correct data.

I haven't really looked at the outputs completely to verify this, but I think what is happening is that the returned data is more lines of data than the Okta Admin panel displays for the same query.  Not that there is more data, but what is considered a "line" is different between what the Admin panel displays as a log item, and what the API rate-limit considers a line.  

This morning, I set the filter in my script to just gather rate-limit violations and warnings, and the returned data was within the expected parameters.  I think for the filter of /api/v1/authn, the amount of data lines returned for that query is higher than what is displayed in the Admin Panel, so much so, that we hit our 60/min api rate-limit for log data api calls.  This is due to the max 100 line pagination of data breaking a single query into multiple API calls to the /logs/ api URI.

I will be at Oktane17, so I plan to discuss with some of the folks in person about this script, and using the logs API to monitor things like authentications per minute and user api calls per minute.  The purpose of the script was to monitor those api usages to ensure we aren't closing in on our rate limits to avoid outages proactively.  

Thanks everyone for replies!