We are working on a solution in Azure Data Factory to auto-deprovision users based on a number of different factors, and are using the API to pull basic information into our Data Warehouse for the first part of the process. I've been able to make the various pipelines work for Users and Groups, but I've run up against a wall when it comes to the App API.

What I want to do is pull all of the User assignments for a single App with their username, but the API only exports 500 users at a time. Unlike the other APIs, the App API "next" link isn't something that I can build using the response from the API call and I am unable to pull the "next" url.

Has anyone been able to get Pagination to work properly for the App API in Data Factory? I don't necessarily need it to work in the initial REST call, if I was able to access the Headers of the response I could in theory find a way to pull the "next" link and pass it back as a variable but everything beyond the Output is obscured.

Ideally, I can make this work inside of Data Factory because then I can be sure that all of the data is updated at the same time.

Thoughts?