This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I have created a workflow which calls an API and downloads a set of data through an elasticsearch and then is converted to a JSON. I would like to maximize the number of records I am able to retrieve however the current limit is set to 10000. Does anyone know a way of increasing this limit? Alternatively, I would like to do an API pagination but unsure how to go about it. One of the fields in my dataset provides the number of records so based on this value, whether it is below 10000, I would only like to call the API once from records 0 to 10000. If the number of records is between 10000 and 20000, I would like to call the API twice and union the results. If the number of records is between 20000 and 30000, I would like to call the API three times and union the results and so on. Would this be an iterative macro?
I don't know elesticsearch, but i came across a very similar scenario with SharePoint with 5K limitation for a single read query.
SharePoint is primarily optimised for browser experience and this is the reason for the limitation - the same is present in Microsoft PowerApps and Looker for example.
The idea i have implemented makes the 1st call to check the number of record (or maximum value of the recordset index) - this is represented as total_recordcount in my workflow.
Provided a single call limit/page size, you can use Generate Rows tool to create increment steps and this is then concatenated with the API call string to materialise a series of calls for your Download tool.
The first response given is great; but if you have a scenario where there are an unknown number of pages of the next page link is not a straight forward incriment, and each download provides you with a next page link, then you will need an iterative macro.
This post gives a handy guide on how they can be used for imagination...