Limiting API requests in iterative macro

Question

I have created an iterative macro that fetches data over an Odata API that is paginated. So after each request I have to get the next link and fetch remaining records and so on.

There are more than 1M records and there are some limitations set by the API gateway.

1. Max 100 requests/minute

2. Response payload size < 10MB

Now I want to dynamically change my page size and slow down/amp up the iterations so that I follow both the rules.

So my questions are

1. Is there any way to calculate the response size from the Download tool?

2. Is there a way to make the iteration halt when it hits the threshold of 100 requests/min and continue when it's a new interval?

Thank you for your time in helping my query.

PS. Can't upload the workflow

FlorianC · Accepted Answer

Hi @fardeen9983,

In order to comply with API constraints:

* Make an initial call to get the total number of records so that you are then able to query for all of them using the pagination parameter;
* Use the throttle tool to limit the number of calls per minutes.

fardeen9983 · Accepted Answer

I do get the count of all records, but the requested logic from me is to keep on running the iterative macro like a True while loop until there are no pagination link in the response.

i will look into the throttle tool.

Thanks for the answer. Much appreciated