API calls using pagination

[lipster26]

Hello,

I've been able to use the download tool to connect to an API. The API call only returns 100 items at a time. I understand I can use pagination and create a macro to make multiple calls. However, once I've run the workflow I've downloaded the data needed for that time. The next time that I run the workflow, I don't want to re-download the data that was already downloaded previously. How can I build a workflow to only download the new information from the API?

Thanks

[apathetichell]

is this an API that you can pass a limit to? Some API's (Oracle Financials springs to mind) set 100 as the base retrieval amount but you can overwrite it by including a limit parameter.

[lipster26]

Hello apathetichell, thanks for the reply. 

I'm not too sure. I'm a noob with API's. From the API documentation there is nothing that references limit parameters.

Sample link that gets to the next set of data is ?skip=100&take=100&status=Approved. Not sure if that helps.

[danilang]

Hi @lipster26 

IF the API in question only adds new records and never deletes older ones, you could use a mechanism where you use a placeholder to keep track.  The general steps are  

1 read the previous placeholder from file, database, etc

2 get the next batch of records using ?skip=placeholder&take=100&status=Approved

3 write the index of the last record to the placeholder file, database etc.

The IF in the first statement is a big one though.  You have to be sure that the records never get deleted or you'll end up missing records in your downloads.  A better option would be to just get all the records available at this time and decide which ones you want to process using some criteria.  You can use the placeholder concept here as well.

Dan