JSON API Data Download Limitations

[pbaba]

I am trying to pull some data down through an API but I seem to be limited to 100 records of data (JSON Name 3) at a time. I believe I need to run an interative macro but I do not have the number of pages only number of total records. Does anyone know how I can get the full data set (or 883 records)? 

[christine_assaad]

Hi @pbaba 

Please check this post: https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Loop-through-API-calls-until-all-data-...

I believe it's related to pagination which could be handled using a macro as described above. Hope this helps. Cheers!

[pbaba]

The issue I have is that I do not have a way of knowing how many pages there are. There is a Cursor field but I am not sure if it plays a role with pagination.

[danilang]

Hi @pbaba 

The methods for handling pagination are API specific.  Sometimes there's a link to the next data chunk in the DownloadHeaders field.  Sometimes there is an offset parameter that you send in the URL.  Look into documentation of the specific API that you're calling. 

Dan 

[pbaba]

@danilang  Not sure if you can help me. This API Uses a cursor for pagination and I need to add cursor to the original URL as a prefix when I run the iterative macro so that it pulls subsequent data until all the data is pulled down. Any assistance you can provide is greatly appreciated

[danilang]

Hi @pbaba 

I did a quick search of the API documentation here and I can't find any details on the correct format of paginated requests.  You'll have to work with them to get more details

Dan

[pbaba]

@danilang  Please see attached documentation.

[danilang]

hi @pbaba 

There's nothing in this document that specifies the format of the payload if you want to start from a record other than 0.  You'll have to reach out to them.

Dan 

[pbaba]

@danilang Thank you very much for your insight. It is appreciated.