Download from API request delivers only 1000 records (?) Need all of them

[Archaeopteryx]

Hi,

Alteryx Designer V2022.1.1, I'm running an API request for data against a Quick Base table. I'm using Alteryx Designer on the laptop. I should receive over 200K records, but I'm getting only an even 1,000 in a standard output tool saving to a csv file on my laptop. I get the same results saving to a csv file on OneDrive - an even 1,000 files. 

How do I overcome this record limit? 

Thanks

Chris

[apathetichell]

99% likelihood this an internal feature of the API you are using. Usually there is either a) a more/count feature of the payload return which tells you how many records there are, how many your received, and where the next link is. b) there may be a limit parameter you can pass to your api. that can be set to a default of 1k records but may be raised. c) the API may not suppor it because of rate limitied. Which specific resource are you hitting in the QuickBase API?

[danilang]

Hi @Archaeopteryx 

Check out the info on the Pagination page of the QuickBase API.  The pagination metadata will either appear embedded in the DownloadData or may be returned in DownloadHeaders.

You'll need to develop an Iterative macro to loop retrieve all the pages.  You can find an example on the Data School UK site 

Dan

[Archaeopteryx]

Hello Apathetic and Danilang,

Thank you for your advise. You're both right. Rather than a query string of something like "?page = ...", there is a parameter in the request body called, "skip:" So, I programmatically incremented the skip: parameter of the request body 1000 records per each request. Thanks. Chris