Retrieving Data from API

[kat53191]

Hello,

I am new to pulling data through API's using Alteryx.  I am trying to retrieve data from the following location:

https://data.bts.gov/Research-and-Statistics/Trips-by-Distance/w96p-f2qv

Looking at the description there should be 2.14M rows of data.  For some reason I am only getting 1000 rows of data so I'm assuming there is some row limit?  Would someone be able to take a look and let me know what I am doing wrong?

Thanks,
Katie

[christine_assaad · Accepted solution]

Hi @kat53191 

Looking at the API documentation https://dev.socrata.com/docs/paging.html

It seems to be a Paging limitation. If no limit is specified, the default is 1000 records

You can adjust the payload limit. I set it to 2000 and it returned 2000 records

[kat53191]

@christine_assaad

Thank you so much! I kept trying that in the Header tab and didn't realize it was supposed to go into the payload limit (which makes way more sense).  One other question, is there a way that unlimited could be passed?  Or with API limits do I need to specify the amount?

Thanks again,
Katie

[christine_assaad]

Hey @kat53191 

I tested it with 1M records and it worked. The Doc is saying 50K for 2.0 endpoint and unlimited for 2.1

Below is a nice post about how to loop thru pages when making API calls that you may find useful 🙂

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Need-to-build-Iteration-Macro-to-pagin...

Cheers!