Download data from Json API

[Muyi]

Hi,

I try to download and parse json data from an API. I successfully did that but I only got the first 1000 items from the total 18481. Is that due to data size limit? I have attached my workflow.

Thank you!

Muyi 

[chukleswk]

What dataset from there are you trying to pull down? I looked at https://data.sfgov.org/resource/veef-x49p.json and there is only 1000 records in there, where do you see 18481 total records? Otherwise, your flow looks fine to me.

[Muyi]

I am trying to download data from this site, https://data.sfgov.org/Geographic-Locations-and-Boundaries/Street-Intersections/gmfx-8h6i. Data I exported in csv or shp format has 18481 records. Is there something wrong with the API? 

I have another workflow (attached) to download another geometry dataset from this organization, which also returned 1000 records.

[chukleswk]

If you aren't using an App Token you are only going to get 1000 records.

The instructions are located here: https://dev.socrata.com/foundry/data.sfgov.org/gmfx-8h6i

You can sign up for an account and create a new APP Token here: https://data.sfgov.org/profile/edit/developer_settings

I modified your flow to add the App Token and query it to pull back all of your records. Once you have created an APP Token you can add it under the app_token column on the input. Once you have created the app_token and added it to the field, all you need to do is change the get_url in order to pull the new dataset.

[chukleswk]

@Muyi Did this help to solve your issue?

[Muyi]

Thank you so much for your solution and workflow. I downloaded your workflow and I saw the app token is null. Why is the null able to get us all the records?

Muyi 

[chukleswk · Accepted solution]

@Muyi That is a good question! According to all of their documentation it requires a token (which you have to generate), why it works without the token is probably an error on their side which they may have to patch. Glad that it works for you though!