Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

General Discussions has some can't miss conversations going on right now! From conversations about automation to sharing your favorite Alteryx memes, there's something for everyone. Make it part of your community routine!

API Download File Size

craigken
5 - Atom

Hi,

 

I ahve create a workflow to connect to a JSON API, and get data, i have tested it retreiving 10 rows and it works fine, as soon as i ask it to get 100 records or 1000, after the API i have JSON parse, from that part onwards the API file size will be 30mb, but the parse and transformation goes up to 100gb in the 100 records instance or over 600gb in the 1000 records instance so i need to kill the process.


Does anyone know why this is?

 

Thanks

neilgallen
12 - Quasar

without seeing the worfklow and the API response it would be hard to diagnose, but one helpful tip would be to drop any unnecessary fields between the download tool and the JSON Parse. In most instances you would have both the URL you requested, as well as the JSON reponse and the returned message (successful, OK, failed, etc). 

 

I'm guessing you won't need those, so just drop them. The JSON parse tool will repeat those fields for each record as it parses fields, so it's a ton of redundancy. If really needed, just have a record ID, and join it back to your data after doing the parse and transformation.

 

Depending on the URL size, other fields, this could help.

VianneyM
Alteryx
Alteryx

hi @craigken,

 

 Since JSON is just simple text, when the JSON construct was developed correctly you can expect to get up to 90% compression. So I don't see it as impossible. 

Best,
Vianney
craigken
5 - Atom

Thanks,


Attached is my workflow, i cant share the API link as this is private work related, i have attached a screen shot of the output. to read in 100 records and put them in the correct order it took over 10 minutes, which is unworkable, as daily i would be reading in 60,000+ records, which has 240 fields. 

 

Any help would be appreciated.

carrjc2
7 - Meteor

Hi Craigken,

 

This post will help.

 

https://community.alteryx.com/t5/Analytics-Blog/SlamData-Alteryx-A-Slam-Dunk-Analytics-Partnership/b...

 

Should be able to easily connect to the API then feed the data into your workflow via the free macro.

 

Jeff

Labels