Processing Large CSV files and long process flows

[hooshiarnejad]

hi

I am very new to alteryx... I am coming from the RStudio world so I am used to dataframes and executing code chunks until you get what you are look for.

In ALteryx how can I avoid repeating the entire process if I am designing and testing new parts of the process.

I have a large dataset consisting of 10 million entries and if I re run the process every-time I make a change it takes several minutes which makes the design experience very frustrating.

In R I would just save the data in the Dataframe and continue to execute new chunks of code until I get it right.

Not sure how to do this in Alteryx.

Thanks

[SPetrie]

You can use a sample tool on your input to sample a smaller set of the data and save the output to a yxdb and use the yxdb for building and testing.

You can also right click one of the tools on your canvas and choose the "cache and run workflow" option which will run and cache the workflow run up to that tool and significantly increase runtime speed. Sometimes that option is not available depending on where you are trying to cache from though.

[ArtApa]

Hi @hooshiarnejad - There are a few ways how you can reduce the time while designing your workflows. Broadly these can be grouped into two categories: Caching and Record Limits. You may learn how to cache your workflow and limit the number or records here: https://community.alteryx.com/t5/Interactive-Lessons/Workflow-Testing/ta-p/747948#done

Also, converting your csv to yxdb will definitely help.

[joesparty]

Hi @hooshiarnejad ,

Also, consider the tool container, to disable parts that are working while you develop new parts.

Cheers