Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.
Don't forget to submit your entry for the Excellence Awards by October 30! | Need more information about the program? Check out the blog here

Handling huge datasets

Highlighted
5 - Atom

Hi, I have a bunch of macros doing some processing on the data. All these macros are strung together in a workflow. The challenge is, this won't work if I have huge datasets (>50M records maybe). What is the best solution to make sure I'm able to overcome this in Alteryx? Should I consider alternatives, such as coding everything in Python instead?

Highlighted
14 - Magnetar

 Hi @akhanna3,

 

what's the exact issue with large amounts of records?

 

I have recently processed 1.7b records in Alteryx and I am currently doing performance tests with the AMP Engine and 170M records. It works just fine. Can your explain you issue in more detail?

 

 

Best

Alex

Highlighted
Alteryx
Alteryx

Hi @akhanna3 ,

 

Alteryx copes well with big data sets. Limitation is your hardware though:

How much RAM does your machine have? No expert here, but minimum should be 16 GB.

 

In case your 50 Mio. records are stored within a database, I recommend to leverage the In-DB tools, which allow you to process the data already within the database:

https://help.alteryx.com/current/designer/database-overview

 

Please let me know whether this helps you.

 

Best regards

Phil

Labels