Processing 1.5 Billion records with Designer

Question

Hello community,

I have been doing some tests with the Designer and BigData lately.

It seems that a great amount of RAM can help to increase the processing speed but CPU doesn't.

I have 10 core (20 logic cores), but the CPU usage is always around 10-15% (peaks at 20%). The RAM on the other side takes as much as I allow him to used (set to 105 GB right now).

My two questions are:

1) Is it possible to enable the Designer to make us of more cores?

2) Would a server make use of more cores? (for a single workflow)

(We have server at work, but I don't want to push 1.5B records to the server if it isn't faster)

Thanks in advance

Alex

danilang · Accepted Answer

hi @grossal

Check out the answer from @KevinP in this post.  He explains the whys and wherefores about the multi-threading strategy in Alteryx.  The post is from 2016 but I think the concepts are still valid.  tl:dr The Alteryx engine multi-threads where it makes sense to.

Dan

MarqueeCrew · Accepted Answer

@grossal ,

RAM and fast local SSD are your best bet. If you joine the beta program (https://betaprogram.alteryx.com) there is a e2 beta that multi-threads.

cheers,

mark