Processing 1.5 Billion records with Designer

[grossal]

Hello community,

I have been doing some tests with the Designer and BigData lately.

It seems that a great amount of RAM can help to increase the processing speed but CPU doesn't. 

I have 10 core (20 logic cores), but the CPU usage is always around 10-15% (peaks at 20%). The RAM on the other side takes as much as I allow him to used (set to 105 GB right now).

My two questions are:

1) Is it possible to enable the Designer to make us of more cores?

2) Would a server make use of more cores? (for a single workflow)

(We have server at work, but I don't want to push 1.5B records to the server if it isn't faster)

Thanks in advance

Alex

[danilang · Accepted solution]

hi @grossal 

Check out the answer from @KevinP in this post.  He explains the whys and wherefores about the multi-threading strategy in Alteryx.  The post is from 2016 but I think the concepts are still valid.  tl:dr The Alteryx engine multi-threads where it makes sense to.

Dan 

[MarqueeCrew · Accepted solution]

@grossal ,

RAM and fast local SSD are your best bet. If you joine the beta program (https://betaprogram.alteryx.com) there is a e2 beta that multi-threads. 

cheers,

 mark

[grossal]

Thanks @MarqueeCrew and @danilang.

I would have loved an option to force it to multi-threading and give the user the chance to decide it.

I have requested access to the Beta on my Partner Account over two month ago, no sign yet. I would love to test it. Especially with my QM background and all the bugs I already found in the 2020.1 it would haven been a good addition, but I guess I have to wait.

Have a great day guys!

Alex