Strange Iterative Macro behaviour

[jevon]

Hello,

Coming across some weird behaviour regarding iterative macros. I've used the same macro in two different cases. First case I've attached a select tool to the output of the iterative macro. The second case, I've attached a sort and select tool to the output of the same macro as shown in the picture below. The top stream (without the sort tool) takes way way longer than the bottom stream. And it's not just for the sort tool. The same happens for the summarise and unique tool, so I'm guessing this behaviour occurs for any tool which performs any sorting (sorting tools: https://community.alteryx.com/t5/Alteryx-Knowledge-Base/Sorting-Processes-in-Alteryx/ta-p/37336). I would've thought the sorting occurs after the iterative macro and therefore would make it slower not faster... Is there a reason why this happens? I've attached the workflow if that helps. Any insight would be greatly appreciated!

@AdamR_AYX 

[danilang]

@Thableaus 

It remains interesting how we're getting different results

If @jevon ever hear's back from Alteryx, maybe, he can let us know what they said.

Dan

[AdamR_AYX · Accepted solution]

This is an interesting one! And you were all close in that it is related to sort/join memory.

The problem comes from the fact that allocating memory can be expensive, so generally we allocate all of the sort/join memory up front and then share it between the tools that need to use it. When the workflow finishes we free that memory.

In the case with the sort in the parent workflow: the sort tool allocates the chunk of sort/join memory and then each time round the iterative macro it "lends" some of that memory to the tools inside the iteration and takes it back at the end of the iteration.

In the case without the sort there is no memory allocated in the parent workflow so each time round the iteration we allocate that memory and at the end of each iteration we have no where to return the memory to so we free it back to the OS. So each iteration has to allocate memory, which is expensive and so the workflow runs slower...

I'll take a look at what I can do about that. Thanks for taking the time to report it!

[Thableaus]

Wow @AdamR_AYX, it makes total sense. Amazing explanation.

I just did a simple example using a single join tool in a workflow (no iterative macro), and added a sort tool in a second workflow with same data.

The second one actually ran faster.

I never thought of that but I'm starting to understand that if you have a Join Tool, adding a Sort Tool is kind of a performance tip for your workflow.

Cheers,

[jevon]

Thanks @AdamR_AYX,

Really great explanation and a really cool insight into how allocating memory works in Alteryx!