Speed up workflow/canvas when using Python tool

[podoseb]

Hello, I'm recently having some trouble managing the canvas of my workflow that uses some Python tools, and I'm not referring to the Run-time.

Basically, I noticed that when I have Python tools active on my canvas (within enabled containers), every single click I make on any tool requires a lot of time to load up the configuration window (up to 10 seconds and more), no matter the tool i choose to edit. This problem goes away if I disable the containers that include all the Python tools when I make the changes. Now the thing is, I'd be fine editing while the container is disabled, but the tools that use the outgoing stream from the Python container don't keep the metadata, making it impossible to actually make the changes before enabling the container again.

I tried running the workflow first, closing the Python container and editing the tools configuration, but as I said it won't let me do it since there is no metadata to work on. I also tried making all connections wireless, which helped me speeding up other gigantic workflows in the past, but not here.

I also tried putting the Python tools in Production mode (no idea if it's useful).

Don't know if I explained myself well. Anybody else got this kind of trouble? And maybe has some advices? It's so frustrating to work with each click needing 10 secs to elaborate (and it doesn't depend on the machine, since it's a pretty good one).

Using Alteryx 2021.1.

[atcodedog05]

Edit : Saw your comment that production mode didnt help.

Production mode should speed up.

[podoseb]

Yeah I read so as well, but from what I've understood Production mode speeds up the Running process, while it shouldn't slow down nor speed up the whole configuration process on the canvas (unless I'm configuring the Python tool itself). Anyway I did try that but made no difference... 😥

[danilang · Accepted solution]

Hi @podoseb 

You have 2 options available 

1.  In User Settings-> Advanced check Disable Auto Configure.  This will stop the metadata refresh that occurs every time you click on a different tool.  You can force a manual refresh when required by pressing F5.

2. Cache the workflow after the Python tool.  This saves a copy of the data to a temp yxdb.   From that point on the metadata should come from the .yxdb and not the previous tools.  I say should because I haven't tried this yet, so post back here with your results

Dan

[atcodedog05]

Why didnt i think of suggesting that cache & run workflow i use it all the time 😅. That will speed up the execution, but dont know about canvas handling lets see 😶

Check and let us know @podoseb 

[podoseb]

Exactly what I was looking for!

Although the Cache trick doesn't work for me in this case, or at least I find it pretty complicated since the data stream goes back and forth for different reasons and there is no clear point to cache the data, the first advice worked as a charm! I can finally click on the canvas without getting the loading icon 😀

But just to check if I understand the function correctly.

By stopping the metadata refresh, does it mean that each tool is in a bubble? So if I add for example a new field with a formula tool, the subsequent Select tool connected to the Formula will not see that new field unless i refresh the metadata by pressing F5? Or am I getting it wrong?

Btw, thanks a lot! Appreciate the advice.

[danilang]

@podoseb 

That's correct.  Any change you make in one tool, is not reflected in the downstream tools until you refresh.

Another trick I've used in the past is to take long running tools and write the output to an explicit xydb file. 

1. Add an output tool after your Python tool and write the data locally. 

2. Put a container around both the python tool and the output tool.   

3. Add an input tool that reads from the yxdb. Put this in a separate container.

4. Add a union tool connected the output of the python tool outside the first container

5. Connect the output of the input tool to the union tool.  

You should have something that looks like this

Disable the input file container and run the workflow once to populate the .yxdb.  Disable the python container and enable the Input container.  You now have the python tool disabled, but the Metadata still updates since it comes from the .yxdb.  Things should speed up enormously

Dan

[atcodedog05]

Hi @danilang 

I dont use python tool much. But these are some great trick to use when working with python tool. Bookmarked them.

Thank you for the knowledge share🙂