Python Tool Dragging Performance

[danrom]

I have seen multiple threads on this and and tried everything I could. I have a pretty basic workflow that has starts off say ~5000 'identifiers'. Some cleaning is done to those identifiers and then they are pushed into a macro. The macro then places each identifier though the python tool which is making API calls (the API is DLL driven / local) and pulling information about the 'identifier' and outputs the results / does a little bit more cleaning of the data / then exports to a tab delimited txt file. Everything is working as intended and very quickly EXCEPT the python tool. Is what I am trying to achieve not suitable for the alteryx python tool? Any tips to materially optimize my performance in python? Also worth nothing - running in the server does make it faster but still not where I need it to be. For example: 1 'identifier' takes ~30 seconds to make it through in the designer. 

[TheOC · Accepted solution]

hey @danrom 

The Alteryx Python tool has what i would call a 'headroom', each time it is used. For instance, every time it is initiated, it takes ~5 seconds to start processing. The issue with your method, is using a batch macro, it will be having that headroom each time the macro runs - or for each record you process.
The solution for this, i think, would be to remove the python tool, and the processing before it, out of a batch macro. You can instead, use a Python for loop, to iterate through the identifiers you are bringing in, and processing them within the python tool as a loop - rather than through a batch macro as a loop.

I hope this helps!,
TheOC

[danrom]

Hi @TheOC,

Thank you for your message - I made the changes in my python script and it made a massive difference in speed. Unfortunately - even in our server it would still take ~3hrs to run everything. Need to hit the drawing board on my python and optimize. 

Thanks again!