community
cancel
Showing results for 
Search instead for 
Did you mean: 

Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.
Community v19.6

Looks aren't everything... But the latest Community refresh looks darn good!

Learn More

Running scheduled workflows in parallel

Meteoroid

Hello Everyone,

 

I am dealing with a scenario in which I will have ~150 workflows that I want to run daily on a scheduled run.  I am able to use the list runner macro to accomplish this but as I understand it, that macro will only run 1 workflow at a time before moving onto the next.

 

My goal is to run multiple workflows in parallel to help with the total run time of all these workflows.  I did read through a few posts about using the Run Command tool and utilizing batch jobs but we don't have the license that includes API & Command Line w/ Scheduler features...

 

I am hoping to find a way to accomplish this with the current license, any insight is greatly appreciated!

Magnetar
Magnetar

If you don't have specific dependency needs, you could potentially split your list up and have multiple list runner's in your process which would each attempt to execute simultaneously.

That may provide you with some ability to do parallel processing, although I'd imagine that a desktop computer would have a lot of trouble running more than a few workflows at a time, so you'll probably see a performance hit either way.

Alteryx Partner

Hi @Claje

 

Thank you for sharing your insight,

 

I'm actually busy with the same issue, whereby i have 3 WFs all which are scheduled to run on a previous WF, however i would like them to run simultaneously and not consecutively. I am currently using a conditional runner for each which looks like this: 

 

Xeew34R

 

However they are not running in parallel, how do i solve this?

 

ideally i could schedule each of these to run at the same time using scheduler but these WFs need to run only on completion of a prior WF

 

Thanks,

Asif Kazi

Highlighted
Magnetar
Magnetar

Hi,

 

The thought I was suggesting beforehand was essentially to split the lists up into three separate workflows to run.

In an instance like the one you have listed below, you'd probably have to get a bit clever with Alteryx.

 

Two thoughts on this:

Assuming you are working with Alteryx Server for scheduling, you could potentially leverage the Alteryx API to run each of your workflows.  Documentation on these APIs is available through the Alteryx Gallery.  That way you could send each of these commands to the server quickly, which might resolve your parallel processing issue.

 

Another option is leveraging the Python tool to build something out.


I don't think I can put together something to effectively test this, but there are some really good posts on doing things like this in Python.

 

All of the CReW "Runner" macros use the AlteryxRunner.exe to run the process.  You could theoretically call this process multiple times via the Python tool, which might work.

 

Here's a Stackoverflow post that covers some information on parallel processing via Python: https://stackoverflow.com/questions/4992400/running-several-system-commands-in-parallel

 

Here's another post talking about using ipyparallel to execute in sequence via Jupyter Notebooks: https://stackoverflow.com/questions/39301966/how-to-enable-ipyparallel-for-jupyter

Labels