Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Parallel simultaneous running of workflows

warrencowan
8 - Asteroid

Hi Everyone, 

 

I have searched the threads on several occasions for direction on how to run multiple workflows in parallel but without much success, so am reaching out to you wiser and wiley'er users for some input if you can help.

 

Some details

Basically, I have a workflow copied for each customer,

The workflow has 7 inputs from a sql database (1 database per customer, identical schemas, same DB server, azure based),

each of the 7 input goes through its own independent flow with no cross flow dependencies, and each then writes to  7 YXDB files on local machine.

How many we're going to St Ives? 🙂 

This is the start point of a longer chain of workflows, and so I also have the reverse of this same scenario at the other end of the workflow when the transformed data goes back to the SQL database.

I have designer running on a very well spec'd machine, and also have access to scheduler on same machine.

The goal is just to parallelize the the input downloads within the workflow, so its can be become part of each customers process. There's no need (at least not yet) to run parallel for everyone, and each customer can run independently.  

 

I've done various tests with the crew list runner and conditional runner macros, which are fab, and often the go to for orchestration, but nothing I've tried seems to work. I've read some scant details that hint at using command line and the command tool but nothing I've seen seems specific enough to follow or apply to my use case.

 

Has anyone managed to crack this with anything yet.

 

Most grateful of any direction.

 

best

 

W

 

 

25 REPLIES 25
warrencowan
8 - Asteroid

thanks @danilang for the threads. The download of data is the main bottleneck in the process, at least for this workflow anyway, and I don't see any parallelism capability there I'm afraid. 

 

The only thing that seems able to do it is the scheduler which is presumably running the alteryxengine.exe, and means breaking the workflow into 7 smaller workflows and putting each of them in individually.

 

Although I'm all ears if you or anyone comes up with something better.

JoeS
Alteryx
Alteryx

@warrencowan wrote:

thanks @danilang for the threads. The download of data is the main bottleneck in the process, at least for this workflow anyway, and I don't see any parallelism capability there I'm afraid. 

 

The only thing that seems able to do it is the scheduler which is presumably running the alteryxengine.exe, and means breaking the workflow into 7 smaller workflows and putting each of them in individually.

 

Although I'm all ears if you or anyone comes up with something better.


I think that's going to be the only way round it if it is the inputs that are the bottleneck.

 

Alteryx Designer within a workflow doesn't have a way of running multiple inputs at once (AKA multi-threading them). So the workaround is for you to split them into different workflows and run them at the same time yourself (or via the scheduler) and ultimately achieve multi-threading in a somewhat manual fashion by creating multiple Alteryx Engine's running at once.

warrencowan
8 - Asteroid
Thanks @JoeS

I guess I needed to hear it was the only way.

Given that were throwing this to the engine, is it possible to run multiple workflows via the command line or even a batch file run by the run command tool.

JoeS
Alteryx
Alteryx

It is possible to run via command line with the desktop automation add on to designer, or on a server.

SteinErikHansen
5 - Atom

If I understood this correctly a potential solution could be to have a sequence of workflows wrapped up in a runner macro workflow or as chained apps:

1. Initial prep

2. Run command tool running a batch file that triggers several jobs for the different processing steps

 

It would be much appreciated if anyone with more experience with the  would be willing to provide an example of triggering several jobs in parallel via a batch file if the idea makes sense. 

 

Maybe a feature request, that at least in some cases would solve the parallel processing issue, be to have the option of triggering multiple apps in a chained app setup (limited by configuration options for max simultaneous jobs)? I'm not quite sure about how the chained apps are executed behind the scenes, but it might be something there.. any thoughts?

 

I guess scheduling and further dependencies would still be a problem .. hmm..

 

EDIT:

Another idea would be to just use Events - Run Command referencing instead of the chained apps setup

RuchikaMangla
8 - Asteroid

Hi @Karam @warrencowan @JoeS ,

 

I have a quick question, does crew runner macros works only on alteryx server? because I am unable to run it on alteryx designer (2019.3). Just curious to know if this also works on server version only similar to Run command tool.

 

Thanks!

JoeS
Alteryx
Alteryx

@RuchikaMangla wrote:

Hi @Karam @warrencowan @JoeS ,

 

I have a quick question, does crew runner macros works only on alteryx server? because I am unable to run it on alteryx designer (2019.3). Just curious to know if this also works on server version only similar to Run command tool.

 

Thanks!


Hi @RuchikaMangla 

 

It's actually the other way round. CReW runner macros are supported for designer but not Alteryx Server.

 

What's the error you are getting?

RuchikaMangla
8 - Asteroid

Hi @JoeS 

 

thanks for your prompt reply!

 

this error I get "CReW macro error: "SupportingExes\AlteryxRunner.exe" returned an error code: -1073741819" error.

 

JoeS
Alteryx
Alteryx

Hi @RuchikaMangla 

 

On looking into the error it looks like it might be due to some missing C++ redistributables.

 

I have found this link that I'd recommend trying initially to see if you have any success: https://answers.microsoft.com/en-us/windows/forum/all/solved-error-code-1073741819-when-running-cert... 

RuchikaMangla
8 - Asteroid

Thanks, @JoeS 

 

I will try it after installing C++ missing redistributable. Thanks for your help, Appreciate it!!

Labels