community
cancel
Showing results for 
Search instead for 
Did you mean: 

Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Parallel simultaneous running of workflows

Asteroid

Hi Everyone, 

 

I have searched the threads on several occasions for direction on how to run multiple workflows in parallel but without much success, so am reaching out to you wiser and wiley'er users for some input if you can help.

 

Some details

Basically, I have a workflow copied for each customer,

The workflow has 7 inputs from a sql database (1 database per customer, identical schemas, same DB server, azure based),

each of the 7 input goes through its own independent flow with no cross flow dependencies, and each then writes to  7 YXDB files on local machine.

How many we're going to St Ives?  

This is the start point of a longer chain of workflows, and so I also have the reverse of this same scenario at the other end of the workflow when the transformed data goes back to the SQL database.

I have designer running on a very well spec'd machine, and also have access to scheduler on same machine.

The goal is just to parallelize the the input downloads within the workflow, so its can be become part of each customers process. There's no need (at least not yet) to run parallel for everyone, and each customer can run independently.  

 

I've done various tests with the crew list runner and conditional runner macros, which are fab, and often the go to for orchestration, but nothing I've tried seems to work. I've read some scant details that hint at using command line and the command tool but nothing I've seen seems specific enough to follow or apply to my use case.

 

Has anyone managed to crack this with anything yet.

 

Most grateful of any direction.

 

best

 

W

 

 

Alteryx Certified Partner

Hi @Dubya ,

 

You mentioned that you looked at the runner macros to go around this, have you tired something like the setup below? Wouldn't this trigger a parallel run of the individual workflows?

 

parrallel workflow.PNG

 

Regards

Karam

Nebula
Nebula

Hi @Dubya 

 

Pardon me if this question seems basic, but are your seven workflow streams identical, i.e. aside from the input and output can you pass any customer data through any of the chained workflows?  If so, you could attach a customer ID column to each of the input streams and then union the data together?  This will mean modifying the internals of the workflow to group by customer whenever necessary, but it leaves you with just one workflow that will work for one or one hundred customers?

 

Dan

 

Asteroid

Hi Karam, thanks for the input. I did try this config, and the 3 test flows do have different start end and write timestamps. See below. So have taken that to mean they did not run in parallel.

 

clipboard_image_0.png

 

 

Would you agree with that, or have I missed something. Flows were all identical, reading same input through identical flows.

Asteroid

Hi @danilang 

 

No the streams are not identical.

 

The workflow is customer specific, and is downloading 7 separate tables of different Google analytics data (ie sales, pages, searches etc) which have been loaded to SQL via another process. 

 

Each table is then processed through a slightly different tool stream in order to render it usable for the next stage of processing for that customer.

 

I'm not sure I completely follow you, but what I think you're suggesting is that I combine some combination of the tables in the import somehow, and then group them by customer?

 

Would it help to elaborate, or has my response disqualified the idea? 

 

best & thanks

 

W

Nebula
Nebula

Hi @Dubya 

 

From what I've been able to gather, the Designer engine is not completely multi-threaded.  There are some tools that will use multiple cores if available, but most are limited to running in a single thread.  In your specific example, the Runner macros are executing in sequence.  There appears to be techniques where you can launch multiple instances of Designer and each of them will run in parallel.  

 

Is your request for parallelism based on the need for speed or the need to have all outputs available at the same time to be picked up by your down stream apps?

 

Dan

Nebula
Nebula

Hi @Dubya 

 

Since you have differences in way you process the various customers, my suggestion wouldn't work.

 

Dan

Asteroid

No probs @danilang . Appreciate you stopping by to help.

 

many thanks

Nebula
Nebula

Hi @Dubya 

 

Did you look at my other post focusing on parallelism in Designer?  The links I posted there may help you along

 

Dan

Asteroid

Hi @danilang 

 

Its speed...

 

tenor

 

I'll check out the thread and see if there's any inspiration there.

 

thnx

 
 
Labels