This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I have searched the threads on several occasions for direction on how to run multiple workflows in parallel but without much success, so am reaching out to you wiser and wiley'er users for some input if you can help.
Basically, I have a workflow copied for each customer,
The workflow has 7 inputs from a sql database (1 database per customer, identical schemas, same DB server, azure based),
each of the 7 input goes through its own independent flow with no cross flow dependencies, and each then writes to 7 YXDB files on local machine.
How many we're going to St Ives?
This is the start point of a longer chain of workflows, and so I also have the reverse of this same scenario at the other end of the workflow when the transformed data goes back to the SQL database.
I have designer running on a very well spec'd machine, and also have access to scheduler on same machine.
The goal is just to parallelize the the input downloads within the workflow, so its can be become part of each customers process. There's no need (at least not yet) to run parallel for everyone, and each customer can run independently.
I've done various tests with the crew list runner and conditional runner macros, which are fab, and often the go to for orchestration, but nothing I've tried seems to work. I've read some scant details that hint at using command line and the command tool but nothing I've seen seems specific enough to follow or apply to my use case.
Has anyone managed to crack this with anything yet.
Pardon me if this question seems basic, but are your seven workflow streams identical, i.e. aside from the input and output can you pass any customer data through any of the chained workflows? If so, you could attach a customer ID column to each of the input streams and then union the data together? This will mean modifying the internals of the workflow to group by customer whenever necessary, but it leaves you with just one workflow that will work for one or one hundred customers?
From what I've been able to gather, the Designer engine is not completely multi-threaded. There are some tools that will use multiple cores if available, but most are limited to running in a single thread. In your specific example, the Runner macros are executing in sequence. There appears to be techniques where you can launch multiple instances of Designer and each of them will run in parallel.
Is your request for parallelism based on the need for speed or the need to have all outputs available at the same time to be picked up by your down stream apps?