Streaming Modules to run in sequence

[jmoney]

I have several modules that need to run in sequence.   Can a ksh or some other program be used to string the modules together?  

So that if I have Step1.yxmd, Step2.yxmd and Step3.yxmd.  2 is dependent on 1, 3 is dependent on 2.  I want to tell it the jobs to run and then to wait until that module completes, then run the next.  


run Step1.yxmd
Wait

run Step2.yxmd
Wait

run Step3.yxmd
Wait


Or, if I have a group of modules that can run concurrently but subsequent jobs are dependent on all of them to finish, i can let all those jobs run and have the next step wait for those to complete to begin.  

run Step1A.yxmd
run Step1B.yxmd
run Step1C.yxmd
run Step1D.yxmd
Wait

run Step2.yxmd
Wait

run Step3.yxmd
Wait


I know this is very simplistic, but hoping it shows the need.  Also, we do have the scheduler set up to use, so if there is a way to set dependencies between modules within Scheduler, that might be helpful.  Just haven't found anything.  

Thanks!

[BenG]

Have you tried using events within a module to kick off an additional module after completion?  This may accomplish some of what you are looking for.  Since you have the server version this is an option for you.  Events can be found on the module properties in the events tab.

[AdamR_AYX]

I am working on a Alteryx macro to solve this extact problem!

So my basic idea is that there would be a parent module which would run each of the child modules in sequence. 

I will try and get my prototype to a stage where I can share it today.  Would love to get your feedback on if it solves your problem and any other ideas for additional features for it.

Adam

[AdamR_AYX]

OK I'm not going to get there today.  I have a design flaw that is going to take more time than I have today to fix.

But my basic idea is this:  A macro with a single input, two outputs and one configuration option:



The behavior is as follows:

• If the macro receives any input data it runs its module exactly once.
• If the macro succeeds it outputs its log to the S (Success) output.
• If the macro has errors it outputs its log to the F (Failure) output.

In this way you can string these macros together to run a series of modules in sequence dependent on the success of the previous module.  You can also deal with error handling by attaching things to the failure output.


I'll try and get the prototype version working next week.

But as always any comments/thoughts/improvement/criticism all very welcome.

Adam

[AdamR_AYX]

Here is my initial prototype for my module runner macros

http://downloads.chaosreignswithin.com/community/RunnerSample.yxzp

I hope to have a more finished version of these in my Q3 release of my macro pack (http://www.chaosreignswithin.com/2014/06/blog-macro-pack.html) in 2 weeks, so if anyone has any feedback prior to that please let me know.

Adam

[sandeep_chayanam]

ModuleRunner is quite useful for one of the requirements I'm currently working on. It almost does my work except one piece of functionality.
Is it possible to pass values from one module to another using this macro?

Scenario :

I got 3 modules (Module1, Module2, Module3) perform inserts/updates data in database tables.

I want Module1 to be executed first & within the Module1 , i need to output a value (say RowCount of data which got inserted into a table in Module1), which is needed as input for Module2. & is the same for Module 3.

I think it will be quite useful to have a macro that controls the sequence of execution of modules & alto to have values passed from one to another.

Looking forward to hear your view on how to handle this scenario & any other suggestions are also appreciated.

Sandeep.

[AdamR_AYX]

Hi Sandeep,

Thanks for the feedback. :-)

So the easiest solution that I can think of is to use a intermediate file to pass data between modules.

So Module1 writes to C:PassData.yxdb and then modules 2 and 3 can both read that file in.

The only problem with this approach is that if multiple "Streams" of modules 1, 2 and 3 are running at the same time, then they could overwrite C:PassData.yxdb from each other which woudl be problematic.

Having said that a better solution is going to be tricky.  It will still need to be an intermediate file, but it will need to be a new intermediate file per run and then passing that between the modules will be difficult.

I have a couple of ideas for this route, but need to try a few of them out.  So watch this space!

Adam

[AdamR_AYX]

I have now released versions of these macros as part of my Blog Macro pack here:

http://www.chaosreignswithin.com/2014/09/blog-macro-pack-2014-q3-release.html

and so have removed the download for the prototype versions above.  (I can't work out how I edit that post)
 

Adam

[sandeep_chayanam]

Hi Adam,

(Chris here- having trouble with my login so have borrowed Sandeep's)

Thank you very much for the ModuleRunner, it has been really useful for our projects, but we are having a bit of trouble with it in one case. One of our colleagues is using the "Sharepoint List Input" tool- her module runs fine when run directly in the designer, but hits an error when she tries to run it through the ModuleRunner.

Have you ever come across this? Or maybe used that tool successfully within ModuleRunner?

This is the error we see:
00:00:0.152 - Error - ToolId 1: Could not load file or assembly 'SrcUtils_DotNet, Version=9.1.0.49438, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

I'd really appreciate it if you had some time to look into this- please let me know if there's any more information you would like from us.

Best,
Chris
 

[AdamR_AYX]

Hi Chris,

I'm afraid you have found a bug.  I am seeing the same behavior here.  Thanks for letting me know.

I'll try and take a look at it this week if I get time.

If you want a short term solution you could try copying the AlteryxRunner.exe from MacrosSupportingExes into the Alteryx bin directory (C:Program FilesAlteryxin) and then editing the run command in the macro CReW_Runner.yxmc to point to the new location.  Not an ideal setup but that seemed to get things working for me.

Cheers
Adam

[sandeep_chayanam]

Hi Adam,

Ah great, thank you very much.

We will try the short term fix tomorrow and let you know how we get on.

Thanks,
Chris