Scheduler to run workflow every time contents of specified folder increase

[AkisM]

I'm not sure if this should be asked in the designer or server forums, but I think this is a fairly common requirement so there should be a few solutions:

In my workflow, I specified a specific folder using directory input+dynamic input, and all files from that folder are pulled each time the workflow is run, and they are output as separate .csv files.

The problem is, I don't want the workflow to run through all files and re-export each time it is run. It would be inefficient. I want it to instead run only for the new files that were dropped in the folder. Is this possible? Would I need to use the filter tool in my workflow and filter by date created and take only the ones that were created in the last minute? And have the scheduler run this workflow every minute? Seems inefficient, especially if i have 15 of these workflows, and the scheduler runs 15 workflows 24/7 every minute.

What would be the best way to solve this problem, or the most simple way?

Thanks in advance.

[MichalM]

@AkisM 

What you described is definitely the most straight forward way of building such logic. Question is, do you need the process to run every minute?

[AkisM]

The reason I want it to run at the shortest interval possible is so that as soon as the user drops the file in the hotfolder which should be transformed by alteryx, they should have the transformed output ready. There shouldn't be a 10 minute delay because the scheduler runs the workflow every 10 minutes.

Can't the scheduler or workflow be event triggered? Like triggered by the event of a folder contents (hosted on a server unrelated to alteryx) changing?

[MichalM]

@AkisM 

You could trigger a workflow published on the server using REST API. Alternatively, you can add a workflow execution to a scheduler queue via a command line. Both of these actions would however need to be triggered by a third party solution.

An alternative could be to build an analytical app which takes the file as an input and provides the desired output via the Gallery web interface. 

[AkisM]

I came across this comment by @RolandSchubert on a relevant thread:

"I think there is no way to implement "event-triggered" execution of a workflow with Designer only.
If you have Scheduler, you could start a workflow from command line and the Windows task 
scheduler provides the ability to start a process triggered by directory/file events such as new
files, file updates.

Roland"

Which seems to imply you don't need a third party app to do this other than maybe the windows scheduler. We do have the alteryx server/gallery/scheduler license, which means we also have the AlteryxEngineCmd.exe utility referenced above.

But does this mean that every user will need alteryx+alteryx engine cmd on their computer? This defeats the whole purpose, as the idea is, user scans document, document ends up directly in hotfolder in excel/csv format, workflow is triggered by detection of new file in hotfolder, workflow is ran automatically from alteryx gallery, workflow spits out transformed excel for the user who doesn't have any alteryx tools installed on their pc.

This is the reason I would like to avoid the "user specifies input file in gallery" solution. I want it to be as no-hands as possible.

[MichalM]

@AkisM 

When you said files were dropped into the folder I thought it was the user dropping them there. By third party I meant an application other than Alteryx. So you can set up a task using Windows talk scheduler on the Alteryx Server machine which will trigger your workflow when a new file lands in the hotfolder. 

[AkisM]

Thanks @MichalM . If I'm understanding correctly, only a specific machine has the server installation files on it and I have to speak with my IT department to set up windows scheduling on that specific machine, correct?

Also, if the scheduler and workflow are triggered on that machine only, will all gallery users still be able to see the workflow's results on their own machines if they navigate to alteryx gallery, workflow results? I thought the workflow results part only shows results of apps that the specific user ran.

[MichalM]

@AkisM 

Correct. The task will need to be configured on the server machine with Alteryx server installed.

Because the process will be executed outside of the Gallery environment and will not be executed by the users, the results of the workflow will not be visible via the Gallery interface.

[AkisM]

What if I create an app that chains 2 workflows together, to create a workflow loop that runs perpetually using events? Would that take up resources and freeze up the server and cause it to not operate correctly with other workflows?

The idea is both workflows do exactly the same thing, except one has the event that triggers the next to begin after it runs, and the next triggers the event to run the previous workflow after it runs.

Also the first workflow counts number of files in folder, and it writes that number to a spreadsheet. Then the second workflow counts number of files in the folder again, and it references the result vs the number on the that spreadsheet. If the file number detected by the 2nd workflow is higher than the one written on the spreadsheet, the workflow would produce an output of the last (n-Difference) files sorted by creation date.

[MichalM · Accepted solution]

The event triggered workflows are executed out of the "field of vision" of the server service layer. If your server is configured to run 2 jobs simultaneously, the event triggered jobs will run outside of this limit which could affect resource consumption and hence performance of the server. 

Running the workflow every minute sounds like the better option here.