File Watcher Jobs using Alteryx Server

[Krishnasn]

I have seen many existing discussions on this topic but I want to understand this with some more clarity.

How teams are implementing the functionality using Alteryx when there is a need to trigger a Alteryx job in the server on receiving the source file in a folder?

Thanks in advance,

Krishna

[fmvizcaino]

Hi @Krishnasn .

The best option is to trigger using an RPA solution, such as UiPath, Automation Anywhere, Power Automate, Mulesoft RPA, etc. 

If you don't have that, you can have the workflow running every X minutes (not ideal, I know) and only proceed if a new file is available in the folder.

Best,

Fernando Vizcaino

[Krishnasn]

Hi Fernando @fmvizcaino ,

Thanks for your response.

Can you please elaborate or provide a sample of how we can put the logic in a workflow to proceed if a new file is available in the folder?

Are you talking about 2 different workflows, then how can we set the dependency

if not, then is this possible in a single workflow?

TIA,

Krishna

[fmvizcaino · Accepted solution]

Hi @Krishnasn ,

This can be done with Control Containers (2023.1+). https://community.alteryx.com/t5/Engine-Works/The-Ins-amp-Outs-of-Control-Containers/ba-p/1314495

You can have a directory tool in the first container, and when a new file appears, the second container is triggered.

The rule to identify a new file could be using one of the dates generated by the directory tool, something in the filename, or you could keep a simple file containing all the files that have already been processed. 

The second container will process everything if you have a file available after the filter tool. If you don`t, the container won`t be triggered and the rest of the process will be skipped.

Best,

Fernando Vizcaino

[Krishnasn]

Thank you very much. 

You are right as it is an overhead for the Alteryx Server to keep running the workflow.

But I will see if this can be implemented as using a control container to execute the logic only when the file is encountered would significantly reduce the job runtime.

Regards,

Krishna

[fmvizcaino]

The good part about the control container is that you will only have the workflow running for a second. It's not ideal because your schedule forecast will add many workflows, but the total runtime will be minimal.

Thanks,

Fernando Vizcaino

[Krishnasn]

I'm curious to know how can an iterative macro be used to implement this functionality. I came across this post but it seems to be archived.

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Event-Driven-Workflow-Triggers-for-Alt...

Do you have any leads on this approach?

[fmvizcaino]

The iterative macro idea is to keep your workflow running endlessly until a file is identified, which would probably trigger another workflow using the server APIs.

The main issue with the approach is that you would have a workflow running 100% of the time.

[Krishnasn]

On keeping the workflow running endlessly, I have the below thoughts --

• Why we tend to say that it is better to have some lightweight orchestration tool to handle this outside of Alteryx?
• How would the other orchestration tool manage this? there are also it has to be a never ending job that would run some scripts  (batch, powershell or python) to keep checking for a file.. so wouldn't that be overhead too?
• If we do this in Alteryx using iterative macro, then what would be the outline of it to find a file? I have the below steps in mind, if you could verify and correct those in case something missed?
  1. Macro will have iteration input with a list of files coming in
  2. It will pick one file at a time and pass it as an output to the main workflow
  3. Macro will have the remaining files as iteration output
  4. Max Iteration has to be defined as 12 (i.e. if we want to check every 5s and our main workflow is scheduled for every 1 min) 
  5. The main workflow will contain:
     • Directory tool (to fetch the list of files) -->
     • Formula & Filter (Check the required file format/name/timestamp) -->
     • Iterative Macro (to take 1 file at a time and then send remaining files for iteration) -->
     • Main Logic (To run logic for the 1 file picked up from macro) -->
     • Archival process (to move the files from source folder to to archive folder)

Thanks & Regards,

Krishna

[fmvizcaino · Accepted solution]

Hi @Krishnasn ,

• Why we tend to say that it is better to have some lightweight orchestration tool to handle this outside of Alteryx?
  • Apples and oranges 😁 Folder trigger is more related to RPA solutions. Alteryx is more of a data manipulation tool that you can customize a lot and achieve some of these things but it is not built for it.
• How would the other orchestration tool manage this? there are also it has to be a never ending job that would run some scripts  (batch, powershell or python) to keep checking for a file.. so wouldn't that be overhead too?
  • Alteryx is licensed by the number of cores (basic server license = 4 cores) and it is costly. By adding an endless process to the server, you would keep other workflows from running simultaneously and being in the queue
  • A batch script or an RPA solution will use less memory and resources to achieve the same results since it was built for it.
• If we do this in Alteryx using iterative macro, then what would be the outline of it to find a file? I have the below steps in mind, if you could verify and correct those in case something missed?
  1. • If you are planning to run the workflow every minute, I`d say it is better to use control containers.
     • If you are planning to have a workflow with an iterative macro, you will also need to have multiple schedules because there is a max 10M iterations allowed, also your schedule would finish after the file is identified, so you would need to use the Gallery APIs to trigger another workflow.

Best,

Fernando Vizcaino