Dynamic Directory Tool

[#1]

1. Challenge:

I needed to poll the Alteryx temp directory for files, to pick up work to process. 

The Directory tool is a "root" tool and runs once before all your other tools, regardless of position in the workflow, so it would never see the files.

Alteryx also has a different temp directory on each run, and the file names I was looking for were also different on each run.

The Directory tool has an interface that takes a static directory location and static file specification pattern, so I had to make a Directory tool that could take input from the stream, and run at the moment it received the stream data.

2. Solution:

The simple solution is to use a batch macro with Control Parameters, and replace the static strings representing the target directory and target file specification.  I also made the sub-directories flag dynamic, whilst I was down there.

3. Value:

I can now pass [Engine.TempFilePath] into the Dynamic Directory tool.  Also, I can create a unique file pattern to ensure that the processing job processes only the relevant files.  I can run two instances of the general processing run, and each instance will create its own job files, and can run it's own polling routine that share the same temp directory but have their own files.

The Dynamic Directory tool is general, and I can re-use it in any situation where the files it is trying to upload are not necessarily there at the start of the run, or where the directory and/or file specification depends on data (ie a file or directory with a date in it). 

Notes:

The Sub-directories field is text, and will take any common positive value like "1", "Y", "True".  To get the full list or expand this, you can peep the macro and change if you need.

The macro is unlocked so you can see how the general principle can be used to make any tool dynamic.

An additional benefit is that you can do directory queries of multiple directories and/or file specifications in the same run by passing multiple records to the tool.  For example, if you have files of interest that might possibly be in different places, you can find them by putting all the possible directories into the tool, one record each.  Similarly, if you have multiple possible file patterns, you can pass those patterns into the tool as records.

Note: I have included an example workflow to show how the tool is used, and an example data file to push data into the example workflow.

[BS_THE_ANALYST]

@Hiblet thanks a bunch for this! This is really useful 👏.

I guess it's worth pointing out that a user will need to ensure one of the following values is used on the main workflow for scanning subdirectories:
("TRUE","T","YES","Y","1")

@Hiblet, this part is useful to know:

"The Directory tool is a "root" tool and runs once before all your other tools, regardless of position in the workflow, so it would never see the files."

Have you played around with Control Containers much? At least this could change the execution order, should you need to force it 🤔.


Question: @Hiblet how do you think this tool would behave on Alteryx Server? 😊.

Again, awesome tool, and a very cool solution ✅

All the best,
BS

[Hiblet]

Hi @BS_THE_ANALYST , Thanks for the feedback, good to know that this is useful to other folks.

I have not used Control Containers much, I must admit.  I think Alteryx sees root tools (the tools that start data streams with no input themselves) as things that should run first to generate records for other tools.  I shamelessly used ChatGPT to work out a solution to this, and it suggested a control parameter as a "kicker", just a passive control to make the tool wait for input, even if you don't use the passed in parameter.  Then I thought, if I am passing in a parameter I may as well use it.  That lead to 3 params and the tool you see there.

I see no reason why this tool would not work on Alteryx Server - famous last words.  The macro just wraps a standard tool, so it should be good.  I wanted to keep the macro open so people could see that it was simple and "vanilla".  Alteryx Server loves vanilla.

I have added a note in the text about the Sub-directories over-ride text value, good suggestion.

Good luck,

Steve