Retrieve list of files in a dynamic subset of folders

[LarryRippe]

I have a number of 'tools' (non-Alteryx applications) that run on a different platform which pick up a .csv file from an 'Inbound' folder, processes it and then moves it to an 'Archive folder.  I am trying to generate a list of files that exist in this dynamically changing subset of folders to see which .csv files have not yet been processed.  This workflow is actually run as an Analytic App where a business user can check the status of the tools (performance can be an issue.)

The folder structure is similar to one of the following where <tool name> will come from an existing list of tools but the list does not include the actual Inbound folder name.  The 'Inbound'  folder will always start with 'Inbound' and may also have a string appended to it.

c:\abc\cde\<tool name>\PROD01\Inbound...

Ex:

c:\abc\cde\TOOL1\PROD01\Inbound

c:\abc\cde\TOOL2\PROD01\Inbound_A

c:\abc\cde\TOOL2\PROD01\Inbound_B

c:\abc\cde\TOOL2\PROD01\Inbound_C

c:\abc\cde\TOOL3\PROD01\Inbound

Note - the tool names do not all start with 'TOOL', the above tool names are just used as an example.

I realize that I could use the Directory tool and list all of the.csv file from c:\abc\cde, including sub directories but there are many folders in the structure that do not qualify because of the <tool name> criteria and the Directory tool is taking an unacceptable amount of time to run.

I tried the Dynamic Input tool I was having an issue when the folder did not contain any files.

[IraWatt]

Hey @LarryRippe,

Does specifying a file type reduce the time taken?

[LarryRippe]

No, it didn't.  In fact, that is how I had the tool set up - it took over a minute to complete.

[danilang]

Hi @LarryRippe 

While your processes are running, open a command prompt from the topmost folder and run the following command 

dir *.csv /s >out.txt

and time how long it takes.  This is pretty much the fastest way to recursively traverse a directory tree.   On my machine this takes about 1 second to walk a tree with about 300 nested subdirectories returning about 2200 files.     Compare this with the time it takes a Directory tool to traverse the same tree.  Again on my machine, it takes about 1 second.  

You may find that it even the DOS command takes a minute to walk the tree, especially if you're traversing a network drive  

Dan

[LarryRippe]

Yes, it is a network folder.  My hope was to be able to only perform the Directory tool against a defined list of folders where the list of folders is retrieved from a table in a database - kind of like a Dynamic Input but in this case, I need a Dynamic Directory tool.

[danilang]

You can try putting in a Directory tool in a Batch macro and passing the list of directories as a control parameter.  Inside the batch macro, have the action tool update the directory attribute

Depending on how many directories you need to search compared to the total number of directories on the drive, this could be a faster option.

Dan

[LarryRippe · Accepted solution]

I ended up retrieving a list of all of the files via a Directory tool and then applying a couple of Filter tools to reduce the number of rows that need additional scrutiny where I am searching the PathName for a defined list of values and finally performing a count to determine the number of outstanding files for each tool.  I was able to get this functionality to run in 20-30 seconds.  While still not ideal for an 'interactive' function, I believe it will be acceptible.

Thank you for your assistance.