Dynamic Input: No Fields Found

[shnbrb]

All - Looking for ideas from the community to tackle an issue i have with trying to dynamically load files

I am looking to pull .txt files on any given day. The workflow uses a formula to construct the filepath, then Dynamic input to load the file. The filepath will be in the format; Directory/Date{variable}/subdirectory1/subdirectory2/subdirectory3/file.txt. This works fine for most cases, however some days the file is empty, and on these days the dynamic Input tool returns a "No Fields Found" error, and the workflow stops.

Ideally I would like to structure the workflow such that it can deal gracefully with days that have no data.

Unfortunately using the Directory tool to query the file size (then having workflow based on if filesize=0 then 'do something else') is unfeasible in this case as there are 1,000s of subdirectories under the root directory which makes the performance very slow.

My other idea is to use run command to query the filesize of the specific file prior to running the dynamic input, however feels a little clunky.

Any other help greatly appreciated.

[DiganP]

@shnbrb I would use the directory tool and use the filter tool to filter out size=0. I just tested with 1000 files, nested in the folder and it ran in one second.

Another method - You can try checking on the box in the dynamic input tool, first row contains data. This is make sure you have atleast one dataset in the file. You can then use the filter tool to filter out that record. 

[Claje · Accepted solution]

Hi,

I'd create a macro for this to create a formula input for your directory tool.  That way you can filter to the specific date and limit the directories.

I've attached an example that should accomplish this - change the formula to include your path, and the macro immediately after it to include the file specifications.

Hope this helps!

[shnbrb]

This is super helpful!

Yes, this is the kind of thing I was looking for, ie a way to 'input' into the directory tool to limit the subdirectories that are searched. I hadn't thought of building a macro to change the input to the directory tool.