Directory input tool - files being grabbed

Question

Hi, first question here in the community so not sure exactly the rules behind adding a new post to the forum.

Anyway the directory input tool and dynamic input tool to pull a file name interests me. I wanted to know which is better. When the tool looks to a folder location to grab my files is it more efficient to grab .yxdb databases or excel files.

We get a file each day and we were selecting the file each day as an input to the flow. We then started making a stacked database each day (prev current) as each day we use a previous days file and current days file to reconcile data. this made an extra step but the time far made up with Alteryx being able to process .YXDB database files faster.

Learning about the Directory input tool and connecting it with a dynamic input tool has shown me it may be possible to pull in new files each day without the operator having to make a selection. Just not sure if we should just pull the original files into the directory tool (possibly making it slower) or continue building the databases each day and have the directory tool bring in .yxdb files each day for faster processing.

Thanks,

Jon

echuong1 · Accepted Answer

The directory and dynamic input are often used in combination with one another. The directory tool will essentially import a list of all of the files in the specified folder(s), as well as the associated metadata - creation time, file extension, etc. That being said, it does not take long to run, and it is not importing a ton of data, so it's not that inefficient to read in an entire folder.

From there, you can narrow down your selection, to only get rows for files that you want to import. For example, if you always want to pick up a file with a certain keyword in the name, and the most recently created, you can filter for the keyword, sort by creation time descending, and use a sample tool to grab the first record.

Once you have your list narrowed down, this would go into a dynamic input to read in each file. This post has some sample set ups in the responses:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Load-multiple-files-from-specific-folders/td-p/393153

Hope this helps!

Jon_Taylor · Accepted Answer

Thank you for the reply and yes that is what I am trying to do. What i was wondering which i guess is self explanatory is that if my directory pulls from a location which contains .yxdb files is the input faster than if the location contained .xlxs files. inherently we know Alteryx can read .yxdb files faster than an excel file. below is a screen shot of my test and the .yxdb files were processed faster. The issue was that we start out with an excel file and make the database .yxdb file to improve the time of the workflow. I have found that by adding a step to make the .yxdb file i can complete the process faster than if I use the raw excel file as my input. I thought maybe using the directory to pull in the excel documents utilizing the Dynamic input tool would make the process faster but in the end it did not. We can use the excel file or add the step to make the .yxdb file. I was looking to try to cut back on adding the extra step but that would add processing time so i have to evaluate user interaction to determine if I want the user to do less but wait longer for results.

Sorry for writing so much as I guess it was more of a discussion than a question.