Schedule hourly without duplicating file loads

dngnc7

Hello: 

I am using the directory tool to load files into a table. I currently have a daily schedule set for 10am but there are times some of the files come in after that time. 

I want to switch to a hourly schedule but stop the flow from processing if the file exists for that day to avoid duplicate files being loaded.

Any ideas on how (if possible) this can be done?

Gawa

@dngnc7 

You can output some log file in yxdb that has information of what files are loaded at that day (name might be "2023-11-07.yxdb")

At the beginning of the WF, you just compare the log file and Directory tool result so as to identify you should process or not.

For this purpose, Detour and Detour End tools can be used for conditional date process.

If you are using 2023.1 or later, and familiar with it, you can also use Dynamic Container tool.