Input Cache: Having static files and new files in the same folder

Question

Hi!

I have an input tool which pools from a folder all Excel files in it; basically "\*.xlsx". However in that folder there are some static files and every week new files get added.

Is there a way to configure the input tool to cache the existing/static files in the same folder while at the same time to read any new file?

To paraphrase:can the input tool somehow be configured to "cache any file after the workflow, and next week use the cache but read any new file, and at the end cache the new file as well"?

Or do you have other ideas how to design such workflow?

HomesickSurfer · Answer

Hi @MRapp

Bingo!

Example workflow attached.  Extract package to C:	emp\ to test.

Essentially, this reads from/writes to a database containing processed filenames.

If new, process records downstream, if not, no records are processed.

Input new files example.yxzp

Luke_C · Answer

Hi @MRapp

I think it depends on the goal, are all of these files the same schema? Are you just looking to reduce the time of reading in the same files every week if there's only one new file?

If that's the case, I'd suggest reading in all of the files once, and outputting as a yxdb file. Alteryx will read the yxdb file much faster. Then each week union the new file to that yxdb file (the yxdb would be an input and an output). You might also want to consider moving the old files to a 'processed' folder.