Single workflow for processing multiple files

[VEDH]

As per business requirement, daily I'll be receiving 6 excel files (1.xlsx, 2 .csv and 3 .xls) from 6 different sources on my email. The challenge here is that the receiving timing of the files is not fixed. Each day file receiving time varies. Hence, I would keep checking the mailbox and whenever the files are received I'd save them down to a particular path.

I wanted to create a single workflow where alteryx will import available excel files, process them and give output whenever I run the workflow. Later when I add new excel files to the path and re-run the workflow, it must only process newly added files and give output.

However, I would also want an option where I can specify if I only want alteryx to process newly added files or process some of the processed file and some of the unprocessed file or all of them. (This is just so that if someday any of the received files was incorrect and later on the same day I receive a revised file, so alteryx must not skip processing it.)

Note: Attached, are few sample files downloaded from google. Hence, data will not be relevant to what I would be receiving and processing.

[danilang · Accepted solution]

Hi @VEDH 

For the automated component, you can use a process similar to this 

The tools in the gray container are used to determine which files to process.  The Directory tool gets the list of files and their last write time.  This is joined with information from the Last Processed Time db and the filter removes the ones that haven't changed since the last run  (see Green container description)   

The tools in the blue container represent your process, i.e. read the files by type, perform additional processing and save the output.   

The green container stores information about the time the process was run.  Union all the file names together and get a unique list.  Add a field with the time that the current workflow started and write the results to ProcessedDB.  I use the time the workflow started as opposed to the current time, since it's possible that a file could be updated between the time it's read and the time the entire workflow finishes.

Replace the Additional Processing tools with the tolls that you require.  Just make sure to connect the select tools in the green container to last tools in your processing container

Note That I couldn't read the .xls file you provided.  I changed the middle stream to filter and treat .xslm files just to get the process to complete, but you'll need to change the filter back to .xls.  

For the User driven process, investigate the Creating Analytic Apps interactive training videos to learn how to add dialog box to capture user input. 

Dan 

[VEDH]

Hi Dan,

Much appreciated on your solution.

Thanks a lot