Need to Build a feeder to update file with incoming data stream

[bryanmason19]

Hi all,

I am currently in the process of building a workspace to update our quarterly reporting. I was wondering if there is a way to create a "feeder" tool where I can drop in a new data stream that will run through a filter and blend with the original data. I am trying to simplify the process going forward so that I do not need to constantly change the input and output files each quarter. Basically I am looking for a way to use 1 workflow that I will not have to continually make changes to when new data arrives. So, in short, I would be able to take the new data from the end of every quarter and simply drop it into my workflow where it will update the output file without needing to add new connections each time. Any help or advice is greatly appreciated.

[michael_treadwell]

It sounds like what you want is possible but the details are too general for me to give you a solution. Could you create an example workflow that would explain your goal?

[bryanmason19]

I created a small example of my workflow....Each input file is new data gathered from the most recent fiscal quarter. I am looking for a way to continually update my output file without having to create a new input stream each quarter. For updating purposes, I input a new excel file to the workflow and simply use the union tool to join the streams together...I am trying to eliminate having to create additional inputs everytime I add data becuase eventually the workflow will become very messy. After doing more research I am guessing the best way would be through the use of a batch macro? I do not need the function to do any mathematical formulas, I simply need it to blend/stack with the data that is currently there. So, my output file will continue to grow as more and more data is entered in every quarter.

[michael_treadwell · Accepted solution]

Would something like this work for you?

In the example I've posted, the Text Input represents the new data from your new fiscal quarter.

The Input tool would be reading all of your old data.

Simply Union new and old, check for duplicates (this could be more complicated for you, I don't know your requirements) and then write back to the same data source that you read from Input.

Each time you run this workflow, it will write any new data to the output and prevent any duplicate data.