Loop through workflow and append new date ranges

[skatofia]

Hi All,

I'm trying to loop through my date ranges to append gradually Redshift input to this output in Oracle. I could just re-run my workflow many times and change the date ranges, but is there a way to go around this and make the job run once and append new date ranges to my data each time?

Thanks.

[ehughes]

Hi @skatofia 

Based on what you described, I wonder if something like this could work:

Input both sets of data. Find the Max date in the Oracle dataset, append this to the RedShift data. Filter to just get values in the RedShift data that are greater than the existing Max in the Oracle data and then Union the new dates/data to the original Oracle data (and output as needed).

I've attached the workflow which may be overly simplified but hopefully something like this could work