Replace and Append to existing data

Question

Im trying to speed up the process of my workflow. So currently i have a workflow that pulls back 2 years of data and outputs to tableau server.

Im trying to only pull back the last 7 days and replace any updates and append these records to 2 years of data. I dont want the L7 days duplicated on the append output. Just update data if changed in the last 7 days.

Thanks looking forward to tips and tricks

mceleavey · Answer

Hi @Nickvv ,

Do the initial load and drop out the file.

Next time you run the workflow, apply the filter for the last two years, load the initial file you dropped out and join on the key. The records you join you already have, so only take the new input records that don't match. These are the new records from the last 7 days.

Union them to the raw input from the file you output initially and output.

This will be the updated file with the new records.

Without seeing your data I can't really give you any more detailed instructions, such as how the data is coming in, is the data a file or record level etc.

M.