Incremental Load concept for Alteryx workflows

[Pankhudri20]

Hello,

I have workflows which will be run daily.

It essentially will load data upto 4 months in past.

However, it takes too long for the workflows to load since data files are huge.

Is there a way that to perform incremental load in alteryx?

Such that every day the workflows append the results to existing output files?

Thank you

Looking forward to a response soon!

Regards,

Pankhudri

[jrgo]

@Pankhudri20 

A setup like the below is probably how you need to set it up. The target file is rewritten every time, but not before it's first read in and union'd with the new records.

[Pankhudri20]

Hi @jrgo ,

Thank you so much for your quick and detailed response.

I have a doubt here:

Can I read past 2 weeks of data every time and append and replace data for dates which are already in output file?

So essentially the new data will be 1 file and the rest 13 days will be re processed. This is because on some days the data file could be missed or uploaded late or the person is on vacation who uploads. So, I want to avoid errors due to missing data.

Can you please tell me how can I handle this situation ?

Looking forward to your response soon!

Regards,

Pankhudri