Please help with simple incremental load

[Ns89]

My input is a yxdb file. My output is a qvx file and a separate yxmd file with the max date of the data.

This max date of the data will be used as an input for the next time the workflow is run (the next day).

How am I able to filter the input the next day so that it uses the max date of the previous day as a filter for min date?
This will pick up all records from the yxdb which have been added since the previous run.

Any help appreciated!

Thanks

Neekul

[Thableaus]

Hi @Ns89 

Could you please share your workflow (at least a picture of it) with us?

Cheers,

[nerces · Accepted solution]

If your workflow is definitely ran EVERY day, you can use the formula below in your filter tool. The formula will get the current date and minuses one day from it. This returns yesterday's date.

[MaxDate] > DateTimeAdd(DatetimeToday(), -1, "days")

NB: I repeat, this will only work if you run the workflow every single day. If you skip a day, then you're going to filter out new records that shouldn't be filtered out.

[Ns89]

Thanks @nerces - It doesn't seem like its possible to use a separate "custom" input as a filter.

I thought this quite basic functionality would exist but apparently not!

However, your solution would work in this case as I would run every day.

Thanks