input large SQL query -Cache and run

Question

Hello,

I am trying to find the best way to run a large sql query and was wondering if there was a way that after the initial run it could create almost like a save point and then when I refresh the following week be able to just add on the new data since it takes so long to run? Is that possible? Does the cache and run function do that on the input tool?

thanks,

Blake

bmcmanu1 · Answer

Hello @Felipe_Ribeir0 is it possible to send a copy of that workflow to me?

Felipe_Ribeir0 · Answer

Hi @bmcmanu1

If you have some column that can be used as reference to separe old data from new data, you could do something like this:

Load all the table data to a yxdb file:

Then, you can use the column that can be used as reference (in this example last_updated_date) to query just new data when the workflow run again. For this, you can use the dynamic input.