Addting Incremental dataset to current dataset in yxdb.

Labhesh

 Hi Alteryx Coummunity

I have a question about how to work on incremental data coming from oracle database.

Data Table is of TIMESHEET.

Every week end let say Saturday i will recieve the full load data ( 100milions rows) that inculdes 1year back data from that date, Then after saturday i will recieve everyday incremental data (15-20million rows) for last 3-6 months that include the delete records, update records & new records from that orcale database.

I am extracting from oracle database and all full load data is loaded into yxdb before processing further.

NOW MY QUESTION:

I want to only delete, update & add those records from incremental data that we are receving everyday in after that we recieved on Saturday, i wanted to keep all the previous records that has not been touched, i wanted to automate this task ?

Note: Their is no unique ID's.

FYI i have never worked with incremnetal set of data processing.Any help would be appreciated.

Thanks in advance

shancmiralles

hi! @Labhesh 

if the only records that you will edit are the latest/current records for the week.. you may want to place a number under options (1. Record Limit) of the Input Data tool configuration.  if the data is thousands.. then you can limit it around 100(? just an estimation i don't know your data ) to represent the current week's data that you will edit/modify. 

so you may check if the only data that were pulled were the data for the latest week  -  by dates ( i assume there is a column/field that represents the date). if your date column comes with time just parse them out first.
for the days and do a formula tool to determine which data came in weekday/weekend:  datetimeformat([date],"%A")