Addting Incremental dataset to current dataset in yxdb.

[Labhesh]

 Hi Alteryx Coummunity

I have a question about how to work on incremental data coming from oracle database.

Data Table is of TIMESHEET.

Every week end let say Saturday i will recieve the full load data ( 100milions rows) that inculdes 1year back data from that date, Then after saturday i will recieve everyday incremental data (15-20million rows) for last 3-6 months that include the delete records, update records & new records from that orcale database.

I am extracting from oracle database and all full load data is loaded into yxdb before processing further.

NOW MY QUESTION:

I want to only delete, update & add those records from incremental data that we are receving everyday in after that we recieved on Saturday, i wanted to keep all the previous records that has not been touched, i wanted to automate this task ?

Note: Their is no unique ID's.

FYI i have never worked with incremnetal set of data processing.Any help would be appreciated.

Thanks in advance :)

[shancmiralles]

output is in oracle too? or xlsx?
i'd use formula tool for this. 
formula tool can help you identify the data came in from mon-fri and sat-(if there is) sun.. 
from there, you can call the data from mon-fri so you can update them. 

@Labhesh 

[Labhesh]

@shancmiralles 
hi

output will be in xlsx,

but the thing is i am extracting data from oracle database both full load and incremental.

i have not worked with incremental data can you elborate the formula use in it 

thanks 

[shancmiralles · Accepted solution]

hi! @Labhesh 

if the only records that you will edit are the latest/current records for the week.. you may want to place a number under options (1. Record Limit) of the Input Data tool configuration.  if the data is thousands.. then you can limit it around 100(? just an estimation i don't know your data :) ) to represent the current week's data that you will edit/modify. 

so you may check if the only data that were pulled were the data for the latest week  -  by dates ( i assume there is a column/field that represents the date). if your date column comes with time just parse them out first.
for the days and do a formula tool to determine which data came in weekday/weekend:  datetimeformat([date],"%A")