INCREMENTAL LOAD

Question

## Incremental load

Iam trying to work on incremental load in Alteryx,iam having two tables with same data and column as we load by full load,suppose one table is dil_supplier_ds from oracle database and another table DIL_SUPPLIER_D from snowflake.
 NOW,IN DIL_SUPPLIER_DS WE UPDATE ONE ROW AND INSERT ONE ROW ,AND COLUMNS WE HAVE SUPPLIER KEY,COMPANY,LAST_UPDATE_DATE.

DIL_SUPPLIER_DS

SUPPLIER_KEYLAST_UPDATE_DATECOMPANYDSP0103/02/2002ABCDSP0204/02/2002ABCDDSP0308/05/2005ABCDEDSP0408/08/2005DELTA

SO USING FULL_INCREMENTAL LOAD I LOAD SAME TABLE INTO SNOWFLAKE,AS DIL_SUPPLIER_D.

DIL_SUPPLIER_D IN SNOWFLAKE

SUPPLIER_KEYLAST_UPDATE_DATECOMPANYDSP0103/02/2002ABCDSP0204/02/2002ABCDDSP0308/05/2005ABCDEDSP0408/08/2005DELTA

NOW ,I ADDED ONE ROW AND UPDATE DSP02 ROW.

THIS UPDATE TABLE IN DIL_SUPPLIER_DS IN ORACLE:

SUPPLIER_KEYLAST_UPDATE_DATECOMPANYDSP0103/02/2002ABCDSP0208/08/2003GIGADSP0308/05/2005ABCDEDSP0408/08/2005DELTADSP0508/08/2023BELTA

SO I WANT A WORKFLOW,WHICH CAN INSERT A NEW ROW AND UPDATE A NEW ROW IN DIL_SUPPLIER_D IN SNOWFLAKE BY USING INCREMENTAL LOAD

###PLEASE PROVIDE A WORKFLOW FOR REFERENCE.

saraswathireddy · Answer

We are loading data from an Oracle database into Snowflake using Alteryx. The full load process is working seamlessly. However, the issue arises when we attempt to implement the incremental load. Our source data contains a "LAST_UPDATE_DATE" column, and our goal is to compare this column in the target table to identify new records and updated records, then only update these specific records in the target table, achieving an incremental load.

We've gone through the Alteryx documentation, forums, and online resources to find a solution, but we have not been successful in resolving this matter. We believe that with your expertise and support, we can swiftly address this issue and ensure a successful incremental load process.

If you are not the appropriate person to address this matter directly, we kindly request that you guide us to the appropriate expert who can assist us in resolving this issue promptly. Given the timeline constraints due to our client demo, any assistance you can provide will be immensely appreciated.

If your appropriate person, please find more info  provided below

DIL_SUPPLIER_DS in oracle

SUPPLIER_KEY

LAST_UPDATE_DATE

COMPANY

DSP01

03/02/2002

ABC

DSP02

04/02/2002

ABCD

DSP03

08/05/2005

ABCDE

DSP04

08/08/2005

DELTA

SO USING FULL_INCREMENTAL LOAD I LOAD THE SAME TABLE INTO SNOWFLAKE, AS DIL_SUPPLIER_D.

DIL_SUPPLIER_D IN SNOWFLAKE

SUPPLIER_KEY

LAST_UPDATE_DATE

COMPANY

DSP01

03/02/2002

ABC

DSP02

04/02/2002

ABCD

DSP03

08/05/2005

ABCDE

DSP04

08/08/2005

DELTA

NOW, I ADDED ONE ROW AND UPDATE DSP05 ROW AND CHANGED THE COMPANY NAME TO GIGA(DSP02) IN THE SOURCE.(DIL_SUPPLIERS_DS)

SUPPLIER_KEY

LAST_UPDATE_DATE

COMPANY

DSP01

03/02/2002

ABC

DSP02

08/08/2023

GIGA

DSP03

08/05/2005

ABCDE

DSP04

08/08/2005

DELTA

DSP05

08/08/2023

BELTA

WE TO COMPARE : SELECT * FROM DIL_SUPPLIER_DS WHERE LAST_UPDATE_DATE>(SELECT MAX(LAST_UPDATE_DATE ) FROM DIL_SUPPLIER_D)

AFTER COMPARING THE ABOVE CONDITION, WE HAVE UPDATED THE DSP02 ROW AND DSP05

So, two Rows are to be  Updated in target based on condition.

Thank you for your attention to this matter. We eagerly await your response, and we are committed to working collaboratively to overcome this challenge.

please help with workflow

ArnaldoSandoval · Answer

Hi @Gopiroa

You schema's tables should include at least these columns: "CREATION_DATE" and "LAST_UPDATE_DATE", you need the creation date column to identify new records in your incremental process, while the last update date assist you updating existing record on the target table.

hth

Arnaldo