Bulk data loading from SQLServer to Greenplum

Question

Need an advise here,

i'm pulling a large dataset from SQLServer (10M rows) and need to move into Greenplum DB

I tried both with connecting using Input Data (SQLServer) and Output Data (GP) and also Connect In-DB (SQLServer) and Write Data In-DB (GP)

Any approach is taking a life to complete at the point that i have to cancel the process (to give an idea, over the weekend it ran for 18hours and advanced no further than 1%)

Any good advice or trick to speed up these sort of massive bulk data loading would be very very highly appreciated!

thanks in advance,

Regards,

Erick

DanM · Accepted Answer

@eamador,

If you are just moving data from SQL to Greenplum, Alteryx really is not the tool to just move data from one database to another. Alteryx brings in the data and creates temporary files to process the changes with your data. This is why you can play in Alteryx and never make changes to the original data. Alteryx is also a memory intensive tool and if your machine does not have the memory capacity to handle large amounts of memory use then you will experience slowness. I would imagine if you checked how your memory availability was on your machine you would most likely see it pegged to 100% when running that workflow. So if you are just attempting to move that data and not doing any changes to it you are essentially moving it twice.

You can find technical specifications here

DanM