Upload data in SQL in batches

Question

I am trying to copy a table from one SQL server to another. The data size is huge 3M+ rows, it will take almost 6+ hours to copy this . The workflow ends abruptly sometimes at 200k rows, 500k or 1M rows whenever my system sleeps, wifi connection issue or anything else. Is there an alternative to copy data from one SQL to another ? Anything that can be done to expedite. Please note I'm very new to Alteryx. Have tried SQL import export wizard as well but that too stops in between. I want a solution within alteryx

caltang · Answer

Is there a dimension which you reference the split? For example, if your data is granular, you can split it by Month-Year. Or by product... etc.

What sort of data are you working with?

The back-up of back-ups would be to stream the data out as YXDBs, then stream the data back via write access in to your 2nd SQL Server.

Designer07 · Answer

There are total 75 Columns. How do I upload in chunks ? Create separate workflows ?

caltang · Answer

Errr.... I would suggest for you to really understand what you're trying to do first. If you're trying to move from one SQL Server to another, I do not think Alteryx is the best tool -> https://www.vinchin.com/en/blog/sql-server-database-move-to-another-server.html

But if you insist on Alteryx, then you will need to break your data into chunks. 3M+ rows is okay, how many fields are we talking about here? The larger your frame, the slower it is.

Refer below:

https://knowledge.alteryx.com/index/s/article/How-To-Migrate-Entire-Databases-Using-Alteryx-1583460623496

EDIT: Here's something relevant for consideration: https://community.alteryx.com/t5/Alteryx-Connect-Discussions/Migrating-Data-Sources/td-p/443175