Upload of 15GB Data - SQL Bulk Upload

Question

Dear Community,

I have a dataset of 15GB which I would like to upload via the bulk load option to a SQL server.

Unfortunately I recognize that t

he bulk load option is incrediable slow. Creating a hyper file takes me about 12 min storing it our company's network drive. But using the bulk loading option is too slow.

I have tried to use as well in-stream function of the in-DB tools, which did not make any improvements in performance.

What I am doing wrong?

Any help is really appreciated!

Thanks, 
Johanna

BrandonB · Accepted Answer

Here is another thread that addresses some ways that you can optimize the SQL server write speeds: https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Increase-Speed-of-SQL-Server-Write/td-p/467493

You should be fine to use the Output Data tool configured with the bulk loader rather than the data stream in tool. The important thing to keep in mind is that you have 15GB of data that you are sending out from your computer, across the network, and into a SQL Server database. This could also be impacted by other things like your network speed and if you are working on a VPN rather than a direct company connection.

An AutoField tool might be useful as well right before the output. This would reduce the data to the smallest size that can accommodate it which could result in faster write times as well.