Upload of 15GB Data - SQL Bulk Upload

[Jo-B]

Dear Community, 

I have a dataset of 15GB which I would like to upload via the bulk load option to a SQL server. 

Unfortunately I recognize that t

he bulk load option is incrediable slow. Creating a hyper file takes me about 12 min storing it our company's network drive. But using the bulk loading option is too slow. 

I have tried to use as well in-stream function of the in-DB tools, which did not make any improvements in performance. 

What I am doing wrong?

Any help is really appreciated! 

Thanks, 
Johanna

[BrandonB · Accepted solution]

Here is another thread that addresses some ways that you can optimize the SQL server write speeds: https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Increase-Speed-of-SQL-Server-Write/td-...

You should be fine to use the Output Data tool configured with the bulk loader rather than the data stream in tool. The important thing to keep in mind is that you have 15GB of data that you are sending out from your computer, across the network, and into a SQL Server database. This could also be impacted by other things like your network speed and if you are working on a VPN rather than a direct company connection. 

An AutoField tool might be useful as well right before the output. This would reduce the data to the smallest size that can accommodate it which could result in faster write times as well. 

[Jo-B]

Dear Brandon, 

many thanks for the help! I have added the auto-fied to it. 

I am now writing the hyper file directly to the Tableau server, which definitely smaller than the data process through the workflow. 

Many thanks and stay healthy!