This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
Check out the latest post: All Models Are Wrong
Note: An AWS S3 bucket can also be used in place of the internal stage.
An easy method is to create a workflow with a Filter Tool.
Use a Customer Filter and set 1=1 in the expression.
Attach an Output Data Tool to the False output.
Configure the output tool with a Snowflake ODBC driver.
The attached v11.7 workflow creates a temporary output file.
The Run Command Tool, creates sql script that is passed as a parameter to the snowsql command.
The script first creates an internal stage in snowflake.
It then copies the temporary output file to the internal storage.
The internal storage is then copied into the table.
Wow, that's pretty exciting and does not require a S3 account, unlike the beta bulk Loader...
Thanks for sharing!
Thanks for creating this! It reduced my upload time to Snowflake from around 30 hours to about 6 minutes. Just a few pointers for others who might be using this.
1) The configuration file for SnowSQL is found C:\Users\XXXXXX\.snowsql where XXXXXX is your user name for login into your computer. I had to put the connection string in there manually. Note that you can put multiple for different connections.
2) The %temp% call in Alteryx didn't work for me as it would truncate the pathway with a ~. Instead I had to hard code it to a location on my desktop. If you need to do this the following changes are needed
Technically, you don't need wither AWS or Azure to Bulk load into Snowflake, since the account comes with a Staging Area.
I developed my own Bulk method using SnowSQL. It is not as streamlined as having that logic built into the Alteryx connector, but it works well:
Someday, perhaps Alteryx will realize that using the Staging Area should be the only way to implement Bulk loading...
I can't find the bulk_load.sql script - should it be in this package or available somewhere else ?
EDIT : Don't worry, I realised the script is created in the last step I just have to get it working 100% now
@cmcclellan - The SQL is in the formula tool inside the macro
@DurgaS - thanks for this. I was planning building something similar so you have saved me a job! One of the features I was planning to build into it was also file splitting (gzip etc) as one of the key features Snowflake offers is the ability to parallel load compressed archives into Snowflake. It would be great if we could have an additional input of "Number of files" or "File Size" and then generate a gzip prior to loading. I'll take a look when I get some time and post it here if I get it working.
Has anyone done this ? - used the logic above to load the data into stage, then create the table with the proper datatypes for each field (taken from Alteryx?) and then load the data from stage.