Alteryx Designer Desktop Knowledge Base
Definitive answers from Designer Desktop experts.Snowflake Bulk Loader - How Local Staging Works
Created
on - edited on by
HenrietteH
The local staging functionality of the Snowflake Bulk Loader allows you to stage files locally before writing them to Snowflake. This is beneficial if you do not have access to an Amazon S3 bucket to stage files. This article describes the manner in which Alteryx writes files to Snowflake utilizing the local staging mechanism.
1. First, Alteryx writes CSV or GZipped CSV files into a folder in the engine temp directory (Typically %ProgramData%\Alteryx\Engine) named ayx_local_staging_SomeHash. The size of these files is determined by the chunking size set by the user in the file format options of the output data tool.
2. Next, Alteryx tells the Snowflake driver where these files are located, and the driver starts transferring them to the stage in Snowflake that was selected during the initial setup of the connection (Internal Named, User, or Table stage). Alteryx utilizes the PUT command here to transfer the files.
3. After the files have been transferred into the Snowflake staging area, Snowflake copies the data from the staging area into the desired table. Alteryx utilizes the COPY command to move data from staging to the table.
4. Before the workflow finishes, the temp files are deleted from engine temp directory, as well as the files located in the Snowflake staging area that were uploaded by Alteryx.
Additional Information:
Internal Named Stage Error troubleshooting
Snowflake Help and File Format Options
For more information on the staging directories in Snowflake, please see this article.
1. First, Alteryx writes CSV or GZipped CSV files into a folder in the engine temp directory (Typically %ProgramData%\Alteryx\Engine) named ayx_local_staging_SomeHash. The size of these files is determined by the chunking size set by the user in the file format options of the output data tool.
2. Next, Alteryx tells the Snowflake driver where these files are located, and the driver starts transferring them to the stage in Snowflake that was selected during the initial setup of the connection (Internal Named, User, or Table stage). Alteryx utilizes the PUT command here to transfer the files.
3. After the files have been transferred into the Snowflake staging area, Snowflake copies the data from the staging area into the desired table. Alteryx utilizes the COPY command to move data from staging to the table.
4. Before the workflow finishes, the temp files are deleted from engine temp directory, as well as the files located in the Snowflake staging area that were uploaded by Alteryx.
Additional Information:
Internal Named Stage Error troubleshooting
Snowflake Help and File Format Options
For more information on the staging directories in Snowflake, please see this article.
