Having trouble writing workflow to snowflake.

Hello all!

I have been running an alteryx query to snowflake (designer), 2.8 millions rows. Previously, I was not doing the bulk upload and it was taking 24 hours just to get to 75%

Now when I do a snowflake bulk load, it gets to 100% quickly, but the query never finishes.

I attached an image of what I am seeing, this has been at this current view or about 35 mins. Ay insight as to what I can do/why this is happening would be helpful.

image.png

Input

Settings

Error Message

Tips and Tricks

Connectors

Common Use Cases

Help

Dynamic Processing

Download

Data Investigation

Documentation

AMP Engine

AMP-Engine

Accepted answers

All comments

apathetichell

1) Turn off Amp.

2) are you staging locally via bulk - or using an S3?

3) Can you try datastream in - vs output data?

4) Do you see the query id in your snowflake ui? What does it say there?

cmcclellan

So in your Output Data tool you're using bulk load to Snowflake?

I use my own macro (that I developed back when Alteryx didn't use internal staging). Snowflake should load 2.8 million records in a few seconds if you've done it the right way, but a single INSERT for each record will take an amazingly long time (and high cost) because that's how Snowflake works.

apathetichell

we should probably ask whats going on upstream? is there something this is waiting on?

kpontarollo

Hi! @apathetichell thanks for the help!

1. I tried turning off AMP and it still appears to be having the same problem.

2. I am staging locally... I am the only person in my company who uses alteryx so there is no s3.

3. When you say datastream in... do you mean then output to snowflake with what?

4. In snowflake, I can see the table name but that there is 0 data in it.

Any further insight is gratefully appreciated! This has been driving me nuts!

kpontarollo

Nothing that its waiting on, its just an alteryx DB to snowflake

kpontarollo

What is a "single insert" and how would I know if I am mistakenly doing that?

apathetichell

A single instert is non-bulk loaded Snowflake data - ie each row is a distinct API call/query. if you limit your sample size to 100, and set up logs in your ODBC 64 - you will be able to open your logs and see either 1 call to write your api (with 1 instert statement with all values) - or 100 insert statements.

cmcclellan

Can you post more details about your Output Data tool configuration please?

In Snowflake terms, you can't stage locally (as far as I know) - there's only internal staging or external staging)

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703