Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Alteryx Designer Desktop Discussions

Find answers, ask questions, and share expertise about Alteryx Designer Desktop and Intelligence Suite.

"COPY failed with error: Unable to remove some files from S3 after the copy operation"

corey_hermanson
5 - Atom

I feel like this is a dumb question, but I just can't figure out the answer. IT at my company changed the S3 bucket I used for loading data into Redshift via the "Data Stream-In" tool. I changed the name and credentials to match this new configuration, but now I'm receiving a COPY error every time I attempt to load data into Redshift. Data does seem to get loaded into Redshift, but since the error is thrown I can't string together multiple Alteryx flows with the "Runner" tool.

 

 

Error: Data Stream In (3): The COPY failed with error: We were unable to remove some files from S3 after the copy operation!

 

 

I've tried searching similar threads and several point to the Alteryx documetation for Redshift: https://help.alteryx.com/current/designer/amazon-redshift

 

There are many configurations listed on that page, but they all refer to connecting via the "Output Data" tool. All of our Alteryx flows use the "Data Stream In" tool to "overwrite table" in Redshift. I then choose (screenshot at end of post): 

 

"Data Stream-In" tool > Configuration > "Manage Connections" > "Amazon Redshift" > "Write" > "Amazon Redshift Bulk Loader" driver

 

 

The connection string contains the updated bucket and credentials. Does anyone know why that COPY error keeps showing up? Are there Amazon S3 permissions which need to be explicitly stated which might have been missed when setting up a new bucket? The IT guy sent me the current S3 permissions, listed below:

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::new-s3-bucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectMetaData",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::new-s3-bucket/*"
            ]
        }
    ]
}

 

 

 

Screenshot 2020-10-21 104345.png

 

 
 

 

 

1 REPLY 1
patrick_mcauliffe
14 - Magnetar
14 - Magnetar

@corey_hermanson when IT changed the bucket, did they copy your permissions over correctly?

Labels