"COPY failed with error: Unable to remove some files from S3 after the copy operation"

[corey_hermanson]

I feel like this is a dumb question, but I just can't figure out the answer. IT at my company changed the S3 bucket I used for loading data into Redshift via the "Data Stream-In" tool. I changed the name and credentials to match this new configuration, but now I'm receiving a COPY error every time I attempt to load data into Redshift. Data does seem to get loaded into Redshift, but since the error is thrown I can't string together multiple Alteryx flows with the "Runner" tool.

Error: Data Stream In (3): The COPY failed with error: We were unable to remove some files from S3 after the copy operation!

I've tried searching similar threads and several point to the Alteryx documetation for Redshift: https://help.alteryx.com/current/designer/amazon-redshift

There are many configurations listed on that page, but they all refer to connecting via the "Output Data" tool. All of our Alteryx flows use the "Data Stream In" tool to "overwrite table" in Redshift. I then choose (screenshot at end of post): 

"Data Stream-In" tool > Configuration > "Manage Connections" > "Amazon Redshift" > "Write" > "Amazon Redshift Bulk Loader" driver

The connection string contains the updated bucket and credentials. Does anyone know why that COPY error keeps showing up? Are there Amazon S3 permissions which need to be explicitly stated which might have been missed when setting up a new bucket? The IT guy sent me the current S3 permissions, listed below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::new-s3-bucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectMetaData",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::new-s3-bucket/*"
            ]
        }
    ]
}

[patrick_mcauliffe]

@corey_hermanson when IT changed the bucket, did they copy your permissions over correctly?

[corey_hermanson · Accepted solution]

I worked around this issue for 4 years, but I came back to it and yes, I was missing a permission in S3.

The solution was to add the "s3:DeleteObject" permission to the user