where the data is loaded in S3 before copying to Redshift

Question

Dear All,

I met a question when trying to load data from my local PC to Redshift. I am using Redshift Bulk and I learned that the mechanism of this tool is to upload the data to S3 and then copy from S3 to Redshift.

I would like to know where the data is uploaded in S3. As I am facing an error as below.

I am assuming it is due to permission issue and I uploaded the data to somewhere in S3 where I do not have permission to delete.

Error: Output Data (3): The COPY failed with error: We were unable to remove some files from S3 after the copy operation!10Error from AWS: Access Denied

Mahadeva · Answer

Hi All,

Im facing the same issue too. I want to control to which folder in s3 these temp files are loaded before performing bulk load to redshift table.

Is there a way I can provide this folder path in the connection settings to achieve this or any other work around.

Thanks in advance.

DiganP · Answer

@francis_oy make sure the bulk loader is set to the following settings. They are stored as temp files and deleted after the bulk upload process is done.