Load the .csv file to ADLS (Azure Data Lake Storage) Folder from Alteryx

Question

Hello,

I'm looking into a solution to upload the .csv file to ADLS Location, I have managed to establish the connection in ADLS with Alteryx. But the issue I'm facing while uploading the .csv file to it. Please note the file size is 3.1 GB. I'm getting the below error.

Error: Azure Data Lake File Output (2): Error while writing to csv file.- Unable to allocate 37.1 MiB for an array with shape (4865664,) and data type int64.

I have tried to upload the data by creating a batch macro to send the data in batches just to reduce the load. Even this method also not working properly. Attached the workflow for the reference.

Note:

I have tried uploading the data that includes 10000 records, I can be able to achieve it.

It would be much appreciated if you can provide your thoughts and inputs in resolving this issue.

Thanks in advance...!

ADLS_Error.jpg

ADLS_TEST.yxmc

ADLS_TEST.yxmd

Sagarloknath · Answer

Hi apathetichell,

Thank you for your response..!

I have tried closing all the browsers and other applications even though I'm getting the same error.

If I manually drag the file to the ADLS folder will works. But, it needs to automate the process. Hence looking up for the solution.

Regards,

Sagar

apathetichell · Answer

This may be a memory error. I don't think a batch macro would necesarily help here - but I'd recommend closing all browser windows and allocating all resources to Alteryx and resource manager. I'd recommend watching memory peaks there. You may also want to see if any of this is controlled by your odbc64 settings.