ALTERYX INSPIRE | Join us this May for for a multi-day virtual analytics + data science experience like no other! Register Now

Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Python Memory Error while replacing data to bq with Macro

harshal98
8 - Asteroid

Getting Memory Error while replacing data on BQ Macro has some 20767242 rows

 

Capture.PNG.

 

Can anyone guide me here?

 

 

TheOC
11 - Bolide

hi @harshal98 
I believe this is simply a Memory limit on your machine. Here i tested with 2.5gb of data in Alteryx, and imported it into the Python tool the same as you:

TheOC_0-1620260398154.png



however, If you check task manager, it is definitely not 2.5gb of data:

TheOC_1-1620260414242.png

 

And Python needs to store it in RAM.

Honestly, my recommendations would be to cleanse the data as much as you can prior to importing into the Python tool. Sadly Python has a much smaller limit to the size of data (RAM 

dependant), than Alteryx, as it is not compressed the same way.

 

The only other thing i can think of, is splitting the data into multiple chunks, and inserting it into the python tool like that, then running a loop within the python tool, to iterate whatever code you are trying to run, on each chunk of data. As you will be writing over the 'df' each time, this should reduce the memory required.

harshal98
8 - Asteroid

My Chunk size was 1 Lakh while uploading 2 Cr Data ....Just Increasing the Chunk size to 50 Lakhs Solved my Error. Thanks

Labels