How can I pass 20M datas into the Python tool？

[Wii]

-20M data (about two or three hundred columns)

-Server 32GB RAM

When I use

Alteryx.read('#1')

Sometimes Python reports a memory error when reading, and the background can see that Python has used up memory. and sometimes it will be directly interrupted, Alteryx flashback.

We also tried to find a solution from Alteryx's python source code, and we find that the key model PyYXDBReader.

but

it  is encrypted and the source code cannot be seen.

P.S. 开源才会有出路！！！

So I wonder if there is any other way to pass this 20M data into the Python tool, or even 100M data...

Hope to get your reply! Thanks!

[TheOC]

hi @Wii 

There are a couple of potential solutions on this threads:
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Alteryx-MemoryError-in-Python-tool/td-...
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Reading-large-files-with-Python-tool/t...

Common theme seems to be splitting the data up, and reading it as multiple chunks. I have had some success with this in the past.

Hope this helps,
TheOC

[sparksun]

Why not use Alteryx to do all the work for u instead of Python coding?

[Wii]

This thing was already done in phase1 of the POC, there were about five hundreds tools in one workflow. and the bosses felt that was very messy to maintain.🤐🤐🤐

so we try the effect using Python interspersed in Alteryx.

[Wii]

Thank you very much for your answer.

I will try it later.