-20M data (about two or three hundred columns)
-Server 32GB RAM
When I use
Alteryx.read('#1')
Sometimes Python reports a memory error when reading, and the background can see that Python has used up memory. and sometimes it will be directly interrupted, Alteryx flashback.
We also tried to find a solution from Alteryx's python source code, and we find that the key model PyYXDBReader.
but
it is encrypted and the source code cannot be seen.
P.S. 开源才会有出路!!!
So I wonder if there is any other way to pass this 20M data into the Python tool, or even 100M data...
Hope to get your reply! Thanks!
hi @Wii
There are a couple of potential solutions on this threads:
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Alteryx-MemoryError-in-Python-tool/td-...
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Reading-large-files-with-Python-tool/t...
Common theme seems to be splitting the data up, and reading it as multiple chunks. I have had some success with this in the past.
Hope this helps,
TheOC
Why not use Alteryx to do all the work for u instead of Python coding?
This thing was already done in phase1 of the POC, there were about five hundreds tools in one workflow. and the bosses felt that was very messy to maintain.🤐🤐🤐
so we try the effect using Python interspersed in Alteryx.
Thank you very much for your answer.
I will try it later.