Computer crashing - dataset too large?

[sartol]

Hello!

I am connecting to a database, doing a little manipulation, and then streaming out a dataset to run a multi-row formula tool.

My dataset has 7 columns and at least a billion rows.

My workflow runs for several hours having completed all the in-database manipulation but never gets past the data stream out tool, and then my computer crashes. This has happened twice.

I can get you specks on my computer, the exact number of rows, etc. but I am wondering if someone knows exactly how much data you can put through Alteryx.

Thanks!

[Jonny]

hi @sartol ,

I do a bunch of data transformation in-DB and extraction from DB using the stream out tool. not in the billion of rows spectrum but a few millions.

I don't know how many rows you can put through Alteryx but I can suggest a few ideas that I generally use when designing my DB workflows.

1/ I use in DB filter, start with one very small sample set and check - Is my logic sound? 

2/ I scale up slowly using the in DB filter + I enable performance profiling and look for where in my workflow it takes the longest to complete and look for optimisation opportunities in the logic or tool config. if you do that you will notice that Data stream out is not a fast process. check this on enabling performance profiling: Performance Profiling in v10 - Alteryx Community

3/ I also have other processes where I write to a table in DB for later consumption, so I do not stream out until I need to. So your case would be splitting the workflow into two parts: your transformation to write data in DB, and then your extraction.

do you think these could help you?

cheers!

Jonny

[AlteryxUserFL · Accepted solution]

There is no row limit for Alteryx software and it is not uncommon to process over a billion rows of data in my experience. My guess is that the issue is likely your computer or the database. I would highly recommend using Alteryx Server for your use case as well to make things run faster.  Also, I would check into any network issues on the database side and on your computer side. A slow internet connection or a VPN will make pulling that many records from a database take a very long time.  I'm surprised the in DB part is taking so long as the workload for those tools is handed by the database, I would check into the database settings to see if anything is wrong there as well.