What was the maximum size of *.YXDB file you used so far?

[vizAlter]

I would like to know that what was the maximum size of *.YXDB file you used so far?

What kind of risk and precautions were needed?

[MarqueeCrew · Accepted solution]

https://help.alteryx.com/current/designer/alteryx-database-file-format  

there is a 2GB limit on the content of a field. 

create as many rows as you want and any size that you want. 

if you put all of your eggs in one basket, you may have a nightmare if your disk fails or if the OS has read/write issues. 

I'd limit the size only if there were long running jobs that needed to be managed for time or restart. 

I've consumed full disks with data. No limit!

cheers,

 mark

[vizAlter]

@MarqueeCrew - Thank you for sharing the great information with URL!

"While there is no limit to the number of rows or file size, there is a size limit of 2GB for a single cell in a 64-bit environment."

"Field Info" tool revels the allocated size of the field (which is maximum value), btw is there anything to track the size of each cell for this situation?

I'd limit the size only if there were long running jobs that needed to be managed for time or restart. 

"Max Records Per File" option is available in the "Output Data" tool, not by the file size.

Hence, would like to know more about your approach along with the tool/steps. Is there any video/article available?

[MarqueeCrew · Accepted solution]

@vizAlter ,

You're going to love my answer:  It depends.

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/So-i-have-a-large-file-Really-large-Wh... 

The post above asked about running using a huge dataset.  One answer was to use CALGARY tools in running the data.  I like me some CALGARY, but it is awful in some use cases.

I regularly work with national data and don't worry about operationalizing the data into subsets.  I'm a fan of performance tuning workflows after building them with a K.I.S.S. architecture.  Is the complexity of performance enhancements worth risking the maintainability of the workflow.

If you'd be willing to co-author a use-case on Alteryx, I'd be happy to review your needs with you and make some suggestions specific to your requirements.  If that sounds reasonable, please PM me with your details and I'll set something up with you.

Cheers,

Mark

[vizAlter]

@MarqueeCrew - I think that *.YXDB will be sufficient/better if dataset is up to 10 million records (in ~250 fields), and Calgary file (.cydb) is for unexceptional size of the data (good to know about "Calgary" tools).

I will test on the Calgary file as well.

Thank you, again! You are the best!

(Sure, I will PM you soon with more information)