Help getting around Cross Tab / 32k field limit

Question

Context: I use the Optimization Tool to fully optimize a distribution network. I have successfully done this before. My current issue is that while generating a SLAM matrix, my production workflow creates too many fields (over 32k).

Attached is a simplified workflow. Any ideas on how to generate the correct output without using the Cross Tab tool?

Sample Slam Matrix Field Limit.yxmd

apathetichell · Answer

Doesn't sound like my area of expertise - so I'll defer to you that there isn't a way to condense into JSON style semi-structured -or place in a DB designed to handle wide data. So one quick question - why go for 2023.X If you weren't having a problem with the earlier version of Alteryx. I stick with 2021.4 and 2022.1.

tonybursey · Answer

@apathetichell  i write out datasets to the Allocate data format which cannot handle a multi-dimensional format, the largest of which contains over 2 million rows of geography by almost 51,000 fields and it has to be in a single table per level of geography for my particular use case. I also write out to several other traditional wide table structures but typically those are split up.

apathetichell · Answer

rereading this - you have 32K fields  in a YXDB -why? Usually I see this in crosstab gone wrong.  you can try to read the discussions on using .yxdb with python which occured a month or so back. I'd a) recommend not using 2023.X b) recommend figuring out a way to get fewer fields in your file.

and original post was on 32k in an optimization which is a nightmare and will not work. At that level you should use a special purpose optimization tool on server.