Adjusting String Column Length when uploading using HDFS

JonnyR — Wed, 18 Jul 2018 13:43:29 GMT

I am having issues loading data to Cloudera via ODBC connection. I found this forum post with what I believe to be the solution but as I do not understand why it is causing the slow write times I am having a hard time convincing our IT guys to change the settings to try the solution given. What is the reason that lowering the string column length from 32727 to 255 improves performance?

It is my understanding that this number is the max character length rather than just the character length.

Thanks for your help

Re: Adjusting String Column Length when uploading using HDFS

Jim7 — Wed, 18 Jul 2018 14:35:04 GMT

The longer your maximum string length, the more memory the system has to allocate to process the data stream. For that reason, it's always a good idea to use the minimum string length necessary to capture your longest string. That's a good rule of thumb when defining all of your fields in Alteryx as well :)

topic Adjusting String Column Length when uploading using HDFS in Alteryx Designer Desktop Discussions

Adjusting String Column Length when uploading using HDFS

Re: Adjusting String Column Length when uploading using HDFS