Alternatives to overcome Alteryx/Hive Table Schema Character Limit 4000

[tdevrie1]

There are a several ways to store the Avro schema in Hive:

1. Literal JSON string stored in the Hive table properties (Alteryx currently supports)
2. Reference to the schema file stored elsewhere
3. Pass in the schema as a run-time property in Hive

Alteryx only supports Option #1, but that runs into a 4000 Character Limitation which is the default schema limit in Hive’s internal DB. Is it possible to have Alteryx support the other two options to be able to support data sets with large schema definitions?

[SUBASH_TE]

We are running into a similar issue in our company. We are pulling data from our data warehouse and most of the times the data is flattened. We need this data in HDFS to create a managed Impala table that can be consumed in Tableau. The 4000 Schema character limitation is stopping us from doing this. We need an enhancement from Alteryx ASAP that would allow storing the schema definition as an external file or that can be passed as a run time property.

[dmichaels71382]

This is also a very big problem for my company.  When we do data science projects, we often need to do very complicated data transformation and then land that data into HDFS so we can run machine learning programs on it.  As a result of this limit, we have to load the data in HDFS or KNOX.  When the data is very large, this is much, much slower, resulting in lost days of work. 

[Community_Admin]

The status of this idea has been changed to 'Inactive'. This status indicates that:

1. The idea has not had activity in the form of likes or comments in over a year.

2. The idea has not reached ten likes.

3. The idea is still in the 'New Idea' status. 

However, this doesn't mean your idea won't be implemented! The Community can still like and comment on this idea. With enough renewed interest, this idea can be brought back into the 'New Idea' status. 

Thank you for contributing to the Alteryx Community and the Alteryx Product Idea Boards!