Parse column from Hadoop using flat file layout

I had designed a somewhat complex alteryx flow which took the latest .txt file in a NAS drive. This file has several lines that need parsed, and based of the lines record key, has specific string lengths. I built using a dynamic input tool, and everything worked out great with the template and multi row logic for looping through the specific record keys.

However, the long term solution was to store these files in Hadoop. I sweep Hadoop by pulling the latest load datetime stamp. When I try to use the dynamic input tool and connect to the Hadoop instance, there is no File/Field Layout selection to bring in the .flat file that specifies what the fixed length for each individual file. What am I missing here? I searched the community, and I apologize if there is a subject on this already.

Many Thanks,

David

Dynamic Processing

Accepted answers

hhaigh1

Hi @David-UHG

Currently we only support csv and avro files for HDFS which is why you are not seeing the option to map in the layout in the .flat file.

Is it possible to store your files in avro format instead?

All comments

hhaigh1

Hi @David-UHG

Currently we only support csv and avro files for HDFS which is why you are not seeing the option to map in the layout in the .flat file.

Is it possible to store your files in avro format instead?

David-UHG

Thanks Henriette,

In anticipation of getting that answer, I have an output tool for this workflow as an ASC file on a secured NAS drive from the Hadoop column. I can then bring that into my parsing workflow that uses the dynamic input tool. I will then need to join back to the original Hadoop table to bring in all the other columns that are associated to the newly parsed rows. Now what I will need to do is work with our IT department to somehow allow us to install the runner and conditional runner macro. Unfortunately, I don't think this will be easy.

Unfortunately, the original flat file is located on a highly secured Linux server of which I don't/can't get access to.

Two part question:

1 - Any future plans to add fixed format files to other data sources? I know I can create formulas and trim per the fixed format spec, but that can be very tedious depending on the amount of tables/columns per use case.

2 - Any future plans to bring in the runner macro or something similar as native in Alteryx?

My apologies for adding another question that will not be easily searchable.

Many Thanks,

David

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15865

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703