Inputting Large String File

Question

I am working to create an workflow in Alteryx but am having trouble pulling in a huge (55Mb) text file. The file is not a csv file but a single string without New Line (
) indicators. It uses “~” and a code to separate segments (~INS, ~MID, ~NM2,…) with “*” as a delimiter to separate fields within the segments. Some of the segments are optional and not necessarily populated.

I have been successful bringing in files with about 52k characters but the monthly update files have 56M characters and are just too large to bring in as a single field to start. Once I get it in Alteryx I can parse it into smaller more manageable fields but getting it in the input tool is the issue. Does anyone have advice or a resource I can look at? Ideally this will be a scheduled workflow in our Gallery so my goal is no manual steps in the process.

Felipe_Ribeir0 · Accepted Answer

Hi @mkneer

You are configuring the input tool as .csv right? Maybe it is a worth try to configure it as flat file and parse it after with the text to column tool. I had a similar situation some time ago and at least for my case, it worked.

If the provided answer helped you to solve the problem/was correct, please accept it as a solution :)

Thanks.

mkneer · Accepted Answer

My issue was with csv and delimiters.  I had to import as a flat file of fixed length, parse into strings, and then reassemble the strings into the original segments.