ALTERYX INSPIRE | Join us this May for for a multi-day virtual analytics + data science experience like no other! Register Now

Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Reading Text File with conditions?

7 - Meteor

How do I read a 5GB text file with conditions. I do not need to read in all the 5GB as it would take time. Is there any option to read the text file with filter conditions


7 - Meteor

Hi @Georgealex ,


Did you get a chance to look into following 2 solutions? What kind of filters are you trying to apply. Do you have a sample file you can upload?



17 - Castor
17 - Castor

Hi there @Georgealex 


Unfortunately, with text files, you don't have a lot of options.  They aren't indexed so even if there was a way to specific some way to narrow down your search to only lines that contain a specific word or phrase, you still need to read through the entire file to find those rows.


From the Input tool, about the only options you have are to Set the record limit which limits the number of rows pulled in and/or Start the import at a specific line




If your text file is fairly static and you need to read it often, consider having one process that that reads the entire file and writes it back out as an alteryx.yxdb.  The data will be highly compressed on write, the size of the file will probably drop down to about 100MB or so and subsequent retrieval speeds should be 10 to 20 times faster.


If your initial file changes often, consider having a server process that runs periodically and updates the .yxdb file