community
cancel
Showing results for 
Search instead for 
Did you mean: 

Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.
Upgrade Alteryx Designer in 10 Steps

Debating whether or not to upgrade to the latest version of Alteryx Designer?

LEARN MORE

Reading Text File with conditions?

Meteor

How do I read a 5GB text file with conditions. I do not need to read in all the 5GB as it would take time. Is there any option to read the text file with filter conditions

 

Alteryx Partner

Hi @Georgealex ,

 

Did you get a chance to look into following 2 solutions? What kind of filters are you trying to apply. Do you have a sample file you can upload?

 

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Function-to-add-column-rule-before-imp...

https://community.alteryx.com/t5/Alteryx-Knowledge-Base/Dynamic-Filter/ta-p/9922

 

 

Magnetar

Hi there @Georgealex 

 

Unfortunately, with text files, you don't have a lot of options.  They aren't indexed so even if there was a way to specific some way to narrow down your search to only lines that contain a specific word or phrase, you still need to read through the entire file to find those rows.

 

From the Input tool, about the only options you have are to Set the record limit which limits the number of rows pulled in and/or Start the import at a specific line

 

Config.png

 

If your text file is fairly static and you need to read it often, consider having one process that that reads the entire file and writes it back out as an alteryx.yxdb.  The data will be highly compressed on write, the size of the file will probably drop down to about 100MB or so and subsequent retrieval speeds should be 10 to 20 times faster.

 

If your initial file changes often, consider having a server process that runs periodically and updates the .yxdb file

 

 

Dan

Labels