Query to import subset of a YXDB file

Question

I have an input YXDB which contains around 6000 shape files and is quite large (~1.5gb).

I want to import this, filter out to a handful of relevant shapefiles and then do some processing. I can do this with a standard input and filter but then Alteryx needs to load the full 1.5gb input every time I run a query and takes ~ 5mins to load.

Is there any way to use a query on the input tool to only import the records of value? Similar to how you could use a SQL statement when importing from a database to pull a subset of all records.

Any help here is appreciated!

mattreynolds · Accepted Answer

This could be a good use case for the Calgary database format which allows for indexing and further compression (see description below). Better compression and indexing means that even if you must read the entire file, the I/O requirements will be likely be lower (accelerating the queries you're running).

Happy to tweak a workflow if you can share.