Does batch macro read source file at each pass?

Question

Hello,

I have two very large files (millions of records). One file has Name and Postal, and one file had Name, Postal and a few additional Attributes. The goal is to do a Join/Fuzzy Match between them (multiple passes), however, when I bump the two full files together the run time is entirely too long. In searching for ways to reduce this I am trying to do a Batch Macro. So, I am trying to segment my data by State such that I can compare each state to one another instead of the full files to one another.

Question: When I run a Batch Macro, do the entire source files need to be read to know which records to grab? In the example below (disregard text) is the best way to set this up to do a filter and update State? And if I do it that way will each source file need to be read in its entirety for each pass?

CharlieS · Accepted Answer

Hi @AlteryxTrev

I often use batch macros to manage processing resources. The batch macro configuration in the image below may be loading the data each time. This would be controlled by the input "GroupBy" settings on the outside of the batch macro. The input "GroupBy" function of batch macros doesn't require a Control Parameter tool.

Build the macro like you're expecting only a subset of records to be present (you could go ahead and filter both of your inputs to a particular batch value and use those as example inputs to build your macro). This means you don't need a Control Parameter tool/filter tool because the inputs will already be filtered at this point. Save as a batch macro and that's it.

Now that you're ready to use the batch macro, The files will be read once in their entirety (so the necessary batches can be determined from the contents of the files), then each batch is processed one at a time inside the macro.

Check out the attached example and you'll note in the results window that "3 iterations (read:batches) were run".

EDIT: I couldn't help but notice that you're matching names at the postal level and are considering batching on state. If you're in the US, postal codes/ZIPs can cross state lines. Because of that, and if you want to rely on only same postal matching, then I suggest batching on postal codes.