Only bring unique entries into one sheet from another

grsomer

Hi!

I am building a workflow that is bringing in data from multiple sources, but is only bringing in entries that are not already in the main stream of the workflow.

As of right now, I am using input tools to bring in all the different inputs, then using a unique tool on each to only pull out unique entries in each input, then using join tools to connect each input to the main stream, and finally using a union tool to join the L output (only entries in each input not found in the main stream) to join all individual streams to the main stream. Unfortunately, it seems as if too many records are being brought into the join with the main stream. 

Is there another way to create this process?

I have attached a photo of the workflow to hopefully make it easier to understand what I am describing (although the workflow does look quite messy). 

Any help or suggestions are appreciated! Thank you!!

Capture.PNG

fharper

@grsomer  I looked at the pic you included and it is a subset of a larger collection of inputs it seems BUT from what I see you are joining the 4 visible inputs after Unique'ing to one or more other inputs below the pic (offscreen if you will).  then you union it all.  there is no way to insure there are not dups "across"  the 4 visible inputs much less others off screen.  Each input was uniqued but if the same key is in another of those files then you get dups in the end because you only joined one other file to the input.

I suggest staging the joins with unions in the pic below. each input is uniqued as before but then each is joined in stages so all are cleansed of dups.  If it were me I would join and merge in the same way to the main stream as a best practice in case the many inputs being joined have a dup of something in the main stream.  Notice I added a source column in that formula tool.  this way when you validate you can see where it came from, capture the filename or in some way make it a unique value per input.