Union Tool not including all records in a field

[ZoeM]

Hello Community.

I have used the Union Tool before but I havent quite understood what its doing this time around.

I am combining two data sets, one old data and one new data and the fields in each data set are the same (same headers).

However, in the new data, one of the fields has additional rows than the older data i.e. Old data has field 'Name' with Mike, John, Sally while new data has Mike, John, Sally, Robert, Dianne.

When I union the two, ONLY Mike, John and Sally are coming through the output. 

Configuration - Auto Config by Name, Output All Fields and Output Order is unchecked. 

Any ideas as to what is going on?

Steps I have taken

1. Deleted Union tool and brought in new one

2. Changed from Auto to Manual, run, change back to Auto.

Thanks for any ideas you can render me. 

ZoeM

[Luke_C]

Hi @ZoeM 

How are you verifying that you're missing records? Are you trying to search the results of the union? Maybe you have partial results (screenshot below), in which case try adding a browse tool and then verifying if the records are there or not

[ZoeM]

Excellent question.

I filter field to see if any of the new columns are coming. I also have a connection to QlikView and I do not see those new line items in there too. 

I filter before the Union to see what is going in and I filter after the union to see what comes out. I also do a test filter of one of the fields I know is in both data sets. 

[apathetichell]

There is a difference in the fieldnames? - take a look at the manually configure fields option in union and match your fields up. It might not be readily clear what the difference is in fieldnames but most likely that is the cause (name and name. or name and _name or name. vs name_ etc.) If that doesn't work - post some sample data because something in your dataflow may be skunky.

If you can post a screengrab of your union - after you have it as manually configure fields that also could help...

[ZoeM · Accepted solution]

This issue has been resolved. 

1. I think there is an issue with the Union Tool filtering option. When filtering to a specific item which we know is contained in one of the fields, it is not producing results. Is this a limit to size?

2. I had a Join Tool after which was fishing out the new records. Once that was updated, I had all records showing as desired.