Duplicate data issue

[binsell]

Hi All, I'm hoping someone can help me as have an issue with duplicate data. I have attached a file where you can see my data and have shown the data BEFORE I have added a 'arrange' tool and the data AFTER the arrange tool. This is necessary due to the different versions of my data and something I need to keep. This issue I have is that after the arrange tool it's duplicating the data and I want to keep the line that has data in the last two columns (STATUSVALIDFROM & STATUSCHANGEDTO). Hopefully the attached will make sense of it and show you want I'm trying to achieve! Thank you in advance :) 

[CoG · Accepted solution]

It appears to me that you do not have a duplicate data issue and the Arrange Tool is doing exactly what you intend for it to. You just want to remove extraneous rows that have nulls. That can be done in many ways and depends on which formats your data can take. Here is one such solution:

Hope this helps and Happy Solving!

[TUSHAR050392]

Hey @binsell If I understand it correctly, you want get rid of duplicates and just keep one value. After arrange, try using a sort tool by STATUSVALIDFROM & STATUSCHANGEDTO descending and then using Unique in which you will group by OWNSTU and SESSIONID. This way if one record is Null, it will be always 2nd in order and will be dropped. If both are null then one will be dropped. Let me know if this helps

[Yogesh1795]

Cant you use summarize tool to get rid of duplicated data?

[binsell]

You are absolutely correct, it' not a duplicate data issue! Thanks for the solution, much appreciated :)