Lineage at Column Level

[Kosi]

Connect currently provides lineage for a datasource at the table level but we would like to be able to have this at a more granular level i.e. the column level so if a specific column changes in the table (e.g. data type change), we are notified of the change and able to identify quickly assets that will be affected.

@SeanAdams @Revathi

[VojtechT]

Hi @Kosi ,

we are currently investigating options how to achieve this goal. It is clear that we cannot get the columns using the current approach, i.e. parsing the workflow definition, but need to get on a lower level. 

This will probably have a significant impact on the whole lineage-making process and the results can be expected in rather long- than short-term.

[Aum]

We can benefit from this too.

[monodera]

My customers also want Lineage at Column Level .

[SeanAdams]

This may also need @AdamR  and his team to contribute - since this would be much easier if the engine had a unique field ID internally (similar to the way that compilers have a symbolic variable table that gives a unique memory address to each declared variable)

Field traceability, if done at the engine level, would also allow for optimisation hints like "you've never used this field in an output - may be worth eliminating early in the flow" - again, similar to the hints you get in code IDEs.

[VojtechT]

Hi @Kosi ,

I definitely agree with you. Unfortunately this improvement is not much about Connect being able to show such information as about retrieving the information. We are looking for ways how to get it straight from engine, but I cannot promise anything at this point. 

[bigdata]

I got the same request from my business partners for column level lineage. For Connect to evolve as a competing data lineage, this is a critical feature to include.