Dynamic Data Cleansing

[Inactive User]

Hi Community,

How can i select field dynamically in data cleansing tool.

My data input is changed every time in terms of column name and and structure. Can anyone help me to make it dynamic ?

Thanks in advance !

[joelmiller66]

@Inactive User  I would use the Multi-Field Formula Tool and leverage the dynamic or unknown fields option. You also can transpose your data and then apply the cleansing steps to the value field.

Do you have a sample dataset?

[JalenBraegelmann]

If your dataset is an excel file, I would check the 'First Row Contains Data' option for its configuration. This will auto generate headers based on the number of columns included. You could then give every row a record id, filter out row 1 (your actual headers), do the cleanse on the defaulted headers (F1, F2, F3...), then union your actual headers with your cleansed data, and use a Dynamic Rename to pull in the 1st row of data as your dataset's field names. The same process would exist for other non-excel sources, but you would have to have a 

See example below. INPUT --> see preview with F1, F2 header names now.

CLEANSING

DYNAMIC RENAME

Full Workflow

[JalenBraegelmann]

Forgot to finish my thought. For non-excel files you would have to find a way to push your actual headers down a row before repeating the same process. This could look like having a static text input with dummy headers and do a union between text input (first in union order) and you dataset. Then repeat using the same exact steps in my original post.

[flying008]

Hi, @Inactive User 

What's happen in your case ? The changes as your field name ? type ? size ? position ?

Maybe you can explain your needs more.

[Inactive User]

Hi @flying008, 

Table structure and column names may vary user to user, i just want to make it fully dynamic in which no user intervention is required. Is there any way to make it dynamic ? .

[flying008]

Hi, @Inactive User 

What do you mean by fully dynamic? Are you always wanting to select all columns or for columns of a certain type or containing a keyword?

If you can give a sample data or snap, It's be clear.