Data Profiling on Multiple Input Files

[Nikesh_ZA]

Hi There. I am trying to perform simple data profiling on a large number of data files but have run into some problems. I hope someone can help me out.

For a single data input file, I am able to use the Field Summary tool to perform the required data profiling and write the results to an Excel workbook. I now wish to extend this to be run on (for example) 100 data input files and output the results into separate sheets in an Excel workbook. I have used a macro to read in my 100 input files (this works fine) but because I feed this into a Select tool, I end up with a single dataset (encompassing the 100 files I inputted) on which the data profiling is being performed. Please see below a screenshot of my current workflow.

To be clear, what I want is as follows:

[input x files] > [perform data profiling on each of the x files] > [output results for each of the x files into separate CSV files/Excel sheets]

Any help would be appreciated.

Thanks!

[NicoleJohnson · Accepted solution]

As part of your macro, integrate the data profiling steps as well so that each time it reads in a file, it will run the data profiling and output the results to a CSV file (you can use the "Take File/Table Name from Field" option to create a new tab or output file for each file read)... and then it will repeat the process for the next file it reads. The way you have it currently configured, it will collect all file data first and then output the combined data from the macro to one data stream in your main workflow... but by integrating that data profiling step into the macro itself, this should get you the results you are looking for since it will now perform that data profiling step each time it reads a file, before moving on to the next one.

Let us know if you need a bit more guidance on implementing this!

Cheers,

NJ

[Nikesh_ZA]

HI Nicole

Thanks for your help and replaying so quick. I had initially tried moving the whole workflow into a macro but had obviously missed something. I've now got it working, although not 100% in the way I need it to. So will work on some final tweaking.

Regards,

Nikesh