Best practice to document total input files?

I have my input tool pointing to a folder where all files are the same format and type. So I have the path with *.csv.

What's a good practice to ensure that I've successfully imported all files? Right now I'm visually checking. I see 89 files in the folder and in Alteryx's results message I see Input Data 89 files.

I worry that if a random file has an extra column or is a .xls file type, there's a risk it'll be missed beyond my visual check process.

Best Practices

Accepted answers

Erin

Have you tried using the Test tool? You could change your directory back to *.* file types to bring in all files - then connect that to the Test tool as input #1. Add a filter for file type .csv - then connect it to the same Test tool after the filter as your Input #2. You can then add a test for Record Count Matches Other Input Record Count. Something like this:

Since your test was a .csv that didn't import, maybe you can add a summarize on file name after your dynamic input and make that your #2 input to your Test tool to check for failed .csv loads.

For your other question, @fharper explains how to export the results messages in this post:
https://community.alteryx.com/t5/Alteryx-Designer-Desktop-Discussions/Log-File-Location-and-Variables/m-p/80884/highlight/false#M3479

All comments

FrederikE

Hey @jaypee1217,

You can use a Directory tool followed by a Count tool. This will show you how many files are in the selected folder and is not dependend on the file type.

This can you use to compare to the files input with your Input tool.

jaypee1217

I misread. I thought there was a compare tool.

So this is still a visual check. I put a file with extra columns so i know it would fail. i had a 90 count from my directory tool. and from the workflow results messages i see Input Data 89 files.

Is there a way to export the Results message? If i could export the Input Data 89 files and the Directory count 90 files, then that would be a sufficient checks/balances to capture failed files.

Erin

Since your test was a .csv that didn't import, maybe you can add a summarize on file name after your dynamic input and make that your #2 input to your Test tool to check for failed .csv loads.

jaypee1217

I didn't even know about the Test tool. I will go down that route and see what i come up with. And thank you for the link on how to export results messages!

Erin

@jaypee1217 - You're welcome! Mind marking my comment as a solution please?

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15867

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703