Input Data Tool | Not detecting all files in directory

[JBO]

I often use the Input Data Tool to input multiple like files from a directory using the * before the file extension, like this:

\\MyDirectory\*.csv

I have a directory with 41 CSV files and the Input Data Tool is only bringing in two of them. This has happened to me before. I usually have a Filename field added on Input and then run a summary to make sure all files came in.

Anyone have thoughts on why this happens and how to solve it?

Is there a better way to bring in multiple like files into a flow?

Thanks for your help.

[Thableaus]

Hi @JBO 

Using Directory Tool + Dynamic Input is a flexible alternative to union multiple files.

Directory Tool pulls file paths and filenames from a directory. Dynamic Input reads them by dynamically changing the path/filename for each file.

Cheers,

[JBO]

Thanks. Unfortunately, I am getting the same result.

The Directory Tool correctly identifies all 41 CSV files, but the Dynamic Input Tool is only bringing in Data from two of the files.

I'm not sure what the issue is. Maybe I need to call support. 

[Thableaus]

@JBO 

Is the structure of your CSV files the same? Same Schema?

Did you set up to change the full file path in the Dynamic Input Tool? Could you provide some screenshots from your workflow?

Cheers,

[JBO]

I just figured it out. It turns out that there are 3 minor variations across the 41 files. I have put them into 3 subfolders and then bringing each of the 3 buckets in separately.

Thank you so much for your help!

[JBO]

I do have one more question, though. 

Sometimes we get many files, way more than 41. 

Is there a tool in Alteryx that would tell me which files have the like fields and group them for me so I don't have to go through every single file to figure out where the problem is?

Or a tool that could run a report of all field names for each file so you can scan to figure out which are different.

That would be an enormous time saver. 

If it's not a tool, it should be.

Thanks so much. 

[Thableaus]

Hi @JBO 

If you're using the Input Tool, you can set this option:

This will add to each record the folder where it came from. Would this help?

Cheers,

[JBO]

No. That just provides the file name. I need to know which files have different fields so I can group them accordingly. 

In other words, I want Alteryx to tell me which files are causing the problem with bulk import so I can isolate them and not have to open every single one to figure out which ones have different fields/headers.

[Thableaus]

@JBO 

Using Input tool in a Batch macro and this option I mentioned you, you can figure this out.

Batch Macro you need to build

Configuration of the Batch Macro

Workflow with Batch Macro on it

This trick is used when you want to union multiple files from a Folder with different schemas.

Using the "Output FileName to a Field" option in the Input Tool, you'll know which fields belong to each file. 

You could also add a Field Info tool ahead of the Input Tool inside your Batch Macro to get Field information to each file. This would work as well.

Cheers,