Wildcard only read first file

Hi everyone,

I have 8 CSV files with same schema in the same folder. When I try to run wildcard on them, it only read one file and the warning goes "Record #2 not enough fields".

Then, I changed delimiters to "\n", but this time it still read one file but the warning is gone. I am confused. Thanks for help.

Input

Accepted answers

All comments

binu_acs

@largychen Connect the input tool to a browse tool and try again?

Yoshiro_Fujimori

Hi @largychen ,

The warning

Record #2: Not enough fields in record.

seems to occur when the number of fields in the record #2 is less than the number of fields in the first row of .csv file.

e.g. I got the same warning when I read the following .csv files with Input Data tool.

Spoiler

>type input_*.csv

Input_1.csv

Col1,Col2,Col3
Data01,Data02,Data03
Data11,Data12
Input_2.csv

Col1,Col2,Col3
Data01,Data02,Data03
Data11,Data12

Input Data tool configuration

But I could read multiple files regardless of the above warnings as the log below.

Spoiler

Designer x64 Started running at 05/05/2023 13:30:18

...
Input Data (1) Record #2: Not enough fields in record.
Input Data (1) 2 records were read from "C:\Users\...\Input_1.csv"
Input Data (1) Record #2: Not enough fields in record.
Input Data (1) 2 records were read from "C:\Users\...\Input_2.csv"
Input Data (1) 2 files and 4 records read
Designer x64 Finished running in 0.4 seconds with 2 warnings

So I guess the problem is not related to the warning.

Could you share your configuration of Input Data tool, with the original delimiter ?

largychen

Thanks for quick replies

@binus I tried but it didn't work

@Yoshiro_Fujimori

here is my configuration

binu_acs

@largychen Can you try to increase the field length to 9999 and try again?

Yoshiro_Fujimori

@largychen ,

You set Record Limit to '1".

Could you delete '1' in the parameter and try again?

largychen

Thank you guys.

Yoshiro_Fujimori I set the record limit because all I want for that CSV is the first and second row. I need the record limit unless I will get below error and warning. I screenshot and attach my source file here

Input Data (223) Record #2: Not enough fields in record.
Input Data (223) Error reading "C:\Users\XXX\Cambium22_MidCase_hourly_p1_2024.csv": Too many fields in record #3

@danilang I tried, but it doesn't work

Cambium22_MidCase_hourly_p4_2026.csv

largychen

To be clear, I have 8 CSV like this, and I will need my output like this

largychen

Hey Dan

Here is the file and screenshot.

Cambium22_MidCase_hourly_p1_2024.csv

danilang

hi @largychen

Neither of the files that you've attached is actually a csv file. These are text files with a .csv extension. A well formatted csv file has exactly one line of field definitions followed by a series of data lines as in

ID,Name,Salary

1,"Jane Doe",20000

2,"Matt Smith",10000

Your files have a header in red followed by the actual data in green

If you start your input on line 6, the files should be read without error.

Dan

largychen

Hey Dan, Thank you. I have actually two inputs one starting in line 1 and the other starting in line 6.

The reason why I still need line 1 is that I need that info in my output sheet. The only thing I need is Row 1&2 in below

danilang

hi @largychen

In that case, set record limit = 2 in your second input tool to only read rows 1&2 and avoid the errors that caused by field differences in row 3.

Dan

largychen

Sorry Dan, record limit = 2 doesn't work I got the same messages. Only record limit =1 will work without warning and error, but the issue is record limit seems not working with wildcard. I want row 2 in all file stack together like this below.

Input Data (223) Record #2: Not enough fields in record.
Input Data (223) Error reading "C:\Users\XXX\Cambium22_MidCase_hourly_p1_2024.csv": Too many fields in record #3

Ideal output

danilang

hi @largychen

This seems to work for the 2 files that you posted

Read all the lines with \n as the delimiter and 1st row as a data row. Take the 1st two rows from each file. Then take the 1st row and parse the header fields using text to columns. On the bottom skip the first row from each file and parse the data fields. Union and use a Dynamic Rename to moved the 1st row to the headers

Dan

WildCardMergeof1stDataRowFromAllFiles.yxmd

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703