The field schema changed between iterations ->The only difference seems to be the metadata

[Chloé]

Deal All, 

I'm using an iterative macro which was running perfectly well for 3 months and since yesterday...

Now I get an error message that the schema changed between iterations. 

I have checked:

-> the number of columns

-> the name of the columns

-> the order of the columns

-> the data type of the content of each columns

No difference at all. 

I used the "Basic Data Profile" Tool in order to compare the metadata of the excel files for which I have no problem to run the workflow with the "changed" excel file of yesterday.

Here we are !!! 

All the previous extracts were flat files without any metadata included whereas since yesterday the excel file I am provided with includes 20 metadata per field / column. 

1.- Are the metadata responsible of the error message "The field schema changed between iterations" ? If not, what can I investigate ? I do not have any idea left ? 

2.- How to "remove" or "desactivate" the metadata ?

Thanks in advance, 

Chloé. 

[LordNeilLord]

Hey @Chloé

This is a common issue and is, as you suggested, mostly likely cause by different metadata.

To avoid this problem, you can build a simple batch macro to import each file. The batch will then union the outputs together (configure it to union on name) and avoid any errors being produced.

[Chloé]

Hi @LordNeilLord, 

Thanks for your answer. 

I have tried to apply your advice but seems that I'm not reaching the goal :S 

Could you have a look to the attached macro ? 

Thanks for your help, 

Chloé. 

[LordNeilLord · Accepted solution]

Hey @Chloé

I've changed your macro slightly, you only need the one control parameter to change the location of the file and in the interface designer settings you need to change the output options to auto config by name.

Note, if you're using XLSX files you need to update your "fullpath" to include the sheet name i.e |||Sheet1

I've included an example for you.

[Chloé]

THANKS !!!! 

It works perfectly now.

One last question: in the interface designer settings, what is the difference between the output options "auto config by name" and the "All iterations will have the same output schema"? 

I would expect the same result but it isn't. I "lose" some data by choosing the latter... 

[LordNeilLord]

In the first option...it does a simple union on the name (ignoring the metadata) but on the second option it checks the metadata as well....which will give you the same error as you got from trying the wildcard import

[Chloé]

You're right ! Sorry, my question was ... stupid :) 

Thanks a lot again. 

[bazilahmed]

Thank you for sharing the solution. I was able to fix my workflow as well. Appreciate it.

[Stuart_Nulty]

Interface Designer setting in the macro did the trick for me, thank you.