"Missing fields" in tools downstream from macro using dynamic input

Hi everyone,

Here's the description of the issue I've been facing:

- I've built a simple macro that takes a file name as input question and uses an "input tool" to import an Excel file. The file can have any schema (it's used to avoid having users type full paths).

- In workflows where that macro is used (green in the screenshot), I've noticed that when I'm connecting a new tool downstream (in the screenshot, the Select), it generates immediately a "missing field" error in the existing operator right after the macro (the Filter in the screenshot). However, the field still exists in the data (which wasn't modified, as we can see in the results table).

- My hypothesis is that the new connection triggers a metadata update in the existing tools, but somehow they don't use the current metadata in the workflow, but another version (e.g. the metadata from the file you have to define in the input tool in the macro)

Anyone has faced the same issue? Right now my answer to people using the macros is to rerun the workflow every time they see the issue, which is often and not convenient.

Thanks!

Amaury

Error.png

Initial situation.png

Macros

Accepted answers

patrick_mcauliffe

@Amaury Your experience matches that which I have also had.

I believe I re-ran the macro with placeholder data and re-saved it so it's most recent run would have some sort of existing metadata.

If you can post your macro?

All comments

patrick_mcauliffe

@Amaury Your experience matches that which I have also had.

I believe I re-ran the macro with placeholder data and re-saved it so it's most recent run would have some sort of existing metadata.

If you can post your macro?

Amaury

Hi Patrick,

Sure, see attached. The macro doesn't have much: the user fills in a file name, and the macro just modifies the current workflow's path and append the user's filename. The goal is to access a datastore without having to worry about where it is. There's no check, so users can point it to any file.

read_from_datastore.yxmc

Amaury

Hi Patrick,

This solved the problem in a particular case: when the macro always outputs the same fields.

In that case, I also had to ensure that these fields are present in the macro output regardless of the file that's being imported inside the macro. This means adding a Text Input with the required columns, without any row, and union it to the input data. This makes the macro checks successful at all times.

However, this didn't solve the issue when the macro can produce any type of output schema. In that case, I noticed that there's a tick box hidden in the interface designer "output fields change when the macro's configuration or input changes".

The result of this setting seems to be that the workflow that uses the macro actually remembers the fields that were output by the macro, making all downstream tools happy.

Amaury

Quick Links

This months top contributors

SophiaF 432

AkimasaKajitani 390

mceleavey 388

mbarone 337

michael_weaver 335