Alteryx Union With Dynamically Switchable Control Containers

Hi everybody,

I've got a question that's been bothering me for a while and I haven't been able to find a solution to. I've included a screenshot with a simplified example of the issue.

It relates to a scenario where I have a set of PDF reads to do and have to dynamically come up with a solution to determine which ones. This works well using the Filter tool to turn containers on and off and if all processing is done inside each control container, all is fine.

However, if I need to combine the results outside the control containers, the Union tool fails with the "No incoming metadata from one or more upstream tools". I've tried various solutions to resolve this issue with no luck. Limiting the number of PDF reads is critical for my use case as one of my routines is already taking over 2 minutes to complete, unacceptably long.

Does anybody have any ideas on how I can resolve this?

Thanks

Screenshot 2025-03-05 110937.png

Workflow

Computer Vision

Accepted answers

All comments

Warcry

Screenshot 2025-03-05 223801.png

Change the configuration to Ignore - Continue Processing Records

bmillar777

Unfortunately, that didn't resolve the problem.

I've also tried to set "Auto Config by Position" and even to "Manually Configure Fields" and enabled the non-blocking option, but nothing from this angle seems to resolve it.

adighe

I have the exact same issue. Have tried all the options for the union tool. Amp on and off did not make a difference. Any clue as to what different approach to take in such situations?

Not all streams will have data. There are only two fields coming into the union tool with same names and datatype and size. I have set it to "Auto Config by name"

Error: Union (458): No incoming metadata from one or more upstream tools. Check the input data and run the workflow again

Union tool Error.png

Union tool columns.png

bmillar777

I never found a solution in the end and maybe there isn't one. The closest I got was adding the Record ID tool to the Union inputs which removed the errors but didn't output any data It might be possible to use the Python tool to make it work, although I haven't had a proper go at that to see.

GMG0241

Hi, I think the issue COULD be that it's due to having a split stream. Please see the attached screenshot/example which I believe should solve your problem by creating a single stream, although I'm not 100% sure as I don't have access to the Computer Vision tool set

solution.yxmd

flying008

Hi, @bmillar777 @adighe

There are some issues that need to be confirmed first:
1- Do all your test*.pdf files exist?
2- Do all your PDF file paths have access permissions and are they set correctly?
3- Do all your PDF to Text tools have normal output? (The output here means at least an output header, or a standard header and data record, not nothing.)

If the answers to the above three questions are all yes, then under normal circumstances, the error in your screenshot will not appear. At the same time, the most likely problem is problem 3. For some reason, the PDF to Text tool will not output anything, so the Union tool will report an error due to the lack of metadata for one or more incoming connections.

**************
Solution:
1- You need to modify the existing workflow, add an Output tool after each of the three PDF to Text tools, and output each data to the file format you specify. Remember that the three output files should be placed in the same folder.
2- If the data structure of the three files is exactly the same, you can use the wildcard option of the Input tool or the Dynamic Input tool to Union all the data, so that the above error will not appear and affect your work.
3- If the data structures of the three files are not exactly the same, you may need to use a batch macro to complete the above Union operation. There are too many such macros in the Gallery, you can download them directly and use them.
4- You can also use 1 Control Contain to control the order of the above two steps relative to the three PDF to Text tools, which should be a very easy thing for you.

That's all, I wish you good luck!

adighe

@flying008,

In my case it's not files but the download tool with various DB or API calls that might not be executed. That causes errors before the flow reaches the join. The errors were hidden in control containers that collapse. But this causes the metadata to not reach the union. I have to figure out how to pass metadata or dummy data through every path that reaches the join and then remove it.

Union tool Error.png

apathetichell

can you share a screen shot of the config of your union tool? union should not require a connection is initialized - but I've seen this be sticky. I think some things to try are:

1) delete the union - re add.

2) in the containers create a text input tool with a dummy column name and no data. union this in your container. run.

the above is in standard Alteryx pre-control containers. It's possible that the control containers deploy is a bit 'special' -> if that's the case maybe throw things in a macros instead of control containers?

basically - what you are trying to do is resync the workflow cache.

adighe

I have used all configs for the union tool with no luck, and current config is auto config by name. Adding the dummy data is what will work. The downloads in red may or may not execute and that results in the formula/filter downstream to error due to missing variables. I added the missing dummy variables with a formula marked in green. That took care of error in that stream. Any errors upstream with or without containers will prevent metadata to flow downstream and union does not like that.

@apathetichell adding text input is a better idea as its very obvious what the purpose is.

Union Solution.png

Union Config.png

apathetichell

@adighe- I use text input - and I'd actually use a field name which must be in the union tool (ie something from your parsed download tool data). It does not have to have rows -> it just needs a field name and it will populate into the union tool.

Reiterating - old school this worked fine -> control containers may be 'unique'

bmillar777

Unfortunately, I don't think any of these solutions will resolve the original problem, I think I've tried them all already. Let me define the issue with bullet-points.

* The original purpose was to read as few PDF's as possible. If the problem could be solved by reading PDF 1, then that's good. If not, then we need to read PDF 2, and then PDF 3 and so on. Containers are necessary for this problem as the PDF tools expect an input if they are not inside a disabled container.

* The screenshot I provided was a simple example I constructed to demonstrate the problem. The following is how things work depending on the situation:-

1) The three Filter tools turn the containers on and off automatically depending on the data in the Text Input tool. However, if we ignore those tools, the same problem still applies. The real issue is with the containers and the Union tool.

2) If we leave all the containers switched on, the workflow runs perfectly. However, if we disable even one, the workflow fails reporting that no "incoming metadata could be found from one of the upstream tools"

2.1) It seems that having no input at all from a disabled container is a problem for the Union tool which generates the error.

2.2) No setting changes in the Union tool change the issue, not even the non-blocking one.

3) The problem doesn't exist if all processing is done inside an individual container. For example, having three Union tools, one in each container doesn't generate any error. But then you can't rejoin the rest of the workflow to do further processing. In my use case, that's a problem as after reading one PDF, I need to exit into the workflow to determine the next PDF container to switch on before doing further processing.

4) Using any kind of macro isn't a solution here for the same reason as the Union tool, you need to leave the container to exit the macro and that can't be done without error.

I think a possible solution is to use nested workflows. Then you could remain in the container, write an output and then read it in the next workflow etc. But it's a very complicated solution which I didn't try. Another possibility is to use nested control containers, but if you have ten PDF's to read, that's going to be very ugly

apathetichell

Hey @bmillar777 - sorry was confusing the two tool sets - I think part of your problem is that the PDF tool you are using uses the Python tool. The Python tool wants to run - > even with no data.My hunch would be:

1) disable all containers.

2) create a text input with whatever downstream schema you need.

3) run -> confirm that it does not error.

4) set up your containers to optionally RUN - not optionally NOT RUN. Re test.

If that does not work -> I'd try 1) putting each container in a separate macro.

2) going old school and using disabled standard containers vs control containers with them set to run with data (and their standard setting being disabled) - again in a (Batch) macro -> albeit you could use one.

flying008

Hi, @bmillar777 @adighe

1- No matter you use PDF to Text tool or Download tool, they will cause no output for some reason. So the problem is not the control container, but the fact that as long as one of the incoming connections of the Union tool is an empty connection (no metadata is passed in by the upstream tool), an error message "incoming metadata could be found from one of the upstream tools" will be reported.

2- So you need to analyze which step before the Union tool may have no metadata output, either create a text input with whatever downstream schema you need as @apathetichell said, or use dynamic input to process the previous data later as I said above.

3- In fact, there is another solution here, which is to use the Branch plug-in similar to Until ...to or If...then logic to implement the branch process. It does not require the use of the Control Contain tool, does not rely on the AMP mode, and does not require the use of multiple nested macros. But since your company may restrict the use of such 3rd external tools, we will not discuss it in detail.

bmillar777

Hi all, thanks for your replies.

@apathetichell, I'm not sure what you mean by optionally RUN, I assume you mean disable the control containers and then enable them using the Filter tool. If that's what you mean, it doesn't unfortunately work as the Filter tool won't enable a disabled container.

@flying008, a control container is necessary for this procedure as the PDF tool needs an input if it's not disabled. So when it's disabled, it will always lead to an empty connection in the Union tool. Adding a Text Input tool to add input doesn't work as any joins or unions will simply receive no input from the disabled PDF tool. It's a circular problem.

A dynamic input solution could work, it's not something I've tried to date. I don't think we can use third party tools, but this is something I've been wondering about lately and might ask the management.

I'm actually no longer working on this problem so I don't need an answer to this question any more, I found the time delay in reading all PDF's for my implementation was not too prohibitive so I went with that. But it's an interesting case to face in Alteryx development.

apathetichell

@bmillar777- I just assumed I was a curmudgeon for hating on control containers - looks like the product has issues. So back to old school containers.

1) you'll need a batch macro.

2) you'll need an old school container in the macro (set it as disabled).

3) you'll turn your filter tool into a boolean.

4) you'll add a control parameter

5) you'll hook up an action tool -> it will feed from the control parameter to your container

6) - I prefer update value with formula here not enable/disable container.

7) you'll update the value which says disabed = true (or enabled=false or whatever it says) using a formula like ![#1] or [#1] whichever you are using based upon your boolean logic.

Test this with one. If it works -> replicate to all of them (or build out the dynamic logic so one macro can process them all. That's your call. I am high 90s% sure that you can do this -> because I have done this. Albeit on like 2021.3 so who knows.

I'd show you a sample but 1) I don't use that PDF tool 2) that is moot because I currently do not have an Alteryx license and do not use Alteryx at my 9-5.

bmillar777

If I understand your meaning correctly, this is the kind of setup you mean?

If so, this does not work because if the container is turned off by the Boolean filter in the workflow, the workflow fails with "The output of the macro is not valid". If the container is turned on, it works fine.

Screenshot 2025-04-02 171336.png

Screenshot 2025-04-02 171350.png

apathetichell

disable the container. The container in the macro should start as disabled.

the action tool should enable the container when you want it to run (and should have required input fields.).

NeoInfiniTech

An alternative solution to this that doesn't require the use of any macros could be turning this into an analytic app, where the full path would have to be provided through a Text Box or File Browse tool from the Interface category, then an Action tool originating from the Interface tool of choice would connect to the Text Input tool to modify the full path provided, while another Action tool that would also originate from the Interface tool of choice would connect to the Union tool to modify the raw XML of the Union tool. If the file names are standard and RegExable in terms of dynamically determining which container(s) would be disabled or enabled, then this information could be used to dynamically delete the incoming metadata line (#1, #2 or #3) and/or any other configuration lines from the Union tool's raw XML.

If you are not familiar with the XML structure of the workflow, you can open the workflow file with Notepad or any other text editing program and/or enable displaying the XML of individual tools from User Settings in Alteryx Designer.

I could prepare a sample workflow but I'm away from my computer right now.

The only caveat with using this method would be to use the Debug Workflow option provided in the Interface Designer each time if you wish to see the progress directly for performance profile purposes.

apathetichell

@NeoInfiniTech- I'm fairly sure (high 90's confidence) that this won't work for python tools.

NeoInfiniTech

@apathetichell, I understand that the PDF tools are Python based, however I think it shouldn't prevent this method from working as the connection numbers are pre-determined and only the Union tool's raw XML is modified.

As another solution, I could recommend using the Mass PDF Input tool, however it might not be possible if Predictive Tools package is not installed or R packages are not installable due to firewall restrictions by IT. Also, only reading text from a "digital" (not OCR scanned but Saved As from a Word document for example) PDF is possible with this macros, as opposed to the OCR abilites of the PDF to Text tool, which is a part of the Intelligence Suite add on. You might also notice minor differences in how the text is parsed (whitespace after paragraphs starting with 1., 2. etc.)

With all that being said, I believe it is worth mentioning that PDF reading is much faster with the macro I provided the link to (as opposed to PDF to Text tool), which I can confirm after having tested and compared both tools performance wise for a project I worked on (might be due to the time it takes to initalize the Python package but cannot tell for sure).

One caveat for this tool - while I didn't encounter it recently - is that you might come across an error (unlike the PDF to Text tool found in Intelligence Suite package which might be more reliable) reporting that it was not able to parse a particular PDF (even if the necessary R packages are installed and you are able to read in other PDF files) - this might happen for a number of different issues (the following being hypothetical) such as the PDF being too large to be parsed by the R tool used by the macro, encoding of the PDF not being compatible or PDF having been created by a program that is not able to generate a PDF properly to make it possible to be read by Alteryx (a similar issue can occur with XLSX files generated by certain program or ERP software, when you try to input them in Alteryx they would cause an error, you would have to open the XLSX files in Excel and Save As the problematic files to solve the issue (whether manually or by PowerShell/VBA), but the same method might not work or be possible for PDF files).

apathetichell

@NeoInfiniTechthe problem is that the union tool is reporting the error - but the error actually is occurring at the PDF tool - but not being flagged until the union tool. having said that -> I'm not sure if that's easier than the classic disable container/enable based upon value method which was used prior to control containers.

I'd recommend that - if possible - when reconfiguring to a batch macro with a container @bmillar777 should replicate the logic in the container so he doesn't need four different containers and can instead reuse the same macro four times with different variables/options being passed in.

apathetichell

Hey @bmillar777 - did you get this to work when you went back to standard containers and configured them initially in disabled mode?

bmillar777

No, sadly it made no difference.

apathetichell

yucky - sorry.

bmillar777

No worries I don't believe there is a simple solution to this problem, but if I face it again, I'll be interested to try out some of the more complicated ideas I have or were suggested here.

CoG

One ugly way to solve this problem is to introduce temp files that act as intermediaries between the output that you need to be able to toggle on/off, and the union tool.

The general approach looks like this:

1. Initialize a temp.yxdb output with a Text Input Tool that has no rows of data inside a control container.

2. Copy paste that output tool into each toggle-able Control Container (assuming that only one should run) and terminate that section of the workflow at the Output Data Tool (i.e. no more connection to the Union Tool)

3. If you need to union other things you can now add an Input Data Tool referencing your temp .yxdb (also Control Container so that it runs last in the list).

4. If you don't need to union other things you could just use the temp.yxdb as the basis for the next step of your workflow.

This removes the need for macros. The problem is 100% in the Union tool and is caused by Runtime disabled Control Containers/Containers. I don't know why it has this behavior, but you will always get that error. In newer versions of Designer, the error can be avoided by ensuring that the "beginning" of a workflow section that connects to the Union is what is disabled, but it still stops the remainder of data from flowing through the Union Tool, which doesn't really solve the problem.

If you ever need this again (or someone else is facing this issue), there are work-arounds.

Hope this helps and Happy Solving!

flying008

Hi, @bmillar777

Hi,
IMHO, What is the purpose of using a control container?

************
1- If the test1.pdf or testN.pdf files do not exist, then the control container will not output any data to the Union tool?
2- Used to manually control the switching of testN.pdf output?
3- If the above guess 1 is correct, does it mean that after deleting all three PDF recognition tools from the canvas, the same error message by union tool as above will be repeated at runtime?
4- Can you avoid using AMP mode?

************
If the answers to all four questions above are yes, congratulations, maybe you don't even need the control container in this scenario.
Please explain your requirements in detail. Perhaps we can implement it in the simplest way possible.

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15867

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703