Alteryx Union With Dynamically Switchable Control Containers

[bmillar777]

Hi everybody, 

I've got a question that's been bothering me for a while and I haven't been able to find a solution to. I've included a screenshot with a simplified example of the issue. 

It relates to a scenario where I have a set of PDF reads to do and have to dynamically come up with a solution to determine which ones. This works well using the Filter tool to turn containers on and off and if all processing is done inside each control container, all is fine. 

However, if I need to combine the results outside the control containers, the Union tool fails with the "No incoming metadata from one or more upstream tools". I've tried various solutions to resolve this issue with no luck. Limiting the number of PDF reads is critical for my use case as one of my routines is already taking over 2 minutes to complete, unacceptably long. 

Does anybody have any ideas on how I can resolve this? 

Thanks :)

[apathetichell]

@adighe- I use text input - and I'd actually use a field name which must be in the union tool (ie something from your parsed download tool data). It does not have to have rows -> it just needs a field name and it will populate into the union tool.

Reiterating - old school this worked fine -> control containers may be 'unique'

[bmillar777]

Unfortunately, I don't think any of these solutions will resolve the original problem, I think I've tried them all already. Let me define the issue with bullet-points.

* The original purpose was to read as few PDF's as possible. If the problem could be solved by reading PDF 1, then that's good. If not, then we need to read PDF 2, and then PDF 3 and so on. Containers are necessary for this problem as the PDF tools expect an input if they are not inside a disabled container.

* The screenshot I provided was a simple example I constructed to demonstrate the problem. The following is how things work depending on the situation:-

1) The three Filter tools turn the containers on and off automatically depending on the data in the Text Input tool. However, if we ignore those tools, the same problem still applies. The real issue is with the containers and the Union tool.

2) If we leave all the containers switched on, the workflow runs perfectly. However, if we disable even one, the workflow fails reporting that no "incoming metadata could be found from one of the upstream tools"

2.1) It seems that having no input at all from a disabled container is a problem for the Union tool which generates the error.

2.2) No setting changes in the Union tool change the issue, not even the non-blocking one.

3) The problem doesn't exist if all processing is done inside an individual container. For example, having three Union tools, one in each container doesn't generate any error. But then you can't rejoin the rest of the workflow to do further processing. In my use case, that's a problem as after reading one PDF, I need to exit into the workflow to determine the next PDF container to switch on before doing further processing.

4) Using any kind of macro isn't a solution here for the same reason as the Union tool, you need to leave the container to exit the macro and that can't be done without error. 

I think a possible solution is to use nested workflows. Then you could remain in the container, write an output and then read it in the next workflow etc. But it's a very complicated solution which I didn't try. Another possibility is to use nested control containers, but if you have ten PDF's to read, that's going to be very ugly :)

[apathetichell]

Hey @bmillar777  - sorry was confusing the two tool sets - I think part of your problem is that the PDF tool you are using uses the Python tool. The Python tool wants to run - > even with no data.My hunch would be:

1) disable all containers.

2) create a text input with whatever downstream schema you need.

3) run -> confirm that it does not error.

4) set up your containers to optionally RUN - not optionally NOT RUN. Re test.

If that does not work -> I'd try 1) putting each container in a separate macro.

2) going old school and using disabled standard containers vs control containers with them set to run with data (and their standard setting being disabled) - again in a (Batch) macro -> albeit you could use one.

[flying008]

Hi, @bmillar777  @adighe 

1- No matter you use PDF to Text tool or Download tool, they will cause no output for some reason. So the problem is not the control container, but the fact that as long as one of the incoming connections of the Union tool is an empty connection (no metadata is passed in by the upstream tool), an error message "incoming metadata could be found from one of the upstream tools" will be reported.

2- So you need to analyze which step before the Union tool may have no metadata output, either create a text input with whatever downstream schema you need as @apathetichell  said, or use dynamic input to process the previous data later as I said above.

3- In fact, there is another solution here, which is to use the Branch plug-in similar to Until ...to or If...then logic to implement the branch process. It does not require the use of the Control Contain tool, does not rely on the AMP mode, and does not require the use of multiple nested macros. But since your company may restrict the use of such 3rd external tools, we will not discuss it in detail.

[bmillar777]

Hi all, thanks for your replies. 

@apathetichell, I'm not sure what you mean by optionally RUN, I assume you mean disable the control containers and then enable them using the Filter tool. If that's what you mean, it doesn't unfortunately work as the Filter tool won't enable a disabled container. 

@flying008, a control container is necessary for this procedure as the PDF tool needs an input if it's not disabled. So when it's disabled, it will always lead to an empty connection in the Union tool. Adding a Text Input tool to add input doesn't work as any joins or unions will simply receive no input from the disabled PDF tool. It's a circular problem.

A dynamic input solution could work, it's not something I've tried to date. I don't think we can use third party tools, but this is something I've been wondering about lately and might ask the management.

I'm actually no longer working on this problem so I don't need an answer to this question any more, I found the time delay in reading all PDF's for my implementation was not too prohibitive so I went with that. But it's an interesting case to face in Alteryx development.

[apathetichell]

@bmillar777- I just assumed I was a curmudgeon for hating on control containers - looks like the product has issues. So back to old school containers.

1) you'll need a batch macro.

2) you'll need an old school container in the macro (set it as disabled).

3) you'll turn your filter tool into a boolean.

4) you'll add a control parameter

5) you'll hook up an action tool -> it will feed from the control parameter to your container

6) - I prefer update value with formula here not enable/disable container.

7) you'll update the value which says disabed = true (or enabled=false or whatever it says)  using a formula like ![#1] or [#1] whichever you are using based upon your boolean logic.

Test this with one. If it works -> replicate to all of them (or build out the dynamic logic so one macro can process them all. That's your call. I am high 90s% sure that you can do this -> because I have done this. Albeit on like 2021.3 so who knows.

I'd show you a sample but 1) I don't use that PDF tool 2) that is moot because I currently do not have an Alteryx license and do not use Alteryx at my 9-5.