Data issue union with an empty input

[Cutugnos]

Hi everyone,

We're encountering an issue during the implementation of a workflow and would appreciate your insights.

Context:
We're pulling data from two tables (MARA and MAKT). Occasionally, one of the two input of the union may be empty. When this happens, the Union tool's output is also empty, and the workflow fails to populate the expected data downstream.

We've tried a few workarounds:

• Using the CacheDataset Macro to materialize the data in Alteryx's cache. However, this requires running the workflow twice, which is not ideal given its complexity and runtime.
• Disabling AMP engine or using an Action tool to cache the dataset in a temporary folder. Unfortunately, these approaches still result in an empty output unless both input streams to the Union tool are populated.

We're looking for a more efficient solution that allows the workflow to handle cases where one of the input tables is empty, without compromising the output or requiring multiple runs.

I've attached a screenshot to help illustrate the issue. There are also some try and workaround we tried!

Thanks in advance for any suggestions or ideas!

 

[jrlindem]

Could you consider using a Control Container to introduce conditional logic.  If the stream is empty go one direction, if not go the other?  Meaning, if no data, then don't union.  If there is data then union?

[Cutugnos]

Unfortunately I can't use it, we're in a Macro envirnoment

[jrlindem]

Interesting.  I'm not sure why being in a "macro environment" is prohibiting you.  Are you saying that where this is occurring is within a macro you don't have access to edit?

For what it's worth, I'm not convinced you're not having another problem that's not associated with the union.  The union tool should be outputting the values from either stream inputting it regardless of whether one or the other contains no values.  I've tested this, best I can, on my end and my union tool is working fine when the two streams are populated, and/or when only one is...

I still think there's a path to leverage conditional logic to insert a dummy row of data when the input stream is found to be empty and then simply filter that dummy row out downstream, simply to ensure your union is allowing data through.  But again, I'm not sure why this is necessary...  I would recommend copying the workflow and removing out other parts to isolate down to the union tool in question and verify, for sure, that it's really that tool that's causing the problem.

Sorry i can't offer more specific solutions, but hope that helps.  -Jay

[dreldrel]

Maybe use the block until done tools if you cannot use the control container?

[Mathias_Nielsen]

I agree with @jrlindem, we have solved similar issues with dummy records and then filtered them out after. We did not use conditional logic for this though, just did it for all inputs that might be empty :)

[Gaurav_Dhama_]

When you say that the data flowing into union is empty from one stream, is it because the container that passes the data is disabled or data is just filtered out due to filter or some join?

[MinhLO]

@Cutugnos  This is a bug I raised to Alteryx a few months ago. Unsure if they are planning to fix it.

For example if a container is disabled before an union, then you will get an error message like 'No incoming metadata from one or more upstream tools'. Can you confirm this is what you see?

To solve this issue, we took two actions:

- on workflows where it was possible, we removed AMP.

- on workflows where it was not possible (for example we were using a custom tool with latest SDK, hence AMP needed), we segregated the workflow in two different workflows. One workflow would conserve AMP and create the dataset, the other one without AMP would consume this dataset.
However this is highly linked to our uses cases, hence it might not be reproducible for you.

In a nutshell:
- If you can, remove AMP

[Cutugnos]

@MinhLO

Hi MinhLO, we've already contacted them, I followed the steps you recommended but unfortunately it still doesn't work. We removed the AMP as well, the situation is the following, really simple, then one of the two Summarize is empty (and we don't know when, it is aleatory) the union doesn't reproduce an output? Do you have any ideas except: 

use a control container, it is not feasible for this workflow split in two

use a container to disable the empty summarize, it still doesn't work and we don't know when the summarize is empty

use a cache, we should run twice every workflow

or we don't know...we're still looking for a workaround

thanks to everybody for the suggestions

[Gaurav_Dhama_]

@Cutugnos 
Since i cannot see whats before the summarise tool, see if the below helps.

If you have a container that gets disabled based on some scenario, then any tool outside this container and before union will fail your union tool.

To handle this scenario, put your summarise inside the container which gets disabled. This should fix your issue.