Limited Data Handling Capabilities

[CBoardy98]

Hi All,

I am creating a pretty large work flow which will need to be able to handle +300k lines.

I am doing this by using Desktop Designer, I have reached a point where a single part of this project is complete. I have continuously reviewed and checked everything is flowing correctly and can confirm the Workflow should work 100%.

I am grabbing from multiple data sources that are pretty large data sets & then doing calculations and mapping the data.

I can see that this Designer is limited to 8 Threads & 4GB of Memory, am I limited due to my Small Laptop?

While review I was struggling to determine why input wasn't matching the output and have reviewed thoroughly.

I discovered that data was somehow being lost due to the shear volume of data, I duplicated the workflow and ran each data source separately through the exact same workflow and the numbers match. Is there any work around this issue without having to create individual workflows for each data source, as this will slow down the project a lot as this is the only first step.

I have been looking through the Results Log and can only see on thing which potentially shows the issue which I will attach below. 

[Raj]

If the data is getting Correct for each individual data set, then you should try converting the workflow to macro and running for each individual data set as this will limit the possibility of getting error.

[CBoardy98]

Thanks for your input.

I am pretty new to Alteryx so don't know many of the Abilities but I also have access to an Alteryx Server.

If I upload it to there is it more likely to work without any errors, a bit hesitant to create a macro but will look into it tomorrow.

[DavidSkaife]

Hey @CBoardy98 

You may have to go through the whole workflow section by section to see where the data is dropping out (if you haven't already identified that). You have a lot of Joins and other tools where there is the potential for some data to drop off.

If the workflow doesn't work on Desktop then its very unlikely to work on the Server. If you're new to Alteryx i'd not try delving into macros yet, identify where the data is going missing first then look to streamline it with macros.

If you're able to upload some dummy data then that would also allow the community to help, but appreciate this may not be possible given the volume!

[Tam · Accepted solution]

Hi @CBoardy98 

There are a lot of join tools, you'll need to check the L & R output anchor. 

I would suggest that you use container combined with caching to debug where you are losing data. 

The alternative to is break the workflow into multiple workflows and saving them as an analytic app and chained each apps after the successful run of the previous, this would require outputting the results at the end of each workflow into a file to be use by the subsequent analytic apps. 

HTH. 

Tam 

[CBoardy98]

Thanks a lot for all your input.

Decide best for now is to split the data sources into similar workflows so that it's not as heavy of a load and just to merge them together later on.

[apathetichell]

TBH - that's not a huge dataset re: Alteryx - and I havne't seen  record loss due to memory issues (I've seen an inability to run the workflow/crashing or the workflow taking forever). so here are the reasons why you may be losing records:

1) you are filtering for a value which you think is there but isn't.

2) you have an inner join where you expect there to be a 1-1 match but something isn't joining (very common)

3) you are brining in a file which is being dropped (or a column you are filtering for downstream is dropping) because of schematic differences (ie field name in one file is TEST and in the other it's TESt and Alteryx is creating a new field - or in once case TEST is a number and in the other it's a string).

4) Amp is doing funky Amp things.

So the questoins to ask are:

1) what version of Alteryx are you using

2) Is amp on (does it have to be)

3) Can you check out any the outer right/left anchors of any joins for missed records.

4) Can you check any filters to see what's being dropped

5) Do you have an detours/control containers?