Massive increase in size of workflow on conversion to analytical app

[olimpio]

Hi,

I have a workflow the size of the workflow is 709kb however, on converting it to an analytical app the workflow size increases to 261MB, slowing down the entire process.

has anyone faced this issue? what am i doing wrong

i have tried to resave the workflow as well but everytime i do it the same thing happens. any help would be really appreciated

[MarqueeCrew]

@olimpio ,

I've done (shot myself) this before by using an IMAGE for the app/macro that is big.  The other cause is if you are storing data in TEXT INPUT tools.

Do either of these causes resonate with you?

Cheers,

Mark

[olimpio]

Hi, im not using an image or a text input. Its a workflow that has multiple queries embedded within, I tried the workflow is just 709kb and it just increases to 250k + MB once i save it as an app really strange

[danilang]

hi @olimpio 

Are you adding any interface tools, when you save it as an app?  Could these interface tools be storing cached versions of their inputs in the XML?

We had the file size increase happen as well with a case where we generate a dataset as xml before passing it as an input to a stored proc in a Dynamic Input tool.   While developing the workflow we were using a small sample xml and the file size was around 70K.   When doing functional testing we ran with an input dataset that generated a 100MB xml file.  When we saved the workflow, we found that the entire xml was saved in the MetaInfo node of the Dynamic Input tool.  If we run the workflow with the sample xml and then save it, the size decreases back to 70KB.  

Have a look at the raw xml of the workflow as an app and see if you can find a very large chunk of text stored in the MetaInfo Node of one of the tools

Dan 

[olimpio]

Hi Dan,

Thank you for responding.

The interface tools added, are the browse tool that pull in an excel input file containing numbers that are queried through a dynamic input tool, if there are no hits in the table 1, that group will be queried in dynamic input 2, eventually creating an excel output

"Have a look at the raw xml of the workflow as an app and see if you can find a very large **bleep** of text stored in the MetaInfo Node of one of the tools 

If we run the workflow with the sample xml and then save it, the size decreases back to 70KB."  

1. How do I look at the metainfo ? 

2. How do I run the workflow with the sample XML?

Regards,

Olly

[olimpio]

Hi Everyone, the issue still continues any solution would be really helpful

[danilang]

Hi @olimpio 

To see the xml of the workflow, open the file using a text editor,  Notepad++ if you have it or just Notepad.  Look for the MetaInfo elements.  Most of them should look like this with one line or maybe two for each Field

The MetaInfo element with all the extra text data looks like this.  The text in the first field extends for 5000 lines.

Once you find what the issue is, you should be able to see where the data is coming from.  In our case it was a massive piece of XML that was passed in as a parameter to a dynamic input tool.  To reduce the size of the workflow, we used a small sample input file, which generated a small XML.   This small XML was saved as the MetaInfo in place of the much larger one. 

You'll have to do the same process.  Find out where the giant data is coming from and use a small sample input instead of the normal large one. 

Dan

[olimpio]

Hey Dan, thanks for the detailed explanation

I did  review the workflow in Notepad++

Few points i would like to mention:

I did find the metainfo, its the output from the dynamic input that seems large if i am reading it correctly. the only this is the worflow is set up to run as follows:

1. Input could be 1 account or 1000 or more and the output could be large between 1000 - over 1m may be.

2. I tried using a smaller input of just 10 accounts but its still the same

[danilang]

@olimpio 

The metainfo isn't used when the workflow is run other than to validate field types and sizes.  It has no impact on data that passes through the workflow when it's actually running.   

What you need to do is run the workflow with an input that generates the smallest possible .yxwz file size and save the workflow after running with that input.   From then on run the workflow using this .yxwz file. 

If you're still developing,  make sure that the last run you do before saving your final copy of the workflow is performed with the small input file.

Dan     

[olimpio]

Hey Dan,

What i just realized is, the workflow  i.e. the yxwz file.

If I click on the run button instead of the magic wand that is when the same thing i.e. increase in the size from a 700kb to a 200MB file occurs, is that the same issue of the meta data info or is this a different issue