Loading a workflow causes massive memory usage before running

[marco_zara]

Hello,

has anyone ever noticed some workflows using over 0.5GB of memory upon loading?

This is before running the workflow, for some reason any change locks the interface up to a minute and it almost feels like there is a memory leak somewhere as RAM usage goes up with every change.

Has anyone experienced anything similar?

[marco_zara]

Hello JPoz,

I can't provide the input easily as it's customer data but I suspect you could reproduce the issue by taking a file with lots of categorical data and using cross-tab to transpose it to columns, a long enough list of cities could do it.

The file has approximately 700 columns produces by a cross-tab tool, which for every created field creates the following:

Spoiler

source="Formula: IF [_CurrentFieldName_] IN("Customer No","Segmento Macro","Segmento Profit","Mercato","CLUSTER FATTURATO CERVED","CLUSTER ADDETTI CERVED","Canone Dati","Canone Fonia","Canone WLR","Traffico Dati","Traffico Fonia","Incoming","Attivazione - CIA","Grand Total","Duration in Minutes IN","Duration in Minutes OUT","Duration in Minutes Total","Rated Amount (euro) IN","Rated Amount (euro) OUT","Rated Amount (euro) Total","CHURN PARZ.","RICONTRATT.","CHURN TOTALE","Ordini_DATA_CENTER","Ordini_ICT_MANAGED_SERVICES","Ordini_MOBILE","Ordini_NMS","Ordini_SECURITY","Ordini_SERVIZI_CORE","Ordini_UC","Euro_DATA_CENTER","Euro_ICT_MANAGED_SERVICES","Euro_MOBILE","Euro_NMS","Euro_SECURITY","Euro_SERVIZI_CORE","Euro_UC") THEN   
       IF IsNull([_CurrentField_]) THEN 
     IF ( [_CurrentFieldType_] IN ('Byte', 'Int16', 'Int32', 'Int64', 'Float', 'Double', 'FixedDecimal')) THEN
        0
         ELSE [_CurrentField_]
     ENDIF
 ELSE [_CurrentField_]
 ENDIF
    ELSE [_CurrentField_]
    ENDIF" type="Int64" />

This is a big block of code that is apparently being parsed and it tends to slow down the interface.

I've got worse offenders than this but they tend to be bigger workflows, in this case it was blatant because a very simple flow is massively slowed down.

A "simple" (in logical terms, code sadly almost never is) solution would be an option to stop formula tools from adding metadata to created fields.

This could also likely help fixing the fact that copying a field name in a select tool also copies all the metadata, which is not exactly handy.

If you're interested in "big, slow" workflows I can provide other examples but sadly no input data.

[jpoz]

You're right, the data files themselves aren't needed, this is entirely an issue with the meta info being stored in the workflow. I think I have a potential solution to this.

Can you or anyone else send me more workflows with similar loading problems? It would help to get this issue fixed. Thanks.

[marco_zara]

Here it is, this is a dataset that I'd have loved to show at Inspire as well...

This one might be justified in its slowness though, although as you can see it's quite big for an XML file.

[jpoz]

Thank you for providing those workflows.

I was able to make several performance improvements to the Designer using these workflows. For example, on my computer the modellol.yxmd file was taking roughly 18 seconds to load. It now opens in about 2 seconds.

I will continue to try to optimize, and any additional workflows you can provide would be helpful.

[marco_zara]

Here are a few more workflows where the designer grinds to a halt when opening or doing any change, even using an SSD and a Xeon E5-2620 with 256GB RAM

Thank you very much for any optimization!

[marco_zara]

Resurrecting this thread for another problematic workflow: unzips to over 400mb and cannot run with 16GB ram as it kills most available memory.