Optimal Workflow Size

Question

I'm working on a project that is turning into a rather large workflow file.  The problem I'm having is that each time I either add a new tool, modify an existing tool, or connect two tools, my system pauses, Designer says its not responding, then after 5-10 seconds Designer comes back to life and I can continue on.  This cycle continues anytime I try to do anything and is very annoying.

If I'm working in a small workflow, I have no response issues.

So I'm wondering is there a practical limit for the size of a workflow before it becomes more of a pain to work on than having several small workflows?

As there are still plenty of process steps to add to the workflow, I'm looking for suggestions how I can make a workflow perform well at design time.

Below is what the workflow looks like.

Many thanks in advance!

danilang · Answer

@Winston

You can also look at caching the data.  If you're making modifications at the end of your workflow, you might be able to cache the data at one of the points before. This might also speed up the metadata refresh, though I'm not sure about that. Check out this article for the conditions where you can and can't cache.

Dan

Winston · Answer

@TheOC thanks for the suggestion, it makes adding in new tools much faster but as @danilang pointed out, it doesn't populate metadata downstream until you run the flow again.  As the flow takes a while to run its quicker to wait for the tools to slowly update than to have to run the flow after adding new tools.  I will have to try using F5 to how that works time wise.

@danilang You are correct that there are several parts that are repetitive and are coming from different sources for matching.  Because of that the data from each source is different and thus requires different data manipulation before being put through fuzzy match testing.

I am working on a Matching Macro that takes data in as generic field names and then its just a matter of what fields I assign to the macro inputs.  Unfortunately, this will only replace my fuzzy match tools  so not a lot of savings other than no longer having to figure out why one match tool works but the next one doesn't.

Thanks for the suggestions!

TheOC · Answer

great shout @danilang!

I couldn't see that much from the images, your eyesight is better than mine!

And I also didn't know you could force a Metadata refresh with f5, that's really useful to know thanks for sharing that insight!