workflow optimisation

[raju_coder]

Hi Team,

I have a workflow which runs for 5 hours, when I see the profiling report , there is no major time consumption tool , however the total time consumed by all the tools is 3 hours approximately including writing the data to a table .But I also see 4500 warning and 1500 conv errors , are they consuming rest of the time?

[Pawel_Paleczny]

Hello Raju,

I saw in your file that you are working on cloud stored data, you cannot do all the transformations using only 'In-Database' tools? This approach might be more efficient.

[MattBSlalom]

Though the errors are concerning, they are not likely consuming the execution time.  Looking at your logs, there's not a single or very few actions that seem to be the culprit.  Instead, I expect this workflow is VERY complex (which is likely why there are so many errors being reported that haven't already been addressed).  Without actually reviewing the workflow, the best suggestion i can make is to try to trim down the volume of data (both rows via Filter tools, and columns via Select tools) wherever possible.  The need to propagate large datasets through a workflow (even if fields / rows aren't used) can cause significant overhead.