How can you speed up workflow

Question

How can you speed up workflow?

As you can see in the figure below: 1.5GB of data (927,582 pieces of data), using the tools in the red circle for batch replacement operation, running for about 30 hours, and then pushing the data into Tableau server took another 20 minutes. Is there anything the community can do to increase the speed?

wlau · Answer

Is it possible to reorganise the workflow to carry out that big text operation in a database engine of some sort?  Ideally, do the processing using in-DB tools upstream, and then bringing into the workflow for Tableau later?  I get the feeling that even in an un-optimal setup of pushing 1.5Gb to a database in the middle of the workflow, just to process the 1mil rows and then downloading it back into the workflow should be less than 30 hours.

danilang · Answer

Hi @pg_two

You have your Dynamic Replace configured to Autofield Field Types.  This means that Designer has to scan all the data to determine the best field type after ever pass of the tool.  If you can, use the preceding Select tool to set the correct output field for all your post processed records and use Keep Original Field Types in the Dynamic Select.

Dan

pg_two · Answer

Thanks，@FrederikE

You may be right about tableau release loss for such a long time. This is the formula I used here.