Cache Option on Every Tool

[JohnJPS]

Idea:

I know cache-related ideas have already been posted (cache macros; cache tools), but I would like it if cache were simply built into every tool, similar to the way it is on the Input Tool.

Reasoning:

During workflow development, I'll run the workflow repeatedly, and especially if there is sizeable data or an R tool involved, it can get really time consuming.

Implementation ideas:

I can see where managing cache could be tricky: in a large workflow processing a lot of data, nobody would want to maintain dozens of copies of that data.  But there may be ways of just monitoring changes to the workflow in order to know if something needs to be rebuilt or not: e.g. suppose I cache a Predictive Tool, and then make no changes to any tool preceeding it in the workflow... the next time I run, the engine should be able to look at "cache flags" and/or "modified tool flags" to determine where it should start: basically start at the "furthest along cache" that has no "modified tools" preceeding it.

Anyway, just a thought.

[JohnJPS]

Idea:

I know cache-related ideas have already been posted (cache macros; cache tools), but I would like it if cache were simply built into every tool, similar to the way it is on the Input Tool.

Reasoning:

During workflow development development, I'll run a workflow repeatedly, and especially if there is sizeable data or an R tool involved, in can get really time consuming.

Implementation ideas:

I can see where managing cache could be tricky: in a large workflow processing a lot of data, nobody would want to maintain dozens of copies of that data.  But there may be ways of just monitoring changes to the workflow in order to know if something needs to be rebuilt or not: e.g. suppose I cache a Predictive Tool, and then make no changes to any tool preceeding it in the workflow... the next time I run, the engine should be able to look at "cache flags" and/or "modified flags" to determine where it should start: basically start at the "furthest along cache" that has no changes preceeding it.

Anyway, just a thought.

[MacRo]

Hey @JohnJPS, turns out this is already on the roadmap according to @BenG!

[Atabarezz]

This boils down to the difference between pentaho and KNIME right?

 I'm looking forward to it...

[Atabarezz]

this is the prime capability of SPSS modeler as well, any updates on this?

[Karlo]

This would be very useful indeed. Hoping this is implemented sooner, rather than later :)

[AlexKo]

Thanks for posting about this! We're currently evaluating this for a future release.

[AlexKo]

Thanks for the idea @JohnJPS! We're releasing a version of caching in the upcoming 2018.3 release, which allows you to right-click on a tool in the workflow and create a cache temp file. Next time the workflow is run, it would start at that cache.

Stay tuned! I'd love to hear what you all think once you get a chance to play with the new feature.

[Karlo]

@AlexKo that is great news! Any idea on the release date for version 2018.3?

Thanks

[AlexKo]

Caching's live in 2018.3!

Thanks for pushing this one up on the priority list for us. We hope it will help streamline your workflow development and rapid prototyping use cases.

[JohnJPS]

Thanks Alex. 2018.3 absolutely rocks!