Automatic (efficient) way to deleting multiple workflows from server that meet criteria

[tdiroff]

Hi

In a heavily used server environment, and depending on how workflows are deployed, it is possible to accumulate a vast number of "one off" workflows that could/should be deleted  as they would never be used again. In one of our environments we have over 1 million of these.

Currently we are manually deleting them 500 at a time but have asked if there is a way to script the delete process to make it more efficient. We have been told that to really delete a workflow you would need to touch at least 4 collections.

Can we have a Delete workflow API in one of the next releases in order to address this issue?

Thank you

Tom Diroff

[tdevrie1]

This is a big issues when automating the creation of workflows (workflows creating workflows).  Clean up is very tiresome, risky (deleting!!!!) and tedious given the current interface restrictions

[KylieF]

Thank you both for your feedback!

We appreciate all the feedback we get for our products. If you haven’t yet, please be sure to review our updated Submission Guidelines as it goes over the criteria and idea boards in a bit greater detail.

[JohnPelletier]

@tdiroff @tdevrie1 Thanks for the feedback. I'd love to know more about your use case! First question is, why are you creating so many workflows that are one-offs? Can you give an example of when you would create one of these and why? What can it do that an ordinary re-usable workflow cannot?

Secondly, I just think it's so cool that you're using a workflow to create other workflows, and that you've created over a million of them. Why do you need so many?

If there are confidential details that you'd rather not post publicly, please reach out to me with a personal message. @JohnPelletier 

Thanks! 

[tdiroff]

Hi John

It is a very cool application (at least I think so)

Background: As a shipper you can save money if you can efficiently pack the semi's and ship containers with your goods so that there is no unused space. If you are a large shipper like Ford, you can save lots of money.

The main workflow in question runs multiple times a day with each run covering a different geographic or functional area.

When the main workflow runs it calculates and spawns over 1000 different workflows .. each spawned workflow pertains to an individual shipping instance (semi or container). 

The application was designed  to use the "addtoqueue" method as there is no Gallery in that environment consequently each run of the main workflow will generate over 1000 single use workflows and over time we can reach over 1000000 workflows registered with the controller.

Does that help?

Tom

[JohnPelletier]

@tdiroff I agree, it sounds very cool!

From the product perspective, I'm trying to understand why there is a need to spawn so many one-off workflows instead of creating wizard-based apps where you can change the parameters and/or input files of a single workflow. How different are these workflows from each other? What are you missing in functionality that necessitates the creation of one-off workflows, instead of the more manageable and reusable kind?

For example, would having more automation logic help?

Why not have one "fill_container" workflow that takes some inputs depending on geo and function?

Also, would it be possible to use the POST /jobs Gallery API instead of the add_to_queue command? If not, why?

Thanks for the valuable feedback! You're giving us great insight into your interesting use case. And if there's any additional info that you'd rather share in a private message, please feel free to message me directly.

[tdiroff]

HI John

In this case there is a "master" workflow that accesses a database with all of the individual container or truck information and based on that information will spawn individualized workflows to process that specific container or truck information. This processing is time sensitive (over 1000 need to be finished within 30 min) therefore, to leverage the entire processing power of the workers in the environment the 1000 workflows are spawned and sent to the available workers.

There is no Gallery in this environment so API calls could not be used

Tom

[JohnPelletier]

@tdiroff Tom, is there a reason why the Gallery and its APIs cannot be used? If you had access to the APIs, then we have more options for how to solve these problems. We could help you to have just one analytic app, with different input files and/or inputs to parameterize the workflow and then you can run several instances of the same workflow at a time. This would eliminate the need for clean up of 1000s of one-time workflows, because you can just use one.

[tdiroff]

Hi John

The Gallery and it's API's can't be used because there is NO Gallery in this environment... we only have controllers and workers. So the workflow was designed to use the add_to_queue method... that's all we have and in a big corp is it surprisingly difficult to get new resources without enormous justifications and expense, so you use what you were given. (and that's not counting the effort to redesign the workflow and the risk involved)

So, forget about our no gallery environment for a moment.. even if we had a Gallery... over time, in any heavily used environment, a certain amount of crud will exist... so even if you don't want to clean up thousands of "wrongly" created workflows what about the need to keep the DB clean of other old stuff?

Tom

[KylieF]

Thank you for your feedback!

Our product team is currently reviewing this idea and looking into the technical requirements as well as most optimal method for potentially implementing this idea. Once we have completed this review we'll be sure to update the status of this idea and if we're able to include this feature on the near future road map.