Real-time insight into Gallery queue

[Frederik1]

Hello Community,

Is it possible to do real time monitoring of the current state of an Alteryx Gallery? I want to be able to inspect the current queue of the server, what jobs are queued, who owns them, when they were queued, etc. such that the admin can better understand what and who causes these queues, possibly in a visualization in Tableau.

I know that Alteryx has built a macro that extracts such data from the Gallery and outputs it in Tableau, but it needs to be relatively real-time to satisfy the purpose of immediate action to manage the current queue. Here

It is to be used on a busy server that is often suffering from long queues and thus it is best if the solution is not related to running an Alteryx Workflow (as this could end up in the queue as well worsening the issue.) 

I have investigated the Gallery API for the local host URL, but it doesn’t seem to include the relevant data in question.

Also, I have read in the Community that log files are locked when the given job is running, so this is also not an option as far as I understand.

I am very curious to hear any input to this. Did anyone succeed in doing something similar?

Best regards

Frederik

[mbarone]

This is much needed and would be very valuable.  The only thing that comes close, out of the box is looking at the Queue via Designer, Options, View Schedules, Queue.  I've heard of people scraping the queue with the Mongo Input tool and doing funky things with that, but it's very complex.  While I've built plenty of Mongo modules using the Mongo Input tool, nothing comes close to what you describe.

[Frederik1]

Thank you for your quick reply mbarone.

I looked into your suggestion of looking at the queue via Designer and it seemed promising, but not quite the thing since it is too much of a requirement to open Designer and look at the jobs there. More information is needed and in a more neat display.

However, from further experimenting with the API it actually seems to be possible with the GET /admin/v1/workflows/jobs/ endpoint.

Then it should be possible to enrich it by joining that with user information from another endpoint and finally getting the computational utilisation from the command line with something like

wmic /node:machinename cpu get loadpercentage

All this can be done in Python and run relatively often to approximate real-time insights.

[mbarone]

Love it, great idea!  If you get it up and running, maybe post a blog about it or even submit it for a presentation at the next Inspire (if accepted, it'll get you into the event for free 🙂 )

[superjesse]

Did you ever come up with a solution? We are searching for a way to see real-time queue metrics to alert us when the queue is getting too large.

[kleofas97]

Hi,

I wanted to get jobs that are currently queued. I have used GET /gallery/api/v1/workflows/subscription/ to retrieve all the Apps and theirs AppIDs.

Unfortunately, when I passed the retrieved ID to GET /gallery/api/v1/workflows/{appId}/jobs/, I have empty list returned. I was told that it might be because of API storing only JobID for runs executed via API. Do you know any way to retreive with this command all jobs history for certain app id?

Thanks,

Adam