Windows Paging file on Alteryx server box

[Steve_Luckcock]

Hi Folks,

does anybody have any information of any settings that should be applied to the paging file in Windows server.

We have been experiencing memory problems on Alteryx server and have been directed to fiddle around with the Minimum sort/join memory usage parameter. Seemed to work - most of the time.

To check if we had fixed the problem, we ran a deliberately hungry workflow and watched the memory usage go up and up both on the server diagnostics and the windows systeminfo command.

The process kicked off with 28GB of available physical memory and 46GB of available virtual memory.

This end up as 700MB and 910MB of available physical and virtual memory before it reached 100%, keeled over and took the server with it.

I was a bit surprised as I expect the Virtual memory to increase as it started paging around 90% usage. to a max of 3 times the physical memory (96GB 3*32).

This is on an AWS EC2 box emulating a 4 core server with 32GB of physical memory.

Does anybody have any experience of these sort of issues?

Any information welcomed

[patrick_mcauliffe]

If the paging started that high, then you definitely have the sort/join memory setting wrong, not the paging.

Change your paging file back to the default system managed.

Go to System Settings --> Engine Configuration --> 

• Default sort/join memory usage (MB)

What is this set to?

[Steve_Luckcock]

Hi Patrick

Paging file has always been system Managed - don't have permission to change it on the machine

The current sort/join memory usage is currently at 5000MB.

Default set of Processing threads is set at 5 

At present trying to fix this we have concurrent workflows set to 1

We have tried setting current sort/join memory usage down to 2000MB without it affecting the result

[patrick_mcauliffe]

@Steve_Luckcock  give this a try...

[Steve_Luckcock]

I tried this configuration. I didn't expect it to work as it is the initial configuration we had back with 2019.1.6  which is when we started having these issues.

According to the deep dive document on this site, for Server with worker a rule of thumb calculation for sort/join memory usage is:-

Physical RAM 

_________________   

2

------------------------------

#concurrent workflows

which i s (((32GB)/2)/4) = 4000MB rather than 8943MB.

However, There was an itial steep clime to about 8943MB then a slow gradual increase to the 100% of available memory and the machine crashed.

This test Workflow is deliberately hostile (gets the mean of about 9.5 million records not using in database), but was put together by a user for this very reason, in order to check out these memory problems we have been experiencing with more sensible workflows.

All these workflows function perfectly on the users desk tops (even this one although takes an excessive amount of time), but they need server to schedule workflows. 

We can accept that an individual  workflow may be ridiculous and fail.We have put a max runtime on 3hrs on the server, but a single workflow shouldn't be able to take down the whole machine and everyone else's workflows with it.

I feel that something isn't right somewhere and the sort/join usage parameter seems to be the only lever we have got.

[patrick_mcauliffe]

See what happens when you enter 8943 in the memory setting and then hover the mouse for a moment:

Try 4160.

[Steve_Luckcock]

That's interesting, so hovering over 8943 get the little message "please enter valid value" but allows me to plough on regardless.

Reducing it to 4160 no message on hover over. Probably not he best user warning but...

Tried it but I am afraid I got he same result. Left monitoring he memory whilst running off, just in case utilizing precious memory inspecting.

Sort of Heisenberg uncertainty thing. Checked it in the morning (Schroedinger's server). Sadly he cat was dead.

We have an Azure installation half the size (16GB). Never had a problem. On Tuesday will deliberately overload it and see what happens. Have to wait till Tuesday to get permission from the users to break their server.

[Steve_Luckcock]

I have had an opportunity to stress test the alteryx server 2020.1.6 on an azure environment.

This is a machine half the size of of the AWS platformed machines I am having issues with. It is 2 core 16GB .

It was set up with 2 concurrent workflows that I kicked up to 3 for testing 4400MB sort/Join memory and 3 threads.

I thrashed it with the heaviest of the users workflows concurrently. Memory usage according to the Administrator diagnostics peaked at 89% and hovered around that for the majority of the testing. Physical memory on the machine peaked at 59% when it was exporting to Tableau but most of the time looked to be around 24-39% never in any danger of crashing the machine. CPU was all over the place as was Disk activity.

Basically it performed as well if not better than I expected and I could not crash it.. So I would suggest the problem is with installation on an AWS platform. Has anybody else got Alteryx server on AWS. Have you experienced these issues?

[patrick_mcauliffe]

Very interesting results.

How long had the AWS instance been running without a reboot?

[Steve_Luckcock]

I am afraid it has to be rebooted every time Alteryx brings it down.