I've had this happen multiple times
What helped: 1) Broke multiple joins into sequential single or multiple joins. I had 10 large files to join, splitting to 2 joins of 5 each (plus through data) made it work. 2) Eliminated the data cleansing tool. This is counterintuitive, but it worked. Instead, I used a formula tool to do some brute force data cleaning on a few key fields. 3) Split workflows that have this issue into multiple parts. For example, bring in the large YXDB file, do a few operations, and write to an interim YXBD. Repeat with a few other steps. Limit joins in each subworkflow.
What did not help: 1) Changing sort-join memory. 2) Closing or changing the priority of applications on my machine.
What I was not able to try: 1) Running the workflow on Alteryx Server. The workflow and necessary databases were too large to upload easily to server. It kept timing out or having other issues.
