Sas7bdat Files Produce Full Disk Space

Question

i am running a workflow that uses a 1GB sas7bdat file and the workflow only has a few tiles,sort and unique tools, but the swap files created (temp files) are more than 500 and those seems to be 1GB of size, has anyone face this issue? Any suggestions to avoid getting the disk space full?

Is it possible to use macros to reduce the generation on temp swap files generated during the workflow execution? Thanks for the suggestions

IgnacioVilaplana · Answer

The behavior you're describing with extremely large string sizes (e.g., 1,073,741,823 or 2,147,483,647) in the Formula tool and Text Input tool is unusual and not typical for Alteryx. Normally, string fields default to a size like 255 unless explicitly set otherwise. This abnormally large string size can cause excessive memory and disk usage, leading to the generation of many large swap/temp files and ultimately filling your disk space.This issue may be related to a bug or corruption in how the workflow or Designer is handling string metadata. It is not expected behavior and could explain why your workflow is generating thousands of large temp files despite only processing a 1GB dataset with relatively simple tools.Things that you could do: Try recreating the problematic tools (Formula, Text Input) from scratch in a new workflow or new tab to see if the string size defaults to a normal value (e.g., 255). Sometimes workflows can get corrupted metadata. If it persists, try installing a different version of Alteryx Designer, and if you can't, put a ticket to Alteryx Support for them to have a closer look.Best!Nacho

Gibranscrw · Answer

Hi Ignacio thanks for the reply, I have done all the procedures except the one of splitting the dataset I to chunks and I still getting the disk space full, however it seems that the workflow is accumulating the number of swap files to generate as the error showed like 15 swap files that were not able to be generated and now it is showing 12000 swap files. I am only ingesting 1 GB dataset to the workflow and I have multiple macros a just a few tiles, sort and unique tools which could  related to generating temp files.

I also noticed that when I use alteryx the formula tool set a string size of 1073741823 and if I add a text input tool before that formula tool with just one record and the number one as a v_string the string size changes to 2147483647. I remember creating the same set of tools in a workflow and the string size displayed would be 255. Is there any issue with my alteryx designer? I am using Alteryx 2024.1.1.178 patch 6.

IgnacioVilaplana · Answer

Hi @Gibranscrw, yes, this is a known issue when working with large files and workflows that generate many temporary swap files, which can quickly consume disk space. Here are some suggestions to help reduce the temp file usage and avoid filling your disk:

* Check the location of your temporary files in Workflow Configuration > Runtime > Temporary Files. If possible, set this to a drive with ample free space, ideally separate from your system drive.
* Reduce the amount of data stored in Browse Everywhere (the data preview in the Results pane). By default, Alteryx stores 1MB per tool anchor, which can add up with many tools. You can lower or disable this in Options > User Settings > Edit > Advanced > Tool Results Settings.
* Disable Browse tools or remove unnecessary Browse tools in your workflow, as they generate additional temp files.
* Consider splitting your large dataset into smaller chunks and processing them in batches using batch macros. This approach can reduce memory and disk usage during execution.
* Running Alteryx Designer at a lower priority (set in User Settings) can help system responsiveness but does not directly reduce temp file size.
* If your workflow is simple (just a few tools like Sort, Unique, and Tiles), ensure you are not inadvertently creating many intermediate outputs or Browse tools that increase temp file generation.

Using macros can help by processing data in smaller groups, but the key is to control the batch size to balance performance and resource usage. Hope this helps!