Is it normal that yxdb gets smaller after sorted?

Hi there! I had a large file and it contained a unique ID. I was trying to see whether this ID had been duplicated in the previous step so I added a unique tool and outputted the results. As you know, a unique tool automatically outputs data sorted. And I discovered that the sorted one is much smaller than the unsorted one (as below) even though they have the same records.

My question is should these two files be the same only except for the order difference? Thanks!

Screen Shot 2024-01-28 at 9.31.47 PM.png

Workflow

Datasets

Best Practices

Accepted answers

Qiu

@chloehong445
I think you are correct.

I opened the sample flow of Sorting tool, the data size changes slightly before and after.
The original data size is 904 bytes, and after the Sorting too, it varies from 891 to 901 bytes.

Given the extremely small size of the sample data, the data size difference can be amplied for the big data.

2024-01-29 135120.png

caltang

Yes it is normal. Nothing gets lost by the way, only order changes.

Gawa

hi @chloehong445 Great observation! I didn't know that.

According to the manual, it says .yxdb format is compressed, so sorting data might help compress data in more compact way. (I'm not sure of the exact principle or algorithm though😅)

https://help.alteryx.com/current/en/designer/data-sources/alteryx-database-file-format.html#alteryx-database-file-format

All comments

Qiu

@chloehong445
I have never observed or thought this way.
The record number is same so maybe the data structure is optimized after sorting?

chloehong445

I think so, just want to make sure 🤣

Qiu

@chloehong445
I think you are correct.

I opened the sample flow of Sorting tool, the data size changes slightly before and after.
The original data size is 904 bytes, and after the Sorting too, it varies from 891 to 901 bytes.

Given the extremely small size of the sample data, the data size difference can be amplied for the big data.

2024-01-29 135120.png

caltang

Yes it is normal. Nothing gets lost by the way, only order changes.

Gawa

hi @chloehong445 Great observation! I didn't know that.

According to the manual, it says .yxdb format is compressed, so sorting data might help compress data in more compact way. (I'm not sure of the exact principle or algorithm though😅)

https://help.alteryx.com/current/en/designer/data-sources/alteryx-database-file-format.html#alteryx-database-file-format

aatalai

you can also use the test tool for extra comfort that they are the same?

aatalai

or atleast things such as number of records etc

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703