Summarize Rounding Error

I have a dataset with large numbers (float) on which I need to calculate averages by group. Floats are not recommended for Summarize/Group Bys, so what data type do I need to convert to in order to preserve accuracy?

In the attached workflow, see group value "0.007916" which has values B = 1095091000 and 1074619000. These are converted into 1095091072 and 1074619392 when using FixedDecimal type.

data_ex_.yxdb

Summarize_RoundingError.yxmd

Preparation

Accepted answers

DavidxL

I found that an earlier data import step had automatically loaded/defined the numbers as floats, and so the numbers displayed by the Browse tool were only exact to 7 digits of precision despite showing more digits. It seems non-intuitive to me as I would expect some sort of indication that while "1074619000" is being displayed, the actual value is "1.074619E9" with no guarantee for the last 3 digits.

All comments

MarqueeCrew

I'd use DOUBLE.

Cheers,

Mark

DavidxL

Using the Select tool to convert to Double results in the numbers changing. Is there a different method of converting?

The highlighted numbers in column B should end in zero.

double.PNG

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15867

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703