Summarize Rounding Error

[DavidxL]

I have a dataset with large numbers (float) on which I need to calculate averages by group. Floats are not recommended for Summarize/Group Bys, so what data type do I need to convert to in order to preserve accuracy?

In the attached workflow, see group value "0.007916" which has values B = 1095091000 and 1074619000. These are converted into 1095091072 and 1074619392 when using FixedDecimal type.

[MarqueeCrew]

I'd use DOUBLE.

Cheers,

Mark

[DavidxL]

Using the Select tool to convert to Double results in the numbers changing. Is there a different method of converting?

The highlighted numbers in column B should end in zero.

[SeanAdams]

easiest way is to use a formula - it gives you far more control over changing types, and there are loads of type conversion functions available (using select, you have no control over how it's converted)

[DavidxL · Accepted solution]

I found that an earlier data import step had automatically loaded/defined the numbers as floats, and so the numbers displayed by the Browse tool were only exact to 7 digits of precision despite showing more digits. It seems non-intuitive to me as I would expect some sort of indication that while "1074619000" is being displayed, the actual value is "1.074619E9" with no guarantee for the last 3 digits.