This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
After reading up on floating point errors, I'm still unable to understand how it's creating the behavior we're seeing in some of our workflows when the same number behaves differently across different records.
For example, we have a double field and two records both have the number 5 in that field. They both pass through a formula that codes them into buckets where they are essentially flagged as >=5 or <5. I understand that this doesn't work perfectly because of floating point rounding, but shouldn't it always work imperfectly in the same way for the same number? In other words, if two of the same number (5) are both passing through the same formula, shouldn't they have the same outcome (regardless of whether the outcome is right or wrong)?
Similar issue when we group by a double and sometimes the same number would show up in different groups.
Hi Charlie - the data is from a database so I can't upload it. When I copy the data into Excel everything ends up correcting itself. I've attached some screenshots of what I'm seeing if that's helpful.
Yes, we're using double but could use a fixed decimal if necessary. I was more curious about why it only happens on certain records - my understanding of floating point is that the number gets stored to as much precision as the field type would allow, but that would mean that every instance of 5 would round the same. Is that incorrect?