Formula Dividing a number not getting expected results.

Question

I am on version 2018.2

I have a simple formula where I am dividing 1 number by 100 and comparing it to another number. It works most of the time, but sometimes I get answers I don't expect. I suspect is has to do with the datatype precision, but It doesn't appear I can use fixed decimal fields in the formula. I've attached an example below.

Let's say I have Field one and Field2 with the following values

.074 7.4
.094 9.4

and the formula

If [Field1] = [Field2]/100
then 1
else 0
endIF

I would expect both to be one, but the first row returns 0.

How do I get around this?

DivisionExample.yxmd

NicoleJohnson · Accepted Answer

I expect it has something to do with rounding and the double field type... try adding this to your formula (appears to work for the examples provided):

If [Field1] = round([Field2]/100,0.001) then 1 else 0 endIF

This will round your Field2/100 value to the thousandths. To be safe, you could also add the rounding logic to the Field1 value. This will round your values to the same decimal place to ensure matches at that decimal level.

Cheers,

NJ