How to handle the Precision Lost Error

[danielkresina]

I'm trying to load some data from CSV files that have numbers with very high precision.  Due to other reasons, I have to load the data as text fields and then convert it to numbers in my workflow.  Whenever I try to convert the string values to numbers, I get a warning at runtime that the numbers "had more precision than a double. Some precision was lost."  This occurs no matter what numeric datatype I try to use.

I'm fine with rounding these rediculously precise numbers, but I'd like to supress the warning message somehow.  I tried explicityly using the ROUND function, but that doesn't work since I need to convert the text to numbers anyway before the round function can be used so I still get this error.



Any ideas?  Thanks.

[cbridges]

Thank you @AdamR_AYX for reducing my conversion errors from 100 + to only 1! My numbers are LAT/LONG and my error now is "ConvError: Create Points (37): LAT: A value was out of range and no point was created."

Any suggestions?

(I have a personal mission to have no conversion errors - not because I care about rounding, but because they're annoying)

[cbridges]

Quick update @danielkresina - turns out my original issue had nothing to do with rounding. I had null values in my raw data and didn't realize I could / needed to convert those to zeroes before proceeding. Solved all my problems in that workflow! Shout out to @BlytheE for that gold nugget!

[AdamR_AYX · Accepted solution]

So you will want to put a filter before the Create Points tool and filter off any out of range latitudes with the following formula

[lat] <= 90 && [lat] >= -90

then you just need to decide what to do with the records that come out of the false side

[cbridges]

That works perfectly! Thanks!

[cbridges]

And, it detected an instance where lat and long were reversed! Well done!

[Jasers]

What if you want to keep the precision?  I have a crazy long number in string format like 568.1500244140625 that I get from an api feed, but when I use tonumber, it rounds it.  How do I keep the precision?  After using tonumber, it goes to 568.15.

[RodL]

Testing out your number, I used a formula tool and changed the Data Type to FixedDecimal with 23.18 as the size.

I still got the conversion message, but it seems the entire number came through.

[Jasers]

I think that would work.  Thanks!!

[jpburrowes]

Thanks for the suggestion to change the data type to fixed decimal RodL. This resolved a problem I was having with a particular longitude value