How to handle the Precision Lost Error
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
I'm trying to load some data from CSV files that have numbers with very high precision. Due to other reasons, I have to load the data as text fields and then convert it to numbers in my workflow. Whenever I try to convert the string values to numbers, I get a warning at runtime that the numbers "had more precision than a double. Some precision was lost." This occurs no matter what numeric datatype I try to use.
I'm fine with rounding these rediculously precise numbers, but I'd like to supress the warning message somehow. I tried explicityly using the ROUND function, but that doesn't work since I need to convert the text to numbers anyway before the round function can be used so I still get this error.
Any ideas? Thanks.
Solved! Go to Solution.
- Labels:
- Expression
- Preparation
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
So a formula like so
ToNumber(StringField, 1)
into a new field of type double should do the trick
Cheers
Adam
https://www.linkedin.com/in/adriley/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
If your goal is to just change the decimal precision, you may want to use the round function directly.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
Feel free to send a sample if you'd like and I can take a look. danielkresina at tabsgroup.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Notify Moderator
