Fixed Decimals

Question

Hello all,

The scenario is associate starts working in night shift at "03-05-2021 23:23" to "04-05-2021 06:40" if the working hours are between 23:00-06:00 then that time should be counted towards night shift hours and rest are day shift Hours, and the break time is 30 mins if total hours grater than 6 hours.

i have made the calculations all in Double format,

but there is 0.01 diff when the calculations are converted to fixed Decimals How to avoid that differences, here. while calculating

?

The output I got is

Total Hours    7.28Day SHift Hours    0.67Night Hours 6.12, and break 0.5

There is 0.01 diff when total hours - (day hours + night Hours + break)

Any leads on this please!

Maskell_Rascal · Answer

Hi @dkma

This is being caused by a rounding issue when you convert a double to a fixed decimal. You totals unconverted are as follows.

Total Hours: 7.283333

Day Shift Hours: 0.666667

Night Shift Hours: 6.116667

Break: 0.5

When you convert them over to a fixed decimal, it rounds everything to the nearest hundredth. There are some complicated ways to get around this, but it requires converting your numeric fields over to a string and then cutting off the tail end of the decimal to the hundredth before converting it back.

cpapaioannou · Answer

The usual problem in these kinds of situations is that one converts to 2-point fixed decimal BEFORE summing up all the values. However you correctly noted that all calculations are indeed made in double format.

My next guess would be that, if this diff must indeed equal to zero, then there is a final rounding issue. A common practice I use is to firstly round the numbers to the wanted decimal point and only then convert them with the select tool. Optimally I only use a multi-field which does both.

However, this is probably not to be applied here, given that you convert minutes to double-format numbers and this in fact can lead to 3 point decimals by itself. Maybe use double-format till the very end?