Int32 works, Double/Float doesnt

Matthew

i'm working on a linear regression macro. and it mostly works, but it only works when the input data type for the x-axis is an integer

if i change the data type to double (without changing any of the values) suddenly the "slope" output is zero.. but i am having a hard time figuring out why..

New Workflow19.yxzp

OllieClarke

@Matthew okay, there's a number of things that are different if you use a double vs. an int. 

The main problem though is your join from NewX onto X is from a double onto an integer (as long as you start with a double). It is incredibly unlikely that a double will be perfectly round, so a lot of data is being lost in this join (tool 97). This is the difference between all 4,987 records coming through the join, or only 49 records...

If you set your new X field to be an integer, then you get non-0 slopes.

There are other issues which come in from rounding though - your first generate rows tool will generate a different number of rows if fed an int or a double (more for int). 

I've made a 'fixed' version which is attached, where I force the X and Y fields to be an int64 in the first select, so you don't end up with these issues - you may of course want to tweak it, but hopefully that makes sense as to what was going wrong. 

I also used the field map option of the macro input, and the 'update select by reverse field map' to remove the need for bonus dropdowns and action tools.

Hope that helps

Ollie

New Workflow19_OC.yxzp