Int32 works, Double/Float doesnt

[Matthew]

i'm working on a linear regression macro. and it mostly works, but it only works when the input data type for the x-axis is an integer

if i change the data type to double (without changing any of the values) suddenly the "slope" output is zero.. but i am having a hard time figuring out why..

[OllieClarke]

Hey @Matthew you didn't package the macro with your workflow, so I can't see what's in it that's causing the issue

[Matthew]

@OllieClarke  sorry about that, this version should have it

[OllieClarke · Accepted solution]

@Matthew okay, there's a number of things that are different if you use a double vs. an int. 

The main problem though is your join from NewX onto X is from a double onto an integer (as long as you start with a double). It is incredibly unlikely that a double will be perfectly round, so a lot of data is being lost in this join (tool 97). This is the difference between all 4,987 records coming through the join, or only 49 records...

If you set your new X field to be an integer, then you get non-0 slopes.

There are other issues which come in from rounding though - your first generate rows tool will generate a different number of rows if fed an int or a double (more for int). 

I've made a 'fixed' version which is attached, where I force the X and Y fields to be an int64 in the first select, so you don't end up with these issues - you may of course want to tweak it, but hopefully that makes sense as to what was going wrong. 

I also used the field map option of the macro input, and the 'update select by reverse field map' to remove the need for bonus dropdowns and action tools.

Hope that helps

Ollie

[Matthew]

@OllieClarke yes, it looks like the join where i try to join the generate rows onto the raw data is exactly where the problem is. thank you for narrowing it down

i can quickly band-aid the situation by doing a full join instead of a left join, but i'll work out a more robust solution