SQL Server - numeric and float data types -
i'm having trouble getting accuracy many decimal places possible. if cast 6 different numbers numeric(36,18)
, calculations between them, resulting value cut off @ 6 decimal places. if use float, can go further, hear cut off well.
any great, thanks.
precision of result: [p1 - s1 + s2 + max(6, s1 + p2 + 1)] -> 36-18+18+max(6,18+36+1)=36+55=91 scale of result : [max(6, s1 + p2 + 1)] -> max(6,18+36+1)=55 in situation precision greater 38 , in case (as stated in documentation)
*the result precision , scale have absolute maximum of 38. when result precision greater 38, corresponding scale reduced prevent integral part of result being truncated.
scale must reduced (91-38=) 53, (55-53=) 2 ... minimum scale length 6 (because of expression scale=[max(6, s1 + p2 + 1)]) , can't reduced lower 6 - have result
Comments
Post a Comment