-
Notifications
You must be signed in to change notification settings - Fork 404
Conversion Precision #1128
-
When converting certain values such as 328.999mV to V (which is a base unit), the answer adds a bit of error resulting in 0.32899900000000004.
Looking at the code I am a bit confused as to why this occurs. Any clarification would be very helpful.
Beta Was this translation helpful? Give feedback.
All reactions
This is caused by the normal double precision: https://en.wikipedia.org/wiki/Double-precision_floating-point_format
If you write 328.999d / 1000 in a normal c# code you will get the same error. This is just how floating point in computers work.
To avoid this in c# code it is possible to use decimal instead: 328.999m / 1000
Replies: 1 comment
-
This is caused by the normal double precision: https://en.wikipedia.org/wiki/Double-precision_floating-point_format
If you write 328.999d / 1000 in a normal c# code you will get the same error. This is just how floating point in computers work.
To avoid this in c# code it is possible to use decimal instead: 328.999m / 1000
Beta Was this translation helpful? Give feedback.