Re: Setting Float Precision in Lua.c
[
Date Prev][
Date Next][
Thread Prev][
Thread Next]
[
Date Index]
[
Thread Index]
- Subject: Re: Setting Float Precision in Lua.c
- From: KHMan <keinhong@...>
- Date: 2018年6月11日 10:35:20 +0800
On 6/11/2018 5:49 AM, Albert Chan wrote:
Double rounding cause the value to be off by +/- 1 ULP
It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.
-- Albert 6/5/18
Turns out, above statement is *wrong* ... Sorry
Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)
(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12
If above is true, the case for 53-bits rounding is even stronger.
Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8
I'm sorry, something has gone wrong with my results:
$ 32bit/lua -i
Lua 5.4.0 (work1) Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000012
$ 64bit/lua -i
Lua 5.4.0 (work1) Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000006
This is obviously unacceptable. You should investigate this.
--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia