5,442 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
9
votes
1
answer
187
views
Does std::to_chars ever really disambiguate using round_to_nearest?
[charconv.to.chars] says the following:
The functions that take a floating-point value but not a precision parameter ensure that the string representation consists of the smallest number of ...
0
votes
1
answer
140
views
Precision loss when migrating data from SQL Server to Snowflake using ADF via CSV staging (but not with Direct Copy) [closed]
I’m migrating SQL Server tables to Snowflake using Azure Data Factory.
Direct Copy from SQL Server to Snowflake works, but I can’t use it because of the 100 MB row size limit. That forces me to take ...
5
votes
2
answers
304
views
Why are C++ and Python giving me different answers, when given (I think) the same precision?
I have this code in C++:
#include <iostream>
#include <iomanip>
#include <limits>
#include <boost/multiprecision/mpfr.hpp>
using namespace boost::multiprecision;
template<...
0
votes
3
answers
186
views
Poor numerical accuracy solving for roots of "simple" cubic equation
my code solves cubic equations with binary search, I have an example of input and output, but results doesn't match. for example input: 1 -3 3 -1, output: 1.000000, but instead of 1.000000 it gives me ...
1
vote
1
answer
173
views
Avoiding duckdb OutOfRangeException when multiplying Decimals
I'm working with DuckDB and have several client-provided SQL expressions that use DECIMAL(38,10) columns (fixed precision with 10 digits after the decimal point).
For example:
SELECT S1__AMOUNT * ...
-4
votes
3
answers
319
views
the pow() function isn't accurate
I read Calculating very large exponents in python , and Ignacio Vazquez-Abrams's answer was to use the pow() function
So , I run the following command in Python 3.9.6 :
print(int(pow(1.5,96)))
and ...
0
votes
0
answers
64
views
TypeOrm softDelete only updates deletedAt column to seconds precision. Can this be made more accurate?
Currently using typeOrm 0.2.37 with a mariaDB 11 database.
I have entities set up using @DeleteDateColumn() which are formed with datetime(6) precision.
On calling softDelete() or softRemove() on an ...
1
vote
0
answers
161
views
How to implement mod for floating point type that is accurate even if given value is extremely large? [duplicate]
This is about questioning implementation, so please do not mention "use fmod" or some other libraries.
I'm trying to implement Normalize function that normalizes radian angle to 0 ≤ θ ≤ 2π. ...
0
votes
1
answer
304
views
Inaccuracy replicating Fortran mixed-precision expression in Rust
I have the following code in my Fortran program, where both a and b are declared as REAL (KIND=8):
a = 0.12497443596150659d0
b = 1.0 + 0.00737 * a
This yields b as 1.0009210615647672
For comparison, ...
2
votes
1
answer
128
views
Different result when passing an entry and the same entry which is inside a matrix in Fortran
The TLDR:
I have a Fortran function, shiftxe_pos that acts on an element fe(i,j,:,:) with
integer, parameter :: dp = kind(1.0d0)
integer, parameter :: nx
integer, parameter :: nv
...
0
votes
1
answer
41
views
How to change `MPFR_MAX_PREC` either via `R` or a re-compile?
The default value (in bits, I believe), as shown via Rmpfr:
.mpfr_maxPrec()
[1] 9.223372e+18
I'd like to find a way to change that value to something larger. Here's a real-world example where it ...
3
votes
2
answers
203
views
How to compare the numerical precision of two different tensors
I'm trying to compare the output of two tensors with different numerical precisions: one in float64 (fp64) and the other in float32 (fp32). Both tensors are computed from the same inputs and represent ...
3
votes
2
answers
212
views
How does Oracle convert decimal values to float?
If I have a float(5) column, why does 7.89 get rounded to 7.9 but 12.79 gets rounded to 13, not 12.8?
Binary forms are as follows for 3 examples:
7.89 0111.01011001 ------ round to------\> 7.9 ...
1
vote
1
answer
118
views
integer exact computation with logs
I need to compute ceil(log_N(i)) where log_N is the log with positive integer base N and i is also a positive integer.
The straight forward python implementation using floating point math fails at ...
-1
votes
1
answer
157
views
Unexpected behaviour of Trunc() function [duplicate]
I am experiencing an unexpected result from the Trunc() function, due to the lack of precision of floating-point numbers, when the float value stored is just below the positive integer value:
var
u1, ...