Questions tagged [floating-accuracy]
Concerning the accuracy of operations performed on floating point numbers.
floating-accuracy
1,368
questions
1
vote
1
answer
70
views
How does rounding works in float multiplication?
The exact value of float 0.7f is 0.69999...
so I thought the result of 0.7f * 100f would be something below 70, like 69.99999...
But the result is exact 70.
Does float multiplication involve such ...
- 13
1
vote
1
answer
188
views
Can Neumaier summation be sped up?
Neumaier summation is an improvement of Kahan summation for accurately summing arrays of floats.
import numba as nb
@nb.njit
def neumaier_sum(arr):
s = arr[0]
c = 0.0
for i in range(1, ...
- 20.9k
0
votes
1
answer
59
views
How does SQL Server store integer values of over 2^24 in a FLOAT(53) column?
I have a SQL Server database from which I can read integer values of (way) over 2^24 from a column that has datatype float. This confuses me since these values should be to big/lengthy to be ...
- 117
-1
votes
1
answer
46
views
Float calculation problem. 12.7 - 20 + 7.3 not equal zero [duplicate]
12.7 - 20 + 7.3 = -8.8817841970013e-016
I got this problem when I read Programming in Lua book, and I cannot figure it out how to get zero as the result of the calculation.
When I formatted the result ...
0
votes
1
answer
51
views
How much accuracy is lost due to representing time as a IEEE 754 binary64 number?
My python program has a busy loop. Within the loop, some delays are calculated in the following fashion:
timeout_s: float = args.timeout_s
timeout_end_s: float = time.time() + timeout_s
while True:
...
- 135k
0
votes
1
answer
117
views
ceil(double(int)*double) or ceil(float(int*double))?
I had this C++ code:
int x;
double y;
// do sth
int z = ceil(x*y);
If x = 410, y = 1.1, then z = 452 when it should've been 451. I changed to z = ceil(float(x*y)) and it works. My friend suggests z ...
- 13
4
votes
5
answers
407
views
Fast implementation of complementary error function with 5-digit accuracy
[While this is a self-answered question, I will happily upvote and accept any alternative answer that either provides superior accuracy for the same amount of computational work or reduces the amount ...
- 25.3k
0
votes
0
answers
38
views
R: floating point errors in simple math operations [duplicate]
So of course:
1023.1 - 0.1 < 1023
[1] FALSE
but...
1024.1 - 0.1 < 1024
[1] TRUE
Is this a bug? Is it expected behaviour?
Is there some way I can avoid this, such as rounding all my numbers to ...
- 297
0
votes
1
answer
51
views
Truncating decimal digits like Excel in python
In my excel file, I've some calculations. Calculation returns 0,345 and excel convert it to 0,35. I made it in python. Python is returning 0,345, but I want to take 0,35 like Excel. I've tried with ...
user22304060
0
votes
1
answer
80
views
Calculating power of (1-1/x)^y for very large x,y
I am thinking about calculating $(1-1/x)^y$ for very large x,y numbers in python.
One way todo so, it to use exp(y*log1p(-1/x)), but I am not sure about its accuracy.
Are they any results on using ...
- 347
0
votes
0
answers
38
views
Model Accuracy Using Transformer
I'm trying to increase the Accuracy of my Model. The model aims to leverage BERT's contextual understanding of language to perform binary classification on IMDb movie reviews. By fine-tuning specific ...
0
votes
0
answers
74
views
Why assignment changes the precision of calculation of MATLAB?
I am writing some scripts using MATLAB R2020a and I am having a problem: there are two values I want to assign to a vector z_samples, namely x_samples(l_indexes)./sum(x_samples)(a vector) and ...
0
votes
1
answer
112
views
Rounding errors when computing intersection between two lines in python
I'm attempting to create a Python script that identifies all the unique intersection points, eliminating duplicates, among all the lines passing through a subset of points of a regular grid in 2D or ...
- 135
1
vote
1
answer
197
views
Workarounds for High Floating-Point Error in NumPy Arrays Raised to Powers
I have data that represents responses from a 24(?)-bit A/D converter. I am setting up the data to go into a polynomial regression in the function below, but I am getting detrimental amounts of ...
- 15
1
vote
2
answers
175
views
imshow plotting very large integers, but "dtype object cannot be converted to float"
I have the following code, plotting a function on a grid, where the function happens to have a very large integer value:
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter, ...
- 117