All Questions
Tagged with floating-point decimal-expansion
7
questions
0
votes
0
answers
40
views
How are results guaranteed in algorithms that include intermediate approximation?
Let me elucidate this vague question with an example. Consider for example the following Gauss sum of roots of unity
$N=(e^{2\pi i/5} + e^{2\pi i/4} e^{4\pi i/5} + e^{4\pi i/4} e^{8\pi i/5} + e^{6\pi ...
- 1,950
2
votes
1
answer
486
views
Correct comparison of real number for n digits precision (absolute vs relative difference)
To compare if $2$ real numbers are equal, we define a desirable precision e.g. $n$ digits and then check if the following condition holds: $-\frac{1}{10^n} \lt x - y \lt \frac{1}{10^n}$
Now I was ...
- 1,609
0
votes
1
answer
207
views
Why is 1.4 - 1.3 == 0.9999+ but 0.4 - 0.3 == 1.000000003
I'm not sure if this is a maths question or a programming question or a how-does-your-computer-work question. Sorry about that.
I remember from university that 0.999999 ... == 1 since 1 - 0.999999 ......
- 111
2
votes
3
answers
987
views
How to interpret fractional number of bits of precision
In double-precision floating-point format there're effective $53$ bits of mantissa stored. This lets us estimate maximum number of decimal digits of precision available:
$$N_{max}=\log_{10}2^{53}\...
- 6,875
2
votes
2
answers
405
views
binary and floating point representation
suppose that we have following binary digits
$00011001.110 $,we can do following thing
$00011001.110=1\cdot2^4+1\cdot2^3+1\cdot2^0+1\cdot2^{-1}+1\cdot2^{-2}=25.75$
then what does means?
We then ...
- 9,254
2
votes
1
answer
334
views
Prove that in any base the number of digits composing the repetitive mantissa of the reciprocal of a prime $p$ never exceeds $p-1$.
I was trying to find bases where the reciprocals of primes have a short repetitive mantissa. Here is what I found:
The bases are on the left. The primes are at the top. The numbers represent the ...
- 864
3
votes
2
answers
227
views
How to identify whether a fractional part of a number contains more that 2 digits.
EX. I want to accept numbers which have maximum of 2 digits after decimal points.
i, e, 10.23 should be allowed and 10.233 should not be allowed. What mathematical operations can be done to ...
- 131