BACKGROUND
I was debating asking this in math exchange but opted for here since it is based on engineering specification.
I was recently going through the specifications for Whitworth threads in BS 84:2007. Interesting read for me being a first timer. It is a relatively short specification with most information given through tables. In says that the values in the tables are calculated with the formulas given in the Annex. When attempting to reproduce the tables in excel using the formulas, then double checking the results with the tables I am noticing some very small discrepancies between the tables and the formula (0.0001"). I am attributing this to some sort of rounding error, but cant figure out why and how to fix it.
The above formulas are used to determine the tolerance at the effective diameter. This calculation is completed with the final results rounded to 0.0001 precision and agree 100% with the table values
The above formula is used to determine the tolerance for the major diameter and is based of the calculation for the tolerance for the effective diameter.
PROBLEM
I have completed this calculation rounding 4 different ways and and there are 4 rows that will have a discrepancy of +/- 0.0001".
WHAT HAS BEEN TRIED
The four different ways I tried to get to the final rounded tolerance were:
- full calculation without rounding and only rounding the final result to 4 decimals;
- Using the rounded T(eff) and adding it to the rounded 0.01*P^0.5;
- Using the rounded T(eff) and adding it 0.01*P^0.5 (not rounded), then rounding the results; and
- Using T(eff) (not rounded) and adding it to the rounded 0.01*P^0.5, then rounding the results.
Data Table
In the table above, the green text is the value given in the table for the tolerance. The black data for the tolerance is calculated using the the formulas above. The pink background rows are rows where the difference between the table tolerance value and the calculated value are not zero.
Sample Data
TPI d Le P
40 0.1250 0.1250 0.025
24 0.1875 0.1875 0.041666667
20 0.2500 0.2500 0.05
18 0.3125 0.3125 0.055555556
16 0.3750 0.3750 0.0625
14 0.4375 0.4375 0.071428571
12 0.5000 0.5000 0.083333333
12 0.5625 0.5625 0.083333333
11 0.6250 0.6250 0.090909091
11 0.6875 0.6875 0.090909091
10 0.7500 0.7500 0.1
9 0.8750 0.8750 0.111111111
8 1.0000 1.0000 0.125
7 1.1250 1.1250 0.142857143
7 1.2500 1.2500 0.142857143
6 1.5000 1.5000 0.166666667
5 1.7500 1.7500 0.2
4.5 2.0000 2.0000 0.222222222
4 2.2500 2.2500 0.25
4 2.5000 2.5000 0.25
3.5 2.7500 2.7500 0.285714286
3.5 3.0000 3.0000 0.285714286
3.25 3.2500 3.2500 0.307692308
3.25 3.5000 3.5000 0.307692308
3 3.7500 3.7500 0.333333333
3 4.0000 4.0000 0.333333333
2.875 4.5000 4.5000 0.347826087
2.75 5.0000 5.0000 0.363636364
2.625 5.5000 5.5000 0.380952381
2.5 6.0000 6.0000 0.4
QUESTION
What am I doing wrong with the rounding and how do I apply a fix that is consistent?