0
$\begingroup$

My launch simulation's producing numbers on the order of 1e-12 and sometimes smaller.

I can't imagine that any sensor would be so precise and therefore that 10+ decimal places would be meaningful or appropriate.

And even if sensor readings were that precise, I can't imagine it being good use of memory to store numbers to the 10th+ decimal place.

This has me wondering (and please excuse the dumb question, as I am not a computer scientist): how precise do GNC numbers get? How many decimal places would position/velocity state vectors retain? Would they be stored with more decimal places than the precision of their sources (sensors) justifies?

(E.g., if position measurements were precise to within 0.1 m, then would position vectors be stored with more than one decimal place when expressed in m?)

Improve this question
$\endgroup$
3
  • $\begingroup$ One would definitely use more bits to store the data than what is available during acquisition. Kind of like the concept of guard bits. Because, it helps to retain the additional bits generated on intermediate results. The truncation, if any, is better done on the final result and not on the intermediate results. $\endgroup$
    – AJN
    Commented Jun 6, 2021 at 2:36
  • $\begingroup$ It also makes sense to use more bits than what the sensor can achieve because, sensor fusion algorithms, sensor averaging algorithms etc increase the effective number of bits on the final fused/averaged result. $\endgroup$
    – AJN
    Commented Jun 6, 2021 at 2:38
  • $\begingroup$ Thanks for explaining! Digits cost memory so I wonder: would they store the maximum of digits allowed (before truncating at the end)? Or would they store just a few more digits than... sensor precision... demands? $\endgroup$
    – user39728
    Commented Jun 6, 2021 at 3:20

1 Answer 1

Reset to default
1
$\begingroup$

The Apollo Guidance Computer used a plus/minus sign and 5 decimal digits. This can be seen in the "List of Nouns Used in Program Colossus" of the AGC manual as well as the number of digits on the DSKY user interface:

DSKY

There are various data types used by the nouns. A table in the source file PINBALL_NOUN_TABLES.agc controls how each noun is formatted for input and output. Each noun can have up to 3 values.

  • Octal values represent one 15-bit memory word. They are rendered as five octal digits for input and output; thus, the internal precision is fully displayed. The 8, 9, +, and - keys are invalid inputs for these types of values (i.e. there is no sign bit for octal values).

  • Whole values are stored as 29 bits and 1 sign bit (two memory words). Many values have an upper limit which is documented in ENTRY_LEXICON.agc, but the internal precision is fully displayed. There are rendered using up to 5 decimal digits.

  • Fractional values represent values less than ±1 and are stored as 29 bits and one sign bit. They are rendered as a ± sign and five decimal digits. Thus, there is some internal precision that is not displayed.

  • Floating point was not used as an internal representation. However, there were a few informational values that were calculated as whole numbers, then displayed in decimal with the decimal point displayed in between the appropriate digits (e.g. XXXX.X nautical miles).

  • Degree values are stored in 29 bits plus one sign bit, and rendered with 5 decimal digits (XXX.XX, except for elevation XX.XXX). Thus, some internal precision is lost in the display.

  • Time nouns are stored in 28 (?!) bits, representing a count of 1/100 seconds (the period of the clock interrupt). They are the only nouns that use more than one line to display a single value: 00XXX hours, 000XX minutes, and 0XX.XX seconds. Note that this format can actually display more than the precision of the internal value.

  • There are some weird "XX by XX" displays, but they seem to be computed values for informational purposes.

Attitudes were represented by three degree values (roll, pitch, yaw) in 6 total memory words. Positions were represented by three whole-number values (X, Y, Z) in 6 total memory words.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Was the number of digits displayed the full internal precision? $\endgroup$
    – Organic Marble
    Commented Jun 6, 2021 at 11:37
  • 1
    $\begingroup$ @OrganicMarble: Answer updated. $\endgroup$
    – DrSheldon
    Commented Jun 6, 2021 at 16:29
  • $\begingroup$ Thank you for this very excellent answer :) $\endgroup$
    – user39728
    Commented Jun 6, 2021 at 16:48