In everyday usage, we rely heavily on percentages to quantify probabilities and proportions: we might say that a prediction is accurate or accurate, that there is a chance of dying from some disease, and so forth. However, for those without extensive mathematical training, it can sometimes be difficult to assess whether a given percentage amounts to a “good” or “bad” outcome, because this depends very much on the context of how the percentage is used. For instance:
- (i) In a two-party election, an outcome of say to might be considered close, but to would probably be viewed as a convincing mandate, and to would likely be viewed as a landslide.
- (ii) Similarly, if one were to poll an upcoming election, a poll of to would be too close to call, to would be an extremely favorable result for the candidate, and to would mean that it would be a major upset if the candidate lost the election.
- (iii) On the other hand, a medical operation that only had a , , or chance of success would be viewed as being incredibly risky, especially if failure meant death or permanent injury to the patient. Even an operation that was or likely to be non-fatal (i.e., a or chance of death) would not be conducted lightly.
- (iv) A weather prediction of, say, chance of rain during a vacation trip might be sufficient cause to pack an umbrella, even though it is more likely than not that rain would not occur. On the other hand, if the prediction was for an chance of rain, and it ended up that the skies remained clear, this does not seriously damage the accuracy of the prediction – indeed, such an outcome would be expected in one out of every five such predictions.
- (v) Even extremely tiny percentages of toxic chemicals in everyday products can be considered unacceptable. For instance, EPA rules require action to be taken when the percentage of lead in drinking water exceeds (15 parts per billion). At the opposite extreme, recycling contamination rates as high as are often considered acceptable.
Because of all the very different ways in which percentages could be used, I think it may make sense to propose an alternate system of units to measure one class of probabilities, namely the probabilities of avoiding some highly undesirable outcome, such as death, accident or illness. The units I propose are that of “nines“, which are already commonly used to measure availability of some service or purity of a material, but can be equally used to measure the safety (i.e., lack of risk) of some activity. Informally, nines measure how many consecutive appearances of the digit are in the probability of successfully avoiding the negative outcome, thus
- success = one nine of safety
- success = two nines of safety
- success = three nines of safety
Definition 1 (Nines of safety) An activity (affecting one or more persons, over some given period of time) that has a probability of the “safe” outcome and probability of the “unsafe” outcome will have nines of safety against the unsafe outcome, where is defined by the formula (where is the logarithm to base ten), or equivalently
Remark 2 Because of the various uncertainties in measuring probabilities, as well as the inaccuracies in some of the assumptions and approximations we will be making later, we will not attempt to measure the number of nines of safety beyond the first decimal point; thus we will round to the nearest tenth of a nine of safety throughout this post.
Here is a conversion table between percentage rates of success (the safe outcome), failure (the unsafe outcome), and the number of nines of safety one has:
Success rate | Failure rate | Number of nines |
infinite |
Thus, if one has no nines of safety whatsoever, one is guaranteed to fail; but each nine of safety one has reduces the failure rate by a factor of . In an ideal world, one would have infinitely many nines of safety against any risk, but in practice there are no guarantees against failure, and so one can only expect a finite amount of nines of safety in any given situation. Realistically, one should thus aim to have as many nines of safety as one can reasonably expect to have, but not to demand an infinite amount.
Remark 3 The number of nines of safety against a certain risk is not absolute; it will depend not only on the risk itself, but (a) the number of people exposed to the risk, and (b) the length of time one is exposed to the risk. Exposing more people or increasing the duration of exposure will reduce the number of nines, and conversely exposing fewer people or reducing the duration will increase the number of nines; see Proposition 7 below for a rough rule of thumb in this regard.
Remark 4 Nines of safety are a logarithmic scale of measurement, rather than a linear scale. Other familiar examples of logarithmic scales of measurement include the Richter scale of earthquake magnitude, the pH scale of acidity, the decibel scale of sound level, octaves in music, and the magnitude scale for stars.
Remark 5 One way to think about nines of safety is via the Swiss cheese model that was created recently to describe pandemic risk management. In this model, each nine of safety can be thought of as a slice of Swiss cheese, with holes occupying of that slice. Having nines of safety is then analogous to standing behind such slices of Swiss cheese. In order for a risk to actually impact you, it must pass through each of these slices. A fractional nine of safety corresponds to a fractional slice of Swiss cheese that covers the amount of space given by the above table. For instance, nines of safety corresponds to a fractional slice that covers about of the given area (leaving uncovered).
Now to give some real-world examples of nines of safety. Using data for deaths in the US in 2019 (without attempting to account for factors such as age and gender), a random US citizen will have had the following amount of safety from dying from some selected causes in that year:
Cause of death | Mortality rate per (approx.) | Nines of safety |
All causes | ||
Heart disease | ||
Cancer | ||
Accidents | ||
Drug overdose | ||
Influenza/Pneumonia | ||
Suicide | ||
Gun violence | ||
Car accident | ||
Murder | ||
Airplane crash | ||
Lightning strike |
The safety of air travel is particularly remarkable: a given hour of flying in general aviation has a fatality rate of , or about nines of safety, while for the major carriers the fatality rate drops down to , or about nines of safety.
Of course, in 2020, COVID-19 deaths became significant. In this year in the US, the mortality rate for COVID-19 (as the underlying or contributing cause of death) was per , corresponding to nines of safety, which was less safe than all other causes of death except for heart disease and cancer. At this time of writing, data for all of 2021 is of course not yet available, but it seems likely that the safety level would be even lower for this year.
Some further illustrations of the concept of nines of safety:
- Each round of Russian roulette has a success rate of , providing only nines of safety. Of course, the safety will decrease with each additional round: one has only nines of safety after two rounds, nines after three rounds, and so forth. (See also Proposition 7 below.)
- The ancient Roman punishment of decimation, by definition, provided exactly one nine of safety to each soldier being punished.
- Rolling a on a -sided die is a risk that carries about nines of safety.
- Rolling a double one (“snake eyes“) from two six-sided dice carries about nines of safety.
- One has about nines of safety against the risk of someone randomly guessing your birthday on the first attempt.
- A null hypothesis has nines of safety against producing a statistically significant result, and nines against producing a statistically significant result. (However, one has to be careful when reversing the conditional; a statistically significant result does not necessarily have nines of safety against the null hypothesis. In Bayesian statistics, the precise relationship between the two risks is given by Bayes’ theorem.)
- If a poker opponent is dealt a five-card hand, one has nines of safety against that opponent being dealt a royal flush, against a straight flush or higher, against four-of-a-kind or higher, against a full house or higher, against a flush or higher, against a straight or higher, against three-of-a-kind or higher, against two pairs or higher, and just against one pair or higher. (This data was converted from this Wikipedia table.)
- A -digit PIN number (or a -digit combination lock) carries nines of safety against each attempt to randomly guess the PIN. A length password that allows for numbers, upper and lower case letters, and punctuation carries about nines of safety against a single guess. (For the reduction in safety caused by multiple guesses, see Proposition 7 below.)
Here is another way to think about nines of safety:
Proposition 6 (Nines of safety extend expected onset of risk) Suppose a certain risky activity has nines of safety. If one repeatedly indulges in this activity until the risk occurs, then the expected number of trials before the risk occurs is .
Proof: The probability that the risk is activated after exactly trials is , which is a geometric distribution of parameter . The claim then follows from the standard properties of that distribution.
Thus, for instance, if one performs some risky activity daily, then the expected length of time before the risk occurs is given by the following table:
Daily nines of safety | Expected onset of risk |
One day | |
One week | |
One month | |
One year | |
Two years | |
Five years | |
Ten years | |
Twenty years | |
Fifty years | |
A century |
Or, if one wants to convert the yearly risks of dying from a specific cause into expected years before that cause of death would occur (assuming for sake of discussion that no other cause of death exists):
Yearly nines of safety | Expected onset of risk |
One year | |
Two years | |
Five years | |
Ten years | |
Twenty years | |
Fifty years | |
A century |
These tables suggest a relationship between the amount of safety one would have in a short timeframe, such as a day, and a longer time frame, such as a year. Here is an approximate formalisation of that relationship:
Proposition 7 (Repeated exposure reduces nines of safety) If a risky activity with nines of safety is (independently) repeated times, then (assuming is large enough depending on ), the repeated activity will have approximately nines of safety. Conversely: if the repeated activity has nines of safety, the individual activity will have approximately nines of safety.
Proof: An activity with nines of safety will be safe with probability , hence safe with probability if repeated independently times. For large, we can approximate
giving the former claim. The latter claim follows from inverting the former.
Remark 8 The hypothesis of independence here is key. If there is a lot of correlation between the risks between different repetitions of the activity, then there can be much less reduction in safety caused by that repetition. As a simple example, suppose that of a workforce are trained to perform some task flawlessly no matter how many times they repeat the task, but the remaining are untrained and will always fail at that task. If one selects a random worker and asks them to perform the task, one has nines of safety against the task failing. If one took that same random worker and asked them to perform the task times, the above proposition might suggest that the number of nines of safety would drop to approximately ; but in this case there is perfect correlation, and in fact the number of nines of safety remains steady at since it is the same of the workforce that would fail each time.Because of this caveat, one should view the above proposition as only a crude first approximation that can be used as a simple rule of thumb, but should not be relied upon for more precise calculations.
One can repeat a risk either in time (extending the time of exposure to the risk, say from a day to a year), or in space (by exposing the risk to more people). The above proposition then gives an additive conversion law for nines of safety in either case. Here are some conversion tables for time:
From/to | Daily | Weekly | Monthly | Yearly |
Daily | 0 | -0.8 | -1.5 | -2.6 |
Weekly | +0.8 | 0 | -0.6 | -1.7 |
Monthly | +1.5 | +0.6 | 0 | -1.1 |
Yearly | +2.6 | +1.7 | +1.1 | 0 |
From/to | Yearly | Per 5 yr | Per decade | Per century |
Yearly | 0 | -0.7 | -1.0 | -2.0 |
Per 5 yr | +0.7 | 0 | -0.3 | -1.3 |
Per decade | +1.0 | + -0.3 | 0 | -1.0 |
Per century | +2.0 | +1.3 | +1.0 | 0 |
For instance, as mentioned before, the yearly amount of safety against cancer is about . Using the above table (and making the somewhat unrealistic hypothesis of independence), we then predict the daily amount of safety against cancer to be about nines, the weekly amount to be about nines, and the amount of safety over five years to drop to about nines.
Now we turn to conversions in space. If one knows the level of safety against a certain risk for an individual, and then one (independently) exposes a group of such individuals to that risk, then the reduction in nines of safety when considering the possibility that at least one group member experiences this risk is given by the following table:
Group | Reduction in safety |
You ( person) | |
You and your partner ( people) | |
You and your parents ( people) | |
You, your partner, and three children ( people) | |
An extended family of people | |
A class of people | |
A workplace of people | |
A school of people | |
A university of people | |
A town of people | |
A city of million people | |
A state of million people | |
A country of million people | |
A continent of billion people | |
The entire planet |
For instance, in a given year (and making the somewhat implausible assumption of independence), you might have nines of safety against cancer, but you and your partner collectively only have about nines of safety against this risk, your family of five might only have about nines of safety, and so forth. By the time one gets to a group of people, it actually becomes very likely that at least one member of the group will die of cancer in that year. (Here the precise conversion table breaks down, because a negative number of nines such as is not possible, but one should interpret a prediction of a negative number of nines as an assertion that failure is very likely to happen. Also, in practice the reduction in safety is less than this rule predicts, due to correlations such as risk factors that are common to the group being considered that are incompatible with the assumption of independence.)
In the opposite direction, any reduction in exposure (either in time or space) to a risk will increase one’s safety level, as per the following table:
Reduction in exposure | Additional nines of safety |
For instance, a five-fold reduction in exposure will reclaim about additional nines of safety.
Here is a slightly different way to view nines of safety:
Proposition 9 Suppose that a group of people are independently exposed to a given risk. If there are at most nines of individual safety against that risk, then there is at least a chance that one member of the group is affected by the risk.
Proof: If individually there are nines of safety, then the probability that all the members of the group avoid the risk is . Since the inequality
is equivalent to the claim follows.Thus, for a group to collectively avoid a risk with at least a chance, one needs the following level of individual safety:
Group | Individual safety level required |
You ( person) | |
You and your partner ( people) | |
You and your parents ( people) | |
You, your partner, and three children ( people) | |
An extended family of people | |
A class of people | |
A workplace of people | |
A school of people | |
A university of people | |
A town of people | |
A city of million people | |
A state of million people | |
A country of million people | |
A continent of billion people | |
The entire planet |
For large , the level of nines of individual safety required to protect a group of size with probability at least is approximately .
Precautions that can work to prevent a certain risk from occurring will add additional nines of safety against that risk, even if the precaution is not effective. Here is the precise rule:
Proposition 10 (Precautions add nines of safety) Suppose an activity carries nines of safety against a certain risk, and a separate precaution can independently protect against that risk with nines of safety (that is to say, the probability that the protection is effective is ). Then applying that precaution increases the number of nines in the activity from to .
Proof: The probability that the precaution fails and the risk then occurs is . The claim now follows from Definition 1.
In particular, we can repurpose the table at the start of this post as a conversion chart for effectiveness of a precaution:
Effectiveness | Failure rate | Additional nines provided |
infinite |
Thus for instance a precaution that is effective will add nines of safety, a precaution that is effective will add nines of safety, and so forth. The mRNA COVID vaccines by Pfizer and Moderna have somewhere between effectiveness against symptomatic COVID illness, providing about nines of safety against that risk, and over effectiveness against severe illness, thus adding at least nines of safety in this regard.
A slight variant of the above rule can be stated using the concept of relative risk:
Proposition 11 (Relative risk and nines of safety) Suppose an activity carries nines of safety against a certain risk, and an action multiplies the chance of failure by some relative risk . Then the action removes nines of safety (if ) or adds nines of safety (if ) to the original activity.
Proof: The additional action adjusts the probability of failure from to . The claim now follows from Definition 1.
Here is a conversion chart between relative risk and change in nines of safety:
Relative risk | Change in nines of safety |
Some examples:
- Smoking increases the fatality rate of lung cancer by a factor of about , thus removing about nines of safety from this particular risk; it also increases the fatality rates of several other diseases, though not quite as dramatically an extent.
- Seatbelts reduce the fatality rate in car accidents by a factor of about two, adding about nines of safety. Airbags achieve a reduction of about , adding about additional nines of safety.
- As far as transmission of COVID is concerned, it seems that constant use of face masks reduces transmission by a factor of about five (thus adding about nines of safety), and similarly for constant adherence to social distancing; whereas for instance a compliance with mask usage reduced transmission by about (adding only or so nines of safety).
The effect of combining multiple (independent) precautions together is cumulative; one can achieve quite a high level of safety by stacking together several precautions that individually have relatively low levels of effectiveness. Again, see the “swiss cheese model” referred to in Remark 5. For instance, if face masks add nines of safety against contracting COVID, social distancing adds another nines, and the vaccine provide another nine of safety, implementing all three mitigation methods would (assuming independence) add a net of nines of safety against contracting COVID.
In summary, when debating the value of a given risk mitigation measure, the correct question to ask is not quite “Is it certain to work” or “Can it fail?”, but rather “How many extra nines of safety does it add?”.
As one final comparison between nines of safety and other standard risk measures, we give the following proposition regarding large deviations from the mean.
Proposition 12 Let be a normally distributed random variable of standard deviation , and let . Then the “one-sided risk” of exceeding its mean by at least (i.e., ) carries nines of safety, the “two-sided risk” of deviating (in either direction) from its mean by at least (i.e., ) carries nines of safety, where is the error function.
Proof: This is a routine calculation using the cumulative distribution function of the normal distribution.
Here is a short table illustrating this proposition:
Number of deviations from the mean | One-sided nines of safety | Two-sided nines of safety |
Thus, for instance, the risk of a five sigma event (deviating by more than five standard deviations from the mean in either direction) should carry nines of safety assuming a normal distribution, and so one would ordinarily feel extremely safe against the possibility of such an event, unless one started doing hundreds of thousands of trials. (However, we caution that this conclusion relies heavily on the assumption that one has a normal distribution!)
See also this older essay I wrote on anonymity on the internet, using bits as a measure of anonymity in much the same way that nines are used here as a measure of safety.
61 comments
Comments feed for this article
3 October, 2021 at 10:07 pm
Hamed
Interesting concept but I’m not clear how it addresses the cases you raised in the beginning about weather prediction and voting. What’s the real advantage of this system rather than elegancy?
4 October, 2021 at 1:12 am
-
The utility of log probabilities is that they can be added and subtracted, instead of multiplied. Hence, they lend themselves to easier mental calculations.
4 October, 2021 at 6:37 am
Terence Tao
It allows for a more apples-to-apples comparison between probabilities and risk. We are willing to use weather predictions that may be unreliable say 25% of the time (0.6 nines) because our risk tolerance for rain when we expect sun (or vice versa) is extremely high. On the other hand most of us would refuse to fly on an aircraft that is expected to crash 0.1% of the time (three nines) because our risk tolerance for death is extremely low. The problem with percentages is that their interpretation is context dependent in a mathematically complicated fashion: an assertion like “98% survival rate” would be very good in some contexts, and terrible in others. A logarithmic scale such as nines allows for easier comparisons, as it spaces out distinct values of risk more properly (e.g., 98% and 99% success rates seem very similar, but actually they differ by 0.3 nines, whereas 50% and 51% success rates, differing by only 0.01 nines, are essentially the same rate), and as mentioned in another comment allow one to use the simpler mental arithmetic of addition and subtraction (of numbers with one digit after the decimal point) to do calculations, rather than multiplication or division of percentages.
4 October, 2021 at 12:20 am
Fred the Cat
seems overly complex for what should be a simple “hunch” test and individual risk management approach. e.g. a paranoid person with a 10% chance of rain, will take an overcoat, umbrella, gumboots; whereas someone else with the same chance of rain will pack their Hawaiian shirt. So do you need to layer different risk profiles on top of this to capture individual risk appetite?
4 October, 2021 at 6:29 am
Terence Tao
One feature of this system is that it allows for easier conversion from individual risk, which as you say people already have reasonable intuition for, to risk on groups. For weather this is not an issue, since in a group outing if it rains on one person it will almost certainly rain on everybody (the risk of rain is almost completely correlated amongst the group). But for say pandemic risk the situation is different and one’s intuition can lead one awry. An individual might take their chances with a 2% case fatality rate (1.7 nines) when contracting COVID (though this is still rather risky and unwise in my view), but may not realise that under a scenario in which say their extended family of ten people all contract COVID then the probability of at least one fatality in the group increases (assuming independence) to 18% (0.7 nines) which is much less acceptable. (This is an oversimplified model, since comorbidities will make the fatalities in this group correlated, but still hopefully illustrates the point that a nines-based perspective can help gauge risk more accurately.)
5 October, 2021 at 7:27 am
Anonymous
IFR for Covid is 2% only in the 70+ age category
5 October, 2021 at 7:32 am
Anonymous
Table 1 is the official IFR for Covid from the CDC
https://www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html
As you can see it is way lower than 2%.
5 October, 2021 at 1:16 pm
Anonymous
The caption of the table you cite literally says “These are *not* predictions or estimates of the expected impact of COVID-19.”
Plus, it is written just above that in this table, scenario 5 “represents the best estimate, based on the latest surveillance data and scientific knowledge.” Scenario 5 gives an IFR of 9% for 65+ years old, not 2% for 70+.
Also, this page was last updated on March 19 2021, way before the delta variant became dominant.
5 October, 2021 at 4:33 pm
Anonymous
The caption of the table you cite literally says:
“The scenarios are intended to advance public health preparedness and planning.”
Cool you are right it’s 9% for 65+ and 0.05% for < 49 years. It is a pity that CDC does not update its information. One would have thought it's important in light of delta, but apparently it's not anymore.
6 October, 2021 at 6:14 am
Anonymous
Sorry if my comment sounded too harsh or nitpicky, just wanted to nuance the data given at the link you shared. I agree with you, it is a pity the CDC page has not been updated (though they do explicitly caution that their table is not meant to give estimates). In general, I don’t think that this specific CDC page is a very good reference for a realistic estimate of the COVID-19’s current IFR in the general population.
This meta-analysis from the European Journal of Epidemiology (2020, before the alpha and delta variants) estimated the IFR of COVID-19 in general population to be between 0.5% and 2.5%, depending on the country (see Fig. 6): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7721859/.
So the 2% given by Prof. Tao do not seem too far off.
Note that in hindsight, these estimates do not seem unrealistic. For instance about 0.18% of the whole population of Peru and 0.35% of the whole population of the Brazilian state of Amazonas officially died of COVID-19 as of today (and of course not everyone in these populations got infected).
6 October, 2021 at 6:32 am
Anonymous
This other independent meta-analysis from a few months before estimates the IFR between 2% and 4.4% : https://www.sciencedirect.com/science/article/pii/S138665322030113X?casa_token=6bKNuR7AkjsAAAAA:DCtkc7gDAhGhJ-Z7YnQBHfCyidnyNIjFBeWCcH4sJ74eaIEopCWBhVY4pdhKzzpg61gP_4Q_9g
6 October, 2021 at 4:12 pm
Anonymous
It’s actually kind of crazy that 3/2 year into a pandemic it is not possible to find an authoritative IFR that we can all agree on. This is why I choose to cite the CDC because it seemed to me to be the estimate that people would be the least likely to contest.
10 October, 2021 at 4:55 pm
David Hilbert
Since covid IFR is exponential with age, unconditional nines of safety are largely irrelevant. But conditional nines are illuminating.
Roughly, covid IFR(%) ~ 10^(age/20)/2,000 (rounding and rearranging the log(IFR) formula in the linked paper below.)
So that’s 5.3 nines of safety for newborns, 4.3 for 20 year-olds, 3.3 for 40 year-olds, 2.3 for 60 year-olds, and 1.3 for 80 year-olds.
40 (or 3.3 nines) is the cutoff around which covid risk is behaviorally relevant.
https://www.ncbi.nlm.nih.gov/labs/pmc/articles/PMC7721859/
11 October, 2021 at 9:21 am
Terence Tao
Given that the mRNA vaccines decrease the risk of death from COVID by a factor of ten or so, perhaps one way we should frame the benefit of the vaccine in view of this data is that it effectively makes you 20 years younger (with respect to the risk of dying from COVID). :-)
15 October, 2021 at 1:06 pm
Anonymous
20 years younger?60 to 40,or 40 to 20?
[As a first approximation, yes to both: a vaccinated person of age has roughly the same risk of dying from a COVID infection as an unvaccinated person of age , for pretty much any . -T]
29 December, 2021 at 1:45 pm
Ben
Hi Terry, be sure not to conflate correlation and causation! Being 20 years younger would also influence other variables that further change the prediction.
30 January, 2022 at 4:57 am
Anonymous
That’s a cool illustration of the usefulness of the concept.
4 October, 2021 at 4:32 am
YahyaAA1
Perhaps some automatic translation app will turn “k nines of safety” into “guard dogs”?
4 October, 2021 at 7:28 am
Mason Porter
Predrag Cvitanovic wrote down a comparative ‘struck by lightning’ metric on his blog years ago: https://chaosbook.blogspot.com/2015/12/how-important-is-this-news-item-to-me.html
4 October, 2021 at 8:12 am
Terence Tao
Nice! The “struck by lightning” measurement has the advantage that it automatically scales for changes in group size or exposure duration, avoiding the need for the additive or subtractive modifiers used here. The equivalent metric for nines would be the relative number of nines compared to being struck by lightning; for instance, dying in a car crash is about 3.2 nines more likely than being struck by lightning (corresponding to the “1400 SBLs” in that blog post), regardless of how large a group or what time period one is considering.
4 October, 2021 at 12:05 pm
Chris J.
When I first started reading this post, I thought it was going to be a single unit of risk that lets you compare across choices where the severity may be different. For example, if with action #1 you have a 10% chance of catching a cold, but with action #2 you have a 0.0001% chance of dying, which should you choose? (Comparing different medical operations / procedures or choosing what foods to eat, etc. would be more concrete examples of this.) I think these types of choices come up a lot in life, and it would be helpful to have an easier way to reason about them.
4 October, 2021 at 6:25 pm
Terence Tao
One could take an economic viewpoint here and assign a monetary cost to any given risk. One crude metric here is the size of insurance policies that people generally take out against a risk: for instance if the average size of a life insurance policy is $1 million, then this would suggest that death would be equivalent to a cost of $1 million, or six orders of magnitude more than a dollar. Each nine of safety would then reduce this cost by an order of magnitude, for instance if one had six nines of safety against death as you suggest then the net cost of action #2 would roughly be a dollar. Doing a similar exercise for the cost of action #1 (figuring out the monetary cost of catching a cold and multiplying by 10%) would then allow for a comparison. So the question is how many orders of magnitude of cost one has for any given negative outcome.
(Of course, one could argue that not all costs can be measured by purely monetary means, but perhaps there are more sophisticated ways to estimate total cost that extend beyond the monetary dimensions. For instance one could use existing life choices of the individual to infer some implied relationships between the relative costs that individual assigns to various risks and thus tease out their implicit risk tolerances, though there may be limitations to such an approach if the individual is not perfectly rational.)
26 October, 2021 at 1:34 pm
arch1
The expected monetary value approach you describe here seems at odds with your earlier observation that the difference between a 98% and a 99% chance of (say) surviving feels much bigger than the difference between 50% and 51%.
Any thoughts on reconciling these?
1 November, 2021 at 12:48 pm
Terence Tao
This comes down to comparing costs in relative terms rather than absolute terms. If we assign a monetary value to death, then an action with a 98% survival rate has twice the expected cost as an action with a 99% survival rate, whereas an action with a 50% survival rate is only about 2% more “expensive” in cost than an action with a 51% survival rate. But in absolute terms they both cost the same amount of additional mortality.
2 November, 2021 at 11:31 am
Anonymous
the ORF is a biological term means opening reading frame,which you cannot simply summarise its shade as arch :-)
3 November, 2021 at 2:28 am
Anonymous
still miss the sunshine that day,see you in overflow:-)
4 October, 2021 at 12:22 pm
Anonymous
“of a the” in defn 1
[Corrected, thanks – T.]
4 October, 2021 at 12:38 pm
Anonymous
Sorry if I missed it, but is this proposal for the general public (and everyday language) or for scientists? If it’s the former, I wonder how well people can adapt their intuition to a logarithmic scale..
4 October, 2021 at 4:38 pm
B Fortner
And all this talk about cheese is making me hungry! But seriously, I LIKE IT. If we use logarithms for earthquakes, and most people are familiar with that (and many understand the scaling of logarithms), they could certainly be useful for describing a pandemic, and comparing it to other pandemics, or even other disasters, if the same variables are used (such as deaths).
Sent from Yahoo Mail on Android
4 October, 2021 at 10:18 pm
Anonymous
Just curious, are you thinking about these things as part of your new role in the White House?
11 October, 2021 at 9:03 pm
KM
This is great news. Congratulations Prof. Tao. It is good to have top researchers in public policy making positions.
5 October, 2021 at 1:41 pm
Tom Matte
Professor Tao,
I wanted to send you an article I wrote for Medium. I think you’ll find the entire paper fascinating, especially the math. You can skip the intro and scroll down to the math section if you like.
View at Medium.com
The way my mind sees mathematical objects doesn’t fit into the nice tidy package that we’re all taught as students. This coordinate system is fluid and has multiple origins, none of which are static.
Feel free to respond if you have any questions or need clarity on some of my work. I’m not a mathematician, but I do my best.
Thanks,
Tom
404-625-4611 tom-matte.com
>
5 October, 2021 at 6:31 pm
Anonymous
You can’t see gravitational waves, Tom, and your coordinate system is nonsense.
5 October, 2021 at 4:18 pm
Casual causal
Causality is the new hype.
6 October, 2021 at 3:26 am
Natanael
Thanks for the nice post Terence. Do you know the book by I. J, Good, Probability and the Weighing of Evidence? Working with A. Turing to break the Enigma codes, Good used ideas similar to the ones you proposed,
Cheers!
6 October, 2021 at 12:18 pm
Ben
Another approach to commicating certain kinds of risk is the micromort & the microlife. 1 micromort = 1 in a million chance of death (i.e. 6 nines of risk), and 1 microlife = a half hour reduction in life expectancy.
However Professor David Spiegelhalter tells me they didn’t gain the widespread use he’d hoped for: https://www.facebook.com/econometricsguru/posts/2307447336003065
6 October, 2021 at 2:36 pm
anonymous
Extending your approach, could we also use confidence intervals and hypothesis tests (chi-square test, Student t test, Fisher-Snedecor test)?
7 October, 2021 at 4:33 pm
Greg
In the proof of proposition 10, the word “and” is not to be written in italic.
8 October, 2021 at 11:32 am
Anonymous
yes it is
8 October, 2021 at 5:40 am
Dara
This could be a great way of managing something I’ve seen as a big issue in Safety Engineering. In Failure Modes and Effect Analysis (FMEA), types of system failure are given a risk score which is calculated by multiplying a Probability score (1-5) by a Severity score (1-5), and sometimes multiplied again by a Detectability score. In the original military standard, MIL-P-1629, the levels of Probability are defined clearly: ‘Frequent’ is >20%, ‘Reasonably Probable’ is 10-20%, ‘Occasional’ is 1-10%, ‘Remote’ is 0.1-1%, ‘Extremely Unlikely’ is <0.01%. In the vulgarised versions of FMEA that crop up in manuals, these definite scores tend not to be included, and the non-linearity of the scale is lost. In my view, this means the tool ends up misrepresenting the nature of risk in these kinds of critical operations (like healthcare or aviation), as it makes the extreme risks under consideration seem more manageable. I really like the idea of the ‘nines’ as a way to make log-scales practicable, and getting an alternative notation to simple probabilities. If anyone knows of anything similar to this concerning risk, I’d love to hear.
9 October, 2021 at 12:09 am
Aditya Guha Roy
Reblogged this on Aditya Guha Roy's weblog.
9 October, 2021 at 12:12 am
Aditya Guha Roy
This reminds me of Shannon entropy which I recently encountered and I am still studying about it, but I cannot figure out a proper connection between these two notions at this moment (must be because I don’t know much about Shannon entropy).
10 October, 2021 at 4:01 pm
Anonymous
The current trend in US governments discourages gun use more, but legalizes drug use. Not a rational response.
10 October, 2021 at 4:05 pm
Anonymous
Speaking of irrational responses, the US is set to be passed in economic output by China in under 10 years, but the current government’s policy abroad emphasizes it’s not “US or China”. The US also made a State visit to Vietnam. A straightforward win instead is a correct response.
10 October, 2021 at 4:26 pm
Anonymous
China gained two years on the US in its race to overtake it economically as a result of the COVID-19 pandemic, according to current sources. Does anyone have an estimate of how this affects the probability of a deliberate attack? I had estimated this at high probabilty.
10 October, 2021 at 5:42 pm
Anonymous
It’s fair to use harder science in decision making, which makes choices that are bright line. Hard analysis is also more useful than soft. When soft ergodic type theorems are proved instead with hard analysis, they have more applications.
[Corrected, thanks – T.]
15 October, 2021 at 3:31 am
Optimus subprime
Reminds me a bit of the “march of nines” that Elon has referred to regarding the development of self-driving cars.
17 October, 2021 at 8:14 am
Donald Richards
A minor typo: In the proof of Proposition 6, I think you need to change “geometric distribution of parameter {10^k}” to “geometric distribution with parameter {10^{-k}}”.
[Corrected, thanks – T.]
24 October, 2021 at 9:02 am
Nines of safety: a proposed unit of measurement of risk - 3 Quarks Daily
[…] More here. […]
24 October, 2021 at 4:38 pm
simnett
Would anything structural (ie aside from the numbers themselves) change if you used natural log, vs log (10)? It feels like 1/e is a nice sort of number, whereas (say) 3 * 1/10 is harder to think about. Related- if I have a 20 sided dice I “expect” (greater than 50%) to get a one after only 14 rolls, but I also have a ~1/e chance to not get a one after 20 rolls (which is much higher than most people expect).
28 October, 2021 at 2:13 am
Three nines risk, ma’am – The nth Root
[…] In everyday usage, we rely heavily on percentages to quantify probabilities and proportions: we might say that a prediction is accurate or accurate, that there is a chance of dying from some disease, and so forth. However, for those without extensive mathematical training, it can sometimes be difficult to assess whether a given percentage amounts to a “good” or “bad” outcome, because this depends very much on the context of how the percentage is used. For instance: … (What’s new/ Terence Tao) […]
3 November, 2021 at 10:19 pm
Anonymous
Dear Pro.Tao,
Last night, I dreamed of a respected prince riding a white horse was coming to near the most beautiful forest in the world. More and more, it iwas very clear! The scence was very splendid like paradise! There was always a beautiful bird flying forward on the prince’ head to make a direction to the forest. How strange it is! Maybe,this seems to be the best news for the world.
13 November, 2021 at 3:48 pm
Christian Haupt
‘In summary, when debating the value of a given risk mitigation measure, the correct question to ask is not quite “Is it certain to work” or “Can it fail?”, but rather “How many extra nines of safety does it add?”.’
In reality, we may face the problem, that a given risk mitigation measure carries itself risk that is at present unquantifiable.
20 November, 2021 at 3:45 am
DrB
If you are in the age range of 30-40. for healthy people, the risk of hospitalization from Covid is 1 in 4500 = 0.00022, source: https://www.qcovid.org/Calculation. Your chance of hospitalization from the covid shot (2 doses) due to myocarditis is 1 in 20000. Given that the vaccine effectiveness is basically zero after 6 months, you will need two boosters to have protection from hospitalization for an entire year. Hence the risk for myocarditis related hospitalization over the course of one year is therefore 0.00015. One additional booster the next year and your risk for hospitalization from the vaccine is greater than your risk of covid related hospitalization (you only get covid once, after which your natural immunity protects you from severe disease, probably a lifetime). This is, I think, an example of a cascade failure of a system. The vaccine effectiveness, over time, turns negative. You further need to consider that the creation of virulent variants, due to selective pressure from the S-specific antibodies in the vaccinated, may reduce the effectiveness of the vaccine even earlier than the 6 month window. Of course, mass vaccination campaigns increase the chances of escape variants of the virus. If, in addition, your waning antibodies from the vaccine lead to antibody dependent enhancement, you get additional increased risk of severe disease due to the vaccine, bringing the effectiveness even more into the negative numbers. Altogether, your risk from the vaccine is, over time, much much greater than your risk from the disease itself.
20 November, 2021 at 7:40 am
Terence Tao
What is the source on your myocarditis risk statistic? In particular, is it for the same (extremely favorable) “age range 30-40, healthy” demographic that the statistic you are trying to compare it with is drawn from?
20 November, 2021 at 7:49 am
DrB
“Myocarditis With COVID-19 mRNA Vaccines”, Journal Circulation, page 3. I guess one can nitpick this, because the background rate is probably included in this estimate 1 / 20000. Age range for Myocarditis is 18-30. So I probably should have made my argument for the 20-30 age range to be more accurate and on target. The risk for Myocarditis from the vaccine decreases with age, so in the range 30-40 it is probably lower than 1 / 20000. I still think the overall argument can be made that way. Also, one should definitely include all other risk factors associated with the vaccines for a true thorough analysis, like the rate of Guillian Barre Syndrome, Appendicitis (this one, if caught early, is very treatable, but still almost always results in hospitalization), etc. All side effects are VERY RARE by themselves, but they are cumulative and that is the point.
20 November, 2021 at 8:07 am
Terence Tao
Your analysis assumes that (a) the probability of an adverse reaction from a vaccine dose is independent for each dose (i.e., ignoring all risk factors that would be common to each dose), and (b) the lifelong COVID reinfection rate is zero. Also (c) you are still underplaying the risk of COVID by restricting to the “healthy” demographic when considering COVID risk, but not when considering vaccine risk.
In any case, the risk analysis you mention was indeed conducted in the journal paper you cite. To quote from its conclusion: “Given the known potential risk of complications with COVID-19 infection, including hospitalizations and death even in younger adults (mortality remains 0.1–1 per 100 000 for persons 12–29 years of age), the risk-benefit decision remains overwhelmingly favorable for vaccination.”
20 November, 2021 at 8:46 am
DrB
Thank you for your reply. Some remarks:
(a) The validity of this assumption requires more data. We don’t know for example the exact etiology of the vaccine induced Myocarditis. But I doubt that an argument of the form “if person x did not get M. from dose n, he will very likely not get M. from dose n+1” is valid. The occurrence of the inflammation could be completely up to the chaotic dynamics of the patients body, in combination with the distribution of the contents of the shot in the system. We do know, the risk is higher after the second dose. This is likely due to the presence of antibodies from the first dose, that can attack heart muscle, when the protein from the second dose is introduced, detected by the immune system and subsequently attacked. This would, I believe, give credence to the idea of assigning similar probabilities to booster shots. But this is for now only conjecture.
(b) I do not assume the lifelong reinfection rate to be zero. I do assume that the t-cell memory of the previously infected recovered patient recognize the pathogen at a later time, mounting a sufficient reaction to render the patient essentially save from a severe case of covid. He may still experience symptoms, but is essentially save from hospitalization. This assumption is backed by a great number of publications.
(c) The particular rate of Myocarditis (at least) is highest, in the very healthy and young demographic. In fact, athletes seem to be disproportionately affected.
The risk analysis of the paper does not take into account my larger point. Namely, that we will need booster shots in very short ranges, over an indefinite amount of time. Covid is very unlikely to disappear. Assuming that a vaccine with better durability does not come along in the near future, the cumulative burden of repeated booster shots should at least be taken into consideration. In particular, we could ask ourselves if potential treatments, using medication, could reduce covid risks more effectively in the long run.
20 November, 2021 at 8:56 am
DrB
I should add that point (b) is much more difficult for the vaccinated. If the vaccinated person gets a breakthrough infection with covid, it may not be the case that they have the same broad protection from severe disease upon reinfection as the unvaccinated, due to a phenomenon called antigenic original sin. Data from public health England seems to support this: The previously infected vaccinated have lower levels of N-specific antibodies than the previously infected unvaccinated. This suggests that the S-specific antibodies of the vaccinated suppress their ability to mount a broad immune response upon infection. Hence a booster may be required EVEN after having had and recovered from a breakthrough infection.
20 November, 2021 at 5:53 pm
Terence Tao
The protective effect of COVID-19 antibodies is expected to wane after a year or so, which is not incomparable with the similar waning effect of the vaccine. But in any event this effect is only available to patients who were already infected at least once with COVID. If the risk to be measured is that of contracting COVID while unvaccinated at least once, this cumulative risk will still grow essentially linearly with time regardless of antibody protection amongst the previously infected; avoiding infection in one year confers no protection whatsoever against being infected the next. Compared against the conjecturally linear cumulative risk of booster shots, the net effect is largely a wash.
Also focusing on hospitalisation data alone seems to be the wrong metric, given that the average unvaccinated COVID hospitalization is significantly more severe than the average vaccine adverse reaction hospitalization (in particular the fatality rate from the former is certainly higher).
Finally, the young and healthy demographic also seems to be the wrong demographic to focus on for this discussion, given that boosters are only recommended at present for high-risk groups such as the aged or those with pre-existing conditions.
Certainly proper risk assessment should be done for all treatment options, but the proper venue for this is a peer-reviewed publication, not a sequence of blog comments.
12 December, 2021 at 12:24 pm
Robi Rahman
Your idea was cross-posted to the Effective Altruism forum,
https://forum.effectivealtruism.org/posts/L9pB9sWTngF4BcHeL/nines-of-safety-terence-tao-s-proposed-unit-of-measurement
where someone pointed out that this idea is already in use for electric grid availability:
https://en.wikipedia.org/wiki/High_availability