Forecasting Terminology
Quantitative Epistemology Optimization
Definition: The use of math, science, and technology to reduce uncertainty on general questions in ways that create decision value. General questions are defined as questions that include both simple quantitative questions (“Given these parameters, calculate the mean”), and questions that require significant human judgement (“Based on this complex historical evidence, how many people lived in historic Rome?”). In particular, Quantitative Epistemic Optimization refers to a cluster of techniques that can be used together to help with these issues.
Alternatives:
- Epistemic Engineering
- Quantitative Epistemology
- Epistemic Optimization
Confidence: 2/5
Decision Information Value (Decision Value for short)
The value that comes from information insofar as that information helps your decision making. Similar to “Value of Information”, except:
- It’s not the “value of perfect information”
- It’s not specific to one decision, but the expectation for all future decisions. Confidence: 2/5
Epistemic Pain:
Economic personal or organizational losses that come from improvement in beliefs.
Confidence: 1/5
Alternatives: Epistemic Loss, Epistemic Sacrifice
Examples:
“After careful analysis, we realize that you have been a terrible parent.”
“It turns out that our business has been dramatically harming the environment”
“Our project is likely to fail”
“I’ve come to believe in Islam, although my family is Christian, and would dissown me if they knew. I could keep it a secret, but doing so is costly.”
Deliberation
The act of an agent doing work (thinking or gathering information) in order to reduce uncertainty on a set of questions.
Confidence: 4/5
Debate
Alternatives: Reconciliation, Discussion, Back and forth
Multiple agents with different estimates discussing things / sharing information, with the goal of reducing uncertainties.
Confidence: 2/5
Note: Gah, I don’t like this.Value vs. accuracy is important for a different distinction.
Prediction-Information-Gain vs. Prediction-Accuracy
Prediction information gain refers to the information gain or that comes from information. Prediction-Accuracy refers to how well that prediction reflected its corresponding answer. It’s possible that these could be not perfectly correlated; for instance, agent A may have poor accuracy, but is highly valuable because their predictions help an aggregate. There can be multiple measures of value and accuracy.
Confidence: ⅗
Other words: Instead of information gain: accuracy gain, benefit, assist, bonus,
One interesting tidbit here: If you actually believe something, you don’t need a proper scoring rule as your loss function. You’re not going to lie to yourself.
Expected Estimation Loss
Alternatives: Expected Error Loss, Expected Loss, Expected Score, Expected Certainty, or we just abandon this.
An alternative name for entropy, when done via a prediction and when using log error. Units may be the same as entropy. The reasoning for this is that entropy is used all over the place, and typically refers to other situations. The phrase “I’m trying to minimize the expected loss of this forecast” seems more intuitive to me than “I’m trying to minimize the entropy of this forecast”. My guess is that “entropy” would cause more confusion.
Confidence: ⅖
Note: If you are estimating a distribution, things get tricky here. You can’t take the differential entropy.
**Estimation Loss:**The actual loss, after the result is known. There are many ways to calculate error.
Alternatives: Ex-post error/loss (compared to ex-ante), Post-evidence error, post-improvement error
Estimation
An estimation is an attempt of a quantification on a specific parameter. In most cases estimations are uncertain, though in the limit they could have no uncertainty.
Prediction
Predictions are estimates that attempt to estimate the results of a verification procedure. Most estimates can be arbitrarily turned into simple predictions, if one is flexible enough with the verification procedure.
Verification
A verification is an estimate of a parameter after the acquisition of evidence. One prediction’s verification could itself be a prediction for a further verification.
Predictions vs. Verifications share a similar relationship as Priors vs. Posteriors; they are a similar shape, but separated by the presence and update of evidence.
Evidence
Information that helps reduce uncertainty on a set of variables. Predictability could be defined pre-evidence and post-evidence for any set of evidence.
Discovery
The release of evidence. This is often a gradual process.
This is similar to discovery in law, which is a specific period where evidence is gathered.
Evaluation
Verifications that require judgemental analysis of information, rather than directly taking a specific statistic or fact.
Estimation / Prediction / Verification Curves
Graphs of expectations of work done vs. expected information accuracy or value. Most of the time there are diminishing returns.
Alternative names: Estimability / Predictivity / Verifiability Curves.
Estimation / Prediction / Verification Limits
The limits of expected information accuracy or value with arbitrarily large resources. “Arbitrarily large” could be defined differently depending on the circumstances; there are some situations that could expend far more resources than others.
Estimation / Prediction / Verification Potential
The remaining potential to improve, from the current point to the limit. Can be specified in percentage of the original, which may be a bit useful.
Work & Effectiveness
Section titled “Work & Effectiveness”Estimation / Prediction / Verification / Evaluation Work
Costs imposed to do better at estimation/prediction/verification.
Note: There should be some way of specifying the cost-effectiveness of these kinds of work, vs. the returns by having them.
Frontier? Efficiency?
Accuracy vs. Value
(Estimation | Prediction | Verification | Evaluation) Work (Accuracy | Value) Effectiveness
The marginal accuracy or value gain of a specific intervention, divided by its’ cost. Either counterfactual or using Shapley values.
(Estimation | Prediction | Verification | Evaluation) Work Accuracy | Value) Frontier
The set of projects that would maximize expected effectiveness
Tractability?
This seems useful, but I’m not sure what on the curves it should refer to.
Prediction Setup
Alternatives: System, Engine*, Machine, Complex, Compound, Network, Procedure, Scheme, Design, Faction, Party, Body, Agency, Company, Body, Rig, Outfit, Unit, Ensemble, Faction, Partnership, Coalition, Vehicle, Setup, Coalition, Market
Note: Greg Lewis was in-favor of engine*.
Prediction System Appraisal
Alternates: Attestation, Pinning, Assessment,
https://nlp.stanford.edu/IR-book/html/htmledition/an-appraisal-of-probabilistic-models-1.html
Probability
Section titled “Probability”Perspective
The probability of X from agent Y’s perspective, given information Z, is the posterior of Y after doing a bayesian update on Z, using Y’s prior.
Ontology
Predictability
Relevant wikipedia pages:
https://en.wikipedia.org/wiki/Entropy_estimation
https://en.wikipedia.org/wiki/Nat_(unit)
https://en.wikipedia.org/wiki/Information_content
https://en.wikipedia.org/wiki/Estimator
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
https://en.wikipedia.org/wiki/Differential_entropy
https://en.wikipedia.org/wiki/Perplexity
https://en.wikipedia.org/wiki/Qualitative_variation
https://en.wikipedia.org/wiki/Level_of_measurement
https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy
https://en.wikipedia.org/wiki/Standard_error
Comments from Nuño Sempere
Section titled “Comments from Nuño Sempere”Restored with permission (Nuño’s comments, with Ozzie’s replies).
On “Quantitative Epistemology Optimization”:
Nuño Sempere: In general, I’d favor a style in which for every definition, you give an example.
Nuño Sempere: Who is this document for?
Ozzie Gooen: At this point, us. Later on I’d want to rewrite it into a longer form with more explanation, but hopefully keep the terminology the same.
On “Confidence: 2/5”:
Nuño Sempere: What does this confidence refer to?
Ozzie Gooen: Sorry; it’s kind of my confidence that this is generally a good term to use; it’s a combination of the specific term and the “idea of the term”; independent on which specific name we choose.
On “Epistemic Pain:”:
Nuño Sempere: Epistemic Loss
Ozzie Gooen: Thanks! Good idea, I’ll think about it.
Ozzie Gooen: “Epistemic loss is typically conceived of as the loss of a corpus of knowledge, or less commonly, as the further loss of epistemic methodologies. ” https://www.tandfonline.com/doi/abs/10.1080/21550085.2017.1342966 I think pain, in comparison, will make more sense with future examples.
On “Prediction-Information-Gain vs. Prediction-Accuracy”:
Nuño Sempere: Here is perhaps a rewrite: Improving the engine vs outperforming the engine. In a prediction engine, like a prediction market, we hope that participants improve the market. They may do so by outperforming the engine, for example by beating a prediction market. By being more accurate than the market, they help move the market in the right direction. But they can also improve the engine without outperforming. In a prediction market, this can happen if a participant is inordinately convinced by a factor which the market hasn’t taken into account at all yet. They would also help the market move in the right direction, while they themselves are not very accurate. This distinction can be expressed in many ways: - Improving the engine vs outperforming the engine - Adding information to a market vs scoring well in that market - Improving the aggregate vs being better than the aggregate - etc.
Ozzie Gooen: Good point. In general, everything in this doc really does need to be explained better before reaching a larger audience. The main goal right now is just to get a bit of consensus/discussion on the key ideas/terms (though this is tricky without much discussion)
On “Loss”:
Nuño Sempere: I prefer Error
Nuño Sempere: In particular, loss may not make sense as a word in this context unless “minimize the loss function” says anything to you. Error is also Latin, i.e., easier to understand by Europeans.
Ozzie Gooen: Good to know about error being latin, that never would have occurred to me.
On “My guess is that “entropy” would cause more confusion”:
Nuño Sempere: If you make this decision, you may want to rewrite the definition so as not to reference entropy?
Ozzie Gooen: There is a lot of overlap/similarity with entropy; it can be the same depending on loss function and setup.
On “Predictability”:
Nuño Sempere: You haven’t defined predictability yet
On “.”:
Nuño Sempere: It may be the case that a question is significantly easier or harder to predict before or after a piece of evidence comes in
Nuño Sempere: Only one set of curves? It’s also not really obvious from the name what the curves refer to. (As opposed to, say, “supply and demand curves”)
Ozzie Gooen: Hm… I guess there’s a question of how many curves we could imagine there being. Happy to be more specific.
Nuño Sempere: Why do you need a technical word for this, as opposed to “prediction accuracy at the limit”?
Ozzie Gooen: It’s quite possible we don’t. But if people end up using this a lot, they may want a tighter term.
Nuño Sempere: Why don’t you just talk about prediction cost, prediction cost-effectiveness, and maybe divide prediction cost in cost of infrastructure vs participant cost / marginal cost?
Ozzie Gooen: Good point, I’m not sure what kind of structure is best here.
Nuño Sempere: Another term I’ve seen used is recalcitrance, i.e., how the difficulty of smth, in this case prediction, increases as you put more resources into it.
On “Engine”:
Nuño Sempere: +1
Ozzie Gooen: I’m curious to understand why you favor engine. I could see the infrastructure as being the “engine”, but the main attribute will be the people, which I’m not used to thinking in terms of “engine”.
Nuño Sempere: Black box intuition? I remember I’ve seen it used in one of Warren Buffet’s yearly letters, and maybe I associate it with prestige and other nice things because of that?