Skip to content
QURI Priors

Epistemics: An Early Guide

Epistemics: An Early Guide

Section titled “Epistemics: An Early Guide”

Meta:

This is meant as an example of an index to a book/course/etc on epistemics. The goal is to represent a comprehensive take on what materials are useful to improve epistemics.

This represents my opinionated take on what the field could be.

This obviously is very much lacking. There’s little on community/organizational work, there are very few mentions of relevant literature. I’d flag that this is inspired heavily by the Rationalist/EA scene.

This covers a great deal of ground. That’s much of the point. This is an early project, we want to lean on making sure that potentially useful things don’t get missed.

Trying to go into depth on each bullet here would take ages. LLMs and independent study could help a lot here. The goal is to provide a high-level take on useful areas for people to investigate using other materials, not outline a textbook to try to comprehensively cover each area.

Opinions welcome! I’d like to spend some more time on this, then post it online.

–

Theory

Section titled “Theory”
  • What is epistemics?
    • “Being correct about things, particularly when being correct is socially difficult.”
    • “The effectiveness of a person or team on finding valuable information.”
    • There’s a history of the term “Epistemology”. It should be clear where the term “epistemics” came from and who uses it.
  • What philosophical background will we use? (“Epistemic optimization” vs. “Epistemology”)
    • Different schools of thought would understand epistemics in different ways. We will draw heavily from:
      • Bayesianism
      • Empiricism
      • Economics, particularly Information Economics
      • Management Theory
      • Analytic Philosophy, as opposed to Continental Philosophy
  • What would better epistemics do?
    • Hedge funds with better epistemics would make more money. They would better prioritize projects and remove outdated takes.
    • Policy advisors with better epistemics would improve policy quality.
    • Individuals with better epistemics would generally be more confident about many potentially-contentious issues in their lives.
    • Broadly speaking, epistemics should help improve decision-making at scale, leading to benefits whenever humans are involved with things. It’s also likely to significantly improve coordination.

Examples in various fields

  • Judgemental Forecasting
    • What practices do top forecasters use?
    • Understand Accuracy, Calibration, Scoring Rules, Prediction Markets, etc.
    • This area acts as a good microcosm of optimizing epistemics.
  • Finance
    • Which hedge funds and market firms represent the big-picture best reasoners? What are those groups doing?
    • What information do hedge funds pay for? Why do they seem to pay for large databases, but not much for subscription journalists or analysts?
  • Philosophy
    • Cover the philosophical schools of thought underlying Bayesianism/Empiricism/Analytic Philosophy.
  • Journalism
    • What skills do top journalists use?
    • What are the main challenges faced by journalists / journalism?
    • Why is there so little money in Journalism? What can be done to change the incentives?
  • History
    • What are examples of those in power, or the masses, attacking the truth? For example, limiting speech, restricting education, criminalizing dissent, surveilling communication? How well do these follow common patterns?
  • Moral Situations / Advocacy
    • How successful were Truth and Reconciliation Commissions in history?
      • What are examples of countries/situations that probably should have had Truth and Reconciliation Commissions, but didn’t? Why didn’t they? How expensive would these have been?
    • How does one speak “Truth to Power?”
  • Business
    • Many successful entrepreneurs (i.e. Steve Jobs, Elon Musk) are remarkable at making some decisions (marketing, product), but dramatically overconfident in other areas (i.e. their product successes). Why is this?
    • How does groupthink impact key organizational decisions?
  • Politics
    • Why do dictators frequently fall into conspiracy theories?

When True Knowledge is Inconvenient, Or False Knowledge is Convenient

Section titled “When True Knowledge is Inconvenient, Or False Knowledge is Convenient”

One major epistemic challenge comes when knowledge of the truth happens to be inconvenient for some set of actors. Arguably most accessible “convenient truths” have been picked, but the “inconvenient truths” are frequently pushed aside. This creates an environment where truth-seekers frequently run into “inconvenient truths”.

In these cases, much of the bottleneck of spreading information will be about how to work with or around the actors who would prefer it not being spread.

Examples include:

  • A government that wants to hide corruption
  • A religious order that wants to hide logical problems with their belief
  • Software companies typically want to hide information about security breaches, to the extent they can legally do so
  • A parent who hurt their child but doesn’t want to admit it to themselves
  • A professor who held their field back 10 years but won’t admit it
  • A person who thinks of themselves as highly altruistic but is actually quite harmful
  • Much of the world has “status quo bias”, or inertia to change. Truth can get in the way of this. For example, say that new scientific metrics would come out that would somewhat accurately estimate the global value of each work of science. I’m sure any attempt at this, especially decent and trusted attempts, would upset a great many people.

There’s no one “rulebook for advancing inconvenient knowledge.” But there is a lot of material on different aspects of this.

  • Journalists have a lot of experience fighting power figures who don’t want certain work published. This involves a mix of [carefully understanding what would upset which people] and [being prepared to fight, when things get adversarial.]
  • Lots of work around moral advocacy. For example, people promoting animal welfare often run into the issue that a lot of their educational work involves making people understand the moral downsides of their actions. This is a clear inconvenience for these people. Similar for most other forms of moral advocacy (racism, environmentalism, etc).
  • Therapy, particularly around difficult conversations. Many patients don’t want to admit many key things about themselves. Getting them to do so can require a great deal of empathy, patience, practice, and skill.
  • In business, many people have challenges giving honest critiques to each other. This can be quite delicate. Bosses are often mediocre at being both honest and empathetic to their subordinates. And many businesses are poorly set up to allow subordinates to relay important information to superiors.
  • Different cultures vary in how candid they are.

It should be noted that the above include examples of true knowledge that would be inconvenient to a certain party, but likely convenient/positive overall. There are also a lot of cases where true knowledge would likely be bad on the whole. For example, areas where privacy is net-valuable.

  • Often, certain privacy enables property rights. If passwords couldn’t be kept public, that would result in a lot of economic damage.
  • A political dissident wants to hide their work from the public or the government.
  • A person gets involved in a small scandal that would become dramatically misunderstood if made public.
  • A person wants to sell their work online, but if it gets distributed for free, they couldn’t do so.

This means that “promoting inconvenient truths” is sometimes pro-social and sometimes anti-social. There are critical disagreements on which cases are pro-social and which are anti-social. (Obviously, inconvenienced parties will typically think and argue that such knowledge is anti-social)

Tools For Convenient Lies

Section titled “Tools For Convenient Lies”

Many tools that can be used for spreading truth can also be used for spreading falsehoods. But some differentially prefer falsehoods. Here we try to focus on the latter.

Motivated Reasoning / Self Deception (Personal, Unintentional)

Motivated reasoning has been extensively studied and outlined.

One interesting question is to study where and when it’s useful to the offending party.

“Being yourself wrong about things” can be pragmatic and useful, but it can also be easy to overshoot. In general, this seems like an incredibly difficult thing to get right.

If “convenient and unusual falsehoods” were a product, it should come with major warning signs. “Using this product is likely irreversible. After using it, the world will make less sense to you and many of your plans won’t work as well as you envisioned. This product is known to be highly addictive.”

Egos (Personal, Unintentional)

It’s clear that many people greatly overestimate their own abilities. This is one of the clearest and largest human biases. Oversized egos lead to untruths, and untruths lead to all the tools of spreading untruths.

As is the case for other reasons for convenient lies, overly large egos carry a bunch of personal advantages. This can be useful for both personal motivation and for convincing others.

There’s a decent amount of literature on narcissism, which is a clinically diagnosed, but particular form of “having an outsized ego.” Narcissism is known for being incredibly difficult or impossible to recover from.

Sycophancy (Group, Unintentional)

Knowledge of convenient falsehoods is often kept alive with large amounts of sycophancy.

  • Executives and politicians who surround themselves with yes-men.
  • People with inflated egos, who marry significant others who go along with it, and sometimes force their children to go along with them. This can lead to a deep web of falsehoods.
  • AI models are clearly sycophantic.
  • Journalists will focus on narratives where their readers are positive and morally standing, and people who their readers don’t like are bad. Most people live in ecosystems that are optimized (largely unknowingly, a la market pressure) to not challenge them.

Sycophancy is often fairly unintentional.

Epistemic Capture / Marketing / Propaganda (Group, Intentional)

Epistemic capture refers to the use of force by those in power to spread beliefs that are convenient to them. This is often done with intention (spreading this false knowledge), though in different situations those in power might themselves believe this false knowledge or not.

  • Rulers who spread (frequently effective) propaganda
  • Business marketing schemes that wind up convincing people of broad things

Arguably, these are broadly the same techniques as are used for spreading true information. Many of the best marketing techniques are equally effective in spreading both true and false information.

Attacks on Truth-Promoting Parties (Group, Intentional)

Agents trying to proliferate untruths will often come into conflict with truth-promoting parties. For example:

  • Governments frequently get frustrated by journalists
  • Conspiracy theorists have to argue against academics
  • Populists commonly attack “the elite”
  • Intellectuals with unique and strong positions often form attacks against most other intellectuals
  • Street gangs will promote distrust of the police/authorities
  • Scientology has a massive beef against psychologists. One reason was that they argued that their techniques solved many issues of psychology, and psychologists disagreed. So it made sense for them to discredit psychologists, for their followers
  • Pundits who disagree with prediction markets, but get asked about it, will often come up with various (typically false) attacks on these prediction markets.

These attacks can range from light verbal arguments to extreme violence, depending on the situation.

Obviously, this also means that it’s a red flag when someone is attacking truth-promoting parties.

Epistemics of Conveniences and Morality

Section titled “Epistemics of Conveniences and Morality”

One critical aspect of researching conveniences and information is that it requires certain moral stances to be taken.

A lot of epistemics can arguably be done without getting into morality. For example, there are a lot of tools that will help a mathematician do useful work faster, or a business find insights more quickly.

Once inconveniences get involved, that changes. People will get hurt. It might well be for the better, but it does involve a degree of uncomfort or pain.

Many technical researchers enjoy staying away from moral issues. But on the flip side, this neglectedness could mean that some of this area is more cost-effective, for those willing to engage with these topics.A large silver lining is that “discussing the theory and mathematics of fighting convenient falsehoods” is typically much less controversial than “directly fighting convenient falsehoods.” Being meta has its benefits.

Epistemics for Individuals

Section titled “Epistemics for Individuals”
  • Common failure modes
    • Incentives & Costs
      • Motivation (A person is externally motivated to be incorrect about an issue)
        • Can we develop methods of measuring the benefits that individuals get from believing things, outside of their truth status?
        • How well do people’s choices about what to believe reflect their basic incentives? For example, if they can choose to be accurate or inaccurate on an issue, will they make those decisions solely with their personal welfare in mind?
      • Ego (A person is highly and systematically overconfident because of a large ego)
        • Is it possible to dampen outsized egos?
        • See narcissism.
      • When is learning “painful”?
        • For example, a person finds out that they were a bad parent, or that their mistake has led to devastating consequences.
        • How can we understand tradeoffs between accuracy and pain avoidance?
        • What tradeoffs happen in practice? What tradeoffs should happen? (Descriptive vs. normative)
        • Can we quantify the human labor required to update a person’s opinion to a certain degree, especially when doing so comes with emotional hurting?
      • Trauma & Signaling
        • How much are people’s beliefs reflective of personal traumas they have experienced, or that were passed down.
        • Looking at beliefs as a way of signaling miscellaneous information. I.E. “I believe in Hitler, because I want to signal that I’m deeply angry.”
    • Failure scenarios
      • Logical Fallacies
      • Cognitive Biases
      • Failures in the use of language (i.e. Pragmatics, late Wittgenstein)
  • Basic Tools
    • There are many tools in one’s toolkit for understanding the world. Strong reasoners should have a skilled understanding of each. Each tool comes with its own pros and cons, these should all be understood.
    • Tools:
      • First-principles Thinking
      • Basic Qualitative Methods
        • Empathy
      • Basic Quantitative Methods
      • Evidence of Historic Use (“The fact that a practice has endured provides some evidence for its effectiveness”)
        • Burkean Conservatism
        • Chesterton’s Fence
      • Evidence of Popular Use
        • What are top companies and startups using?
      • Scientific Analyses
      • Expert Opinion
  • Power Tools
    • Advanced quantitative methods
    • Fermi Analysis
    • Advanced Qualitative Methods
    • Advanced diagrams of decision-makers, reasoning, etc
    • Discernment
    • Using AI, now in in the future
  • Balances (Things you want to do at the right amount)
    • Agreeableness. You want to be disagreeable enough to be interesting, but not overconfident. This is tough to get right. You want to be a combination of being [right when others are wrong] and not [wrong when others are right].
    • Candidness. You want to convey information, but at the same time, it’s easy to be too candid in certain situations.

Epistemics for Collectives

  • Effective communication
    • How do different techniques fall in terms of being effective, and in terms of being truth-seeking?
      • Start with the “Basic Tools” listed above.
      • Add in marketing/messaging techniques. Such as:
        • Emotional appeals
        • Memes
        • Sarcasm
        • Clever headlines
        • (This is tricky to neatly categorize)
  • Community culture
    • How candid is a certain culture? How does it rank on Power Distance?
      • Cultures low on candidness will have higher communication costs for uncomfortable topics.
      • Is this changeable? Yes, to some degree.
      • Bonus topic: How to make uncomfortable information less scary.
  • Epistemic Capture
    • How do those with power control others beliefs?
  • Community sophistication
    • Do community members understand enough about mathematics/epistemics/philosophy/etc to make use of advanced reasoning techniques?
  • Playbook for “Handling difficult conversations”
  • Incentive systems

Evaluations (particularly, Systematized Evaluations)

(There’s a whole lot to say here, might add later)

Epistemic Capture

Epistemic Cooperation

Epistemic Dominance

Epistemic Risks

Epistemic Dynamics

  • Epistemic Lock-in, vs. change.
  • When do communities lock-in false beliefs? When do they change? How can we get them to change?