Accuracy Agreements

Note

Ozzie Gooen · April 2023 · Completeness: ★☆☆ (sketch)

Unpublished QURI draft, posted as-is — rough and unfinished; collaborator comments have been removed.

Ozzie Gooen’s take

I was very interested in how philanthropists could spend millions of dollars on good estimation. The going assumption was that the main way was to pay Metaculus or Good Judgment a lot of money; I thought this was a clever alternative that might be more scalable. I still feel pretty good about it, though it would take some work to implement.

Opus 4.8’s summary

A brief, explicitly preliminary proposal for a financial instrument called an “Accuracy Score Agreement.” The mechanism: a client publishes an agreement with a “Type Specification” and a scoring rule; bidders submit forecasts (e.g., probability distributions over many continuous variables) and bid prices; the highest bidder buys the contract and is paid at resolution according to a scoring rule (such as a log-score-based linear payout). The stated motivation is that prediction markets handle binary questions well but struggle with distributions and functions. A worked example with 50 variables resolved in a year is given, followed by a list of possible extensions (a secondary trading market, fractional shares, daily markets, updatable forecasts, third-party resolution, rewards for explanations). One note raises that requiring money down might count as gambling and proposes splitting payment into a fixed fee plus an accuracy agreement. The document is a short sketch of mechanism design with no analysis of incentives, pricing, or failure modes.

Title: Accuracy Agreements (Draft)

Note: This is very early and preliminary.

The problem: While prediction markets excel at determining outcomes for binary questions, they face difficulties when it comes to probability distributions and more complex scenarios, such as those involving functions.

Proposed solution: A client publishes an “Accuracy Score Agreement” with an associated “Type Specification.” The agreement’s terms are as follows:

• A purchaser of the agreement is required to provide a forecast that complies with the established Type Specification.
• Upon reaching a prearranged future date, the Agreement will undergo Resolution, which entails:
  • The client will determine the answers to the forecasting questions.
  • The client will evaluate and assign a score to the submitted forecast.
  • The contract holder at the time of resolution will receive payment based on a linear function of a scoring rule. An example of this could be: “The average log score of the forecast minus the prior, multiplied by $10,000.”
• Holding an “Accuracy Agreement” with a specific forecast has a calculable value. If the holder trusts the forecast, the value would be equal to the expected resolution payment. (Easily calculated using expected loss).
• When a question is proposed, potential buyers bid for the agreement. Each bidder submits their own forecast (which should be requested in advance via a private bidding system). If a bidder believes their forecast has an EV of “$25,000,” they might bid “$23,000” for the agreement.
• The highest bidder is chosen and they buy the contract.

An example:

• A client publishes:
  • A list of 50 continuous variables, to be resolved in 1 year.
  • A set of (weak) priors for those variables.
  • A simple scoring function, e.g., ($1K * total logScore sum)
  • A future date for bidding to commence (1 month out).
• Bidders submit:
  • A list of forecasts for the 50 variables, each as a probability distribution.
  • A maximum price for purchasing the agreement with their forecast.
• The top bidder is chosen and sold the contract.
• In 1 year, the contract resolution occurs.
  • All (or some) of the variables are resolved.
  • The contracts’ forecasts get scored.
  • The agreement owner is paid according to the predetermined scoring rule

Potential Changes / Improvements

• An active marketplace. Purchasers can trade these contracts with each other over time, after the initial purchase. This can work like a stock market, with many submitting prices at which they will buy or sell.
• Introduce a market with multiple shares, allowing buyers and sellers to purchase only a portion of the market.
• Implement “daily markets” where agreements pertain only to a specific date’s forecast value. The same question would be asked daily, with different bidders potentially purchasing the corresponding agreement on different days.
• Buyers can update their forecasts over time. The contracts cover the average accuracy over that time. This incentivizes buyers to continue to improve their contracts.
• Forecasting questions can be resolved and scored by agreed-upon third parties.
• Maybe there’s an extra reward for a forecast that comes with a good explanation or similar. This could be part of the scoring function.
• In some cases, it might make sense for bidders not to have to put any money down. For instance, this might be considered gambling. One alternative, if the bidders are trusted, is that they get payments in two parts: one fixed fee, and one accuracy agreement. For example, Open Philanthropy puts out a call for a forecasting consultancy to forecast 1000 variables - but most of the payment is in the form of an accuracy agreement.