Hybrid General Intelligences

Note

Ozzie Gooen · November 2018 · Completeness: ★★☆ (developing)

Unpublished QURI draft, posted as-is — rough and unfinished; collaborator comments have been removed.

Ozzie Gooen’s take

In 2018, LLMs were promising but still very basic. Alongside was the superforecasting world, which showed a lot of promise but was expensive. Some of this was a picture of how to combine the two, for general-purpose abilities. At the time, we weren’t sure whether it was an infohazard, or who would find it interesting, so publishing it didn’t seem high-priority.

Opus 4.8’s summary

A November 2018 draft introducing the concept of “Hybrid General Intelligences” (HGIs): organizations made up of human talent, methods, and narrow AIs, framed as collective agents analogous to AGIs and potentially capable of recursive self-improvement. The draft proposes a unit of strategic ability called the “Neumann” (calibrated to John von Neumann), gives speculative Neumann estimates for organizations like Facebook and the US Defence Department, and introduces an “amplification” factor distinguishing augmented from unaugmented capability. It distinguishes method-focused HGIs (MHGIs) from AI-focused HGIs (AHGIs), arguing the former could be safer, slower-takeoff, and might encourage consequentialist value-convergence, and discusses instrumental convergence and the possibility of an HGI singleton. The draft relies on several figures (referenced but image-based), includes a multi-phase narrative “story” of a hypothetical $100M MHGI build-out, an embedded self-critical “skeptical take,” and ends with a list of open questions soliciting feedback. It is explicitly speculative and exploratory throughout.

Hybrid General Intelligences (HGIs) are agent-acting systems made up of human talent, methods, and narrow AIs, understood with a focus on their collective intellectual capabilities. While it should be obvious that their intelligence is “general”, as that is true for human intelligence, the phrase helps understand these systems in terms similar to Artificial General Intelligences (AGIs). I believe that it’s possible (>5% probability) that HGIs have the potential to recursively self-improve at significant rates (enough that would seriously concern us). While they may not be as impressive as AGIs, they may be more important to consider on a short time-scale.

Definitions & Capabilities

Any organization now would qualify as at least a very basic HGI. Generally, the more easily modeled the organization is as a strategic agent, the better the frame of HGI. This is important insofar that it’s useful to think about HGIs in conversations like, “what are the main agents we should be concerned with, and what are their goals?” If there’s a powerful organization, but it’s much better modeled as a few competing subagents, then the primary reference perhaps should instead be those subagents.

To help put some numbers on intellectual abilities, we may think in units of “Neumanns” or “VNs.” Here, 1 Neumann-year is equivalent to the strategic abilities of John von Neumann for one year. John von Neumann could be a good candidate for this, not just because of his general intellectual ability at topics around mathematical strategy, but for the frequency and magnitude of his work. While he did not do much direct strategy work, I think it’s fair to reason that he could have been at least decent at it if he tried.

Ability to improve strategy is more important than direct strategic ability in the limit of people; if you put 30 Steve Jobs equivalents together, they may be great individually, but may do a poor job scaling their abilities 30 times. Some evidence for this lies in the fact that few organizations have co-CEOs, and no prominent ones have more than 2 people in that position. Even having a COO is considered a risk because of egos and incentives.

This is obviously incredibly speculative, but let’s imagine a table of the strategic capabilities of the following organizations.

Facebook	0.1 to 3	Mark Zuckerberg and Sheryl Sandberg seem highly intelligent. I would imagine they have 3-20 very close strategical confidants.
US Defence Department	1 to 14	The US Defence Department is perhaps the largest unified-ish department in the US. The intelligence may be quite distributed on many different operational problems.
Chinese Government	2 to 25	The Chinese Communist Party is very large, technocratic, and homogeneous.
Average VC-backed startup	0.05 to 0.2	In many of these situations, the founders do most of the strategic work (in their spare time), perhaps with one or two business advisors.
Hedge fund, managing $1B	0.1 to 1.5	Hedge funds typically manage $50-300M per employee. This work seems quite scalable, as long as you have the necessary vision / interesting trading strategies. I think the average hedge fund employee may be something like 0.02 to 0.2 Neumanns.

If such a table is reasonable, here’s one approach to classify HGIs based on their Neumans.

HGI Ability Level	Neumann Count (Augmented)
Prosaic	1-20
Weak	20-100
Strong	100+

The next factor is that of “amplification.” We could imagine that strategic talent combined with very basic levels of methods and tooling would act as a neutral point, with an “amplification” factor of 1. At this point it’s important to distinguish different types of Neumanns. We could imagine “Unaugmented Neumanns” to mean the neumanns an organization would have if it were amplified with a factor of 1x, or pre-amplification. “Augmented Neumanns” means the equivalent Neumman count post-amplification.

This distinction is important for understanding changes in amplification; even with a constant set of inputs (Unaugmented Neumanns), an HGI could output increasing total neumanns.

Understanding HGIs

HGIs are made up of three basic elements: talent, methods, and narrow AIs.

Methods:

The specific methods and business practices used by organizations seem very important. We consider all technology that doesn’t primarily AI-based to be under “methods”. Some examples of techniques that could be useful in a considerable HGI include:

• Forecasting on a sophisticated software platform
• Using powerful knowledge management tools for shared information
  • Creating and collaborating on useful ontologies
  • Managing lists of expected values and similar for important factors
• Understanding & using Bayesian analysis and probabilistic graphical models
• Correct epistemologies
• An effective culture
  • Epistemic humility
  • An intense work ethic
  • Honesty
  • A focus on quality

Talent:

Talent consists of the humans involved in an intelligence effort. Some kinds of talent I imagine would be valuable include:

• Forecasters with different areas of expertise
• Operations support to organize knowledge work
• Innovators of different types to come up with new ideas
• Researchers and research assistants
• Engineers to build relevant software
  • Note that internal software is far cheaper to make than external-facing software, so it’s sometimes possible to make surprisingly powerful tools for relatively small efforts.
• Engineers to make AI systems

There is a bit of a fine line between “talent” and “methods”, for instance, would group training count as “improving talent” or “implementing methods”? I don’t think the specifics here matter too much for the general argument.

Narrow AIs:

This covers most use of AIs to augment individual or group decision processes. I’ve separated this from the methods because I suspect it will be a crucial factor for the next 30 years of HGI progress. This is the one area we should expect strong exogenous effects.

The work that Ought is doing seems like one useful way that somewhat narrow AIs could help with general reasoning.

MHGIs (Method-Focussed HGIs)

It seems like there’s more progress to be made with methods and narrow AIs than with talent. I could imagine alternative HGIs focussed either on methods or narrow AIs, so wanted to give them some attention. This means that these HGIs could have significantly more advances in either methods or narrow AIs. For the sake of shortness, I’m going to call Method-focussed HGIs, “MHGIs,” and AI-focussed HGIs, “AHGIs.”

The below diagram represents the abilities of a MHGI; I predict that in order to be powerful it would have to have much better methods than we currently have, and somewhat more advanced narrow AI’s than we currently have. The blue triangle represents the capabilities of a powerful example MHGI, with distance towards the elements proportional in size to ability in that area.

I think that method-focused HGIs could have a lot of potential and may be relatively safe. Strong methods could lead to introspectibility, more similar to principled Bayesian approaches to reasoning than more black-box neural net approaches. However, it may also be less likely.

There’s a story about the hypothetical development of an MHGI on the bottom of post that you may want to skip to if that seems most interesting.

Can methods combined with weak narrow AIs self-improve?

My current thinking is that there are some relatively straightforward (though expensive) ways for a research team to become quite a bit more intelligent & capable. I don’t see any obvious bounds outside of cost and willingness. I would imagine that many of these techniques will be useful for improving the system itself, leading to feedback loops. I would give a >20% chance that most of the value that would come from a well designed $2-million dollar MHGI effort would come from recursive improvement. This is covered a bit in the “story” section.

Little Interest

I don’t know of any external actors who seem interested in and capable of making a method-focused HGI anytime soon. A MHGI may require creators to both be knowledgeable and excited about fundamental decision-making tools such as Bayesian epistemology, probabilistic programming, forecasting, expected value maximization, and perhaps even philosophies like consequentialism. In fact, some of these things have very long histories of seeming to be dramatically overlooked, so perhaps even if groups were to publicly proclaim MHGIs as a possibly powerful tool, very few others would listen. Outside of the Effective Altruism / rationalism communities, I know of no obvious groups who care a lot about more than two of the above-mentioned tools.

Some of the work around DAOs may be related. These are autonomous organizations theoretically managed by clever voting methods and prediction markets. However, I have not seen this work focus on recursive method improvements, and generally do have high expectations of the feasibility of basically all DAO-related projects.

Value-convergence towards consequentialism

If it were true that the creation of a powerful MHGI would require its creators to use principled approaches to decision-making, then it may feature value-convergence, rather than orthogonality. I would expect there to be a strong bias towards consequentialist thinking. It’s hard to me to imagine what a strong system that optimizes without a strong consequentialist utility function would look like, especially for the goals that many actors seem to have. Much of utilitarianism/consequentialism is simply the application of mathematical optimization to high levels of decision making instead of lower levels, so a human-led system of principled decision making may well resemble or converge on somewhat consequentialist beliefs.

A slow takeoff

A MHGI seems like it would have a relatively slow takeoff (~3-20 years). Humans would be doing much of the work and may do many of the most innovative steps.

Alignment problems, but relatively safe ones

A powerful MHGI may be given some sort of utility function, and thus may have similar challenges as have been discussed for AGI with utility functions. However, it would be running on a longer time scale, and humans would be in-the-loop in many situations. I imagine there will be problems with incentives, but think that these could be expected to generally be fixed.

The most important question to ask an MHGI at every stage of its development may be something like, “would it be higher-EV for us to focus next steps on safety or on capacity?” A gradual and intentional ramp up could occur until it was no longer deemed expected to be safe.

Hypothetically this could be a good testing environment for understanding similar issues in AGIs.

Possibly high visibility

It may be possible to bring in talent from a wide and diverse set of experts, perhaps for forecasting or other interventions. Methods may also require a large number of people, and correspondingly a large amount of money. Therefore it’s likely that the program would eventually become quite apparent to outsiders. It may be very difficult to do in secret.

A substantial moat

A MHGI may require a lot of resources and time. Because of the current lack of interest, it seems possible that if there is any major success, it would be difficult for others to quickly catch up. This is especially the case if the methods could be kept secret, though this may be challenging. This monopoly gravitation may mean that groups that develop MHGIs would feel relatively secure, and wouldn’t have to take as many dramatic measures to prevent other MHGIs as may otherwise occur.

AHGIs (AI-Focussed HGIs)

Rather than being method-focused, HGIs may be able to become very powerful mainly with advanced narrow AIs. Unlike MHGIs, I have less certainty about how these could be created or what it would look like. There could be many ways to augment human reasoning ability using narrow AIs, and is not yet obvious which specific approaches will be the most useful.

Riding the wave of AI progress

I imagine that AHGIs would focus on developments close to that of public AI progress. How much is somewhat a matter of available budget and innovation. This could make things both “democratic” and unstable. It may be hard to tell what types of AHGIs are possible until new public innovations are announced. Once those do happen, it may not be obvious how much of an edge any single player could have and maintain. If one actor is able to take some recent breakthroughs, innovate on them, and create an AHGI, it may only be a short while until either independent competitors or the public research community catches up.

This lack of a “moat” may encourage AHGIs to take more risky and aggressive strategies, to better ensure their possible rivals will not stop them. On the flip side, they may be able to develop faster (faster takeoffs) than MHGIs, being similar to AGIs. If this is true, while competition could come soon, their additional intelligence may make it relatively easy to prevent competition.

AHGI vs. Intelligence Amplification

The concept of AHGI is really meant to emphasize a system that uses multiple humans, some powerful methods, and narrow AIs, to have powerful strategic capabilities. There are some scenarios where AHGIs could look much like this, but others where they wouldn’t. One could imagine a case where the important strategic actors would look much more like a few very smart individuals with a few really powerful in-house AIs. For these, a better concept may be that of “intelligence amplification”, which is more individual-focussed than group-focussed.

For discussion, we can regard HGIs as requiring at least 2+ people working closely together, in an organization that would be usefully thought of as an agent.

HGIs as Agents

I think one of the key aspects of categorizing an HGI as such is to identify it as having agency. When I imagine a human with “intelligence amplification”, I imagine one person agent with additional AI. But if I imagine “a dedicated team following a written down decision optimization procedure”, then any individual in that team seems less important. The system itself would gain a sort of agency. Of course, parameters could be controlled or decided by a particular human, but on the whole, most decision-making would come from a collective.

One could also think of what I call an HGI as a “recursively-self-improving strategic organization assisted by AI”. In this case, one should still focus on the organization more than any individual within it.

If we can consider HGIs as agents, then we may be able to refer to them as “having goals.” So we could say things like, “The HGI will desire X, and seek to use methods A, B, and C, to get X.”

Instrumental Convergence

If we think any kind of HGI could gain more abilities than existing human groups, I think we should consider that their abilities may lead to them acquiring more abilities. If this seems high-EV, I would expect it to be the default action for these groups.

There’s already a lot of “instrumental convergence” between many human actors. Many humans seek status, money, and power, even if they have different goals (Effective Altruists included.) I think this makes sense pragmatically; these things do seem very useful for achieving many sorts of different goals.

Likewise, I would assume that most HGIs would desire goal-content integrity, resource acquisition, cognitive enhancement, technological perfection, and self-preservation (some of the goals stated for AGIs).

HGI Singletons

It seems possible that a powerful HGI may be able to create a singleton in ways similar to an AGI. I’ve written about this a bit before in my similarly-private post on “Pre-AGI singletons.” One potentially nice property of an HGI creating a singleton is that it could provide a safe period to deliberate next steps. A powerful HGI may be safer to create than a powerful AGI. I would imagine HGIs to generally desire to create AGIs (instrumental convergence), but if one were to have a singleton first it could take time and be careful when attempting to create one.

A story of an (M)HGI’s development

A multi-millionaire entrepreneur commits $100 million to build a powerful MHGI. It begins in the highly experimental state, where it’s not at all obvious what it will be useful for in the initial stages. This entrepreneur leads the venture, hiring a team of:

• 10 engineers for web tooling & data collection
• 3 operations staff
• 3 machine learning engineers
• 4 strategy analysts from RAND or similar
• 10 junior researchers (relatively inexpensive but intelligent humanities or technical graduates)
• 1-2 library science specialists or similar for ontology creation
• 1-4 experienced superforecasters or equivalents
• 4 managers to run things These people would be chosen in part for their willingness to work in a pragmatic and extremely honest way, similar to Bridgewater. I estimate a team of this size would cost between $5-15 million per year.

Phase 1: Initial Infrastructure & Basic Testing

The engineers begin making shared numeric multidimensional knowledge bases for structured data. Tools that can organize structured data, like “what has the GDP been like for every country in every year for the last 30 years?” This would be similar to Google’s Knowledge Vault and Wolfram Research’s backend data infrastructure for Wolfram Mathematica, though it would allow for probability distributions instead of just raw numbers, and be focused more on user input (similar to WikiData). The engineers would then work with the researchers to fill this with important historic data.

Rather than just being about historic data, this tooling would later allow for predictions on future data. The machine learning engineers would use various tools (DataRobot and similar comes to mind) to do lots of trend analysis to predict future probability distributions from this historic data. These would be somewhat wide.

The experience forecasters would help train the researchers to begin making forecasts on the future probability distributions. Those from ML systems would produce distributions with low resolution (though high calibration). There are many areas where these could be narrowed with human judgment. These forecasters will try to narrow many of these distributions.

Initial testing would be done on generic global events that would happen in the near term. Many kinds of things would be tested, for the purposes of better understand what the system could do ok with.

Phase 2: Meta-use

After 4-10 months of setup, researchers will start using the system on itself. They would set a goal, such as “legally maximize profits from this system over 10 years.” People will start work on investigating strategies and using the platform to estimate the success of those strategies. Meta-questions start being formed. Some examples of meta-questions include:

• What is the EV of spending 5 quality-adjusted research hours estimating this variable? (EV based on an evaluation by person X after 3 months)
• What is the EV of adding feature Y to the application?
• What is the EV of researching topic Z to better understand how to estimate EV?
• What are the EVs of every possible action that could be taken to improve this organization?
  • Examples include employee habits, culture, practices, hiring, etc.

All the employees would also be rated on several metrics based on their expected and existing EVs to the system. This would be broken down into specific things they do, making it somewhat clear how they can improve.

Eventually, the use of EVs will be replaced with other ways of doing structured reasoning that are expected to be more pragmatic.

If human forecasters can be effectively combined with many small AI systems and used with a secure technical architecture to structure the knowledge, they may be able to be very effective at in total estimating many parameters with degrees of accuracy hard to beat by any single human.

Inefficiencies and poor clarity of incentives and value creation are traditionally major hurdles to productivity. One goal with this is to make sure that employee work incentives are aligned as closely as possible to expected value maximization. The usefulness of their work will be estimated as precisely as feasible, and those estimates will be made known to various people. This setup will probably make some people uncomfortable; therefore the hiring process would be selecting for people who would accept it, and salaries may increase to help compensate for that.

Phase 3: Resource Generation

After 2-25 months of phase 2, the system would determine that some resources would be optimally spent on efforts to gain money. This is done by targeting the financial industry. The comparative advantage of such a system in the financial industry would initially be medium to long-term trades, where historic data and expectations about the global future matter more. A hedge fund is set up to do trades. After 6-24 months, the group makes some significant profits.

Phase 4: Global Influence

The really important aspect of this story is how such a system would rise to power, rather than what it would do with it. There are many ways it could go once it has power, that is for another document.

The system would effectively be following the stabilize-reflect-execute strategy, so the execution should be understood to be difficult to predict.

How should knowledge of HGIs change our decisions?

• We may want to attempt to make a strong HGI for EA purposes. Even if we fail to make something that’s very self-improving, any work in this area could still be useful for our strategic efforts. We would also like to be in a place whereby if strong narrow AIs were to make HGIs far more promising, we could quickly take advantage of that.
• We may want to watch out for others making HGIs and consider risks they would create.
• If we think that HGIs are more much more likely to be important than AGIs, then perhaps we may not have to worry about AGI-specific issues as much.

The skeptical take

There’s definitely an outside-view voice in my head that thinks this is all extremely unlikely. It’s basically ranting something like,

“

Here’s a techy who’s really into rationality, Bayesian reasoning, and forecasting. Of course he’s going to say that those specific things will happen to be the most important things in the world, even though basically every single organization outside of the EA/rationalist community is not at all interested. Probabilistic programming is mostly hype, forecasting has almost never been successfully used within organizations, and expected values sound nice but aren’t actually used by major strategists. Recursive improvement would require not only that these things rationalists love could be done well together (which hasn’t been shown yet), but that they would magically create a hyperintelligence in a way barely even specified.

The whole thing is named “HGI”, like he’s trying to get some quick fame, riding on the popular wave of Superintelligence, even though the proposed entity is really just ‘A smart organization.’

”

I think such views should be taken into consideration. The outside view of much of this work is probably quite pessimistic.

That said, it could of course be the case that even if we are pessimistic, the expected value could still be positive of working on it further.

Primary Open Questions (please leave comments in response)

Much of this document is highly speculative. This work is mainly around trying to find useful frames and hopefully getting feedback.

• Is the above framing (HGIs, MHGIs, AHGIs) a useful one? If it is, could it be improved?
• Should future versions work be public? Are there any specific parts that should not be public? If it should not be public, should it be shared with anyone in particular?
• Should we expect MHGIs to be relatively safe?
• Should we expect MHGIs to both require and encourage consequentialist-leaning beliefs?
• How feasible is recursive self-improvement in MHGI and AHGI systems?
• What does the space of possible HGIs look like?
• What should the expected timelines be for HGIs?
• What kinds of actors do we think may attempt to make HGIs in the future, if any? Will they generally be aligned with EA principles?
• Should EAs try to build MHGIs for our own use? If so, what should the strategies be?

Primitive vs. Weak vs. Strong HGIs

Interesting Links:

https://www.mergersandinquisitions.com/start-hedge-fund-hiring-team-organization/

https://www.winton.com/research/how-big-is-the-hedge-fund-industry

https://alphacution.com/top-hedge-funds-aum-per-employee-trading-strategy/