Pre-AGI Singletons
If all violent conflicts were decided by games of Go, then Google could conquer the world.^1 While we don’t live in that world, is not obvious how much harder it would for Google or another single organization to become a world dominant force.
Singletons were discussed in Superintelligence^2 as an end-all state from which the resulting leaders (human and/or machine) would control global future activity. The main argument there is that an AGI (artificial general intelligence) would allow it or its creators the ability to manipulate or overpower other human actors. In the case of a multipolar scenario (multiple actors with similar AGIs), it seems likely that eventually one agent will eventually outmaneuver the others. Once one superintelligence begins to dominate, it seems incredibly difficult for anyone else to stop it.
The idea of AGI seems like a solid proof of a more general singleton argument. If a sufficiently advanced AGI were developed (compared to other actors and to existing technology), a singleton seems quite straightforward. But of course, this doesn’t mean that a singleton won’t happen before an AGI is created. If narrow capabilities are powerful enough, it may be quite possible to create a singleton before AGI creation. We call these “Pre-AGI Singletons.”
So the resulting question is when can we expect a singleton to occur. And to answer this we should probably ask what technology may be necessary to make one.
A singleton-desiring agent doesn’t need to control the entire world right away, they just would have to get started on something with a very likely progression to that outcome. The real goal in an uncertain world isn’t to have immediate power, but instead to have power in high probability in the future. If an agent is on a long-term but inevitable road to this power, then the future of humanity has mostly been decided.
Because of this, we may want to categorize singletons with probabilities. A 20% singleton agent would describe an agent that is estimated to produce a singleton with a 20% probability^3. A singleton-seeking agent would estimate its’ own probability, and do what it thinks to maximize that probability in the future. Given the significant perceived rewards of a singleton, this may be it’s only pre-singleton goal.
The main strategies for increasing an agents’ singleton chances include:
- Increasing their own capabilities. (Capability amplification)
- Decreasing the probabilities of other agents producing singletons first. (Threat reduction)
- Preventing other organizations from stopping actions 1 and 2. (Interference reduction)
If these three goals can be achieved, then an agent’s singleton probability will increase in time. If they can be ensured, than the agent’s singleton probability would be very high.
The Hidden Singleton Motive
Section titled “The Hidden Singleton Motive”The Messianic Secret refers to a motif where Jesus commanded his followers to hide the fact that he considered himself the Messiah. Revealing oneself to think themselves the messiah was a pretty radical act against other authorities, so it would make sense to keep that secret until they could be pretty sure they would be successful.
Likewise, an singleton-intent agent would probably want to hide their intent until their chances were extremely good. Their intent would be a threat to other singleton-intent agents, who would be incentivized to stop them.
A singleton-intent agent not wishing to reveal itself as such would have to limit itself to reveal-limiting actions. Actions that increase its capabilities would have to either justified for other purposes or hidden. It would probably be difficult to completely hide the work needed to become at least a 1%-singleton agent, so it seems more likely that some kind of false front would be needed. The benefits of increasing general capabilities may be generic enough that many fronts would be believable, and even pragmatic without the singleton motive.
Actions that would harm or prevent other actors may be harder to disguise. Anonymous hacking attacks seem preferable, although they may have to be extremely anonymous.
If a singleton-intent agent gets more desperate or certain, it could be expected to do more revealing actions. Their desperation could be caused by either a sense that either other organizations may stop them, or a sense that other organizations may be getting close to making a singleton themselves.
If one were to witness an agent do something that revealed it may be singleton-intent, then this could be evidence that either the agent is very confident in its own likelihood, or that it’s very worried that another singleton-intent agent is very likely to succeed. Either of these cases would be very concerning for an external observer.
It’s not clear what organizations, if any, are singleton-intent at this point. Some explicitly state that they are aiming to create AGIs, but it is not obvious what their strategy would be if they either do or get close enough to make a singleton. As singleton-allowing technologies become more obtainable, more organizations may acquire the goal.
It would seem like singleton-intent agents in many cases would attempt to identify and estimate the chances of other actors either being singleton-intent or eventually becoming singleton-intent, and what their probabilities of success are. Given that each organization would want to adjust it’s perception by others, but care a lot about it’s understanding of others, one may expect an arms race of surveillance, espionage, misinformation, and security.
Types of Singleton-Intent Actors
Section titled “Types of Singleton-Intent Actors”Global Alliances
If a singleton were inevitable, the least deceptive or antagonistic actor may be a global alliance. Carrick Flynn at the Future of Humanity Institute discussed the idea of an “IGO for Common Good,” a research organization that would act as a cross-collaboration between many nations.^4 This is one of the few actors that may not need to keep its intent a secret. There would have to be a very significant amount of global coordination in order for a global alliance to succeed, given that it would be competing against all other actors.
Governments
Perhaps the most likely actor type to create a singleton is a national government. Governments have very large pools of resources, intelligence on other actors, and expertise keeping projects secret. If a government desires to create a singleton, it would probably want to keep both its desires and its capabilities a secret.
It may be so secretive as to significantly sacrifice recruiting potential in order to appear to have fewer capabilities. To clarify, AI development highly depends on great talent, and one productive way to obtain great talent is with announcements of capability (for instance, writing papers on Alpha Go). However, if an organization is confident in its ability to get top talent without publicity, then it may be reasonable for it to strategically appear naive. This may be the case with some governments, who would have competitive advantages of hiding research and paying extra for it.
Corporations
The main advantages corporations may have over governments are leadership and talent. Many top tech companies are led by founding CEOs who have become very experienced at leading large technical projects, and who have been able to create decision structures with very little bureaucracy. On the talent side, several top tech companies may have comparable or more AI and engineering talent than most governments.
If a corporation were able to establish a singleton, it seems unlikely it would continue to be beholden to shareholders. As shareholder control is only enforced by governments, and a singleton would eventually control of those governments, then the corporation would not have to be accountable to shareholders post-singleton. Perhaps a “corporate singleton” could better be looked at as a sort of coup of executives and select parties within a corporation, rather than considering it to represent corporations as we formally know them. A corporate singleton would be an adversarial goal to most shareholders, so would likely be done in secrecy from them.
Private Organizations
If a small group had enough money, it may be able to hire many researchers with a false front purpose. Right now there are over 2,000 billionaires with total assets of over $7 trillion.1
Setting up a sophisticated AI lab may be achievable to the wealthy. DeepMind, one of the primary AI research facilities, was purchased for £400 million, then has had losses of £54 million in 2015 and £164 million in 2016. Their head count is reported to be only 400; if each cost $500,000 including expenses, that would total $200 million per year. In comparison, Jeff Bezos has been personally spending $1 billion annually on funding Blue Origin2.
As an aside, Blue Origin was an interesting example of a secretive technology product. It was kept secret for three years and only discovered because of its’ purchase of a large amount of land in Texas3. An AI lab may not need many resources outside of humans and compute, so may be able to be much more secretive. The creator of an AI lab may also have much greater incentive to keep it private than Jeff Bezos did with Blue Origin. To be specific though, it was secret from the public; it is possible government intelligence was familiar with it.
Rogue Agents
A “rogue agent” is one that doesn’t fit into one of the other categories. These are thought to be smaller or more private organizations funded by individuals or small groups. Rogue agents would be expected to generally have less money than governments or corporations, but there may be many more of them, and they would have the advantage of significantly more secrecy.
A different possibility would be that a very small team without significant funding could simply develop the necessary technology themselves. This team would have to be either incredibly skilled or lucky, or both.
Tactics
Section titled “Tactics”A singleton-desiring agent would generally seek a strategy that promises accelerating returns, in order to increase its chances of success. The powerful thing about AI development is that it could be useful to improve almost all tactics for success, many of which could then help efforts to improve AI. The most obvious feedback loop is that between AI and capital. If narrow AIs could be used to gain capital, then that capital could be used to improve and expand on those AIs. Narrow AIs and capital could also be used on threat reduction and interference reduction. This process may be much slower than AI “FOOM” scenarios, but it may be similarly difficult to stop after it reaches a threshold.
Capability amplification
The most obvious way to dramatically improve one’s capabilities is by advancing useful narrow AIs. Narrow AIs and infrastructure development that makes it easier to create better narrow AIs and infrastructure development would be broadly useful and may lead to accelerating returns. Otherwise, the goals would be to create narrow AIs that help implement the other tactics.
Income generation
Companies and small groups would likely be constrained by income. In comparison, global alliances and governments may already be funded enough for potential revenues from narrow AIs to not be relevant. For instance, the annual budget of Russia is around $200 billion. It could borrow money if needed.
AI in general is considered to be one of the top, if not the top, most exciting focuses for established and new tech companies. It’s expected to help make a lot of money. This could be from competing in the stock market, producing new media content, being sold as a service for other applications, augmenting robotics, the list is very long.
Perhaps the most difficult aspect of narrow AI income generation is that the largest opportunities will be competitive. A narrow AI working on the stock market would have to significantly outcompete all other AIs doing the same. It may come out that successes there aren’t defensible; perhaps, similar to types of algo-trading algorithms, degrees of intelligence may not matter much after a certain point. Of course, with even a few-month advantage, a large amount of money be be possible to obtain.
-------------------------- The following is more notes for future writing than actual writing ----------------
Threat reduction
Talent poaching
Manipulative coordination
Offensive cyberattacks
Public Revealing
Interference reduction
Plan Secrecy
Computer security
Offensive cyberattacks
Political manipulation -> Terrible attacks
Social manipulation
System breaches
Singleton classification
Section titled “Singleton classification”To help understand the possibilities of singletons, let’s consider a few terms. We can call an antagonistic effort to create a singleton a singleton attack. This is defined as an explicit attempt to prevent counter-attack vectors. Attack vector amplification itself is not an attack, but the effort to prepare one. So for instance, a chemical weapons group could spend years making advanced substances, but it would only be the execution of them that would classify.
Because any attack is uncertain, the specific level of certainty probably matters a lot before the attack happens. Discovered failed attacks would likely come with severe social costs, so trade-offs would have to be weighed before they are executed. Therefore, for attacks that haven’t happened yet, it can be useful to prepend their descriptions with the percentage chance that they are perceived to be successful. For instance, one may wait until they have a “20% singleton attack” before execution. If it becomes possible that other actors may have high-possibility attack vectors, the threshold may go down for other organizations.
An AGI may enable a 100% singleton attack, but if even 0.01% singleton attacks are possible before, they may be attempted.
Attack Vectors
Section titled “Attack Vectors”Ending
Singleton Attempt:
Singleton Attack Vectors:
X% Singleton Attack:
What are the likelihoods of singleton-
Pre-AGI attack vectors
- Of course this isn’t exactly the case. If wars were decided by games of Go, governments would probably have been much more cautious of Google’s developments, and may have tried to either stop them or out-develop them.
- https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0198739834/ref=sr_1_1?ie=UTF8&qid=1513039030&sr=8-1&keywords=superintelligence
- https://www.youtube.com/watch?v=HN7GfRPxUEg&list=PLwp9xeoX5p8M8GfsQqbkXwQpiiBtIRC4c&index=12
There could be many kinds of control vectors.
If machine learning comes out to be essential for a singleton, then this may entail something like:
- Amassing an accelerating amount of compute power and talent.
- Removing compute power and talent from competing organizations.
- Preventing military involvement.
If a group could manage to do those three things, it could be on a pretty good trajectory to becoming a singleton.
Comments from Nuño Sempere
Section titled “Comments from Nuño Sempere”Restored with permission (Nuño’s comments, with Ozzie’s replies).
On “I wrote this document a few months ago. I wrote it to explore ideas, but stopped working on it once I got to some specific details I thought might be…”:
Nuño Sempere: Consider that, if starting from this point you can get to an info-hazard, other people might be able to take the same steps.