Existential Risk and Growth

TL;DR

Almost every economic model of “growth vs. existential risk” quietly assumes that if we stop developing technology, we are perfectly safe — so the only thing growth does is add risk. This paper rips out that assumption. If nuclear weapons, engineered pathogens, or runaway climate feedbacks already pose a per-year chance of catastrophe, then stagnation is not safe: it just means you live under that hazard rate forever, and “forever” guarantees the catastrophe eventually happens. Once you take that seriously, acceleration through a dangerous era can lower the total lifetime probability of catastrophe, because (1) it pulls forward the arrival of safer technology, and (2) it makes society richer, and rich societies treat safety as a luxury good they suddenly want to buy (an “existential-risk Kuznets curve”). The risk-minimizing growth rate, the authors show formally, is typically positive and “may easily be high” (in one calibration, ~100% per year). Faster is only riskier in a narrow case: when the danger of running many experiments in parallel scales worse than linearly, or when policy can’t keep up with rapid change.

Problem & Motivation

The concrete pain: the existing literature on growth and existential risk (“x-risk”) is built on a load-bearing assumption that is almost certainly false. Every prior economic model assumes stagnation is perfectly safe. Under that assumption the math is trivial and depressing — any technological progress only adds danger, so if you care about the long-run survival of humanity, you should slow down or stop.

But look at the actual world. We already have nuclear arsenals, the biotech to engineer pathogens, and the industrial base to push the climate past tipping points. Freeze technology exactly where it is today and the hazard rate — the probability of an existential catastrophe per year — does not drop to zero. It stays positive. And a positive constant hazard rate, integrated over an infinite future, gives you a 100% chance of catastrophe eventually. The authors’ vivid version: a shock that reset the world to its 1925 technology state would not save us — it would condemn us to replay the entire nuclear age, the emissions-heavy industrialization, and the bio-hazards of the last century, over and over, until one of those replays kills us.

So the real question isn’t “growth vs. safety.” It’s: given that we’re already in a dangerous region of technology-space, is it safer to move through it fast or slow? Prior work assumed the answer was always “slow.” This paper shows it’s usually “fast.”

What’s New (Core Contribution)

The novelty is in dropping one assumption and rigorously tracing the consequences. Four genuine contributions:

• Stagnation is no longer assumed safe. Before: hazard rate = 0 when growth = 0. Now: the hazard rate is a function of the technology state you’re sitting in, so standing still in a dangerous state accrues risk indefinitely. This single change flips the headline result.
• “State risk” vs. “transition risk” cleanly separated. Before: the literature (notably Jones’s “Russian roulette” model) lumped danger into the act of developing tech. Now: the paper distinguishes danger from existing in a tech state (state risk, e.g. nukes sitting in silos) from danger created by the act of experimenting to advance (transition risk, e.g. the gain-of-function experiment itself). This distinction is what makes the answer “it depends” instead of “always slow.”
• A formula for the risk-minimizing growth rate. Before: no characterization of an optimal speed through danger. Now: a closed-form expression (Eq. 10) showing the safest growth rate balances marginal state risk against marginal transition risk — and proving it’s strictly positive whenever any state risk exists.
• The Existential-Risk Kuznets Curve. Before: policy/safety spending treated as exogenous or absent. Now: a planner optimally chooses how much consumption to sacrifice for safety, and because safety is a luxury good (you pay more for it as you get richer), faster growth makes society want to be safer sooner — which strengthens the “go fast” conclusion rather than weakening it.

How It Works (Technically)

This is a continuous-time theory paper, so the “mechanism” is a sequence of models, each adding one layer of realism. The trick to reading it is to keep one quantity in your head the whole time: cumulative risk, X — the total area under the hazard curve over all of future time. Survival probability is S∞ = e^(−X). Minimizing lifetime catastrophe probability is exactly the same thing as minimizing X. Everything else is bookkeeping about what makes X bigger or smaller.

The central reframing (the “change of variables” trick). The hazard rate δ_t is risk per unit time. But the authors integrate risk with respect to technology A instead of time t. Why? Because on any growing path, technology crosses each value exactly once, so you can ask: “how much risk do we accumulate while passing through technology level A?” That quantity is the risk density:

x(A, Ȧ) = δ(A) / Ȧ
         = (risk per unit time at state A) × (time spent at state A)

Ȧ (read “A-dot”) is the growth rate — how fast you move through technology-space. The killer insight is right here: risk density is inversely proportional to growth rate. Spend half as long in a dangerous state (grow twice as fast) and you eat half the risk from that state. This is why, with only state risk, faster is always safer (Proposition 1: acceleration always lowers X).

Layer 1 — State risk only. δ depends only on the tech state: δ_t = δ(A_t). Accelerating through any band of technology strictly lowers cumulative risk (unless that band was already perfectly safe). Stagnating at a dangerous state makes X infinite — guaranteed death. Conclusion: grow as fast as possible.

Layer 2 — Transition risk. Now danger comes from the act of developing tech, not just having it. The hazard rate gets a growth-rate term:

δ(A, Ȧ) = f(A) · Ȧ^γ ,   γ > 0

Here γ (gamma) is the crucial knob — the elasticity of hazard to growth speed. It answers: if you run experiments in parallel (faster) instead of in sequence, does that scale risk better or worse than linearly?

• γ < 1: parallel is safer than sequential → still go fast.
• γ = 1: speed doesn’t matter for cumulative risk at all. (This is Jones’s classic “Russian roulette” model — each technology is one trigger-pull regardless of timing.)
• γ > 1: running many experiments at once is disproportionately dangerous (society can survive disasters one at a time but not all at once) → now speed genuinely costs you.

Layer 3 — Both risks together (the money result). Combine them:

δ(A, Ȧ) = h(A) + f(A) · Ȧ^γ
          ╰──┬──╯   ╰────┬────╯
        state risk   transition risk

The risk density to minimize is h(A)/Ȧ + f(A)·Ȧ^(γ−1). The first term falls with speed (less time in the dangerous state); the second rises with speed (more dangerous parallel experimentation, when γ>1). Take the derivative, set to zero, and you get the risk-minimizing growth rate (Eq. 10):

Ȧ*(A) = ( 1/(γ−1) · h(A)/f(A) )^(1/γ)

In plain English: the safest speed is faster when state risk (h) is large relative to transition risk (f), and slower when transition risk dominates. And critically — this is positive whenever h(A) > 0. As long as there’s any danger from merely existing in your current tech state, full stagnation is never the safest option. The authors’ bicycle analogy nails it: you’re cycling next to a busy road with some chance of being hit at any speed including zero. Stopping doesn’t save you — it guarantees you eventually get hit. Unless your accident-rate more than doubles when your speed doubles (γ > 2 territory locally), the safest move is to pedal home as fast as you can.

Layer 4 — Endogenous policy (the Kuznets curve). Now a planner chooses, each year, a safety share B ∈ [0,1] — the fraction of potential consumption to sacrifice for safety (pandemic monitoring, bans on risky processes, etc.). Consumption is C = A(1−B). The planner maximizes discounted expected utility with η > 1 (diminishing marginal utility of consumption that’s steep enough that survival dominates in the long run). The result: because utility is concave, as society gets richer, the marginal value of more consumption falls but the value of staying alive to enjoy it stays high — so the optimal safety share rises with wealth. Safety is a luxury good. Faster growth → richer society sooner → more safety spending sooner → lower hazard. This is the “Environmental Kuznets Curve” logic (Stokey 1998) applied to extinction. It strengthens the go-fast conclusion. The only thing that can reintroduce a “go slow” force is policy friction: if regulators simply can’t respond effectively when technology is changing too fast, that acts like transition risk and can cap the optimal growth rate.

Architecture & data flow

flowchart TD
  A[Technology state A_t<br/>= potential consumption] --> H["Hazard rate δ(A, Ȧ, B)"]
  G["Growth rate Ȧ<br/>how fast we move"] --> H
  B[Safety share B<br/>planner's choice] --> H
  H --> X["Cumulative risk X = ∫δ dt<br/>area under hazard curve"]
  X --> S["Survival probability<br/>S∞ = e^−X"]
  subgraph mechanisms[Three forces on X]
    M1["State risk h A<br/>danger of EXISTING in a state<br/>→ go FAST to escape it"]
    M2["Transition risk f A·Ȧ^γ<br/>danger of the EXPERIMENT itself<br/>→ go SLOW if γ>1"]
    M3["Kuznets policy<br/>richer ⇒ buy more safety<br/>→ go FAST to get rich sooner"]
  end
  M1 --> H
  M2 --> H
  M3 --> B
  X --> OPT["Risk-minimizing growth Ȧ*<br/>balances M1 vs M2,<br/>boosted by M3"]

Schematic of the core change-of-variables idea: drag the growth rate. The hazard curve over time stretches/compresses, but the area under it (cumulative risk X) shrinks as you move faster through a dangerous-but-improving tech landscape — illustrating Proposition 1.

The state-vs-transition tradeoff (Eq. 9–10). The blue curve is state risk (falls with speed), the orange is transition risk (rises with speed when γ>1), and the black total has a minimum at the risk-minimizing growth rate Ȧ*. Drag γ and the h/f ratio to see Ȧ* move; at γ≤1 the minimum runs off to infinity (go as fast as possible).

The algorithm, simplified

There’s no training loop here — the “algorithm” is the optimization that finds the risk-minimizing growth path. Here it is as a concrete numerical computation you could actually run, which is also the skeleton of how you’d turn this paper into a tool:

import numpy as np
from scipy.optimize import minimize_scalar

# Hazard model (Eq. 8):  δ(A, Ȧ) = h(A) + f(A)·Ȧ^γ
# h = "state risk": danger of merely sitting in tech state A
# f = "transition risk": danger created by the act of advancing
def hazard(A, Adot, h, f, gamma):
    return h(A) + f(A) * Adot**gamma

def cumulative_risk(growth_fn, A0, A_inf, h, f, gamma, n=2000):
    """X = ∫ δ dt, re-expressed as ∫ (δ / Ȧ) dA  — the change-of-variables trick.
    risk_density = δ(A,Ȧ)/Ȧ  is risk accrued WHILE PASSING THROUGH state A."""
    A = np.linspace(A0, A_inf, n)
    Adot = growth_fn(A)                          # speed as a function of where we are
    delta = hazard(A, Adot, h, f, gamma)
    risk_density = delta / Adot                   # time spent at A is 1/Ȧ
    return np.trapz(risk_density, A)              # area under the hazard curve

def risk_minimizing_growth(A, h, f, gamma):
    """Closed form (Eq. 10): the safest SPEED at each state A.
    Positive whenever state risk h(A) > 0  →  stagnation is never safest."""
    if gamma <= 1:
        return np.inf                            # go as fast as possible
    return (1/(gamma - 1) * h(A) / f(A)) ** (1/gamma)

# Example: h(A) ∝ A^α (state risk), f(A) ∝ A^ζ (transition risk)
h = lambda A: 1.0 * A**(-0.5)     # state risk falls as tech matures (α<0): survival feasible
f = lambda A: 1.0 * A**(0.0)      # flat transition risk
gamma = 2.0                        # parallel experiments hurt: safe speed is finite

A_grid = np.linspace(1, 50, 50)
safe_speed = [risk_minimizing_growth(A, h, f, gamma) for A in A_grid]
# safe_speed[i] tells you how fast to move through each tech level to minimize lifetime risk

The whole paper, computationally, is: pick functional forms for h, f, and the policy response; integrate the hazard curve under different growth paths; find the path that minimizes X. Everything else is proving this has nice properties (existence, uniqueness, when X is finite at all).

Built on Prior Work

Results & Evidence

This is a theory paper — the “results” are theorems and one calibrated illustration, not experiments on data.

Headline theoretical results:

• Proposition 1: with state risk only, acceleration always weakly lowers cumulative risk. Faster = safer, unconditionally.
• Proposition 2: with both state and transition risk and γ>1, there’s a unique finite risk-minimizing growth rate, and it’s positive whenever any state risk exists.
• Corollary 2.1: for power-law forms h ∝ A^α, f ∝ A^ζ, survival is even feasible (X finite) iff α(γ−1) + γ + ζ < 0 — i.e. only if the tech landscape eventually gets safer or you grow ever-faster.
• The Kuznets result (§4): with an optimizing planner, faster growth is more safe than the no-policy case, because wealth pulls forward stringent safety spending.

The one number worth quoting: in the calibration where transition and state risk coefficients are equal (f̄ = h̄) and γ=2, the risk-minimizing growth rate is 100% per year. That’s the paper’s “may easily be high” claim made concrete.

What the evidence does NOT establish — read this part carefully, because the policy stakes are real:

• Everything rides on unmeasured parameters. Is γ above or below 1? Is state risk or transition risk bigger for AI/bio specifically? The paper assumes functional forms; it does not estimate them. The authors are explicit that quantifying h, f, and γ empirically is the key open task.
• Technology is modeled as one-dimensional. Real progress isn’t a single dial you turn faster or slower; some techs (vaccines) lower risk, others (bioweapons) raise it. The paper explicitly restricts itself to “how fast along a given path,” not “which direction.” Targeted slowdowns of specific dangerous capabilities are entirely consistent with the result.
• “Existential” is binary and one-shot. The model excludes gradual/slow-takeover scenarios (it cites Christiano’s “What failure looks like” as out of scope). If the real AI risk is slow drift rather than a sudden binary event, this framing may not apply.
• Policy frictions are the escape hatch. The whole go-fast conclusion can flip if regulators genuinely can’t respond effectively to fast change — and whether that’s true is, again, unmeasured.

So the honest read: this is a conceptual correction to a field that was making an unjustified assumption. It does not prove “accelerate AI.” It proves “the naive ‘slow down for safety’ argument doesn’t survive once you admit stagnation isn’t safe — and whether to actually go fast depends on parameters nobody has measured yet.”

How You’d Use It

You run an AI services company, so the relevant question is: where does a 42-page economics theorem touch what you build and sell? Three places.

1. As a framing weapon in AI-governance / risk consulting. If any client work touches AI safety posture, responsible-AI policy, or “should we pause,” this paper is the rigorous counterweight to reflexive “slow down” arguments. The distinction you can sell is state risk vs. transition risk: is the danger in deploying the model (state) or in the training run / capability jump itself (transition)? That single reframe changes what controls actually reduce risk. State-risk-dominated problems are reduced by moving through them faster to better tools; transition-risk-dominated problems are reduced by spacing out big jumps. Most “AI safety” discussions never make this distinction — you can.

2. As a model template for any “speed vs. risk” decision a client faces. The math is domain-agnostic. A client deciding how aggressively to roll out an automation, migrate infrastructure, or ship a risky feature is choosing a “growth rate through a dangerous state.” The reusable insight: staying in the dangerous transitional state longer is itself a cost — the risk density δ/Ȧ framing reframes “go slow to be safe” as “you’re just sitting in the danger zone longer.” That’s a genuinely useful decision lens for migration risk, security-debt remediation pacing, etc.

3. The Kuznets insight as a product thesis. “Safety is a luxury good that demand for rises with wealth/capability” predicts that as AI capability (and the value at stake) climbs, willingness to pay for AI safety/governance tooling rises faster than linearly. If you’re deciding whether to build a responsible-AI / eval / monitoring offering, this paper is the theoretical argument that that market grows super-linearly with capability — exactly when your clients get richer and more exposed, they want to buy more safety.

Build Your Own (Minimal Recipe)

You can’t “implement” a theorem, but you can build the thing that makes it useful to clients: an interactive risk-vs-growth model that lets a decision-maker plug in their beliefs and see the risk-minimizing speed. ~80% of the value in a weekend:

Components, in build order:

1. Parameterized hazard function δ(A, Ȧ, B) = h(A) + f(A)·Ȧ^γ with the h ∝ A^α, f ∝ A^ζ power-law forms. (~20 lines, the pseudocode above.)
2. Cumulative-risk integrator — np.trapz of δ/Ȧ over the technology grid. This is the whole engine.
3. Optimizer — for transition+state risk just evaluate the closed-form Eq. 10 per state; for the policy version you need a real optimal-control solve (the hard part — see below).
4. A UI (Streamlit/Observable) with sliders for α, ζ, γ, and the h/f ratio, outputting: the risk-minimizing growth path, the resulting survival probability e^−X, and the survival-feasibility check α(γ−1)+γ+ζ < 0.

Models/libraries to reach for: numpy + scipy.optimize for the static cases; for the policy/Kuznets version, the optimal-control problem needs either a Hamiltonian/Pontryagin solve or a numerical dynamic-programming sweep — reach for scipy.integrate.solve_bvp (it’s a boundary value problem) or a discretized Bellman backward-induction.

The 1–2 genuinely hard parts:

• The endogenous-policy optimization (§4). Once B is chosen optimally each period, you have a coupled system: the optimal safety share depends on future survival, which depends on future safety shares. That’s a fixed-point / optimal-control problem, not a one-liner. This is where 80% of the engineering effort lives.
• Choosing defensible parameter values. The model is only as good as your guesses for γ and the h/f ratio. The honest move is to make these explicitly user-set sliders and present ranges of conclusions, not a single number.

How to Improve It

Limitations are leverage. Five concrete, testable directions:

1. Estimate γ empirically for a real domain. The entire fast/slow conclusion hinges on whether γ ≷ 1 (does parallel experimentation scale risk super-linearly?). Pick one domain — say, frontier-AI training runs or gain-of-function bio — and try to estimate the elasticity of incident rate to concurrency from historical near-miss / incident data. This is the single highest-value follow-up and the authors basically beg for it.
2. Make technology multi-dimensional. Replace the one-dial A with a vector (capability vs. safety tech) and ask about differential acceleration — speed up defensive tech, slow offensive. This turns “how fast” into “how fast in which direction,” which is the actually-actionable policy question.
3. Model slow/gradual catastrophe. The binary one-shot assumption excludes the most-discussed AI failure mode (gradual disempowerment). Generalize “catastrophe” to a continuous, possibly-reversible welfare loss and re-derive — the go-fast result may not survive, which is itself important to know.
4. Endogenize the policy friction term. The paper treats “policy can’t keep up with fast change” as an assumption. Model regulatory capacity as its own state variable that also benefits from growth (better tools to monitor) but degrades with surprise — and see whether the friction escape-hatch actually closes.
5. Add strategic/multi-agent dynamics. The single-planner setup ignores arms-race incentives (Aschenbrenner’s own other work is all about this). Replace the planner with competing actors who each choose growth rates; the risk-minimizing global rate and the Nash rate will diverge, and the gap is the policy problem worth quantifying.

Glossary

• Existential risk (x-risk) — the probability of human extinction or an equally complete, permanent welfare loss; modeled as a binary event that can happen at most once.
• Hazard rate (δ) — the flow probability of catastrophe per unit time. The thing you’re trying to keep low.
• Cumulative risk (X) — total integrated hazard over all future time, ∫δ dt = the area under the hazard curve. Survival probability is e^−X; minimizing lifetime catastrophe risk ≡ minimizing X.
• Survival curve / S∞ — probability of surviving to time t; S∞ = e^−X is the probability of avoiding catastrophe forever. Positive only if X is finite.
• Technology state (A) — a one-dimensional index of how advanced technology is; in the policy model, equated with potential consumption per capita.
• Growth rate (Ȧ, “A-dot”) — how fast you move through technology-space; the decision variable. “Acceleration” = raising it over some band.
• Risk density (δ/Ȧ) — risk accrued while passing through a given technology state = (hazard there) × (time spent there). Falls as you move faster.
• State risk (h(A)) — danger from existing in a technology state (nukes in silos, pathogens that are now buildable). Stagnation can’t escape it.
• Transition risk (f(A)·Ȧ^γ) — danger created by the act of developing/deploying new tech (the experiment itself). Stagnation avoids it.
• γ (gamma) — elasticity of hazard to growth speed. γ<1: parallel is safer; γ=1: speed irrelevant (Jones’s model); γ>1: parallel experimentation is disproportionately dangerous.
• Risk-minimizing growth rate (Ȧ)* — the speed that minimizes cumulative risk (Eq. 10); positive whenever any state risk exists.
• Existential-risk Kuznets curve — the prediction that x-risk first rises then falls with wealth, because safety is a luxury good a richer society chooses to buy more of.
• Safety share (B) — fraction of potential consumption a planner forgoes to lower the hazard rate; C = A(1−B).
• Isoelastic / CRRA utility, η>1 — a utility function with constant relative risk aversion where marginal value of consumption diminishes fast enough that long-run survival dominates the planner’s objective.
• “Time of perils” (Sagan) — the hypothesis that humanity is passing through a uniquely dangerous transitional window that will be safer on the other side — exactly what the Kuznets curve formalizes.