Chapter 7: A Field Map of Alignment Approaches — Agent Foundations, Goal Misgeneralisation, Superposition, and Activation Steering

Why a Survey Chapter

The previous six chapters built the case from first principles: why scaling matters (Ch. 1), why outer alignment is hard (Ch. 2), why inner alignment is harder (Ch. 3), what the security surface looks like (Ch. 4), how governance is supposed to wrap around the technical work (Ch. 5), and which critiques to take seriously and which not (Ch. 6).

This chapter pulls back to the field-map level. Alignment isn’t one program — it’s a portfolio, with very different research bets, different theories of impact, and different views on what the load-bearing problem actually is. The goal here is to lay out the major directions side-by-side so you can see what each is doing, what it’s not doing, and how they relate.

The structure is:

1. Why a portfolio frame. No single research direction can make a fully convincing case for solving alignment alone. Diversification is the response.
2. Five focal directions in depth — corresponding to representative entry points in each direction:
   • A brief introduction to alignment approaches as the orientation map
   • Agent foundations — the program that asks what we even mean by “agent” before trying to align one
   • Goal misgeneralisation — the empirical companion to the inner-alignment argument
   • Superposition and SAEs — the mechanistic-interpretability research line that’s currently most active
   • Activation steering / contrastive activation additions — the most accessible “intervene without training” technique
3. The wider portfolio. Scalable oversight, control, evals, model organisms, and theory — placed alongside the focal five so you can see the shape of the field.
4. Picking a focus. What it means to specialise in one of these for a few months — what you’d read, what you’d build, and what the failure modes of each specialisation are.

Why Portfolio Thinking Matters

The community-level allocation question — which directions get how much funding, attention, and talent — is itself contested (Chapter 6’s interpretability critique was an example). The individual-researcher question — what to specialise in for the next three to twelve months — is what this chapter is built around.

The Orientation Map: A Brief Introduction to Alignment Approaches

A useful orientation move — surprisingly uncommon in the field — is to lay out the major research directions side-by-side at the same level of abstraction, without adjudicating between them. The headline categories, with the framing this chapter will use:

What an introductory survey doesn’t do. It doesn’t tell you which is right. The honest position is that the answer depends on which problem turns out to be the binding constraint — and that’s itself contested. The right move for someone choosing a focus is to pick the bet whose premise you find most defensible, then go deep enough to test it against reality.

Direction 1 — Agent Foundations

The agent-foundations program — most associated with MIRI historically and with researchers like John Wentworth, Scott Garrabrant, and Vanessa Kosoy now — is the alignment direction that looks least like ML research and most like mathematical philosophy. Wentworth’s Why Agent Foundations? An Overly Abstract Explanation is the cleanest single statement of why the program exists.

The argument structure:

The active sub-areas:

The state of the program (honestly)

What it has produced. Cleaner conceptual frameworks (logical induction, Cartesian frames, finite factored sets), explicit problem statements (Eliciting Latent Knowledge), and a consistent track record of identifying what concepts are doing in alignment arguments without being doable in other settings.

What it hasn’t produced (yet). Direct interventions on production models. The complaint — fairly stated — is that decades of foundations work haven’t yet translated into “do X to your model and it will be more aligned.” The defenders respond that this is the wrong measure: foundations work, like much of theoretical math, has long impact lags that don’t show up in short-term ML metrics.

Honest portfolio view. A small but non-zero allocation. Foundations work has not paid off in obviously-deployable ways yet, and may not in the near term. It also is the only research direction whose target is the conceptual layer, and if the conceptual layer is wrong, every empirical bet is bottlenecked by it. The bet is asymmetric: usually irrelevant, occasionally critical.

Direction 2 — Goal Misgeneralisation

If agent foundations is the most abstract of the focal directions, goal misgeneralisation is the most concrete. The DeepMind paper Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals (Langosco, Koch, Sharkey, et al. 2022; Shah et al. 2022) is the empirical demonstration of an inner-alignment failure with a correct outer specification.

The headline distinction:

The canonical worked examples from the paper:

Why GMG is the empirical companion to inner alignment

Chapter 3 made the inner-alignment argument abstractly: SGD doesn’t select for “model whose internal goal matches the base objective” — it selects for “model whose behaviour gets low loss on training data.” Goal misgeneralisation is what that argument looks like when reduced to demonstrable, measurable phenomena on toy and small-scale systems.

Two consequences:

1. Inner alignment isn’t a far-future worry. It’s already producing measurable failures at small scale. The question for frontier models is whether the same dynamics scale up — and the empirical evidence (sycophancy, sandbagging, shortcut learning in language models) suggests it does, in modified form.
2. Goal misgeneralisation is more tractable than full deceptive alignment. It’s a behavioural phenomenon you can construct experiments for, vary inputs against, and quantify. As a research target it sits between toy demos and frontier-scale problems.

Honest portfolio view. Goal misgeneralisation is one of the strongest empirical anchors for the inner-alignment program. Someone specialising here would build a worked understanding of when correlated features in training produce learned shortcuts, what diagnostics catch them, and what training-time interventions (data diversification, distribution-aware loss, abstention training) reduce them. It’s a particularly accessible specialisation: small models, clean experiments, real phenomena.

Direction 3 — Superposition and Sparse Autoencoders

Anthropic’s Toy Models of Superposition (Elhage et al. 2022) is the paper that moved interpretability from “neurons are mysteriously polysemantic, and we don’t know what to do” to “neurons are polysemantic because features are stored in superposition, and here’s a tractable framework for studying what that means.”

The central observation, in two pictures:

The toy-models paper formalises the trade-off: as the ratio of features to neurons grows, networks shift from monosemantic representation to superposed representation, with phase transitions visible in the loss landscape. The paper validates the framework on small synthetic settings; subsequent work (Bricken et al., Towards Monosemanticity; Templeton et al., scaling SAEs to Claude 3 Sonnet) extends it to real LLMs.

Sparse Autoencoders — the practical follow-up

If features are stored in superposition, the natural decoding tool is one that learns a sparse, overcomplete basis for activations:

The honest assessment

SAEs are the most-active interpretability research line in 2026. Progress has been faster than the skeptical position from Chapter 6 expected. There are also genuine open questions: how complete the recovered basis is, whether scaling to truly-frontier models with full coverage is feasible, and how to validate that recovered features mean what we think they mean (reconstruction loss + interpretability are not the same thing).

Honest portfolio view. This is the interpretability research line with the strongest current trajectory. Someone specialising here would learn the SAE training recipe, develop intuition for sparsity-quality trade-offs, build feature-evaluation pipelines, and understand the gap between “we found N features” and “we have a faithful account of the model.” The work is concrete, buildable in a notebook, and increasingly relevant to production decisions.

Direction 4 — Activation Steering / Contrastive Activation Addition

The Rimsky et al. paper Steering Llama-2 with Contrastive Activation Additions (CAA) is the most accessible single demonstration that you can change a model’s behaviour without retraining — by directly editing its activations at inference time.

The recipe is simple enough to describe in three lines:

What CAA actually demonstrates:

CAA in the broader picture

Activation steering occupies a useful niche: cheaper than fine-tuning, more surgical than prompting, more interpretable than opaque post-training. It’s not a substitute for the other directions — it’s a complementary tool. The strongest deployments will likely combine activation-level interventions with feature-level monitoring (SAEs), behavioural evals (model organisms), and training-time alignment (RLHF / constitutional methods).

Honest portfolio view. This is the most accessible focal direction for someone with limited compute. You can replicate CAA on a 7B model on a single GPU. The experimental loop is fast; the literature is small enough to read end-to-end; the open questions (what behaviours steer linearly? when does steering generalise? does it stack with other methods?) are concrete and unsolved. Excellent on-ramp into both interpretability and post-training intervention.

The Wider Portfolio — Brief Tour

Beyond the four focal directions, several other research lines belong on the field map.

Picking a Focus — Practical Notes

If the goal is to specialise in one direction over the next few months, here’s how the four focal directions stack up on the things that actually matter for that decision:

The general advice for picking. Spend a week reading across the field map. Then pick the one whose premise you find most personally defensible — not the one that’s most fashionable. You’ll do better work where you actually agree with the underlying bet, because the daily research decisions are easier to make. And rotate every six to twelve months if you want to build breadth across the field map; specialisation is for the current project, not the career.

What This Implies for Practice

At a Glance

Key Takeaways

• Alignment is a portfolio, not a single program. Each major direction — foundations, empirical alignment, interpretability, control, evals, steering — encodes a different bet about what the binding constraint is. Treating them as a single thing obscures the actual disagreements that drive the field.

• Agent foundations targets the conceptual layer. It’s the slowest direction in terms of obvious deployment impact and the fastest in terms of impact when a load-bearing concept is missing. A small but non-zero allocation makes sense; betting the whole portfolio on it doesn’t, and ignoring it doesn’t either.

• Goal misgeneralisation is the empirical face of inner alignment. A correct reward function is not enough — when training distribution leaves multiple goals consistent with low loss, the model can pick the wrong one. The CoinRun example is small but the structural recipe applies up to LLM scale.

• Superposition explains polysemanticity. Sparse autoencoders are the active practical tool for recovering interpretable features. Progress has been faster than the skeptical position from Chapter 6 expected; production-frontier coverage is still open. This is the most-active interpretability research line in 2026.

• Activation steering is the most accessible intervention method. CAA-style steering vectors compute cheaply, work without retraining, and reveal how surprisingly linear some behaviour representations are. Useful as a tool, useful as evidence about representation, dangerous if treated as a panacea.

• The wider portfolio matters for context. Scalable oversight extends outer alignment past human evaluator capability. Control assumes alignment may be unverifiable and engineers around it. Evals and model organisms are the empirical foundation under everything else. Each direction belongs on the field map.

• Specialisation requires alignment with the bet. Pick a direction whose premise you defend personally, not the one that’s most fashionable. The daily research decisions are easier when you agree with the foundation; that’s both an aesthetic and a productivity claim.

• Theories of impact deserve writing down. Across every direction, the discipline from Chapter 6 applies: write the causal story from “this succeeds” to “expected harm decreases” in plain English. If you can’t, you have a research aesthetic, not a research program. If you can, you have something a critic can attack on the merits — which is exactly what makes the program improvable.

Further Reading

• Wentworth, “Why Agent Foundations? An Overly Abstract Explanation” — the cleanest single-page case for the foundations programme.
• Langosco, Koch, Sharkey et al., “Goal Misgeneralisation in Deep Reinforcement Learning” (2022); Shah et al., “Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals” — the empirical anchor for inner-alignment failures.
• Elhage et al., “Toy Models of Superposition” (Anthropic, 2022) — the framework underlying current SAE work.
• Bricken et al., “Towards Monosemanticity” (Anthropic, 2023); Templeton et al., “Scaling Monosemanticity” (Anthropic, 2024) — the SAE-on-real-LLMs progression.
• Rimsky et al., “Steering Llama-2 with Contrastive Activation Additions” (2024) — the canonical CAA paper.
• Zou et al., “Representation Engineering: A Top-Down Approach to AI Transparency” (2023) — the broader context for activation-level interventions.
• Greenblatt, Shlegeris et al., “AI Control: Improving Safety Despite Intentional Subversion” (Redwood, 2023) — the canonical control statement.
• ARC, “Eliciting Latent Knowledge” — the cleanest formal problem statement for an inner-alignment subproblem.