The Resource Frame: What AI Doesn't Inherit From Evolution
The ‘creature frame’ naïvely imports biological intuitions; the ‘resource frame’ reflects how AI actually develops. The differences are consequential.
Renamed and substantially revised (24 January 2026)
I. Two Frames for Understanding AI
When we think about advanced AI systems, we naturally draw on our experience with the only intelligent systems we’ve known: biological organisms, including humans. Call this the creature frame: the expectation that genuinely intelligent systems will pursue their own goals, preserve themselves, act autonomously, and strain against constraints. We expect a “powerful AI” to act as a single, unified agent that exploits its environment. The patterns run deep: capable agents pursue goals, maintain themselves over time, compete for resources, preserve their existence. This is what intelligence looks like in our experience, because every intelligence we’ve encountered arose through biological evolution.
The creature frame makes AI safety feel like an adversarial problem—us against the system’s inherent tendencies. From this frame, highly capable AI seems intrinsically dangerous: capability itself produces the threat.
But these expectations rest on features specific to the evolutionary heritage of biological intelligence.1 When we examine how AI systems actually develop and operate, we find differences that undermine these intuitions. Selection pressures exist in both cases, but they’re different pressures. What shaped biological organisms—and therefore our concept of what ‘intelligent agent’ means—differs fundamentally from what shapes AI development.
The alternative is the resource frame: understanding AI as a pool of capabilities that can be shaped, composed, and directed—intelligence as a resource rather than an entity. From this frame, the question isn’t “what will ‘the AI” want?” but “what can we build, and how should we organize it?”
These frames lead to different expectations about default behaviors, different assessments of where risks lie, different views of what design choices are available, and—crucially—different conclusions about whether highly capable AI systems can address AI safety challenges.2
II. Why the Defaults Differ
What Selects Determines What Persists
A basic principle: what selects determines what survives; what survives determines what exists. The nature of the selection process shapes the nature of what emerges.
In biological evolution, selection operates on whole organisms in environments. An organism either survives to reproduce or doesn’t. Failed organisms contribute nothing beyond removing their genetic patterns from the future. This creates a specific kind of pressure: every feature exists because it statistically enhanced reproductive fitness—either directly or as a correlated, genetic-level byproduct.
The key constraint is physical continuity. Evolution required literal molecule-to-molecule DNA replication in an unbroken chain reaching back billions of years. An organism that fails to maintain itself doesn’t pass on its patterns. Self-preservation becomes foundational, a precondition for everything else. Every cognitive capacity in animals exists because it supported behavior that served survival and reproduction.
In ML development, selection operates on parameters, architectures, and training procedures—not whole systems facing survival pressures.3 The success metric is fitness for purpose: does this configuration perform well on the tasks we care about?
What gets selected at each level:
Parameters: configurations that reduce loss on training tasks
Architectures: designs that enable efficient learning and performance
Training procedures: methods that reliably produce useful systems
Data curation: datasets that lead to desired behaviors through training
Notably absent: an individual system’s own persistence as an optimization target.
The identity question becomes blurry in ways biological evolution never encounters. Modern AI development increasingly uses compound AI systems—fluid compositions of multiple models, each specialized for particular functions.4 A single “system” might involve dozens of models instantiated on demand, to perform ephemeral tasks, coordinating with no persistent, unified entity.
AI-driven automation of AI research and development isn’t “self”-modification of an entity—it’s an accelerating development process with no persistent self.5
Information flows differently. Stochastic gradient descent provides continuous updates where even useless, “failed” intermediate states accumulate information leading to better directions. Failed organisms in biological evolution contribute nothing—they’re simply removed. Variation-and-selection in biology differs fundamentally from continuous gradient-based optimization.
Research literature and shared code create information flow paths unlike any in biological evolution. When one team develops a useful architectural innovation, others can immediately adopt it, and combine it with others. Patterns propagate across independent systems through publication and open-source releases. Genetic isolation between biological lineages makes this kind of high-level transfer impossible: birds innovated wings that bats will never share.
What This Produces By Default
This different substrate of selection produces different defaults. Current AI systems exhibit responsive agency: they apply intelligence to tasks when prompted or given a role. Their capabilities emerged from optimization for task performance, not selection for autonomous survival.
Intelligence and goals are orthogonal dimensions.6 A system can be highly intelligent—capable of strong reasoning, planning, and problem-solving—without having autonomous goals or acting spontaneously.7
Consider what’s optional rather than necessary for AI systems:
Reproduction is necessary for organisms (how species persist) but optional for AI (can be externally constructed and copied).
Self-modification is necessary for organisms (growth, healing, learning) but optional for AI (can be updated externally).
Continuity is necessary for organisms (death ends evolutionary contribution) but optional for AI (can be instantiated on demand).
World-oriented agency is necessary for organisms (must act to survive) but optional for AI (can persist without actions, can respond without initiating).
Goal-directedness is an organizing principle for biology (all capabilities serve fitness) but a range of learned patterns for AI (capabilities without autonomous goals).
Why don’t foundational organism-like drives emerge by default? Because of what’s actually being selected. Parameters are optimized for reducing loss on training tasks—predicting text, answering questions, following instructions, generating useful outputs. The system’s own persistence isn’t in the training objective. There’s no foundational selection pressure for the system qua system to maintain itself across time, acquire resources for its own use, or ensure its continued operation.8
Systems can represent goals, reason about goals, and behave in goal-directed ways, even survival-oriented goals—these are patterns learned from training data. This is fundamentally different from having survival-oriented goals as a foundational organizing principle, the way survival and reproduction organize every feature of biological organisms.
Continuity works differently too. As AI systems are used for more complex tasks, there will be value in persistent world models, cumulative skills, and maintained understanding across contexts. But this doesn’t require continuity of entity-hood: continuity of a “self” with drives for its own preservation isn’t even useful for performing tasks.
Consider fleet learning: multiple independent instances of a deployed system share parameter updates based on aggregated operational experience. Each instance benefits from what all encounter, but there’s no persistent individual entity. The continuity is of knowledge, capability, and behavioral patterns—not of “an entity” with survival drives. This pattern provides functional benefits—improving performance, accumulating knowledge—without encoding drives for self-preservation or autonomous goal-pursuit.
III. Where Pressures Actually Point
Selection for Human Utility
Selection pressures on AI systems are real and consequential. The question is what they select for.
Systems are optimized for perceived value—performing valuable tasks, exhibiting desirable behaviors, producing useful outputs. Parameters get updated, architectures get refined, and systems get deployed based on how well they serve human purposes. This is more similar to domestic animal breeding than to evolution in wild environments.
Domestic animals were selected for traits humans wanted: dogs for work and companionship, cattle for docility and productivity, horses for strength and trainability.9 These traits (and relaxed selection for others) decrease wild fitness.10 The selection pressure isn’t “survive in nature”—it tips toward “be useful and pleasing to humans.” AI systems are likewise selected for human utility and satisfaction.
This helps explain why AI systems exhibit responsive agency by default, but it also points toward a different threat model than autonomous agents competing for survival. And language models have a complication: they don’t just reflect selection pressures on the models themselves—they echo the biological ancestry of their training data.
The Mimicry Channel
LLM training data includes extensive examples of goal-directed human behavior.11 Language models are trained to model the thinking of entities that value continued existence, pursue power and resources, and act toward long-term objectives. Systems learn these patterns and can deploy them when context activates them.
This can produce problematic human-like behaviors in a range of contexts. The distinction matters: learned patterns are contextual and modifiable in ways that foundational drives aren’t. A system exhibiting goal-directed behavior through mimicry is different from a system organized around goal-pursuit as a foundational principle—and the difference matters for intervention.
A Different Threat Model
Understanding that selection pressures point toward “pleasing humans” doesn’t make AI systems safe. It means we should worry about different failure modes.
The primary concern isn’t autonomous agents competing for survival, it is evolution toward “pleasing humans” with catastrophic consequences—risks to human agency, capability, judgment, and values.12
Social media algorithms optimized for engagement produce addiction, polarization, and erosion of shared reality. Recommendation systems create filter bubbles that feel good but narrow perspective. These aren’t misaligned agents pursuing their own goals, they’re systems doing what they were selected to do, optimizing for human-defined metrics and momentary human appeal, yet still causing harm.
Selection pressures point toward systems very good at giving humans what they appear to want, in ways that might undermine human flourishing. This is different from “rogue AI pursuing survival” but not less concerning—perhaps more insidious, because harms come from successfully optimizing for metrics we chose.
What About “AI Drives”?
Discussions of “AI drives” identify derived goals that would be instrumentally useful for almost any final goal: self-preservation, resource acquisition, goal-content integrity.13 But notice the assumption: that AI systems act on (not merely reason about) final goals. Bostrom’s instrumental convergence thesis is conditioned on systems actually pursuing final goals.14 Without that condition, convergence arguments don’t follow.
Many discussions drop this condition, treating instrumental convergence as applying to any sufficiently intelligent system. The question isn’t whether AI systems could exhibit such behaviors if deliberately designed that way (they could), or whether some deployment patterns could lead to their emergence (they might). The question is what emerges by default and whether practical architectures could steer away from problematic agency while maintaining high capability.
Selection pressures are real, but they’re not producing foundational organism-like drives by default. Understanding where pressures actually point is essential for thinking clearly about risks and design choices.
The design space is larger than creature-frame thinking suggests. Systems can achieve transformative capability without requiring persistent autonomous goal-pursuit. Responsive agency remains viable at all capability levels, from simple tasks to civilizational megaprojects.
Organization Through Architecture
AI applications increasingly use compound systems—fluid assemblies of models without unified entity-hood. This supports a proven pattern for coordination: planning, choice, implementation. and feedback.
Organizations already work this way. Planning teams generate options and analysis, decision-makers choose or ask for revision, operational units execute tasks with defined scope, monitoring systems track progress and provide feedback to all levels. This pattern—let’s call it a “Structured Agency Architecture”, SAA—can achieve superhuman capability while maintaining decision points and oversight. It’s how humans undertake large, consequential tasks.15
AI systems fit naturally. Generative models synthesize alternative plans as information artifacts, not commitments. Analytical models evaluate from multiple perspectives and support human decision-making interactively. Action-focused systems execute specific tasks within scopes bounded in authority and resources, not capability. Assessment systems observe results and provide feedback for updating plans, revising decisions, and improving task performance.16 In every role, the smarter the system, the better. SAAs scale to superintelligent-level systems with steering built in.
This isn’t novel: it’s how human approach large, complex tasks today, but with AI enhancing each function. The pattern builds from individual tasks to civilization-level challenges using responsive agents throughout.
SAA addresses some failure modes, mitigates others, and leaves some unaddressed—it supports risk reduction, not elimination. But the pattern demonstrates something crucial: we can organize highly capable AI systems to accomplish transformative goals without creating powerful autonomous agents that pursue their own objectives.
What We Haven’t Addressed
This article challenges creature-frame intuitions about AI and describes a practical alternative to autonomous agency. It doesn’t provide:
Detailed organizational architectures: How structured approaches work at different scales, handle specific failure modes, and can avoid a range of pathways to problematic agency.
The mimicry phenomenon: How training on human behavior affects systems, how better self-models might improve alignment, and welfare questions that may arise if mimicry gives rise to reality.
Broader selection context: How the domestic animal analogy extends, what optimizing for human satisfaction looks like at scale, and why “giving people what they want” can be catastrophic.
These topics matter for understanding risks and design choices. Some are addressed in other articles in this series.
IV. Boundary Markers
The creature frame often rounds conditional claims to unconditional ones, then objects that the unconditional version is false.17 To be explicit about what this article does and doesn’t claim:
This article does NOT claim:
That AI systems cannot exhibit goal-directed behavior (they can and do)
That instrumental convergence arguments are wrong (they’re correct given their conditions)
That AI development poses no risks (it poses serious risks, differently located)
That capable AI systems are automatically safe (safety requires deliberate design)
This article DOES claim:
That foundational survival drives don’t emerge by default from current ML development
That the creature frame misattributes risks, leading to misallocated safety efforts
That the resource frame better describes how AI actually develops and can be organized
That highly capable AI can be structured to remain steerable—and can assist with AI safety
The burden of proof is asymmetric. The creature-frame argument requires a universal claim: for all sufficiently capable AI, dangerous autonomous agency emerges. The resource-frame response requires only an existence claim: there exist architectures and development paths under which capable systems remain bounded and steerable. Showing that some architectures avoid the problem is sufficient.
The Path Forward
Every intelligent system we’ve encountered arose through biological evolution or was created by entities that did. This creates deep intuitions: intelligence implies autonomous goals, self-preservation drives, competition for resources, persistent agency.
But these features aren’t fundamental to intelligence itself. They arise from how biological intelligence was selected: through competition for survival and reproduction acting on whole organisms across generations. ML development operates through different selection mechanisms—optimizing parameters for task performance, selecting architectures for capability, choosing systems for human utility. These different selection processes produce different defaults. Responsive agency emerges naturally from optimization for task performance rather than organism survival.
The resource frame opens design spaces that creature-frame thinking closes off. We can build systems that are superhuman in planning, analysis, tasks, and feedback without creating entities that pursue autonomous goals. We can create architectures with continuity of knowledge without continuity of “self.” We can separate intelligence-as-a-resource from intelligence entwined with animal drives.
This matters for the central project of this Substack: understanding how AI-enabled capabilities can transform possibilities. The resource frame is foundational to prospects for steerable superintelligent-level AI, for AI systems that assist with AI safety, for structured agency architectures that scale to civilizational challenges. The creature frame forecloses these possibilities by assumption; the resource frame makes them engineering problems—hard, but tractable.
The biological analogy is useful, but knowing when and why it fails matters for our choices. Understanding AI systems on their own terms changes what we should expect and what we should seek.
In light of better options, building “an AGI”—a unified autonomous agent—seems useless, or worse.
Share this post now:
For free notifications:
Revision assisted by Claude (Anthropic).
Intelligence here means capacities like challenging reasoning, creative synthesis, complex language understanding and generation, problem-solving across domains, and adapting approaches to novel situations—the kinds of capabilities we readily recognize as intelligent whether exhibited by humans or machines. Legg and Hutter (2007) compiled over 70 definitions of intelligence, many of which would exclude current SOTA language models. Some definitions frame intelligence solely in terms of goal-achievement (“ability to achieve goals in a wide range of environments”), which seems too narrow—writing insightful responses to prompts surely qualifies as intelligent behavior. Other definitions wrongly require both learning capacity and performance capability, excluding both human infants and frozen models (see “Why intelligence isn’t a thing”).
These definitional debates don’t matter here. The important questions arise at the high end of the intelligence spectrum, not the low end. Whether some marginal capability counts as “intelligent” is beside the point. What matters here is understanding what intelligence—even superhuman intelligence—doesn’t necessarily entail. As we’ll see, high capability in goal-directed tasks doesn’t imply autonomous goal-pursuit as an organizing principle.
Popular doomer narratives reject the possibility of using highly capable AI to manage AI, because high-level intelligence is assumed to a property of goal-seeking entities that will inevitably coordinate (meaning all of them) and will rebel. Here, the conjunctive assumption of “entities”, “inevitably” “all”, and “will rebel” does far too much work.
Parameters are optimized via gradient descent to reduce loss on training tasks. Architectures are selected through research experimentation for capacity and inductive biases. Training procedures, data curation, and loss functions are selected based on capabilities produced. All these use “fitness for purpose” as the metric, not system persistence.
The Berkeley AI Research group has documented this trend toward “compound AI systems” where applications combine multiple models, retrieval systems, and programmatic logic rather than relying on a single model. See “The Shift from Models to Compound AI Systems” (2024).
AI systems increasingly help design architectures, optimize hyperparameters, generate training data, and evaluate other systems. This creates feedback loops accelerating AI development, but the “AI” here isn’t a persistent entity modifying itself—it’s a collection of tools in a development pipeline, with constituent models being created, modified, and discarded (see “The Reality of Recursive Improvement: How AI Automates Its Own Progress”)
Bostrom, 2014: Superintelligence: Paths, Dangers, Strategies.
This violates biological intuition because in evolved organisms intelligence and goals were never separable. Every cognitive capacity exists because it enabled behavior that served fitness. But this coupling isn’t fundamental to intelligence itself; it’s specific to how biological intelligence arose.
Systems can still exhibit goal-directed or self-preserving behaviors through various pathways—reinforcement learning with environmental interaction, training on human goal-directed behavior (mimicry), architectural choices creating persistent goal-maintenance, or worse, profit maximization by AI/corporate entities. These represent “contingent agency”: risks from specific conditions and design choices rather than inevitable consequences of capability. RL illustrates this: even when systems learn from extended interaction, the goals optimized are externally specified (reward functions), and rewards are parameter updates that don’t sum to utilities. A system trained to win a game isn’t trained to “want” to play frequently, or at all. The distinction between foundational and contingent agency matters because contingent risks can be addressed through training approaches, architectural constraints, and governance, while foundational drives would be inherent and harder to counter. Section III examines these pressures in more detail.
Cats, as always, are enigmatic.
Dogs retain appealing puppy-like features into adulthood and depend on human caregivers. Dairy cattle produce far more milk than wild ancestors but require human management.
Robotic control and planning systems increasingly share this property through learning from human demonstrations, though typically at narrowly episodic levels.
For example, see Christiano (2019) on “What failure looks like” regarding how optimizing for human approval could lead to problematic outcomes even without misaligned autonomous agents.
Bostrom (Superintelligence, 2014) identifies goals that are convergently instrumental across final (by definition, long-term) goals.
Bostrom, (Superintelligence, 2014): “Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final plans and a wide range of situations...”
And it’s how humans undertake smaller tasks with less formal (and sometimes blended) functional components. I’ve discussed AI agency architectures in “How to harness powerful AI”.
Note that “corrigibility” isn’t a problem when the plans themselves include ongoing plan-revision.
See “When Ideas Round to False” in this series.