Geek AI Lab
Go back

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Edit page

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Table of Contents

Open Table of Contents

Introduction

Dream-RSI is a framework for recursive self-improvement (RSI) in autonomous AI agents.

The work, Dream-RSI: Recursive Self-Improvement through Evolving Worlds, was released in September 2026 by researchers from Google, Google DeepMind, the University of Maryland, and the University of Virginia.

The central problem is not simply making an AI agent better at solving a single task.

Instead, the question is:

How can an AI agent improve the strategy it uses to discover solutions?

For long-horizon discovery problems, an agent may need to perform thousands of proposal-and-evaluation cycles.

A typical process looks like:

propose→execute→evaluate→select→repeat\text{propose} \rightarrow \text{execute} \rightarrow \text{evaluate} \rightarrow \text{select} \rightarrow \text{repeat}

The difficulty is that the quality of an exploration strategy may only become visible after many iterations.

Consequently, directly optimizing the exploration policy online can be extremely expensive.

Dream-RSI introduces a different approach.

Instead of repeatedly evaluating exploration strategies in the real environment, it uses previously collected discovery histories as replay simulators.

The agent can then “dream” inside these historical worlds, evaluate alternative exploration strategies, improve them, and deploy the improved strategy back into the real search process.

This creates a recursive loop:

Explore→Record→Replay→Improve→Explore\text{Explore} \rightarrow \text{Record} \rightarrow \text{Replay} \rightarrow \text{Improve} \rightarrow \text{Explore}

The important idea is that history itself becomes a reusable computational environment.

Historical Background

Recursive self-improvement has long been discussed in artificial intelligence.

Traditional machine learning generally follows:

data→training→model\text{data} \rightarrow \text{training} \rightarrow \text{model}

An improved model requires additional data, optimization, or architectural changes.

Agentic systems introduce another dimension:

model→strategy→actions→outcomes\text{model} \rightarrow \text{strategy} \rightarrow \text{actions} \rightarrow \text{outcomes}

Here, improving the model is not necessarily the only way to improve the system.

The strategy used by the agent can also become an optimization target.

This leads to a hierarchy:

Object-Level Optimization

Improve the solution to a task.

Meta-Level Optimization

Improve the strategy used to search for solutions.

Recursive Self-Improvement

Use the results of previous optimization processes to improve the optimization strategy itself.

Dream-RSI focuses primarily on this meta-level exploration problem.

The Exploration Problem

Consider an autonomous coding agent searching for an optimized program.

Suppose the agent explores a tree:

T=(V,E)T = (V,E)

where each node represents a candidate program or discovery state.

A simplified search process is:

s0→s1→s2→⋯→sTs_0 \rightarrow s_1 \rightarrow s_2 \rightarrow \cdots \rightarrow s_T

At each state, the agent must decide:

These decisions define an exploration policy.

Let the policy be:

πθ\pi_\theta

where θ\theta represents the parameters or logic controlling exploration.

The objective is to maximize the final discovery quality:

θ∗=arg⁡max⁡θJ(πθ)\theta^* = \arg\max_\theta J(\pi_\theta)

The problem is that evaluating J(πθ)J(\pi_\theta) may require executing the entire discovery process again.

If a single discovery process requires thousands of expensive agent calls, then optimizing θ\theta online becomes extremely costly.

The Core Insight of Dream-RSI

Dream-RSI makes a key observation:

A completed discovery process already contains a large amount of information about the search space.

Suppose an agent previously explored:

                Root
              /      \
             A        B
           /  \      / \
          A1  A2    B1  B2
              |          |
             A2a        B2a

The tree contains:

Instead of discarding this information after the search finishes, Dream-RSI turns it into a replay simulator.

The historical tree becomes a world in which alternative exploration policies can be tested.

History as a Replay Simulator

The concept can be represented as:

H={(st,at,rt,st+1)}t=1T\mathcal{H} = \{(s_t,a_t,r_t,s_{t+1})\}_{t=1}^{T}

where:

Traditional online optimization would execute:

πθ→environment→new trajectory\pi_\theta \rightarrow \text{environment} \rightarrow \text{new trajectory}

Dream-RSI instead evaluates:

πθ→H→replayed trajectory\pi_\theta \rightarrow \mathcal{H} \rightarrow \text{replayed trajectory}

Because the outcomes of historical branches have already been recorded, many alternative decisions can be evaluated without rerunning the expensive underlying agent or evaluator.

This transforms historical data from passive logs into an active optimization environment.

The Dreaming Process

The name Dream-RSI comes from the idea that the agent can improve itself by “dreaming” inside historical worlds.

A simplified process is:

Real World
    |
    v
Online Discovery
    |
    v
Discovery History
    |
    v
Replay Simulator
    |
    v
Dreaming
    |
    v
Improved Exploration Policy
    |
    v
Real World

The important distinction is that the dreaming phase does not need to repeatedly execute the original expensive discovery process.

Instead, the agent explores a previously recorded search space.

Dream-RSI Architecture

The framework can be divided into several components.

1. Coding Agent

The underlying coding agent performs the actual discovery process.

Importantly, Dream-RSI does not require modifying the underlying coding agent.

The framework introduces a lightweight orchestration layer around it.

2. Exploration Policy

The orchestration layer determines how exploration is performed.

It can control:

Formally:

at∼πθ(a∣st)a_t \sim \pi_\theta(a|s_t)

The policy determines which action should be taken at each discovery state.

3. Discovery History

The system records the discovery process.

A history may contain:

H={st,at,rt,resultt}\mathcal{H} = \{s_t,a_t,r_t,\text{result}_t\}

This history becomes reusable after the online search finishes.

4. Replay Simulator

The discovery history is transformed into a replayable environment.

Instead of evaluating a new strategy against the real environment, the strategy can traverse the historical search tree.

The simulator therefore provides relatively cheap feedback:

πθ→H→JH(πθ)\pi_\theta \rightarrow \mathcal{H} \rightarrow J_{\mathcal{H}}(\pi_\theta)

5. Policy Improvement

Candidate exploration strategies are evaluated inside the replay simulator.

The system can then select or refine better strategies.

Conceptually:

θk+1=Improve⁡(θk,H)\theta_{k+1} = \operatorname{Improve} \left( \theta_k, \mathcal{H} \right)

The improved policy is subsequently deployed online.

The Recursive Self-Improvement Loop

The full Dream-RSI process can be written as:

π0→H1→Dream⁡(H1)→π1→H2→Dream⁡(H2)→π2→⋯\pi_0 \rightarrow \mathcal{H}_1 \rightarrow \operatorname{Dream}(\mathcal{H}_1) \rightarrow \pi_1 \rightarrow \mathcal{H}_2 \rightarrow \operatorname{Dream}(\mathcal{H}_2) \rightarrow \pi_2 \rightarrow \cdots

At iteration kk:

Step 1: Online Exploration

Use the current policy:

πk\pi_k

to perform discovery.

Store the resulting discovery tree:

Hk+1\mathcal{H}_{k+1}

Step 3: Build Replay Simulators

Convert the historical discovery process into a reusable simulator.

Step 4: Dream

Evaluate alternative exploration strategies inside the simulator.

Step 5: Improve

Produce a better exploration policy:

πk+1\pi_{k+1}

Step 6: Redeploy

Use πk+1\pi_{k+1} for the next real discovery process.

This creates a feedback loop:

Online Discovery→History→Dreaming→Policy Improvement→Online Discovery\boxed{ \text{Online Discovery} \rightarrow \text{History} \rightarrow \text{Dreaming} \rightarrow \text{Policy Improvement} \rightarrow \text{Online Discovery} }

Why This Is Different from Standard Reinforcement Learning

Dream-RSI has similarities to reinforcement learning, but the optimization problem is different.

In conventional RL, an agent typically learns a policy through interactions with an environment:

πθ→E→τ→θ′\pi_\theta \rightarrow E \rightarrow \tau \rightarrow \theta'

where τ\tau is an observed trajectory.

Dream-RSI instead focuses on optimizing the exploration process itself.

The historical discovery tree becomes an offline environment.

This resembles a combination of:

However, the key abstraction is particularly important:

History≈Replayable Search World\text{History} \approx \text{Replayable Search World}

The system does not necessarily need to learn a neural world model.

The already observed search tree provides the environment.

Off-Policy Evaluation

One of the major advantages of the approach is off-policy evaluation.

Suppose historical data was generated using:

πold\pi_{\text{old}}

Dream-RSI wants to evaluate:

πnew\pi_{\text{new}}

without executing the entire discovery process again.

This gives:

πold→H→Evaluate⁡(πnew)\pi_{\text{old}} \rightarrow \mathcal{H} \rightarrow \operatorname{Evaluate} (\pi_{\text{new}})

The historical data therefore becomes useful for evaluating policies that were not originally responsible for generating the trajectory.

This is especially valuable when evaluation is expensive.

Why Exploration Is the Bottleneck

In many agentic systems, generating one candidate is not the fundamental problem.

The harder problem is deciding where to search next.

Consider a discovery tree containing thousands of possible branches.

A fixed strategy might repeatedly allocate resources according to a predefined rule.

For example:

Explore A
Explore B
Explore C
Explore D
...

But as the search space grows, fixed strategies become increasingly inefficient.

The agent needs to learn:

Which branch?
How deeply?
How many samples?
When to stop?
When to backtrack?

This is a meta-search problem.

The search space therefore has two levels:

Object Search+Meta Search\text{Object Search} + \text{Meta Search}

Dream-RSI primarily targets the second layer.

Experimental Domains

The authors evaluate Dream-RSI in several discovery settings.

The reported experiments cover:

The project reports that Dream-RSI can improve discovery quality and reduce discovery cost in several settings.

Algorithm Engineering

Algorithm engineering treats the coding agent as a mechanism for discovering improved algorithms or implementations.

The search process can involve:

Idea→Code→Execution→Benchmark→Modification\text{Idea} \rightarrow \text{Code} \rightarrow \text{Execution} \rightarrow \text{Benchmark} \rightarrow \text{Modification}

A poor exploration policy may spend large amounts of computation on unproductive branches.

According to the project’s reported results, Dream-RSI achieved:

The project notes that these results were obtained without gradient updates to the underlying coding agent.

This distinction is important.

The system improves how the agent searches, rather than directly retraining the underlying model.

Mathematical Optimization

Mathematical optimization provides another natural environment.

A discovery system may explore candidate solutions:

x1,x2,…,xnx_1,x_2,\ldots,x_n

and evaluate:

f(x)f(x)

The challenge is allocating search resources efficiently.

A better exploration policy can potentially identify useful regions of the search space with fewer evaluations.

The Dream-RSI project reports competitive or improved results across its mathematical optimization experiments.

GPU Kernel Engineering

GPU kernel optimization is particularly suitable for this framework because candidate programs can be automatically benchmarked.

A simplified search loop is:

Generate Kernel→Compile→Execute→Measure→Modify\text{Generate Kernel} \rightarrow \text{Compile} \rightarrow \text{Execute} \rightarrow \text{Measure} \rightarrow \text{Modify}

The objective could be:

min⁡Truntime\min T_{\text{runtime}}

subject to correctness constraints.

According to the project repository, all four evaluated kernels were improved, with:

2.09×2.09\times

higher performance at an equal budget, and:

2.43×2.43\times

fewer generations at equal performance in the reported experiments.

A Simplified Mathematical View

Dream-RSI can be described using three functions.

Discovery

H=Explore⁡(π,E)\mathcal{H} = \operatorname{Explore} (\pi,E)

where:

Dreaming

JH(π′)=Replay⁡(H,π′)J_{\mathcal{H}}(\pi') = \operatorname{Replay} (\mathcal{H},\pi')

The historical search process is used to estimate the performance of an alternative policy.

Improvement

π′=Improve⁡(π,JH)\pi' = \operatorname{Improve} ( \pi, J_{\mathcal{H}} )

The resulting policy is then deployed:

H′=Explore⁡(π′,E)\mathcal{H}' = \operatorname{Explore} (\pi',E)

This produces new historical data:

H→H′→H′′→⋯\mathcal{H} \rightarrow \mathcal{H}' \rightarrow \mathcal{H}'' \rightarrow \cdots

The system therefore creates an evolving collection of reusable discovery worlds.

The Evolving Worlds Perspective

The word “worlds” in Dream-RSI is significant.

A conventional agent interacts with a single external environment.

Dream-RSI instead maintains a growing collection of historical environments:

W={W1,W2,…,Wn}\mathcal{W} = \{ \mathcal{W}_1, \mathcal{W}_2, \ldots, \mathcal{W}_n \}

Each world represents a previously observed discovery process.

The policy can therefore be evaluated across multiple historical worlds:

J(π)=1N∑i=1NJWi(π)J(\pi) = \frac{1}{N} \sum_{i=1}^{N} J_{\mathcal{W}_i}(\pi)

As more real-world exploration occurs:

∣W∣↑|\mathcal{W}| \uparrow

The simulator pool becomes richer.

This gives recursive improvement a data-generating mechanism:

Better Exploration→Better Histories→Better Simulators→Better Exploration\text{Better Exploration} \rightarrow \text{Better Histories} \rightarrow \text{Better Simulators} \rightarrow \text{Better Exploration}

Dream-RSI vs. Traditional Fine-Tuning

It is useful to distinguish Dream-RSI from conventional model training.

Traditional fine-tuning:

Dataset→Gradient Updates→New Model\text{Dataset} \rightarrow \text{Gradient Updates} \rightarrow \text{New Model}

Dream-RSI:

Discovery History→Replay→Policy Improvement→New Exploration Strategy\text{Discovery History} \rightarrow \text{Replay} \rightarrow \text{Policy Improvement} \rightarrow \text{New Exploration Strategy}

The underlying coding model can remain unchanged.

This makes the approach an example of system-level self-improvement rather than conventional parameter-level self-improvement.

System-Level vs. Model-Level Self-Improvement

There are at least two distinct ways an AI system can improve itself.

Model-Level Improvement

Modify model parameters:

θmodel→θmodel′\theta_{\text{model}} \rightarrow \theta'_{\text{model}}

Examples include:

System-Level Improvement

Keep the underlying model fixed but improve:

Dream-RSI primarily operates at the second level.

This is important because system-level optimization can potentially improve behavior without requiring expensive retraining of a frontier model.

Relationship to Experience Replay

Experience replay is a familiar concept in reinforcement learning.

A replay buffer stores:

(st,at,rt,st+1)(s_t,a_t,r_t,s_{t+1})

and allows an agent to reuse historical experience.

Dream-RSI extends the idea from individual transitions toward structured discovery histories.

Instead of merely replaying:

state → action → reward

the system can replay an entire search structure:

                 Discovery
                /         \
          Branch A       Branch B
          /    \          /    \
        A1     A2       B1     B2

The structure itself becomes valuable.

This makes the approach closer to search-tree replay than conventional experience replay.

Relationship to World Models

World models attempt to construct an internal model of an environment.

Conceptually:

E^≈E\hat{E} \approx E

An agent can then simulate future trajectories inside:

E^\hat{E}

Dream-RSI takes a different route.

Rather than necessarily learning a generalized world model, it directly reuses realized historical search trajectories:

H→Replay World\mathcal{H} \rightarrow \text{Replay World}

Therefore, Dream-RSI can be viewed as a form of empirical world modeling over the already explored search space.

The limitation is equally clear:

Replay World⊆True World\text{Replay World} \subseteq \text{True World}

The simulator knows about what has already been observed.

It cannot automatically represent arbitrary unseen branches.

Computational Efficiency

The main motivation for Dream-RSI is computational efficiency.

Suppose online evaluation costs:

ConlineC_{\text{online}}

and historical replay costs:

CreplayC_{\text{replay}}

with:

Creplay≪ConlineC_{\text{replay}} \ll C_{\text{online}}

Then evaluating NN candidate policies gives:

Traditional approach:

NConlineN C_{\text{online}}

Dream-RSI:

Chistory+NCreplayC_{\text{history}} + N C_{\text{replay}}

If:

Creplay≪ConlineC_{\text{replay}} \ll C_{\text{online}}

the savings can become substantial.

This is especially important for agentic systems where each evaluation may require:

Limitations

The approach also introduces important limitations.

Limited Coverage

A replay simulator only contains previously observed branches.

If a new policy selects a branch that was never explored, the historical simulator cannot necessarily provide the true outcome.

Therefore:

Replay Coverage≠Full Search Space\text{Replay Coverage} \neq \text{Full Search Space}

Distribution Shift

The new policy may behave differently from the policy that generated the historical data.

This creates an off-policy distribution gap:

πnew≠πbehavior\pi_{\text{new}} \neq \pi_{\text{behavior}}

As the policies diverge, historical data may become less representative.

Simulator Staleness

Old discovery histories may become less useful as the agent discovers new regions of the search space.

This motivates continuously expanding the simulator pool.

Evaluation Bias

A policy optimized entirely against historical worlds may learn to exploit artifacts of those worlds.

Therefore, online validation remains necessary.

A practical system should maintain:

Offline Improvement+Online Verification\text{Offline Improvement} + \text{Online Verification}

rather than relying exclusively on replay.

A General Architecture for Recursive Agent Improvement

Dream-RSI suggests a broader architecture for autonomous research agents.

                    ┌──────────────────────┐
                    │   Exploration Policy │
                    └──────────┬───────────┘
                               │
                               v
                    ┌──────────────────────┐
                    │   Online Discovery   │
                    └──────────┬───────────┘
                               │
                               v
                    ┌──────────────────────┐
                    │ Discovery Histories  │
                    └──────────┬───────────┘
                               │
                               v
                    ┌──────────────────────┐
                    │ Replay Simulator Pool│
                    └──────────┬───────────┘
                               │
                               v
                    ┌──────────────────────┐
                    │      Dreaming        │
                    └──────────┬───────────┘
                               │
                               v
                    ┌──────────────────────┐
                    │ Policy Improvement   │
                    └──────────┬───────────┘
                               │
                               └──────────────┐
                                              │
                                              v
                                      Online Discovery

This architecture separates:

  1. Task solving
  2. Search orchestration
  3. Experience collection
  4. Offline policy evaluation
  5. Meta-level improvement

That separation is one of the most interesting aspects of the framework.

Applications

The same principle could potentially be applied to many autonomous discovery systems.

Scientific Discovery

An AI scientist could record:

Historical experiments could then become replayable scientific search environments.

Program Synthesis

For program synthesis:

Specification→Program→Test→Score\text{Specification} \rightarrow \text{Program} \rightarrow \text{Test} \rightarrow \text{Score}

Historical synthesis trees could be reused to optimize search strategies.

Compiler Optimization

Compiler optimization naturally produces large search spaces.

A system could explore:

Program→Optimization→Benchmark\text{Program} \rightarrow \text{Optimization} \rightarrow \text{Benchmark}

and reuse previous optimization trajectories to improve the search policy.

GPU Kernel Optimization

GPU kernel search is particularly compatible with this framework because performance can be objectively measured.

The loop becomes:

Kernel→Compile→Benchmark→Replay→Better Search\text{Kernel} \rightarrow \text{Compile} \rightarrow \text{Benchmark} \rightarrow \text{Replay} \rightarrow \text{Better Search}

Mathematical Discovery

Mathematical research can be represented as a branching search process:

Conjecture→Proof Attempt→Result→New Direction\text{Conjecture} \rightarrow \text{Proof Attempt} \rightarrow \text{Result} \rightarrow \text{New Direction}

Historical proof attempts could potentially become reusable search trajectories.

Broader Significance

The most important idea in Dream-RSI is not a new neural network architecture.

It is a change in how we think about experience.

Traditional machine learning often treats experience as data:

experience→dataset\text{experience} \rightarrow \text{dataset}

Dream-RSI treats experience as an executable search environment:

experience→simulator\text{experience} \rightarrow \text{simulator}

This distinction is fundamental.

A log records what happened.

A simulator allows alternative decisions to be evaluated against what happened.

Therefore:

History becomes a computational resource\boxed{ \text{History becomes a computational resource} }

Conclusion

Dream-RSI proposes a framework for recursive self-improvement based on a simple but powerful idea:

An agent’s previous discovery history can become a world in which the agent learns how to explore better.

The framework separates the underlying coding agent from the exploration strategy and introduces a lightweight orchestration layer.

Its recursive loop is:

Explore→Record→Replay→Improve→Explore\boxed{ \text{Explore} \rightarrow \text{Record} \rightarrow \text{Replay} \rightarrow \text{Improve} \rightarrow \text{Explore} }

This approach addresses an important bottleneck in autonomous AI systems: optimizing long-horizon exploration without repeatedly paying the full cost of online evaluation.

Rather than continuously retraining the underlying model, Dream-RSI improves the system at the level of search strategy.

The broader research direction is therefore not simply:

Build a better model\text{Build a better model}

but:

Build a system that becomes better at searching\text{Build a system that becomes better at searching}

As autonomous agents move toward scientific discovery, program synthesis, algorithm design, and engineering optimization, this distinction may become increasingly important.

Dream-RSI provides one concrete framework for studying this problem: use the accumulated history of discovery to construct evolving worlds in which better exploration policies can be developed.

Further Reading

Paper

Project

Citation

@article{zheng2026dreamrsi,
  title   = {Dream-RSI: Recursive Self-Improvement through Evolving Worlds},
  author  = {Zheng, Tong and Wu, Xidong and Zhang, Zheng and
             He, Zhankui and Zhang, Chaoyi and Coleman, Benjamin and
             Wei, Ruoqiao and Bai, Di and Liu, Haolin and Liu, Rui and
             Wang, Xue and Zhuan, Yue and Kang, Wang-Cheng and
             Xiang, Renkai and Huang, Heng and Cheng, Xinwu and Guo, Yunsong},
  journal = {arXiv preprint arXiv:2609.14858},
  year    = {2026}
}

References

  1. Zheng, T. et al. Dream-RSI: Recursive Self-Improvement through Evolving Worlds, 2026.
  2. Dream-RSI official project repository.
  3. Dream-RSI citation metadata and project information.
Edit page
Share this post:
Share this post on WhatsappShare this post on FacebookShare this post on XShare this post on TelegramShare this post on PinterestShare this post via email
Previous Post
AI Agents Are Becoming the New Computing Paradigm
Next Post
DSec: The Infrastructure Layer Behind Scalable AI Agents