How Dream-RSI learns from attempts
The simple idea is not “ask one agent for the answer.” It is to let several attempts explore the task, learn from measured experience, and spend the next improvement effort where the evidence says it is most useful.
The short version
This describes the algorithm. It does not claim that the current Fabric PDF study has already proved an advantage.
Step by step
Define the task
Specify the process, inputs, expected output, safety rules, and evaluator. In the Fabric study, the task is to inspect synthetic account-opening PDFs and decide whether each document is ready or needs review.
Create parallel attempts
Run several independent attempts against the same task. Each attempt can develop a different skill, prompt strategy, tool sequence, or way of handling ambiguity. These are the root branches.
Record experience
Do not keep only the final answer. Record the execution trace, tool calls, artifacts, errors, runtime, cost, and evaluator score. This recorded history is the experience the controller can use.
Evaluate generalization
Use training cases to develop the skill, validation cases to compare branches, and holdout cases only for the final measurement. The evaluator owns the answer key and keeps it away from the agent.
Select a branch to replay
Replay means using the recorded results to choose where to spend the next improvement opportunity. A strong validation score, useful failure evidence, low cost, or a specific recovery pattern can make a branch worth revisiting.
Improve the selected branch
Create a child from the selected parent. The child inherits the parent skill or BP strategy and gets a new improvement attempt. The parent-child relationship preserves exactly where the new attempt came from.
Compare against the baseline
Give the fixed policy and replay policy the same improvement budget. Compare final holdout quality first, then runtime, tool calls, tokens, and cost. Replay is only useful if it improves quality or efficiency.
What the tree looks like
What this means for Fabric
In a Fabric experiment, a branch does not have to be a Python program. It can be a frozen BP-memory snapshot, procedure prompt, system prompt, tool strategy, or combination of those. Each branch runs through the real Worker and leaves behind a trace and artifacts.
For PDF work, the experience can include whether the worker used introspect_pdf_file, rendered pages to images, retried after an extraction problem, flagged uncertainty, and produced the correct structured result.
What is still unproven
The local study demonstrated the mechanics on synthetic JSON. The Fabric PDF study demonstrated real Worker and PDF-tool execution. Neither has yet shown that replay beats a fixed branch policy on a sealed Fabric document holdout. That requires a larger corpus, a true train/validation/holdout split, a deterministic field-level grader, repeated runs, and complete cost accounting.