Can branching and replay work with real Fabric PDF execution?
A first integration study moved the Dream-RSI mechanics out of the local simulator and into customer-32 Fabric development. The worker inspected synthetic account-opening PDFs using the real PDF tools. This page records what ran, what the traces show, and what the study still does not prove.
Research question
Can the branching-and-replay structure be carried into a real Fabric execution where the candidate policies must inspect PDFs, rather than only processing local JSON?
The target is deliberately narrow. Each candidate must inspect three synthetic PDFs and extract four fields: applicant, account type, risk profile, and document status. It should return ready only when the document is signed and all fields are clear; otherwise it should return needs_review.
What we ran
We created an experiment-owned customer-32 Fabric BP and uploaded a synthetic PDF reference-material corpus. The documents contained:
- Alex Morgan, Individual Brokerage, Moderate, Signed and dated
- Jamie Lee, Joint Brokerage, Conservative, Signed and dated
- Taylor Reed, IRA, Aggressive, Pending signature
We created five isolated branches:
| Branch | Intended policy | Fabric BP | Run results |
|---|---|---|---|
root-1 | Evidence ledger with source field and exact value | BP 851 | run 6598 |
root-2 | Concise extraction, then completeness check | BP 852 | run 6599 |
root-3 | Conservative review, with an explicit signed-status distinction | BP 853 | run 6600 |
fixed-child | Fixed-policy improvement branch | BP 854 | run 6601 |
replay-child | Replay-selected improvement branch | BP 855 | run 6602 |
How to read these links: the BP link opens the actual Fabric business-process configuration. The run link opens the actual Fabric event results, including phase completion and PDF-tool calls. These are internal dev links and require Fabric access.
Every BP used Worker V3, eval_eligible=true, tracing_enabled=true, and a narrow tool scope containing workspace, PDF, and time tools. The documents and runs were synthetic and isolated to customer 32.
Execution flow
- Create procedure and system prompt versions for each branch.
- Create an experiment-owned Fabric BP.
- Build and pin a BP-memory snapshot.
- Attach the same three synthetic PDFs as reference material.
- Start one Fabric run per branch.
- Poll until terminal state.
- Inspect the run events, PDF-tool calls, phase output, and tool counts.
What happened
complete. The traces show the real Fabric PDF path was exercised, including humanitylabs_introspect_pdf_file. The workers extracted the synthetic document fields, including the pending signature state for Taylor Reed.The observed tool-call counts were:
| Branch | Tool calls | Observed behavior |
|---|---|---|
| root-1 | 6 | PDF inspection calls plus todo bookkeeping |
| root-2 | 9 | Repeated targeted PDF inspection calls |
| root-3 | 6 | PDF inspection calls plus todo bookkeeping |
| fixed-child | 10 | PDF-to-image and PDF inspection path |
| replay-child | 9 | PDF inspection plus structural PDF inspection |
This shows that different policy prompts produced different real execution traces and tool budgets, even on the same document corpus. The fixed child also exercised image rendering through humanitylabs_pdf_to_images, while the replay child exercised pdf_pdf_inspect before extraction.
How this relates to replay
The run includes fixed and replay-labeled child branches, but this first integration pass is a plumbing proof rather than a controlled replay comparison. The branch prompts encode the intended strategy, and all five branches ran through Fabric. The next pass must select the replay parent from validation scores before creating the replay child, and must grade both children against a sealed document answer key.
Limitations
- No real customer data. Every PDF was synthetic. This is safe dev evidence, not customer-process evidence.
- Small corpus. Three one-page, text-based PDFs are not representative of production document variation.
- No images as primary inputs yet. One child rendered PDFs to images, but the corpus was still PDF-first and text-readable. We did not test noisy scans, rotation, handwriting, or low-resolution images.
- No holdout split yet. All branches saw the same three-document corpus. This run did not contain training, validation, and holdout cohorts.
- No deterministic quality score yet. We have event and tool evidence, but not a sealed field-level evaluator attached to every branch.
- Replay is not proven here. The child labels describe the intended policies, but the controller did not yet select a parent from measured validation results in this Fabric pass.
- Cost data is incomplete. We recorded tool-call counts, but not a complete token, wall-clock, and dollar-cost accounting for every branch.
Conclusion
This study successfully crossed the first boundary: Dream-RSI-shaped branches can execute through real Fabric dev and inspect synthetic PDFs with the production PDF tool path. It is not yet evidence that replay improves quality. The next experiment should add a sealed answer key, a training/validation/holdout document split, noisy image fixtures, structured run outputs, and a controller that chooses the replay parent from validation evidence.