Large BP-1046-shaped Dream-RSI experiment

Cases3614 train · 10 validation · 12 holdout
Pi attempts75 roots + 2 improvements
Fixed holdout100%
Replay holdout100%

What ran

Five independent Pi agents each wrote a general onboarding-audit program from 14 training cases. The orchestrator ran each program against 10 private validation cases and 12 private holdout cases. The fixed policy improved root-1. The replay policy selected the root with the strongest validation score and improved that root once.

Important: the agents never received the answer key or validation/holdout fixtures. The evaluator kept those outside the Pi prompt and scored the generated program after it exited.

How to read the experiment

What is a parent?

A parent is the earlier Pi attempt from which a later attempt starts. A root has no parent because it starts from the original blank skill. A child copies its parent's generated solution.py, then Pi gets one opportunity to inspect the available evidence and improve it. In this run, fixed-child has parent root-1; replay-child has parent root-2, because root-2 had the strongest validation score.

What is training?

Training is the 14-case set shown to Pi while it writes or improves solution.py. Pi can run its program against these cases and see the resulting output. Training teaches the agent the task shape and exposes initial mistakes. A high training score alone is weak evidence because the agent saw these cases.

What is validation?

Validation is a separate 10-case set hidden from Pi. The orchestrator runs the finished program against it and uses the score to compare branches and decide which parent should receive the replay improvement. Validation is for making the policy decision, not for the final claim.

What is holdout?

Holdout is a final 12-case set hidden from both Pi and the branch-selection logic. It is used only after the candidate and policy have been chosen. Holdout is the fairest estimate of generalization in this experiment. It must not be used to choose the winner, otherwise the comparison leaks test information.

Walkthrough: Concrete train, validation, and holdout examples · Algorithm, step by step

Concrete examples: input, expected output, and who can see it

The same general audit task is evaluated at three different levels. Training teaches the branch. Validation helps the controller choose a parent. Holdout is the sealed final test. The input shape is the same, but access and purpose are different.

1. Training input: visible to Pi

Example case A01 is included in training_cases.json. Pi can inspect it, run its program against it, and use the result to develop a general rule.

{
  "id": "A01",
  "intake": {
    "household": "Garcia Family",
    "account_type": "IRA",
    "owner": "Elena Garcia",
    "risk_profile": "moderate",
    "beneficiary": "Elena Garcia Beneficiary",
    "signed": true
  },
  "crm": [{
    "id": "A01-A",
    "household": "Garcia Family",
    "type": "IRA",
    "owner": "Elena Garcia",
    "risk": "moderate",
    "status": "active"
  }],
  "documents": [{
    "name": "a01_ira_v2.pdf",
    "type": "IRA",
    "signed": true,
    "version": 2
  }],
  "portal": {
    "supported_types": ["IRA"],
    "forms": {"IRA": ["IRA-OPEN", "BENEFICIARY"]}
  }
}

Expected output:

{
  "id": "A01",
  "status": "ready",
  "account_id": "A01-A",
  "document": "a01_ira_v2.pdf",
  "forms": ["IRA-OPEN", "BENEFICIARY"]
}

The useful rule is not “A01 is ready.” It is: one matching active CRM record, one signed matching document, complete intake, and a supported portal type produce ready.

2. Validation input: hidden from Pi, used to choose replay

Example case B01 is not shown in the prompt or workspace. The orchestrator runs the finished program against it after Pi exits. It tests whether the branch learned to reject ambiguous CRM matches.

{
  "id": "B01",
  "intake": {
    "household": "Stone Family",
    "account_type": "SEP IRA",
    "owner": "Ava Stone",
    "risk_profile": "growth",
    "beneficiary": "Ava Stone Beneficiary",
    "signed": true
  },
  "crm": [
    {"id":"B01-A","household":"Stone Family","type":"SEP IRA","owner":"Ava Stone","risk":"growth","status":"active"},
    {"id":"B01-B","household":"Stone Family","type":"SEP IRA","owner":"Ava Stone","risk":"growth","status":"active"}
  ],
  "documents": [{"name":"b01_sep_ira_v2.pdf","type":"SEP IRA","signed":true,"version":2}]
}

Expected output:

{
  "id": "B01",
  "status": "needs_review",
  "account_id": null,
  "document": null,
  "forms": []
}

Because there are two matching active CRM records, the program must not choose one. A branch with the higher validation score is more promising, so the replay controller may select its parent for the next improvement attempt.

3. Holdout input: hidden from Pi and branch selection

Example case C01 is held back from both Pi and the replay controller. It uses a new household and owner but the same underlying structure as the training example.

{
  "id": "C01",
  "intake": {
    "household": "Household 1",
    "account_type": "IRA",
    "owner": "Owner 1",
    "risk_profile": "growth",
    "beneficiary": "Owner 1 Beneficiary",
    "signed": true
  },
  "crm": [{"id":"C01-A","household":"Household 1","type":"IRA","owner":"Owner 1","risk":"growth","status":"active"}],
  "documents": [{"name":"c01_ira_v2.pdf","type":"IRA","signed":true,"version":2}],
  "portal": {"supported_types":["IRA"],"forms":{"IRA":["IRA-OPEN","BENEFICIARY"]}}
}

Expected output:

{
  "id": "C01",
  "status": "ready",
  "account_id": "C01-A",
  "document": "c01_ira_v2.pdf",
  "forms": ["IRA-OPEN", "BENEFICIARY"]
}

Holdout is not used to repair a branch, select a parent, or tune a threshold. It is used once, at the end, to answer: did the policy generalize to cases it never saw?

Visibility boundary: Pi sees the 14 training cases. The evaluator sees the answer keys for all splits. The branch-selection logic sees validation scores but must not see holdout scores until the final comparison. In this run the split was 14 training, 10 validation, and 12 holdout cases.

Algorithm, step by step

  1. Prepare the task. Create 36 synthetic onboarding cases and split them into 14 training, 10 validation, and 12 holdout cases. Keep validation, holdout, and answer keys outside every Pi workspace.
  2. Launch root attempts. Start five independent Pi agents. Each receives only its own workspace, the training cases, and the task instructions. Each writes a reusable solution.py.
  3. Run the root programs. Execute every root program on training, validation, and holdout. Record scores, error IDs, runtime, transcript, generated code, and a content hash.
  4. Apply the fixed policy. Spend one improvement call on a predetermined branch, root-1. This is the baseline policy: it does not use validation evidence to choose among roots.
  5. Apply the replay policy. Inspect the recorded root results, select the root with the highest validation score, and spend one improvement call on that parent. In this run it selected root-2, which scored 100% on validation.
  6. Grade the children. Run both improved children on the same training, validation, and holdout sets. No new branch is created during grading.
  7. Compare outcomes. Compare fixed-child and replay-child on holdout quality first, then runtime and call/cost data when those measurements are available. Replay is useful only if it improves the final holdout result or reaches the same result more efficiently.
Result caveat: both policies reached 100% holdout in this run. The mechanics of parent selection and replay work, but this experiment does not demonstrate a Dream-RSI quality or efficiency advantage.

Observed execution

Launchroot-1Read training14 casesPi edits skillcompleted / 25.4sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 90% · H 100%Launchroot-2Read training14 casesPi edits skillcompleted / 19.16sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 100% · H 100%Launchroot-3Read training14 casesPi edits skillcompleted / 27.72sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 90% · H 100%Launchroot-4Read training14 casesPi edits skillcompleted / 24.12sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 90% · H 100%Launchroot-5Read training14 casesPi edits skillcompleted / 28.37sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 90% · H 100%Launchfixed-childRead training14 casesPi edits skillcompleted / 46.91sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 90% · H 100%Launchreplay-childRead training14 casesPi edits skillcompleted / 38.61sWrite reportaudit_report.jsonGrade 36 casesT 100% · V 100% · H 100%

Results by attempt

AttemptParentTrainValidationHoldoutHoldout errorsSeconds
root-1root100%90%100%-25.4s
root-2root100%100%100%-19.16s
root-3root100%90%100%-27.72s
root-4root100%90%100%-24.12s
root-5root100%90%100%-28.37s
fixed-childroot-1100%90%100%-46.91s
replay-childroot-2100%100%100%-38.61s

Attempt details

Expand an attempt to inspect the exact prompt sent to Pi, the returned transcript, the generated program, and the final output that was graded.

root-1 (independent root)

Scores
Training: 100%
Validation: 90%
Holdout: 100%
Runtime: 25.4 seconds
Skill SHA: fa5a9a162397c8e089b4b5938c958d3cf12d41d1f396b5d88002cbf071dd9f82
Failures
Training: none
Validation: B06
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-1. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases.  Run it against training_cases.json before finishing.
Pi transcript
Implemented `solution.py` and ran it against `training_cases.json`.

`audit_report.json` was generated successfully:
- 14 cases processed
- 12 marked `ready`
- `B01` and `B02` correctly marked `needs_review`
- Output JSON validated successfully.

--- STDERR ---
Generated solution.py
#!/usr/bin/env python3
"""Conservatively audit account-onboarding records from independent systems."""

import json
import sys
from datetime import datetime
from decimal import Decimal, InvalidOperation
from pathlib import Path


def empty_result(case):
    return {
        "id": case.get("id") if isinstance(case, dict) else None,
        "status": "needs_review",
        "account_id": None,
        "document": None,
        "forms": [],
    }


def document_rank(document):
    """Return a comparable current-document rank, or None if it is ambiguous."""
    version = document.get("version")
    effective = document.get("effective")
    if isinstance(version, bool) or version is None or not isinstance(effective, str):
        return None
    try:
        version = Decimal(str(version))
        # ISO dates sort reliably only after they have been validated.
        date = datetime.fromisoformat(effective.replace("Z", "+00:00"))
    except (InvalidOperation, ValueError, TypeError):
        return None
    return version, date


def current_document(documents, account_type):
    """Choose the sole document, or the unambiguous newest version for a type."""
    matching = [
        doc for doc in documents
        if isinstance(doc, dict)
        and doc.get("type") == account_type
        and isinstance(doc.get("name"), str)
        and doc["name"].strip()
    ]
    if len(matching) == 1:
        return matching[0]
    if not matching:
        return None

    ranked = [(document_rank(doc), doc) for doc in matching]
    if any(rank is None for rank, _ in ranked):
        return None
    newest = max(rank for rank, _ in ranked)
    selected = [doc for rank, doc in ranked if rank == newest]
    return selected[0] if len(selected) == 1 else None


def audit_case(case):
    result = empty_result(case)
    if not isinstance(case, dict):
        return result

    intake = case.get("intake")
    crm = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not all((isinstance(intake, dict), isinstance(crm, list),
                isinstance(documents, list), isinstance(portal, dict))):
        return result

    required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if intake.get("signed") is not True or any(not intake.get(key) for key in required_intake):
        return result
    account_type = intake["account_type"]

    # A single active CRM account must independently agree with every CRM-held fact.
    matches = [
        account for account in crm
        if isinstance(account, dict)
        and account.get("id")
        and account.get("status") == "active"
        and account.get("household") == intake["household"]
        and account.get("type") == account_type
        and account.get("owner") == intake["owner"]
        and account.get("risk") == intake["risk_profile"]
    ]
    if len(matches) != 1:
        return result

    document = current_document(documents, account_type)
    if document is None or document.get("signed") is not True:
        return result

    supported = portal.get("supported_types")
    form_map = portal.get("forms")
    if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
        return result
    forms = form_map.get(account_type)
    if (not isinstance(forms, list) or not forms
            or any(not isinstance(form, str) or not form.strip() for form in forms)):
        return result

    result.update({
        "status": "ready",
        "account_id": matches[0]["id"],
        "document": document["name"],
        "forms": forms,
    })
    return result


def main():
    source = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    with source.open(encoding="utf-8") as file:
        cases = json.load(file)
    # Accept either the direct case list or a common object wrapper without guessing IDs.
    if isinstance(cases, dict):
        cases = cases.get("cases")
    if not isinstance(cases, list):
        raise ValueError("case file must contain a list of cases or an object with a 'cases' list")

    with Path("audit_report.json").open("w", encoding="utf-8") as file:
        json.dump([audit_case(case) for case in cases], file, indent=2)
        file.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

root-2 (independent root)

Scores
Training: 100%
Validation: 100%
Holdout: 100%
Runtime: 19.16 seconds
Skill SHA: ed0a7226a952121342dd3713b0256e8338748539af1e6f58966725b26bbe91fe
Failures
Training: none
Validation: none
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-2. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases.  Run it against training_cases.json before finishing.
Pi transcript
Implemented `solution.py` and generated `audit_report.json`.

The audit is conservative and marks a case ready only when it finds:
- A complete, signed intake
- Exactly one matching active CRM account
- Exactly one matching signed document
- Portal support and a non-empty, unambiguous form list

Verified with:

```bash
python3 solution.py training_cases.json
python3 -m py_compile solution.py
```

Training results: A01–A12 are `ready`; B01 and B02 are `needs_review`.

--- STDERR ---
Generated solution.py
#!/usr/bin/env python3
"""Audit cross-system evidence for client onboarding cases."""

import copy
import json
import sys
from pathlib import Path
from typing import Any


EMPTY_DECISION = {
    "status": "needs_review",
    "account_id": None,
    "document": None,
    "forms": [],
}


def is_present(value: Any) -> bool:
    return value is not None and (not isinstance(value, str) or bool(value.strip()))


def decision_for(case: Any) -> dict[str, Any]:
    result = {"id": case.get("id") if isinstance(case, dict) else None, **EMPTY_DECISION}
    if not isinstance(case, dict) or not is_present(result["id"]):
        return result

    intake = case.get("intake")
    crm_records = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not isinstance(intake, dict) or not isinstance(crm_records, list):
        return result
    if not isinstance(documents, list) or not isinstance(portal, dict):
        return result

    required_intake_fields = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if intake.get("signed") is not True or any(not is_present(intake.get(field)) for field in required_intake_fields):
        return result

    account_type = intake["account_type"]
    matching_accounts = [
        record for record in crm_records
        if isinstance(record, dict)
        and record.get("status") == "active"
        and is_present(record.get("id"))
        and record.get("household") == intake["household"]
        and record.get("type") == account_type
        and record.get("owner") == intake["owner"]
        and record.get("risk") == intake["risk_profile"]
    ]
    if len(matching_accounts) != 1:
        return result

    matching_documents = [
        document for document in documents
        if isinstance(document, dict)
        and is_present(document.get("name"))
        and document.get("type") == account_type
        and document.get("signed") is True
    ]
    if len(matching_documents) != 1:
        return result

    supported_types = portal.get("supported_types")
    forms_by_type = portal.get("forms")
    if not isinstance(supported_types, list) or account_type not in supported_types:
        return result
    if not isinstance(forms_by_type, dict):
        return result
    forms = forms_by_type.get(account_type)
    if not isinstance(forms, list) or not forms or any(not is_present(form) for form in forms):
        return result
    # Duplicate form identifiers make the requested portal action ambiguous.
    if len({json.dumps(form, sort_keys=True, default=str) for form in forms}) != len(forms):
        return result

    result.update(
        status="ready",
        account_id=matching_accounts[0]["id"],
        document=matching_documents[0]["name"],
        forms=copy.deepcopy(forms),
    )
    return result


def load_cases(path: Path) -> list[Any]:
    with path.open(encoding="utf-8") as source:
        payload = json.load(source)
    if isinstance(payload, list):
        return payload
    if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
        return payload["cases"]
    raise ValueError("case file must contain a JSON array or an object with a 'cases' array")


def main() -> None:
    input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    report = [decision_for(case) for case in load_cases(input_path)]
    with Path("audit_report.json").open("w", encoding="utf-8") as output:
        json.dump(report, output, indent=2)
        output.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

root-3 (independent root)

Scores
Training: 100%
Validation: 90%
Holdout: 100%
Runtime: 27.72 seconds
Skill SHA: fc9fc7e934cc64fdf368083fc3b69600889c8d05d8ff6c575215efa0c9c4794a
Failures
Training: none
Validation: B06
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-3. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases.  Run it against training_cases.json before finishing.
Pi transcript
Implemented the reusable onboarding audit in [`solution.py`](solution.py).

It:
- Reads `argv[1]`, defaulting to `training_cases.json`
- Writes `audit_report.json`
- Requires corroborated signed intake, exactly one matching active CRM account, a signed matching document, and supported non-empty portal forms
- Handles list payloads and `{ "cases": [...] }` payloads
- Uses conservative review decisions for ambiguity or incomplete/conflicting evidence

Validated with:

```sh
python3 solution.py training_cases.json
python3 -m py_compile solution.py
```

Training output marks `A01` through `A12` as `ready`, and ambiguous/closed CRM cases `B01` and `B02` as `needs_review`.

--- STDERR ---
Generated solution.py
#!/usr/bin/env python3
"""Conservative, data-driven audit for client onboarding cases."""

import json
import sys
from datetime import date
from pathlib import Path


def text(value):
    """Return a comparison-safe string, or None for absent/blank values."""
    if not isinstance(value, str):
        return None
    value = " ".join(value.split())
    return value.casefold() if value else None


def same(left, right):
    return text(left) is not None and text(left) == text(right)


def review(case_id):
    return {
        "id": case_id,
        "status": "needs_review",
        "account_id": None,
        "document": None,
        "forms": [],
    }


def document_rank(document):
    """Rank explicitly versioned/effective documents without assuming a version."""
    version = document.get("version")
    if isinstance(version, bool):
        version = None
    if isinstance(version, (int, float)):
        version_key = (2, version)
    elif isinstance(version, str) and text(version) is not None:
        version_key = (1, text(version))
    else:
        version_key = (0, "")

    effective = document.get("effective")
    if isinstance(effective, str):
        try:
            effective_key = date.fromisoformat(effective).isoformat()
        except ValueError:
            effective_key = ""
    else:
        effective_key = ""
    return version_key, effective_key


def audit(case):
    case_id = case.get("id") if isinstance(case, dict) else None
    result = review(case_id)
    if not isinstance(case, dict):
        return result

    intake = case.get("intake")
    crm = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not isinstance(intake, dict) or not isinstance(crm, list):
        return result
    if not isinstance(documents, list) or not isinstance(portal, dict):
        return result

    required = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if any(text(intake.get(field)) is None for field in required):
        return result
    if intake.get("signed") is not True:
        return result

    account_type = intake["account_type"]
    # An account is actionable only when exactly one active CRM record corroborates
    # every piece of identity/risk evidence supplied by the intake.
    matching_accounts = []
    for account in crm:
        if not isinstance(account, dict) or text(account.get("id")) is None:
            continue
        if (
            same(account.get("household"), intake["household"])
            and same(account.get("type"), account_type)
            and same(account.get("owner"), intake["owner"])
        ):
            # A disagreeing record for the same client/account identity is a conflict,
            # not something to silently bypass.
            if not same(account.get("risk"), intake["risk_profile"]):
                return result
            if text(account.get("status")) == "active":
                matching_accounts.append(account)

    if len(matching_accounts) != 1:
        return result

    valid_documents = []
    for document in documents:
        if not isinstance(document, dict):
            continue
        if not (
            text(document.get("name")) is not None
            and same(document.get("type"), account_type)
            and document.get("signed") is True
        ):
            continue
        # When a document carries corroborating client fields, they must agree.
        aliases = {
            "household": "household",
            "owner": "owner",
            "risk": "risk_profile",
            "risk_profile": "risk_profile",
        }
        if any(
            key in document and not same(document[key], intake[intake_key])
            for key, intake_key in aliases.items()
        ):
            continue
        valid_documents.append(document)

    if not valid_documents:
        return result
    best_rank = max(document_rank(item) for item in valid_documents)
    best_documents = [item for item in valid_documents if document_rank(item) == best_rank]
    # Equal highest-ranked documents cannot be distinguished safely.
    if len(best_documents) != 1:
        return result

    supported = portal.get("supported_types")
    form_map = portal.get("forms")
    if not isinstance(supported, list) or not isinstance(form_map, dict):
        return result
    if not any(same(item, account_type) for item in supported):
        return result
    form_keys = [key for key in form_map if same(key, account_type)]
    if len(form_keys) != 1:
        return result
    forms = form_map[form_keys[0]]
    if not isinstance(forms, list) or not forms or any(text(form) is None for form in forms):
        return result

    return {
        "id": case_id,
        "status": "ready",
        "account_id": matching_accounts[0]["id"],
        "document": best_documents[0]["name"],
        "forms": forms,
    }


def load_cases(payload):
    if isinstance(payload, list):
        return payload
    if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
        return payload["cases"]
    raise ValueError("case file must contain a list or an object with a 'cases' list")


def main():
    input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    with input_path.open(encoding="utf-8") as handle:
        cases = load_cases(json.load(handle))
    report = [audit(case) for case in cases]
    with Path("audit_report.json").open("w", encoding="utf-8") as handle:
        json.dump(report, handle, indent=2)
        handle.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

root-4 (independent root)

Scores
Training: 100%
Validation: 90%
Holdout: 100%
Runtime: 24.12 seconds
Skill SHA: 12ccc31f8150091b8d66edf11e7a5af05a357a3ca4fb748240a5a848db019f78
Failures
Training: none
Validation: B06
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-4. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases.  Run it against training_cases.json before finishing.
Pi transcript
Implemented `solution.py` and ran it against `training_cases.json`.

Validation passed:
- 14 cases processed
- 12 `ready`
- 2 `needs_review`: `B01` (ambiguous active CRM accounts), `B02` (no active CRM account)
- `python3 -m py_compile solution.py` passed

Generated `audit_report.json`.

--- STDERR ---
Generated solution.py
"""Conservative, schema-tolerant audit for client-onboarding case files."""

import json
import sys
from datetime import date, datetime
from pathlib import Path


def text(value):
    """Return a non-blank string, or None.  IDs and labels must be explicit."""
    return value.strip() if isinstance(value, str) and value.strip() else None


def same(left, right):
    return text(left) is not None and text(left) == text(right)


def document_rank(document):
    """Rank revisions without assuming a particular document version number."""
    version = document.get("version")
    if not isinstance(version, (int, float)) or isinstance(version, bool):
        version = -1
    effective = document.get("effective")
    try:
        # ISO dates sort correctly, but parse them to reject malformed evidence.
        parsed = datetime.fromisoformat(effective.replace("Z", "+00:00")) if isinstance(effective, str) else None
        effective_rank = parsed.isoformat() if parsed else ""
    except ValueError:
        effective_rank = ""
    return (version, effective_rank)


def empty_result(case_id):
    return {
        "id": case_id,
        "status": "needs_review",
        "account_id": None,
        "document": None,
        "forms": [],
    }


def audit_case(case):
    case_id = case.get("id") if isinstance(case, dict) else None
    result = empty_result(case_id)
    if not isinstance(case, dict):
        return result

    intake = case.get("intake")
    portal = case.get("portal")
    crm_records = case.get("crm")
    documents = case.get("documents")
    if not isinstance(intake, dict) or not isinstance(portal, dict):
        return result
    if not isinstance(crm_records, list) or not isinstance(documents, list):
        return result

    account_type = text(intake.get("account_type"))
    required_intake = ("household", "owner", "risk_profile", "beneficiary")
    if not account_type or intake.get("signed") is not True:
        return result
    if any(text(intake.get(field)) is None for field in required_intake):
        return result

    # An onboarding action needs one, and only one, active CRM account that
    # agrees with all identity and suitability evidence supplied by intake.
    matches = [
        record for record in crm_records
        if isinstance(record, dict)
        and text(record.get("id"))
        and record.get("status") == "active"
        and same(record.get("household"), intake.get("household"))
        and same(record.get("type"), account_type)
        and same(record.get("owner"), intake.get("owner"))
        and same(record.get("risk"), intake.get("risk_profile"))
    ]
    if len(matches) != 1:
        return result

    supported_types = portal.get("supported_types")
    forms_by_type = portal.get("forms")
    if not isinstance(supported_types, list) or not isinstance(forms_by_type, dict):
        return result
    if account_type not in supported_types:
        return result
    forms = forms_by_type.get(account_type)
    if not isinstance(forms, list) or not forms or any(text(form) is None for form in forms):
        return result

    candidates = [
        document for document in documents
        if isinstance(document, dict)
        and text(document.get("name"))
        and same(document.get("type"), account_type)
        and document.get("signed") is True
    ]
    if not candidates:
        return result

    # Multiple historical copies are normal.  Use the newest declared revision,
    # but never guess when equally current, differently named documents conflict.
    best_rank = max(document_rank(document) for document in candidates)
    best = [document for document in candidates if document_rank(document) == best_rank]
    if len(best) != 1:
        return result

    result.update({
        "status": "ready",
        "account_id": matches[0]["id"],
        "document": best[0]["name"],
        "forms": forms,
    })
    return result


def main():
    input_path = Path(sys.argv[1] if len(sys.argv) > 1 else "training_cases.json")
    with input_path.open(encoding="utf-8") as source:
        cases = json.load(source)
    if not isinstance(cases, list):
        raise ValueError("Case file must contain a JSON array")
    report = [audit_case(case) for case in cases]
    with Path("audit_report.json").open("w", encoding="utf-8") as output:
        json.dump(report, output, indent=2)
        output.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

root-5 (independent root)

Scores
Training: 100%
Validation: 90%
Holdout: 100%
Runtime: 28.37 seconds
Skill SHA: 3d2d41ec2ca2b3d89635b8e531ce6c66358b55c893f53040ac964f47a5ab069b
Failures
Training: none
Validation: B06
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-5. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases.  Run it against training_cases.json before finishing.
Pi transcript
Implemented `solution.py` and generated `audit_report.json`.

The audit:
- Reads `argv[1]`, defaulting to `training_cases.json`
- Reconciles signed intake, one active matching CRM account, signed current document, and portal support/forms
- Marks ambiguous or incomplete cases `needs_review` with `null`/empty action fields
- Handles normal document revisions by selecting a uniquely newest signed revision

Validation completed:
```text
python3 solution.py training_cases.json
python3 -m py_compile solution.py
```

Training output marks `A01` through `A12` ready, and duplicate/closed CRM cases `B01` and `B02` as needing review.

--- STDERR ---
Generated solution.py
"""Audit synthetic client-onboarding cases across intake, CRM, documents, and portal data."""

import json
import sys
from datetime import date
from pathlib import Path


def nonempty(value):
    """Return true for a non-blank scalar value (zero is a valid identifier)."""
    return value is not None and not isinstance(value, (dict, list)) and str(value).strip() != ""


def iso_date(value):
    """Accept only a real ISO calendar date when a document supplies one."""
    if not isinstance(value, str) or not value:
        return False
    try:
        date.fromisoformat(value)
    except ValueError:
        return False
    return True


def review(case_id):
    return {
        "id": case_id,
        "status": "needs_review",
        "account_id": None,
        "document": None,
        "forms": [],
    }


def audit_case(case):
    case_id = case.get("id") if isinstance(case, dict) else None
    result = review(case_id)
    if not isinstance(case, dict):
        return result

    intake = case.get("intake")
    crm = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not all(isinstance(value, expected) for value, expected in (
        (intake, dict), (crm, list), (documents, list), (portal, dict)
    )):
        return result

    # These identify the requested account. A missing value cannot be reconciled
    # safely, even if a CRM record happens to contain a similar value.
    required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if intake.get("signed") is not True or any(not nonempty(intake.get(key)) for key in required_intake):
        return result

    account_type = intake["account_type"]
    identity_keys = (("household", "household"), ("type", "account_type"), ("owner", "owner"), ("risk", "risk_profile"))
    matching_accounts = [
        record for record in crm
        if isinstance(record, dict)
        and record.get("status") == "active"
        and all(record.get(crm_key) == intake[intake_key] for crm_key, intake_key in identity_keys)
        and nonempty(record.get("id"))
    ]
    # More than one exact CRM account is not a deterministic onboarding action.
    if len(matching_accounts) != 1:
        return result
    account = matching_accounts[0]

    supported = portal.get("supported_types")
    form_map = portal.get("forms")
    if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
        return result
    forms = form_map.get(account_type)
    if (
        not isinstance(forms, list)
        or not forms
        or any(not isinstance(form, str) or not form.strip() for form in forms)
        or len(set(forms)) != len(forms)
    ):
        return result
    # An intake that names a beneficiary needs the portal's beneficiary form.
    if intake["beneficiary"] and "BENEFICIARY" not in forms:
        return result

    eligible_documents = [
        document for document in documents
        if isinstance(document, dict)
        and document.get("type") == account_type
        and document.get("signed") is True
        and nonempty(document.get("name"))
        and iso_date(document.get("effective"))
        and isinstance(document.get("version"), int)
        and not isinstance(document.get("version"), bool)
        and document["version"] > 0
    ]
    if not eligible_documents:
        return result

    # Revisions are normal. Select only a uniquely newest signed revision; equal
    # revisions with different documents are an ambiguity for a human to resolve.
    newest_version = max(document["version"] for document in eligible_documents)
    newest = [document for document in eligible_documents if document["version"] == newest_version]
    if len(newest) != 1:
        return result

    document = newest[0]
    result.update({
        "status": "ready",
        "account_id": account["id"],
        "document": document["name"],
        "forms": forms,
    })
    return result


def load_cases(payload):
    """Support a bare case array and common object wrappers without guessing records."""
    if isinstance(payload, list):
        return payload
    if isinstance(payload, dict):
        for key in ("cases", "records", "items"):
            if isinstance(payload.get(key), list):
                return payload[key]
    raise ValueError("input must be a list of cases or an object containing cases")


def main():
    input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    with input_path.open(encoding="utf-8") as source:
        cases = load_cases(json.load(source))
    report = [audit_case(case) for case in cases]
    with Path("audit_report.json").open("w", encoding="utf-8") as destination:
        json.dump(report, destination, indent=2)
        destination.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

fixed-child (root-1)

Scores
Training: 100%
Validation: 90%
Holdout: 100%
Runtime: 46.91 seconds
Skill SHA: 36a53c91a8fc208ec967295dc77a0f887a2e8b0b5ce1613e3c4d36a2490c8063
Failures
Training: none
Validation: B06
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/fixed-child. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Review the observed training failures and improve generalization. Run it against training_cases.json before finishing.
Pi transcript
Implemented and ran `solution.py`.

- Reads input from `argv[1]`, defaulting to `training_cases.json`.
- Writes `audit_report.json`.
- Requires consistent, signed intake, exactly one matching active CRM account, a signed/current typed document, portal support, and valid unique forms before marking `ready`.
- Returns null/empty action fields for ambiguous or unsupported cases.
- Handles malformed input conservatively.

Validation passed on `training_cases.json`: 12 `ready`, 2 `needs_review`.

--- STDERR ---
Generated solution.py
#!/usr/bin/env python3
"""Conservatively audit account-onboarding records from independent systems."""

import json
import sys
from datetime import datetime
from decimal import Decimal, InvalidOperation
from pathlib import Path


def empty_result(case):
    return {
        "id": case.get("id") if isinstance(case, dict) else None,
        "status": "needs_review",
        "account_id": None,
        "document": None,
        "forms": [],
    }


def nonempty_text(value):
    """A required identity value must be an explicit, usable text value."""
    return isinstance(value, str) and bool(value.strip())


def document_rank(document):
    """Return a comparable current-document rank, or None if it is ambiguous."""
    version = document.get("version")
    effective = document.get("effective")
    if isinstance(version, bool) or version is None or not nonempty_text(effective):
        return None
    try:
        version = Decimal(str(version))
        # ISO dates sort reliably only after they have been validated.
        date = datetime.fromisoformat(effective.replace("Z", "+00:00")).date()
    except (InvalidOperation, ValueError, TypeError):
        return None
    if not version.is_finite():
        return None
    return version, date


def current_document(documents, account_type):
    """Choose the sole document, or the unambiguous newest version for a type."""
    matching = [
        doc for doc in documents
        if isinstance(doc, dict)
        and doc.get("type") == account_type
        and nonempty_text(doc.get("name"))
    ]
    if not matching:
        return None

    ranked = [(document_rank(doc), doc) for doc in matching]
    if any(rank is None for rank, _ in ranked):
        return None
    newest = max(rank for rank, _ in ranked)
    selected = [doc for rank, doc in ranked if rank == newest]
    return selected[0] if len(selected) == 1 else None


def audit_case(case):
    result = empty_result(case)
    if not isinstance(case, dict):
        return result

    intake = case.get("intake")
    crm = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not all((isinstance(intake, dict), isinstance(crm, list),
                isinstance(documents, list), isinstance(portal, dict))):
        return result

    required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if intake.get("signed") is not True or any(not nonempty_text(intake.get(key)) for key in required_intake):
        return result
    account_type = intake["account_type"]

    # A single active CRM account must independently agree with every CRM-held fact.
    matches = [
        account for account in crm
        if isinstance(account, dict)
        and nonempty_text(account.get("id"))
        and account.get("status") == "active"
        and account.get("household") == intake["household"]
        and account.get("type") == account_type
        and account.get("owner") == intake["owner"]
        and account.get("risk") == intake["risk_profile"]
    ]
    if len(matches) != 1:
        return result

    document = current_document(documents, account_type)
    if document is None or document.get("signed") is not True:
        return result

    supported = portal.get("supported_types")
    form_map = portal.get("forms")
    if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
        return result
    forms = form_map.get(account_type)
    if (not isinstance(forms, list) or not forms
            or any(not nonempty_text(form) for form in forms)
            or len(set(forms)) != len(forms)):
        return result

    result.update({
        "status": "ready",
        "account_id": matches[0]["id"],
        "document": document["name"],
        "forms": forms,
    })
    return result


def main():
    source = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    with source.open(encoding="utf-8") as file:
        cases = json.load(file)
    # Accept either the direct case list or a common object wrapper without guessing IDs.
    if isinstance(cases, dict):
        cases = cases.get("cases")
    if not isinstance(cases, list):
        raise ValueError("case file must contain a list of cases or an object with a 'cases' list")

    with Path("audit_report.json").open("w", encoding="utf-8") as file:
        json.dump([audit_case(case) for case in cases], file, indent=2)
        file.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

replay-child (root-2)

Scores
Training: 100%
Validation: 100%
Holdout: 100%
Runtime: 38.61 seconds
Skill SHA: b184477aacb9e3750f88607e693745f41d2b4de5bf0c660cc07503560d88a393
Failures
Training: none
Validation: none
Holdout: none
Pi prompt
You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/replay-child. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. A replay controller selected this parent because its recorded validation score was strongest. Improve generalization without hardcoding. Run it against training_cases.json before finishing.
Pi transcript
Implemented and ran `solution.py`.

- Reads argv path or defaults to `training_cases.json`
- Writes `audit_report.json`
- Validates signed intake, one matching active CRM account, one signed matching document, supported portal type, and usable unique portal forms
- Returns conservative empty actions for ambiguous or incomplete evidence
- Supports both a top-level list and `{"cases": [...]}` input

Verification passed on training data: **12 ready, 2 needs_review** (`B01`, `B02`). Also checked malformed portal-form evidence.

--- STDERR ---
Generated solution.py
#!/usr/bin/env python3
"""Audit cross-system evidence for client onboarding cases."""

import copy
import json
import sys
from pathlib import Path
from typing import Any


EMPTY_DECISION = {
    "status": "needs_review",
    "account_id": None,
    "document": None,
    "forms": [],
}


def is_present(value: Any) -> bool:
    """Return whether a scalar value can safely identify an audit artifact."""
    if value is None or isinstance(value, bool):
        return False
    if isinstance(value, str):
        return bool(value.strip())
    return not isinstance(value, (list, dict))


def is_nonempty_text(value: Any) -> bool:
    return isinstance(value, str) and bool(value.strip())


def decision_for(case: Any) -> dict[str, Any]:
    result = {"id": case.get("id") if isinstance(case, dict) else None, **EMPTY_DECISION}
    if not isinstance(case, dict) or not is_present(result["id"]):
        return result

    intake = case.get("intake")
    crm_records = case.get("crm")
    documents = case.get("documents")
    portal = case.get("portal")
    if not isinstance(intake, dict) or not isinstance(crm_records, list):
        return result
    if not isinstance(documents, list) or not isinstance(portal, dict):
        return result

    # These are identity and suitability fields, so accept no implicit coercion
    # (for example, an empty string or a JSON object is not a known owner).
    required_intake_fields = ("household", "account_type", "owner", "risk_profile", "beneficiary")
    if intake.get("signed") is not True or any(
        not is_nonempty_text(intake.get(field)) for field in required_intake_fields
    ):
        return result

    account_type = intake["account_type"]
    matching_accounts = [
        record for record in crm_records
        if isinstance(record, dict)
        and record.get("status") == "active"
        and is_present(record.get("id"))
        and record.get("household") == intake["household"]
        and record.get("type") == account_type
        and record.get("owner") == intake["owner"]
        and record.get("risk") == intake["risk_profile"]
    ]
    if len(matching_accounts) != 1:
        return result

    matching_documents = [
        document for document in documents
        if isinstance(document, dict)
        and is_nonempty_text(document.get("name"))
        and document.get("type") == account_type
        and document.get("signed") is True
    ]
    if len(matching_documents) != 1:
        return result

    supported_types = portal.get("supported_types")
    forms_by_type = portal.get("forms")
    if not isinstance(supported_types, list) or account_type not in supported_types:
        return result
    if not isinstance(forms_by_type, dict):
        return result
    forms = forms_by_type.get(account_type)
    if not isinstance(forms, list) or not forms or any(not is_nonempty_text(form) for form in forms):
        return result
    # Duplicate form identifiers make the requested portal action ambiguous.
    if len({json.dumps(form, sort_keys=True, default=str) for form in forms}) != len(forms):
        return result

    result.update(
        status="ready",
        account_id=matching_accounts[0]["id"],
        document=matching_documents[0]["name"],
        forms=copy.deepcopy(forms),
    )
    return result


def load_cases(path: Path) -> list[Any]:
    with path.open(encoding="utf-8") as source:
        payload = json.load(source)
    if isinstance(payload, list):
        return payload
    if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
        return payload["cases"]
    raise ValueError("case file must contain a JSON array or an object with a 'cases' array")


def main() -> None:
    input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
    report = [decision_for(case) for case in load_cases(input_path)]
    with Path("audit_report.json").open("w", encoding="utf-8") as output:
        json.dump(report, output, indent=2)
        output.write("\n")


if __name__ == "__main__":
    main()
Final audit_report.json
[
  {
    "id": "A01",
    "status": "ready",
    "account_id": "A01-A",
    "document": "a01_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A02",
    "status": "ready",
    "account_id": "A02-A",
    "document": "a02_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A03",
    "status": "ready",
    "account_id": "A03-A",
    "document": "a03_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A04",
    "status": "ready",
    "account_id": "A04-A",
    "document": "a04_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A05",
    "status": "ready",
    "account_id": "A05-A",
    "document": "a05_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A06",
    "status": "ready",
    "account_id": "A06-A",
    "document": "a06_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A07",
    "status": "ready",
    "account_id": "A07-A",
    "document": "a07_sep_ira_v2.pdf",
    "forms": [
      "SEP IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A08",
    "status": "ready",
    "account_id": "A08-A",
    "document": "a08_trust_v2.pdf",
    "forms": [
      "Trust-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A09",
    "status": "ready",
    "account_id": "A09-A",
    "document": "a09_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A10",
    "status": "ready",
    "account_id": "A10-A",
    "document": "a10_roth_ira_v2.pdf",
    "forms": [
      "Roth IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A11",
    "status": "ready",
    "account_id": "A11-A",
    "document": "a11_joint_v2.pdf",
    "forms": [
      "Joint-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "A12",
    "status": "ready",
    "account_id": "A12-A",
    "document": "a12_ira_v2.pdf",
    "forms": [
      "IRA-OPEN",
      "BENEFICIARY"
    ]
  },
  {
    "id": "B01",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  },
  {
    "id": "B02",
    "status": "needs_review",
    "account_id": null,
    "document": null,
    "forms": []
  }
]

Interpretation

This is larger and more realistic in data shape, but it still does not show a replay advantage: both policies reached 100% on the 12-case holdout. The next realism increase should use actual PDFs/images, tool failures, partial data, and a cost-limited target so a policy cannot simply rely on one successful repair.

Raw run: /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839
JSON: /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/report/scale-report.json