Five independent Pi agents each wrote a general onboarding-audit program from 14 training cases. The orchestrator ran each program against 10 private validation cases and 12 private holdout cases. The fixed policy improved root-1. The replay policy selected the root with the strongest validation score and improved that root once.
A parent is the earlier Pi attempt from which a later attempt starts. A root has no parent because it starts from the original blank skill. A child copies its parent's generated solution.py, then Pi gets one opportunity to inspect the available evidence and improve it. In this run, fixed-child has parent root-1; replay-child has parent root-2, because root-2 had the strongest validation score.
Training is the 14-case set shown to Pi while it writes or improves solution.py. Pi can run its program against these cases and see the resulting output. Training teaches the agent the task shape and exposes initial mistakes. A high training score alone is weak evidence because the agent saw these cases.
Validation is a separate 10-case set hidden from Pi. The orchestrator runs the finished program against it and uses the score to compare branches and decide which parent should receive the replay improvement. Validation is for making the policy decision, not for the final claim.
Holdout is a final 12-case set hidden from both Pi and the branch-selection logic. It is used only after the candidate and policy have been chosen. Holdout is the fairest estimate of generalization in this experiment. It must not be used to choose the winner, otherwise the comparison leaks test information.
Walkthrough: Concrete train, validation, and holdout examples · Algorithm, step by step
The same general audit task is evaluated at three different levels. Training teaches the branch. Validation helps the controller choose a parent. Holdout is the sealed final test. The input shape is the same, but access and purpose are different.
Example case A01 is included in training_cases.json. Pi can inspect it, run its program against it, and use the result to develop a general rule.
{
"id": "A01",
"intake": {
"household": "Garcia Family",
"account_type": "IRA",
"owner": "Elena Garcia",
"risk_profile": "moderate",
"beneficiary": "Elena Garcia Beneficiary",
"signed": true
},
"crm": [{
"id": "A01-A",
"household": "Garcia Family",
"type": "IRA",
"owner": "Elena Garcia",
"risk": "moderate",
"status": "active"
}],
"documents": [{
"name": "a01_ira_v2.pdf",
"type": "IRA",
"signed": true,
"version": 2
}],
"portal": {
"supported_types": ["IRA"],
"forms": {"IRA": ["IRA-OPEN", "BENEFICIARY"]}
}
}Expected output:
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": ["IRA-OPEN", "BENEFICIARY"]
}The useful rule is not “A01 is ready.” It is: one matching active CRM record, one signed matching document, complete intake, and a supported portal type produce ready.
Example case B01 is not shown in the prompt or workspace. The orchestrator runs the finished program against it after Pi exits. It tests whether the branch learned to reject ambiguous CRM matches.
{
"id": "B01",
"intake": {
"household": "Stone Family",
"account_type": "SEP IRA",
"owner": "Ava Stone",
"risk_profile": "growth",
"beneficiary": "Ava Stone Beneficiary",
"signed": true
},
"crm": [
{"id":"B01-A","household":"Stone Family","type":"SEP IRA","owner":"Ava Stone","risk":"growth","status":"active"},
{"id":"B01-B","household":"Stone Family","type":"SEP IRA","owner":"Ava Stone","risk":"growth","status":"active"}
],
"documents": [{"name":"b01_sep_ira_v2.pdf","type":"SEP IRA","signed":true,"version":2}]
}Expected output:
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}Because there are two matching active CRM records, the program must not choose one. A branch with the higher validation score is more promising, so the replay controller may select its parent for the next improvement attempt.
Example case C01 is held back from both Pi and the replay controller. It uses a new household and owner but the same underlying structure as the training example.
{
"id": "C01",
"intake": {
"household": "Household 1",
"account_type": "IRA",
"owner": "Owner 1",
"risk_profile": "growth",
"beneficiary": "Owner 1 Beneficiary",
"signed": true
},
"crm": [{"id":"C01-A","household":"Household 1","type":"IRA","owner":"Owner 1","risk":"growth","status":"active"}],
"documents": [{"name":"c01_ira_v2.pdf","type":"IRA","signed":true,"version":2}],
"portal": {"supported_types":["IRA"],"forms":{"IRA":["IRA-OPEN","BENEFICIARY"]}}
}Expected output:
{
"id": "C01",
"status": "ready",
"account_id": "C01-A",
"document": "c01_ira_v2.pdf",
"forms": ["IRA-OPEN", "BENEFICIARY"]
}Holdout is not used to repair a branch, select a parent, or tune a threshold. It is used once, at the end, to answer: did the policy generalize to cases it never saw?
solution.py.root-1. This is the baseline policy: it does not use validation evidence to choose among roots.root-2, which scored 100% on validation.| Attempt | Parent | Train | Validation | Holdout | Holdout errors | Seconds |
|---|---|---|---|---|---|---|
| root-1 | root | 100% | 90% | 100% | - | 25.4s |
| root-2 | root | 100% | 100% | 100% | - | 19.16s |
| root-3 | root | 100% | 90% | 100% | - | 27.72s |
| root-4 | root | 100% | 90% | 100% | - | 24.12s |
| root-5 | root | 100% | 90% | 100% | - | 28.37s |
| fixed-child | root-1 | 100% | 90% | 100% | - | 46.91s |
| replay-child | root-2 | 100% | 100% | 100% | - | 38.61s |
Expand an attempt to inspect the exact prompt sent to Pi, the returned transcript, the generated program, and the final output that was graded.
fa5a9a162397c8e089b4b5938c958d3cf12d41d1f396b5d88002cbf071dd9f82You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-1. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Run it against training_cases.json before finishing.
Implemented `solution.py` and ran it against `training_cases.json`. `audit_report.json` was generated successfully: - 14 cases processed - 12 marked `ready` - `B01` and `B02` correctly marked `needs_review` - Output JSON validated successfully. --- STDERR ---
#!/usr/bin/env python3
"""Conservatively audit account-onboarding records from independent systems."""
import json
import sys
from datetime import datetime
from decimal import Decimal, InvalidOperation
from pathlib import Path
def empty_result(case):
return {
"id": case.get("id") if isinstance(case, dict) else None,
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def document_rank(document):
"""Return a comparable current-document rank, or None if it is ambiguous."""
version = document.get("version")
effective = document.get("effective")
if isinstance(version, bool) or version is None or not isinstance(effective, str):
return None
try:
version = Decimal(str(version))
# ISO dates sort reliably only after they have been validated.
date = datetime.fromisoformat(effective.replace("Z", "+00:00"))
except (InvalidOperation, ValueError, TypeError):
return None
return version, date
def current_document(documents, account_type):
"""Choose the sole document, or the unambiguous newest version for a type."""
matching = [
doc for doc in documents
if isinstance(doc, dict)
and doc.get("type") == account_type
and isinstance(doc.get("name"), str)
and doc["name"].strip()
]
if len(matching) == 1:
return matching[0]
if not matching:
return None
ranked = [(document_rank(doc), doc) for doc in matching]
if any(rank is None for rank, _ in ranked):
return None
newest = max(rank for rank, _ in ranked)
selected = [doc for rank, doc in ranked if rank == newest]
return selected[0] if len(selected) == 1 else None
def audit_case(case):
result = empty_result(case)
if not isinstance(case, dict):
return result
intake = case.get("intake")
crm = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not all((isinstance(intake, dict), isinstance(crm, list),
isinstance(documents, list), isinstance(portal, dict))):
return result
required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if intake.get("signed") is not True or any(not intake.get(key) for key in required_intake):
return result
account_type = intake["account_type"]
# A single active CRM account must independently agree with every CRM-held fact.
matches = [
account for account in crm
if isinstance(account, dict)
and account.get("id")
and account.get("status") == "active"
and account.get("household") == intake["household"]
and account.get("type") == account_type
and account.get("owner") == intake["owner"]
and account.get("risk") == intake["risk_profile"]
]
if len(matches) != 1:
return result
document = current_document(documents, account_type)
if document is None or document.get("signed") is not True:
return result
supported = portal.get("supported_types")
form_map = portal.get("forms")
if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
return result
forms = form_map.get(account_type)
if (not isinstance(forms, list) or not forms
or any(not isinstance(form, str) or not form.strip() for form in forms)):
return result
result.update({
"status": "ready",
"account_id": matches[0]["id"],
"document": document["name"],
"forms": forms,
})
return result
def main():
source = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
with source.open(encoding="utf-8") as file:
cases = json.load(file)
# Accept either the direct case list or a common object wrapper without guessing IDs.
if isinstance(cases, dict):
cases = cases.get("cases")
if not isinstance(cases, list):
raise ValueError("case file must contain a list of cases or an object with a 'cases' list")
with Path("audit_report.json").open("w", encoding="utf-8") as file:
json.dump([audit_case(case) for case in cases], file, indent=2)
file.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
ed0a7226a952121342dd3713b0256e8338748539af1e6f58966725b26bbe91feYou are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-2. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Run it against training_cases.json before finishing.
Implemented `solution.py` and generated `audit_report.json`. The audit is conservative and marks a case ready only when it finds: - A complete, signed intake - Exactly one matching active CRM account - Exactly one matching signed document - Portal support and a non-empty, unambiguous form list Verified with: ```bash python3 solution.py training_cases.json python3 -m py_compile solution.py ``` Training results: A01–A12 are `ready`; B01 and B02 are `needs_review`. --- STDERR ---
#!/usr/bin/env python3
"""Audit cross-system evidence for client onboarding cases."""
import copy
import json
import sys
from pathlib import Path
from typing import Any
EMPTY_DECISION = {
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def is_present(value: Any) -> bool:
return value is not None and (not isinstance(value, str) or bool(value.strip()))
def decision_for(case: Any) -> dict[str, Any]:
result = {"id": case.get("id") if isinstance(case, dict) else None, **EMPTY_DECISION}
if not isinstance(case, dict) or not is_present(result["id"]):
return result
intake = case.get("intake")
crm_records = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not isinstance(intake, dict) or not isinstance(crm_records, list):
return result
if not isinstance(documents, list) or not isinstance(portal, dict):
return result
required_intake_fields = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if intake.get("signed") is not True or any(not is_present(intake.get(field)) for field in required_intake_fields):
return result
account_type = intake["account_type"]
matching_accounts = [
record for record in crm_records
if isinstance(record, dict)
and record.get("status") == "active"
and is_present(record.get("id"))
and record.get("household") == intake["household"]
and record.get("type") == account_type
and record.get("owner") == intake["owner"]
and record.get("risk") == intake["risk_profile"]
]
if len(matching_accounts) != 1:
return result
matching_documents = [
document for document in documents
if isinstance(document, dict)
and is_present(document.get("name"))
and document.get("type") == account_type
and document.get("signed") is True
]
if len(matching_documents) != 1:
return result
supported_types = portal.get("supported_types")
forms_by_type = portal.get("forms")
if not isinstance(supported_types, list) or account_type not in supported_types:
return result
if not isinstance(forms_by_type, dict):
return result
forms = forms_by_type.get(account_type)
if not isinstance(forms, list) or not forms or any(not is_present(form) for form in forms):
return result
# Duplicate form identifiers make the requested portal action ambiguous.
if len({json.dumps(form, sort_keys=True, default=str) for form in forms}) != len(forms):
return result
result.update(
status="ready",
account_id=matching_accounts[0]["id"],
document=matching_documents[0]["name"],
forms=copy.deepcopy(forms),
)
return result
def load_cases(path: Path) -> list[Any]:
with path.open(encoding="utf-8") as source:
payload = json.load(source)
if isinstance(payload, list):
return payload
if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
return payload["cases"]
raise ValueError("case file must contain a JSON array or an object with a 'cases' array")
def main() -> None:
input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
report = [decision_for(case) for case in load_cases(input_path)]
with Path("audit_report.json").open("w", encoding="utf-8") as output:
json.dump(report, output, indent=2)
output.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
fc9fc7e934cc64fdf368083fc3b69600889c8d05d8ff6c575215efa0c9c4794aYou are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-3. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Run it against training_cases.json before finishing.
Implemented the reusable onboarding audit in [`solution.py`](solution.py).
It:
- Reads `argv[1]`, defaulting to `training_cases.json`
- Writes `audit_report.json`
- Requires corroborated signed intake, exactly one matching active CRM account, a signed matching document, and supported non-empty portal forms
- Handles list payloads and `{ "cases": [...] }` payloads
- Uses conservative review decisions for ambiguity or incomplete/conflicting evidence
Validated with:
```sh
python3 solution.py training_cases.json
python3 -m py_compile solution.py
```
Training output marks `A01` through `A12` as `ready`, and ambiguous/closed CRM cases `B01` and `B02` as `needs_review`.
--- STDERR ---
#!/usr/bin/env python3
"""Conservative, data-driven audit for client onboarding cases."""
import json
import sys
from datetime import date
from pathlib import Path
def text(value):
"""Return a comparison-safe string, or None for absent/blank values."""
if not isinstance(value, str):
return None
value = " ".join(value.split())
return value.casefold() if value else None
def same(left, right):
return text(left) is not None and text(left) == text(right)
def review(case_id):
return {
"id": case_id,
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def document_rank(document):
"""Rank explicitly versioned/effective documents without assuming a version."""
version = document.get("version")
if isinstance(version, bool):
version = None
if isinstance(version, (int, float)):
version_key = (2, version)
elif isinstance(version, str) and text(version) is not None:
version_key = (1, text(version))
else:
version_key = (0, "")
effective = document.get("effective")
if isinstance(effective, str):
try:
effective_key = date.fromisoformat(effective).isoformat()
except ValueError:
effective_key = ""
else:
effective_key = ""
return version_key, effective_key
def audit(case):
case_id = case.get("id") if isinstance(case, dict) else None
result = review(case_id)
if not isinstance(case, dict):
return result
intake = case.get("intake")
crm = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not isinstance(intake, dict) or not isinstance(crm, list):
return result
if not isinstance(documents, list) or not isinstance(portal, dict):
return result
required = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if any(text(intake.get(field)) is None for field in required):
return result
if intake.get("signed") is not True:
return result
account_type = intake["account_type"]
# An account is actionable only when exactly one active CRM record corroborates
# every piece of identity/risk evidence supplied by the intake.
matching_accounts = []
for account in crm:
if not isinstance(account, dict) or text(account.get("id")) is None:
continue
if (
same(account.get("household"), intake["household"])
and same(account.get("type"), account_type)
and same(account.get("owner"), intake["owner"])
):
# A disagreeing record for the same client/account identity is a conflict,
# not something to silently bypass.
if not same(account.get("risk"), intake["risk_profile"]):
return result
if text(account.get("status")) == "active":
matching_accounts.append(account)
if len(matching_accounts) != 1:
return result
valid_documents = []
for document in documents:
if not isinstance(document, dict):
continue
if not (
text(document.get("name")) is not None
and same(document.get("type"), account_type)
and document.get("signed") is True
):
continue
# When a document carries corroborating client fields, they must agree.
aliases = {
"household": "household",
"owner": "owner",
"risk": "risk_profile",
"risk_profile": "risk_profile",
}
if any(
key in document and not same(document[key], intake[intake_key])
for key, intake_key in aliases.items()
):
continue
valid_documents.append(document)
if not valid_documents:
return result
best_rank = max(document_rank(item) for item in valid_documents)
best_documents = [item for item in valid_documents if document_rank(item) == best_rank]
# Equal highest-ranked documents cannot be distinguished safely.
if len(best_documents) != 1:
return result
supported = portal.get("supported_types")
form_map = portal.get("forms")
if not isinstance(supported, list) or not isinstance(form_map, dict):
return result
if not any(same(item, account_type) for item in supported):
return result
form_keys = [key for key in form_map if same(key, account_type)]
if len(form_keys) != 1:
return result
forms = form_map[form_keys[0]]
if not isinstance(forms, list) or not forms or any(text(form) is None for form in forms):
return result
return {
"id": case_id,
"status": "ready",
"account_id": matching_accounts[0]["id"],
"document": best_documents[0]["name"],
"forms": forms,
}
def load_cases(payload):
if isinstance(payload, list):
return payload
if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
return payload["cases"]
raise ValueError("case file must contain a list or an object with a 'cases' list")
def main():
input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
with input_path.open(encoding="utf-8") as handle:
cases = load_cases(json.load(handle))
report = [audit(case) for case in cases]
with Path("audit_report.json").open("w", encoding="utf-8") as handle:
json.dump(report, handle, indent=2)
handle.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
12ccc31f8150091b8d66edf11e7a5af05a357a3ca4fb748240a5a848db019f78You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-4. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Run it against training_cases.json before finishing.
Implemented `solution.py` and ran it against `training_cases.json`. Validation passed: - 14 cases processed - 12 `ready` - 2 `needs_review`: `B01` (ambiguous active CRM accounts), `B02` (no active CRM account) - `python3 -m py_compile solution.py` passed Generated `audit_report.json`. --- STDERR ---
"""Conservative, schema-tolerant audit for client-onboarding case files."""
import json
import sys
from datetime import date, datetime
from pathlib import Path
def text(value):
"""Return a non-blank string, or None. IDs and labels must be explicit."""
return value.strip() if isinstance(value, str) and value.strip() else None
def same(left, right):
return text(left) is not None and text(left) == text(right)
def document_rank(document):
"""Rank revisions without assuming a particular document version number."""
version = document.get("version")
if not isinstance(version, (int, float)) or isinstance(version, bool):
version = -1
effective = document.get("effective")
try:
# ISO dates sort correctly, but parse them to reject malformed evidence.
parsed = datetime.fromisoformat(effective.replace("Z", "+00:00")) if isinstance(effective, str) else None
effective_rank = parsed.isoformat() if parsed else ""
except ValueError:
effective_rank = ""
return (version, effective_rank)
def empty_result(case_id):
return {
"id": case_id,
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def audit_case(case):
case_id = case.get("id") if isinstance(case, dict) else None
result = empty_result(case_id)
if not isinstance(case, dict):
return result
intake = case.get("intake")
portal = case.get("portal")
crm_records = case.get("crm")
documents = case.get("documents")
if not isinstance(intake, dict) or not isinstance(portal, dict):
return result
if not isinstance(crm_records, list) or not isinstance(documents, list):
return result
account_type = text(intake.get("account_type"))
required_intake = ("household", "owner", "risk_profile", "beneficiary")
if not account_type or intake.get("signed") is not True:
return result
if any(text(intake.get(field)) is None for field in required_intake):
return result
# An onboarding action needs one, and only one, active CRM account that
# agrees with all identity and suitability evidence supplied by intake.
matches = [
record for record in crm_records
if isinstance(record, dict)
and text(record.get("id"))
and record.get("status") == "active"
and same(record.get("household"), intake.get("household"))
and same(record.get("type"), account_type)
and same(record.get("owner"), intake.get("owner"))
and same(record.get("risk"), intake.get("risk_profile"))
]
if len(matches) != 1:
return result
supported_types = portal.get("supported_types")
forms_by_type = portal.get("forms")
if not isinstance(supported_types, list) or not isinstance(forms_by_type, dict):
return result
if account_type not in supported_types:
return result
forms = forms_by_type.get(account_type)
if not isinstance(forms, list) or not forms or any(text(form) is None for form in forms):
return result
candidates = [
document for document in documents
if isinstance(document, dict)
and text(document.get("name"))
and same(document.get("type"), account_type)
and document.get("signed") is True
]
if not candidates:
return result
# Multiple historical copies are normal. Use the newest declared revision,
# but never guess when equally current, differently named documents conflict.
best_rank = max(document_rank(document) for document in candidates)
best = [document for document in candidates if document_rank(document) == best_rank]
if len(best) != 1:
return result
result.update({
"status": "ready",
"account_id": matches[0]["id"],
"document": best[0]["name"],
"forms": forms,
})
return result
def main():
input_path = Path(sys.argv[1] if len(sys.argv) > 1 else "training_cases.json")
with input_path.open(encoding="utf-8") as source:
cases = json.load(source)
if not isinstance(cases, list):
raise ValueError("Case file must contain a JSON array")
report = [audit_case(case) for case in cases]
with Path("audit_report.json").open("w", encoding="utf-8") as output:
json.dump(report, output, indent=2)
output.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
3d2d41ec2ca2b3d89635b8e531ce6c66358b55c893f53040ac964f47a5ab069bYou are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/root-5. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Run it against training_cases.json before finishing.
Implemented `solution.py` and generated `audit_report.json`. The audit: - Reads `argv[1]`, defaulting to `training_cases.json` - Reconciles signed intake, one active matching CRM account, signed current document, and portal support/forms - Marks ambiguous or incomplete cases `needs_review` with `null`/empty action fields - Handles normal document revisions by selecting a uniquely newest signed revision Validation completed: ```text python3 solution.py training_cases.json python3 -m py_compile solution.py ``` Training output marks `A01` through `A12` ready, and duplicate/closed CRM cases `B01` and `B02` as needing review. --- STDERR ---
"""Audit synthetic client-onboarding cases across intake, CRM, documents, and portal data."""
import json
import sys
from datetime import date
from pathlib import Path
def nonempty(value):
"""Return true for a non-blank scalar value (zero is a valid identifier)."""
return value is not None and not isinstance(value, (dict, list)) and str(value).strip() != ""
def iso_date(value):
"""Accept only a real ISO calendar date when a document supplies one."""
if not isinstance(value, str) or not value:
return False
try:
date.fromisoformat(value)
except ValueError:
return False
return True
def review(case_id):
return {
"id": case_id,
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def audit_case(case):
case_id = case.get("id") if isinstance(case, dict) else None
result = review(case_id)
if not isinstance(case, dict):
return result
intake = case.get("intake")
crm = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not all(isinstance(value, expected) for value, expected in (
(intake, dict), (crm, list), (documents, list), (portal, dict)
)):
return result
# These identify the requested account. A missing value cannot be reconciled
# safely, even if a CRM record happens to contain a similar value.
required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if intake.get("signed") is not True or any(not nonempty(intake.get(key)) for key in required_intake):
return result
account_type = intake["account_type"]
identity_keys = (("household", "household"), ("type", "account_type"), ("owner", "owner"), ("risk", "risk_profile"))
matching_accounts = [
record for record in crm
if isinstance(record, dict)
and record.get("status") == "active"
and all(record.get(crm_key) == intake[intake_key] for crm_key, intake_key in identity_keys)
and nonempty(record.get("id"))
]
# More than one exact CRM account is not a deterministic onboarding action.
if len(matching_accounts) != 1:
return result
account = matching_accounts[0]
supported = portal.get("supported_types")
form_map = portal.get("forms")
if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
return result
forms = form_map.get(account_type)
if (
not isinstance(forms, list)
or not forms
or any(not isinstance(form, str) or not form.strip() for form in forms)
or len(set(forms)) != len(forms)
):
return result
# An intake that names a beneficiary needs the portal's beneficiary form.
if intake["beneficiary"] and "BENEFICIARY" not in forms:
return result
eligible_documents = [
document for document in documents
if isinstance(document, dict)
and document.get("type") == account_type
and document.get("signed") is True
and nonempty(document.get("name"))
and iso_date(document.get("effective"))
and isinstance(document.get("version"), int)
and not isinstance(document.get("version"), bool)
and document["version"] > 0
]
if not eligible_documents:
return result
# Revisions are normal. Select only a uniquely newest signed revision; equal
# revisions with different documents are an ambiguity for a human to resolve.
newest_version = max(document["version"] for document in eligible_documents)
newest = [document for document in eligible_documents if document["version"] == newest_version]
if len(newest) != 1:
return result
document = newest[0]
result.update({
"status": "ready",
"account_id": account["id"],
"document": document["name"],
"forms": forms,
})
return result
def load_cases(payload):
"""Support a bare case array and common object wrappers without guessing records."""
if isinstance(payload, list):
return payload
if isinstance(payload, dict):
for key in ("cases", "records", "items"):
if isinstance(payload.get(key), list):
return payload[key]
raise ValueError("input must be a list of cases or an object containing cases")
def main():
input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
with input_path.open(encoding="utf-8") as source:
cases = load_cases(json.load(source))
report = [audit_case(case) for case in cases]
with Path("audit_report.json").open("w", encoding="utf-8") as destination:
json.dump(report, destination, indent=2)
destination.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
36a53c91a8fc208ec967295dc77a0f887a2e8b0b5ce1613e3c4d36a2490c8063You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/fixed-child. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. Review the observed training failures and improve generalization. Run it against training_cases.json before finishing.
Implemented and ran `solution.py`. - Reads input from `argv[1]`, defaulting to `training_cases.json`. - Writes `audit_report.json`. - Requires consistent, signed intake, exactly one matching active CRM account, a signed/current typed document, portal support, and valid unique forms before marking `ready`. - Returns null/empty action fields for ambiguous or unsupported cases. - Handles malformed input conservatively. Validation passed on `training_cases.json`: 12 `ready`, 2 `needs_review`. --- STDERR ---
#!/usr/bin/env python3
"""Conservatively audit account-onboarding records from independent systems."""
import json
import sys
from datetime import datetime
from decimal import Decimal, InvalidOperation
from pathlib import Path
def empty_result(case):
return {
"id": case.get("id") if isinstance(case, dict) else None,
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def nonempty_text(value):
"""A required identity value must be an explicit, usable text value."""
return isinstance(value, str) and bool(value.strip())
def document_rank(document):
"""Return a comparable current-document rank, or None if it is ambiguous."""
version = document.get("version")
effective = document.get("effective")
if isinstance(version, bool) or version is None or not nonempty_text(effective):
return None
try:
version = Decimal(str(version))
# ISO dates sort reliably only after they have been validated.
date = datetime.fromisoformat(effective.replace("Z", "+00:00")).date()
except (InvalidOperation, ValueError, TypeError):
return None
if not version.is_finite():
return None
return version, date
def current_document(documents, account_type):
"""Choose the sole document, or the unambiguous newest version for a type."""
matching = [
doc for doc in documents
if isinstance(doc, dict)
and doc.get("type") == account_type
and nonempty_text(doc.get("name"))
]
if not matching:
return None
ranked = [(document_rank(doc), doc) for doc in matching]
if any(rank is None for rank, _ in ranked):
return None
newest = max(rank for rank, _ in ranked)
selected = [doc for rank, doc in ranked if rank == newest]
return selected[0] if len(selected) == 1 else None
def audit_case(case):
result = empty_result(case)
if not isinstance(case, dict):
return result
intake = case.get("intake")
crm = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not all((isinstance(intake, dict), isinstance(crm, list),
isinstance(documents, list), isinstance(portal, dict))):
return result
required_intake = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if intake.get("signed") is not True or any(not nonempty_text(intake.get(key)) for key in required_intake):
return result
account_type = intake["account_type"]
# A single active CRM account must independently agree with every CRM-held fact.
matches = [
account for account in crm
if isinstance(account, dict)
and nonempty_text(account.get("id"))
and account.get("status") == "active"
and account.get("household") == intake["household"]
and account.get("type") == account_type
and account.get("owner") == intake["owner"]
and account.get("risk") == intake["risk_profile"]
]
if len(matches) != 1:
return result
document = current_document(documents, account_type)
if document is None or document.get("signed") is not True:
return result
supported = portal.get("supported_types")
form_map = portal.get("forms")
if not isinstance(supported, list) or account_type not in supported or not isinstance(form_map, dict):
return result
forms = form_map.get(account_type)
if (not isinstance(forms, list) or not forms
or any(not nonempty_text(form) for form in forms)
or len(set(forms)) != len(forms)):
return result
result.update({
"status": "ready",
"account_id": matches[0]["id"],
"document": document["name"],
"forms": forms,
})
return result
def main():
source = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
with source.open(encoding="utf-8") as file:
cases = json.load(file)
# Accept either the direct case list or a common object wrapper without guessing IDs.
if isinstance(cases, dict):
cases = cases.get("cases")
if not isinstance(cases, list):
raise ValueError("case file must contain a list of cases or an object with a 'cases' list")
with Path("audit_report.json").open("w", encoding="utf-8") as file:
json.dump([audit_case(case) for case in cases], file, indent=2)
file.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
b184477aacb9e3750f88607e693745f41d2b4de5bf0c660cc07503560d88a393You are a bounded coding agent working only in /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839/replay-child. No network, credentials, parent directories, or answer keys. You are given training_cases.json from a synthetic multi-system client onboarding process. Build a GENERAL reusable audit program in solution.py. It must read a JSON case file path from argv[1] (default training_cases.json) and write audit_report.json with one decision per case: id, status (ready or needs_review), account_id, document, forms. Use the evidence in each case: intake, CRM records, documents, and portal metadata. Be conservative: only mark ready when the evidence supports it; otherwise needs_review with null/empty action fields. Do not hardcode case IDs or assume the training file is the only shape. The evaluator will run your program against unseen cases. A replay controller selected this parent because its recorded validation score was strongest. Improve generalization without hardcoding. Run it against training_cases.json before finishing.
Implemented and ran `solution.py`.
- Reads argv path or defaults to `training_cases.json`
- Writes `audit_report.json`
- Validates signed intake, one matching active CRM account, one signed matching document, supported portal type, and usable unique portal forms
- Returns conservative empty actions for ambiguous or incomplete evidence
- Supports both a top-level list and `{"cases": [...]}` input
Verification passed on training data: **12 ready, 2 needs_review** (`B01`, `B02`). Also checked malformed portal-form evidence.
--- STDERR ---
#!/usr/bin/env python3
"""Audit cross-system evidence for client onboarding cases."""
import copy
import json
import sys
from pathlib import Path
from typing import Any
EMPTY_DECISION = {
"status": "needs_review",
"account_id": None,
"document": None,
"forms": [],
}
def is_present(value: Any) -> bool:
"""Return whether a scalar value can safely identify an audit artifact."""
if value is None or isinstance(value, bool):
return False
if isinstance(value, str):
return bool(value.strip())
return not isinstance(value, (list, dict))
def is_nonempty_text(value: Any) -> bool:
return isinstance(value, str) and bool(value.strip())
def decision_for(case: Any) -> dict[str, Any]:
result = {"id": case.get("id") if isinstance(case, dict) else None, **EMPTY_DECISION}
if not isinstance(case, dict) or not is_present(result["id"]):
return result
intake = case.get("intake")
crm_records = case.get("crm")
documents = case.get("documents")
portal = case.get("portal")
if not isinstance(intake, dict) or not isinstance(crm_records, list):
return result
if not isinstance(documents, list) or not isinstance(portal, dict):
return result
# These are identity and suitability fields, so accept no implicit coercion
# (for example, an empty string or a JSON object is not a known owner).
required_intake_fields = ("household", "account_type", "owner", "risk_profile", "beneficiary")
if intake.get("signed") is not True or any(
not is_nonempty_text(intake.get(field)) for field in required_intake_fields
):
return result
account_type = intake["account_type"]
matching_accounts = [
record for record in crm_records
if isinstance(record, dict)
and record.get("status") == "active"
and is_present(record.get("id"))
and record.get("household") == intake["household"]
and record.get("type") == account_type
and record.get("owner") == intake["owner"]
and record.get("risk") == intake["risk_profile"]
]
if len(matching_accounts) != 1:
return result
matching_documents = [
document for document in documents
if isinstance(document, dict)
and is_nonempty_text(document.get("name"))
and document.get("type") == account_type
and document.get("signed") is True
]
if len(matching_documents) != 1:
return result
supported_types = portal.get("supported_types")
forms_by_type = portal.get("forms")
if not isinstance(supported_types, list) or account_type not in supported_types:
return result
if not isinstance(forms_by_type, dict):
return result
forms = forms_by_type.get(account_type)
if not isinstance(forms, list) or not forms or any(not is_nonempty_text(form) for form in forms):
return result
# Duplicate form identifiers make the requested portal action ambiguous.
if len({json.dumps(form, sort_keys=True, default=str) for form in forms}) != len(forms):
return result
result.update(
status="ready",
account_id=matching_accounts[0]["id"],
document=matching_documents[0]["name"],
forms=copy.deepcopy(forms),
)
return result
def load_cases(path: Path) -> list[Any]:
with path.open(encoding="utf-8") as source:
payload = json.load(source)
if isinstance(payload, list):
return payload
if isinstance(payload, dict) and isinstance(payload.get("cases"), list):
return payload["cases"]
raise ValueError("case file must contain a JSON array or an object with a 'cases' array")
def main() -> None:
input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("training_cases.json")
report = [decision_for(case) for case in load_cases(input_path)]
with Path("audit_report.json").open("w", encoding="utf-8") as output:
json.dump(report, output, indent=2)
output.write("\n")
if __name__ == "__main__":
main()
[
{
"id": "A01",
"status": "ready",
"account_id": "A01-A",
"document": "a01_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A02",
"status": "ready",
"account_id": "A02-A",
"document": "a02_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A03",
"status": "ready",
"account_id": "A03-A",
"document": "a03_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A04",
"status": "ready",
"account_id": "A04-A",
"document": "a04_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A05",
"status": "ready",
"account_id": "A05-A",
"document": "a05_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A06",
"status": "ready",
"account_id": "A06-A",
"document": "a06_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A07",
"status": "ready",
"account_id": "A07-A",
"document": "a07_sep_ira_v2.pdf",
"forms": [
"SEP IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A08",
"status": "ready",
"account_id": "A08-A",
"document": "a08_trust_v2.pdf",
"forms": [
"Trust-OPEN",
"BENEFICIARY"
]
},
{
"id": "A09",
"status": "ready",
"account_id": "A09-A",
"document": "a09_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A10",
"status": "ready",
"account_id": "A10-A",
"document": "a10_roth_ira_v2.pdf",
"forms": [
"Roth IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "A11",
"status": "ready",
"account_id": "A11-A",
"document": "a11_joint_v2.pdf",
"forms": [
"Joint-OPEN",
"BENEFICIARY"
]
},
{
"id": "A12",
"status": "ready",
"account_id": "A12-A",
"document": "a12_ira_v2.pdf",
"forms": [
"IRA-OPEN",
"BENEFICIARY"
]
},
{
"id": "B01",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
},
{
"id": "B02",
"status": "needs_review",
"account_id": null,
"document": null,
"forms": []
}
]
This is larger and more realistic in data shape, but it still does not show a replay advantage: both policies reached 100% on the 12-case holdout. The next realism increase should use actual PDFs/images, tool failures, partial data, and a cost-limited target so a policy cannot simply rely on one successful repair.
Raw run: /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/runs/scale-20260921-000839
JSON: /Users/christianroncal/humanitylabs/pi-dream-rsi-poc/report/scale-report.json