Phase 1 confidence-write path is dormant — no prompt instructs the agent to set confidence #54

New issue

Closed

opened 2026-04-07 05:26:12 +00:00 by archeious · 0 comments

archeious commented

2026-04-07 05:26:12 +00:00

Owner

Phase 1 is marked shipped but only the read side is wired up.

cache.py:115 validates an optional confidence field on cache writes
cache.py:191 provides low_confidence_entries(threshold=0.7)
No prompt in prompts.py instructs the agent to set confidence when calling write_cache

Result: every entry has missing confidence, low_confidence_entries() returns everything (since e.get("confidence", 0.0) < 0.7 is always true), and Phase 8 (refinement) will have no useful signal to prioritize from when it lands.

Fix

Update _DIR_SYSTEM_PROMPT to instruct the agent to set confidence (0.0–1.0) and confidence_reason on every write_cache call, with calibration guidance (e.g. high = read the file end-to-end and understood it; low = inferred from filename only)
Update the write_cache tool description in _DIR_TOOLS (ai.py:228) to mention the confidence fields
Verify on a real --ai run that confidence values are landing in cache entries

Notes

Categorical (high/medium/low → 0.9/0.6/0.3) is probably more reliable than asking for free-form floats — see PLAN.md Known Unknowns → Confidence calibration
Discovered during Session 5 documentation deep dive (#53)
Should land before Phase 8 (refinement) so that pass has a real signal to consume

Phase 1 is marked shipped but only the read side is wired up. - `cache.py:115` validates an optional `confidence` field on cache writes - `cache.py:191` provides `low_confidence_entries(threshold=0.7)` - **No prompt in `prompts.py` instructs the agent to set confidence when calling `write_cache`** Result: every entry has missing confidence, `low_confidence_entries()` returns everything (since `e.get("confidence", 0.0) < 0.7` is always true), and Phase 8 (refinement) will have no useful signal to prioritize from when it lands. ## Fix - Update `_DIR_SYSTEM_PROMPT` to instruct the agent to set `confidence` (0.0–1.0) and `confidence_reason` on every `write_cache` call, with calibration guidance (e.g. high = read the file end-to-end and understood it; low = inferred from filename only) - Update the `write_cache` tool description in `_DIR_TOOLS` (`ai.py:228`) to mention the confidence fields - Verify on a real `--ai` run that confidence values are landing in cache entries ## Notes - Categorical (high/medium/low → 0.9/0.6/0.3) is probably more reliable than asking for free-form floats — see PLAN.md `Known Unknowns → Confidence calibration` - Discovered during Session 5 documentation deep dive (#53) - Should land before Phase 8 (refinement) so that pass has a real signal to consume