Table of Contents

Session 5

What was done

New wiki page
Wiki fixes
PLAN.md fixes (main repo)

Discoveries and observations
Decisions
Raw thinking
What's next

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Session 5

Date: 2026-04-06 Focus: Documentation deep dive — orient a new developer to the project before Phase 3 begins Duration estimate: ~1 hour

What was done

Issue #53 — Developer onboarding: deep-dive code tour + reconcile architecture/roadmap docs. Single branch docs/issue-53-onboarding-internals, merged in commit 8c0e29b.

New wiki page

Internals.md (~500 lines) — code-level walkthrough of how Luminos actually works, with file:line references throughout. Sections: the two layers, base scan walkthrough, AI pipeline orchestrator, survey pass, dir loop in depth (message history, tool dispatch, pre-loaded context, token tracker, budget check, partial flush), cache model, prompts, synthesis pass, flags, where-to-make-common-changes cookbook (add a tool / add a pass / change a prompt / change cache schema / add a CLI flag), token budget and cost, glossary. Targeted at a developer who knows basic Python but has never built an agent loop.

Wiki fixes

Architecture.md — added Internals link at top; rewrote the AI pipeline diagram to include _run_investigation, the survey pass, _filter_dir_tools, and the partial-flush early-exit; fixed the cache section which incorrectly described file/dir entries as JSONL (they are sha256-keyed JSON files; only flags.jsonl and investigation.log are JSONL); added partial/partial_reason to the dir entry schema; documented investigations.json; added a note about _TokenTracker and the per-call (not cumulative) budget logic.
Roadmap.md — deleted and replaced. The previous version still listed Phase 1 and Phase 2 as "Not started" months after they shipped, and was missing Phases 2.5, 3.5, and 4.5 entirely. New version is a thin pointer to PLAN.md and open issues, plus a status-at-a-glance table. Three sources of truth (CLAUDE.md / PLAN.md / Roadmap.md) collapsed to two.
DevelopmentGuide.md — added a header note pointing to Internals for code-level questions. Otherwise unchanged.
Home.md — added Internals to the quick-links table.

PLAN.md fixes (main repo)

Added a status snapshot at the top with the same phase table as Roadmap.md.
Fixed the "File Map" section: it claimed a luminos_lib/domain.py would be created for the survey pass, but domain.py was never created — survey logic landed inside ai.py (_run_survey, _format_survey_block, _filter_dir_tools). Marked the row as struck-through with a note explaining what actually happened, and tagged prompts.py and ai.py rows with ✅ for survey work that's already done.

Discoveries and observations

Architecture.md cache section was wrong. I had assumed JSONL based on the wiki, but reading cache.py:106–141 showed sha256-keyed JSON files in files/ and dirs/ subdirs. The reason for the design (O(1) has_entry() check) is genuinely good and worth documenting — it's now in Internals §5.4. This is the kind of thing a deep-dive doc pass catches that incremental edits never would.
PLAN.md File Map drift is structural, not editorial. PLAN.md was written before implementation. The implementation rationally chose a different file layout. The plan was never updated. This will keep happening to PLAN.md as long as it pretends to be both a design doc and a current-state doc. The fix in this session was minimal (a struck-through row + a note at the top), but the broader question — how to keep PLAN.md from rotting — remains.
The survey pass is the cleanest reference implementation in the codebase. It's short, single-purpose, has a clear submit-tool exit, degrades gracefully on failure. Any new pass (planning, refinement) should be modeled on it. Internals §9.2 says exactly this.
_call_api_streaming doesn't actually stream anything to the user. It uses the streaming endpoint but discards events and pulls the final message at the end. There's a latent capability to print live progress that nobody is using. Not a bug — just a thing I noticed.
Confidence support exists in the cache but no prompt sets it. cache.py:191 has low_confidence_entries(threshold=0.7) ready to go. The agent never writes confidence values because no prompt instructs it to. Phase 1 was marked "shipped" but the wiring is half-finished — the schema is there, the read side is there, the write side is dormant. Worth a follow-up issue.

Decisions

One bundled PR for all the doc work, not split per file. The changes are tightly coupled (Architecture cache section ⇄ Internals cache section ⇄ PLAN.md status ⇄ Roadmap status table) and reviewing them together is easier than reviewing them separately.
Replace Roadmap.md, don't fix it. The user explicitly chose this. Fixing it in place just resets the clock until the next drift; replacing it with a pointer to two living sources (PLAN.md design + issues status) removes the failure mode entirely.
Internals.md targets "basic Python, no agent-loop background" per the user. This shaped the depth of explanation in §3 (orchestrator) and §4 (dir loop). I expanded the message-history-grows-monotonically point because that's exactly the kind of thing a non-agent-loop person would not anticipate.
Out of scope on purpose: no code changes, no PLAN.md restructuring beyond status + file-map fix, no fix for the dormant confidence-write path. All confined to issue #53; anything that came up gets a follow-up issue or a Raw Thinking note here.

Raw thinking

Phase 1 completion is technically a lie. The cache schema accepts confidence; low_confidence_entries() exists; the prompt never asks the agent to set it; therefore no entries ever have it; therefore low_confidence_entries() returns everything (since e.get("confidence", 0.0) < 0.7 is always true for missing-confidence entries). The function works as documented but is currently not useful. This isn't a bug worth fixing in isolation — it'll be wired up when refinement (Phase 8) actually consumes the signal — but Phase 1's "shipped" status hides that the read path is starving.
Internals.md is going to drift. Every file:line reference is a hostage to future commits. I added a disclaimer at the top but disclaimers don't actually keep docs current. A lighter-touch alternative would have been file references without line numbers (e.g. "see _run_dir_loop in ai.py"). The user wanted concrete refs and they're more useful for a new dev — but we should expect to see "Internals.md is stale" issues in a few sessions.
The MCP pivot (Phase 3.5) makes some of Internals.md temporary. Once the dir loop's tools are discovered dynamically from an MCP server, §4.2 (Tool dispatch) and §9.1 (Add a new tool) become wrong. Internals will need a major revision after Phase 3.5 lands. Not a reason to delay the doc — better to have it right now and revise then than to delay until "the architecture stabilizes" (which never happens).
_run_dir_loop is 160 lines and four conceptual layers (budget check + early flush, API call + history append, tool dispatch + result append, done check). It's near its readability ceiling. Phase 3 will add planning-pass-driven turn allocation, which is more state in the same function. Worth a refactor before that lands or the function becomes the kind of thing nobody wants to touch.
There's no test for ai.py because it requires a live API. This is documented as policy. But several pieces of ai.py are pure functions (_filter_dir_tools, _format_survey_block, _format_survey_signals, _default_survey, _should_skip_dir, _path_is_safe) that could absolutely be unit-tested. Worth a follow-up issue to extract these into _test_helpers or just add tests/test_ai_pure.py.
The "tools registered in two places" smell. A new tool needs an entry in _TOOL_DISPATCH and an entry in _DIR_TOOLS. Easy to add one without the other. A small registration helper (@dir_tool(name=..., description=..., schema=...)) would collapse this. Not urgent — there are only ~10 tools — but the size of §9.1 in Internals (5 numbered steps for adding one tool) suggests this is friction.
Roadmap.md is now a 30-line pointer. It's fine but slightly silly as a standalone wiki page. Could be folded into Home.md. Left it standalone for now because external links to wiki/Roadmap would break.

What's next

User-stated next focus: Phase 3 — Investigation planning. The relevant issues are:

#8 Add _PLANNING_SYSTEM_PROMPT to prompts.py
#9 Implement planning pass with submit_plan tool in ai.py
#10 Add dynamic turn allocation from plan output
#11 Save investigation plan to cache for resumed runs

The natural order is #8 → #9 → #10 → #11. #8 and #9 are sized like the survey pass work was — small, single-pass loops. #10 is the harder one because it changes how _run_dir_loop consumes max_turns (currently a constant, would become per-dir from the plan). #11 is mostly a cache schema change.

Follow-ups noticed during this session that should become issues:

Phase 1 confidence-write path is dormant (no prompt sets it)
Pure helpers in ai.py could be unit-tested without a live API
Tool registration is duplicated between _TOOL_DISPATCH and _DIR_TOOLS
_run_dir_loop is approaching its readability ceiling — refactor before Phase 3 adds more state