Architecture

This page is the high-level map. For a code-level walkthrough with file:line references — how the dir loop actually works, where to add a tool, what the cache really stores — read Internals.

Overview

Luminos is an agentic Claude investigation tool. Every invocation runs the full pipeline: a base scan first to feed the agent its initial picture, then a survey pass, a planning pass, per-directory dir loops with dynamic turn allocation, then a final synthesis pass. The base scan is not a standalone product, it is the agent's input.

Entry point: luminos.py — argument parsing, scan orchestration, AI pipeline kickoff, output routing.

Module Map

Module	Purpose	External commands
`luminos.py`	Entry point — arg parsing, scan(), AI kickoff, main()	None
`luminos_lib/tree.py`	Recursive directory tree with file sizes	None (os)
`luminos_lib/filetypes.py`	Classifies files into 7 categories	`file --brief`
`luminos_lib/code.py`	Language detection, LOC counting, large file flagging	`wc -l`
`luminos_lib/recency.py`	Finds N most recently modified files	`find -printf`
`luminos_lib/disk.py`	Per-directory disk usage	`du -b`
`luminos_lib/report.py`	Formats report dict as terminal output	None
`luminos_lib/cache.py`	Investigation cache — read/write/clear/flush	None
`luminos_lib/ast_parser.py`	tree-sitter code structure parsing	tree-sitter
`luminos_lib/prompts.py`	System prompt templates for AI loops	None
`luminos_lib/ai.py`	Multi-pass agentic analysis via Claude API	anthropic, python-magic

Base Scan Data Flow

scan(target)
    build_tree()           → report["tree"], report["tree_rendered"]
    classify_files()       → report["file_categories"], report["classified_files"]
    detect_languages()     → report["languages"], report["lines_of_code"]
    find_large_files()     → report["large_files"]
    find_recent_files()    → report["recent_files"]
    get_disk_usage()       → report["disk_usage"], report["top_directories"]
    └── returns report dict

AI Pipeline

analyze_directory(report, target)
    │
    └── _run_investigation()
            │
            ├── _get_investigation_id()    new UUID, or resume an existing one
            │
            ├── _discover_directories()    find all dirs, sort leaves-first
            │
            ├── _run_survey()              single short loop, max 3 turns
            │       inputs: survey_signals + 2-level tree preview
            │       Tools: submit_survey
            │       output: shared description, approach, relevant_tools,
            │       skip_tools, domain_notes, confidence
            │       (skipped via _default_survey() on tiny targets)
            │
            ├── _filter_dir_tools(survey)  remove skip_tools (if confidence ≥ 0.5)
            │
            ├── _run_planning()            single loop, max 3 turns
            │       inputs: survey output + full tree + file signals
            │       Tools: submit_plan
            │       output: plan dict (priority/shallow/skip dirs,
            │               turn allocations, investigation order)
            │       (skipped on tiny targets or loaded from plan.json
            │        on resumed runs)
            │
            ├── _apply_plan()              sort dirs into bands, build turn map
            │
            ├── per-directory loop (ordered by plan, dynamic max_turns)
            │       _build_dir_context()    list files + sizes + MIME
            │       _get_child_summaries()  read cached child summaries
            │       _format_survey_block()  inject survey context into prompt
            │       _run_dir_loop()         agent loop with budget check on
            │                               every iteration; flushes a partial
            │                               cache entry on budget breach
            │       Tools: read_file, list_directory, run_command,
            │              parse_structure, write_cache, think, checkpoint,
            │              flag, submit_report (with completeness)
            │
            ├── _write_plan_evaluation()   plan_evaluation.json quality metrics
            │
            ├── _run_synthesis()            single loop, max 5 turns
            │       reads all "dir" cache entries
            │       produces brief (2-4 sentences) + detailed (free-form)
            │       Tools: read_cache, list_cache, flag, submit_report
            │       fallback: _synthesize_from_cache() if out of turns
            │
            └── returns (brief, detailed, flags)

Token usage and the context budget are tracked by _TokenTracker in ai.py. The budget check uses the most recent call's input_tokens, not the cumulative sum across turns — see #44 and the Internals page §4.4 for why.

Cache

Location: /tmp/luminos/<investigation_id>/

Layout:

meta.json              investigation metadata
plan.json              planning pass output (cached for resumed runs)
plan_evaluation.json   quality metrics: plan predictions vs outcomes
files/<sha256>.json    one JSON file per cached file entry
dirs/<sha256>.json     one JSON file per cached directory entry
flags.jsonl            JSONL — appended on every flag tool call
investigation.log     JSONL — appended on every tool call

File and dir entries are stored as one sha256-keyed JSON file per entry (not as JSONL) so that has_entry(path) is an O(1) os.path.exists() check rather than a file scan. Only flags.jsonl and investigation.log are JSONL.

File entries (files/<sha256>.json):

{path, relative_path, size_bytes, category, summary, cached_at,
 [confidence], [confidence_reason], [notable], [notable_reason]}

Dir entries (dirs/<sha256>.json):

{path, relative_path, child_count, dominant_category, summary, cached_at,
 [confidence], [confidence_reason], [notable_files],
 [partial], [partial_reason]}

partial: true marks a dir entry written by the budget-breach early-exit path — the agent didn't reach submit_report and the summary was synthesized from already-cached file entries.

Flags (flags.jsonl):

{path, finding, severity}   severity: info | concern | critical

Investigation IDs are persisted in /tmp/luminos/investigations.json keyed by absolute target path. Cache is reused across runs for the same target. --fresh mints a new investigation ID. --clear-cache deletes the entire cache root.

Key Constraints

AI investigation is the product. The base scan exists to feed the agent. There is no --ai flag. AI runs unconditionally on every invocation.
Anthropic API key is required. If ANTHROPIC_API_KEY is unset, luminos exits cleanly (exit 0) with a one-line hint instead of running.
Dependencies installed via requirements.txt. anthropic, tree-sitter + grammars, and python-magic are normal pip dependencies. setup_env.sh creates a venv and installs them.
Subprocess for OS tools. LOC counting, file detection, disk usage, and recency shell out to GNU coreutils. Do not reimplement in pure Python.
Graceful degradation everywhere. Permission denied, subprocess timeouts, individual dir-loop failures — all handled without crashing the run.

AI Model

claude-sonnet-4-20250514

Context budget. 70% of 200,000 tokens (140,000) — Sonnet 4's real context window with a 30% safety margin. The budget is checked against the latest per-call input_tokens reading (the actual size of the context window in use), not the cumulative sum across turns. Early exit flushes partial cache on budget breach. See #44.

Per-loop turn cap. The planning pass assigns each directory a turn budget: priority dirs get 15-20 (capped at 25), shallow dirs get 5, default dirs get 10. This replaced the old fixed max_turns=14. The cap exists to prevent runaway loops when the agent gets stuck. The plan_evaluation.json quality report tracks turns used vs allocated per directory. See Planning Pass for the full design.

Per-loop message history growth. Tool results are appended to the message history and never evicted, so per-turn input_tokens grows roughly linearly across a loop (~1.5-2k per turn observed on codebase targets). At the current caps (max 25 turns for priority dirs) this stays under 200k. Raising caps significantly would expose this further. See #51.

Pricing tracked and reported at end of each run.