archeious/marchwarden
1
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions
marchwarden/docs/stress-tests/M3.3-runs/16-scope.log
Jeff Smith 13215d7ddb docs(stress-tests): M3.3 Phase A — calibration data collection 
Issue #46 (Phase A only — Phase B human rating still pending, issue stays open).

Adds the data-collection half of the calibration milestone:

- scripts/calibration_runner.sh — runs 20 fixed balanced-depth queries
  across 4 categories (factual, comparative, contradiction-prone,
  scope-edge), 5 each, capturing per-run logs to docs/stress-tests/M3.3-runs/.
- scripts/calibration_collect.py — loads every persisted ResearchResult
  under ~/.marchwarden/traces/*.result.json and emits a markdown rating
  worksheet with one row per run. Recovers question text from each
  trace's start event and category from the run-log filename.
- docs/stress-tests/M3.3-rating-worksheet.md — 22 runs (20 calibration
  + caffeine smoke + M3.2 multi-axis), with empty actual_rating columns
  for the human-in-the-loop scoring step.
- docs/stress-tests/M3.3-runs/*.log — runtime logs from the calibration
  runner, kept as provenance. Gitignore updated with an exception
  carving stress-test logs out of the global *.log ignore.

Note: M3.1's 4 runs predate #54 (full result persistence) and so are
unrecoverable to the worksheet — only post-#54 runs have a result.json
sibling. 22 rateable runs is still within the milestone target of 20–30.

Phases B (human rating) and C (analysis + rubric + wiki update) follow
in a later session. This issue stays open until both are done.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 20:21:47 -06:00

321 lines
36 KiB
Text
Raw Permalink Blame History

Researching: What proprietary indexing strategies do high-frequency trading
firms use for order book reconstruction?
{"question": "What proprietary indexing strategies do high-frequency trading firms use for order book reconstruction?", "depth": "balanced", "max_iterations": null, "token_budget": null, "event": "ask_started", "logger": "marchwarden.cli", "level": "info", "timestamp": "2026-04-09T02:11:11.888630Z"}
{"transport": "stdio", "server": "marchwarden-web-researcher", "event": "mcp_server_starting", "logger": "marchwarden.mcp", "level": "info", "timestamp": "2026-04-09T02:11:12.816801Z"}
{"event": "Processing request of type CallToolRequest", "logger": "mcp.server.lowlevel.server", "level": "info", "timestamp": "2026-04-09T02:11:12.829566Z"}
{"question": "What proprietary indexing strategies do high-frequency trading firms use for order book reconstruction?", "depth": "balanced", "max_iterations": 5, "token_budget": 20000, "model_id": "claude-sonnet-4-6", "event": "research_started", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.web", "level": "info", "timestamp": "2026-04-09T02:11:12.871225Z"}
{"step": 1, "decision": "Beginning research: depth=balanced", "question": "What proprietary indexing strategies do high-frequency trading firms use for order book reconstruction?", "context": "", "max_iterations": 5, "token_budget": 20000, "event": "start", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:12.871693Z"}
{"step": 2, "decision": "Starting iteration 1/5", "tokens_so_far": 0, "event": "iteration_start", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:12.872051Z"}
{"step": 9, "decision": "Starting iteration 2/5", "tokens_so_far": 1212, "event": "iteration_start", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:27.416025Z"}
{"step": 16, "decision": "Starting iteration 3/5", "tokens_so_far": 15135, "event": "iteration_start", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:33.632271Z"}
{"step": 23, "decision": "Token budget reached before iteration 4: 35581/20000", "event": "budget_exhausted", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:40.723229Z"}
{"step": 24, "decision": "Beginning synthesis of gathered evidence", "evidence_count": 35, "iterations_run": 3, "tokens_used": 35581, "event": "synthesis_start", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:11:40.723491Z"}
{"step": 25, "decision": "Parsed synthesis JSON successfully", "duration_ms": 72229, "event": "synthesis_complete", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:12:50.638239Z"}
{"step": 42, "decision": "Research complete", "confidence": 0.72, "citation_count": 8, "gap_count": 4, "discovery_count": 4, "total_duration_sec": 101.111, "event": "complete", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.trace", "level": "info", "timestamp": "2026-04-09T02:12:50.639828Z"}
{"confidence": 0.72, "citations": 8, "gaps": 4, "discovery_events": 4, "tokens_used": 70892, "iterations_run": 3, "wall_time_sec": 97.76683187484741, "budget_exhausted": true, "event": "research_completed", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.web", "level": "info", "timestamp": "2026-04-09T02:12:50.639933Z"}
{"error": "[Errno 13] Permission denied: '/home/micro/.marchwarden/costs.jsonl'", "event": "cost_ledger_write_failed", "researcher": "web", "trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "logger": "marchwarden.researcher.web", "level": "warning", "timestamp": "2026-04-09T02:12:50.640430Z"}
{"event": "Processing request of type ListToolsRequest", "logger": "mcp.server.lowlevel.server", "level": "info", "timestamp": "2026-04-09T02:12:50.648897Z"}
{"trace_id": "f4c43973-7cac-4193-a249-cbb1302de4f7", "confidence": 0.72, "citations": 8, "tokens_used": 70892, "wall_time_sec": 97.76683187484741, "event": "ask_completed", "logger": "marchwarden.cli", "level": "info", "timestamp": "2026-04-09T02:12:50.931342Z"}
╭─────────────────────────────────── Answer ───────────────────────────────────╮
│ High-frequency trading firms use several proprietary and semi-documented │
│ indexing strategies for order book reconstruction, though most production │
│ details remain trade secrets. Based on available evidence: │
│ │
│ 1. **Hash Table + Array Hybrid**: The most commonly cited production │
│ approach combines plain arrays (for cache-friendly sequential memory access │
│ minimizing cache misses) with hash tables (for O(1) lookup of specific price │
│ levels). This codesign optimizes both speed and cache locality. [Sources 15, │
│ 16, 28] │
│ │
│ 2. **B-Tree / ISAM Indexing**: The historically significant Island ECN │
│ (1996), built by Josh Levine, used in-memory B-tree indexing via an ISAM │
│ storage engine with zero disk access during matching, achieving O(log N) │
│ access per price level. This is considered the documented proof-of-concept │
│ for production-grade LOB indexing. [Source 29] │
│ │
│ 3. **Hybrid Binary-Linear Search**: A IEEE-documented approach proposes a │
│ simple linear data structure for tracking the order book combined with a │
│ hybrid binary-linear search algorithm to maintain top bid/ask with minimal │
│ latency. [Source 19] │
│ │
│ 4. **ROI Vector (Region-of-Interest Vector)**: Used in backtesting │
│ frameworks like HftBacktest, this approach restricts the active price range │
│ to a bounded region of interest, enabling vector-based O(1) access within │
│ the ROI while avoiding full-book scanning. [Source 25, 35] │
│ │
│ 5. **Lock-Free Concurrent Data Structures**: To handle concurrent updates │
│ without mutex overhead, firms implement lock-free data structures allowing │
│ multiple threads to update the LOB simultaneously. [Sources 15, 16] │
│ │
│ 6. **Event-Driven with Selective Polling Hybrid**: The LOB primarily │
│ operates event-driven but incorporates high-frequency polling for the most │
│ latency-sensitive execution pathways, ensuring sub-microsecond │
│ responsiveness. [Sources 15, 16] │
│ │
│ 7. **Order Record Reuse (Object Pooling)**: Levine's Island engine reused │
│ recently freed order records for new orders—described as 'hugely │
│ important'—a form of memory pooling that avoids allocation overhead during │
│ high-throughput periods. [Source 29] │
│ │
│ 8. **Structural Filtration for Signal Quality**: Recent research (2025) │
│ proposes filtering transient LOB events by order lifetime, update count, or │
│ inter-update delay before indexing, improving directional signal quality │
│ (OBI) extracted from the reconstructed book. [Source 6] │
│ │
│ Notably, red-black trees—frequently cited in academic literature—are rarely │
│ used in production due to poor cache behavior versus simpler arrays at │
│ realistic market depths. The key insight from practitioners is that │
│ algorithmic data structure choice (O(log N) vs O(N)) dominates hardware │
│ investment: a $2M co-location/FPGA upgrade produced no measurable latency │
│ improvement when the underlying order book used a sorted array with O(N) │
│ inserts. [Source 23, 29] │
╰──────────────────────────────────────────────────────────────────────────────╯
Citations
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ # ┃ Title / Locator ┃ Excerpt ┃ Conf ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ 1 │ Matching Engine Architecture: │ Josh Levine built the Island │ 0.95 │
│ │ Why Your Order Book Data │ matching engine in FoxPro for │ │
│ │ Structure Is the Real Latency │ MS-DOS... The order book used │ │
│ │ Bottleneck │ in-memory B-tree indexing via │ │
│ │ https://electronictradinghub. │ an ISAM storage engine. Zero │ │
│ │ com/matching-engine-architect │ disk access during matching. │ │
│ │ ure-why-your-order-book-data- │ Every price level accessed in │ │
│ │ structure-is-the-real-latency │ O(log N) time. Levine's │ │
│ │ -bottleneck/ │ optimization for new-order │ │
│ │ │ entry latency: reuse recently │ │
│ │ │ freed order records for new │ │
│ │ │ orders — a detail he called │ │
│ │ │ 'hugely important' │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 2 │ Optimizing Limit Order Book │ I use a combination of plain │ 0.88 │
│ │ for HFT Systems │ arrays and hash tables to │ │
│ │ https://www.linkedin.com/post │ manage the LOB. Arrays are │ │
│ │ s/silahian_hft-hft-trading-ac │ highly effective with CPU │ │
│ │ tivity-7351226537301417988-ei │ caches, offering sequential │ │
│ │ cX │ memory access that minimizes │ │
│ │ │ cache misses. The integration │ │
│ │ │ of hash tables provides quick │ │
│ │ │ access to specific entries, │ │
│ │ │ ensuring that both speed and │ │
│ │ │ cache locality are optimized. │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 3 │ Red Black Trees for Limit │ They're not necessarily ideal. │ 0.92 │
│ │ Order Book - Quantitative │ In fact, they're rarely used │ │
│ │ Finance Stack Exchange │ in production trading systems │ │
│ │ https://quant.stackexchange.c │ with low latency │ │
│ │ om/questions/63140/red-black- │ requirements... a simple array │ │
│ │ trees-for-limit-order-book │ or vector with linear access │ │
│ │ │ patterns will often outperform │ │
│ │ │ any complex data structure │ │
│ │ │ with better asymptotic runtime │ │
│ │ │ because a simple array │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 4 │ Order Book Reconstruction - │ HashMapMarketDepth... │ 0.85 │
│ │ HftBacktest │ BTreeMarketDepth... │ │
│ │ https://mintlify.com/nkaz001/ │ ROIVectorMarketDepth::new(tick │ │
│ │ hftbacktest/concepts/order-bo │ _size, lot_size, roi_lb, │ │
│ │ ok │ roi_ub)... │ │
│ │ │ FusedHashMapMarketDepth │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 5 │ Order Book Filtration and │ Three real-time, observable │ 0.82 │
│ │ Directional Signal Extraction │ filtration schemes: based on │ │
│ │ at High Frequency │ order lifetime, update count, │ │
│ │ https://arxiv.org/html/2507.2 │ and inter-update delay. These │ │
│ │ 2712v1 │ are used to recompute OBI on │ │
│ │ │ structurally filtered event │ │
│ │ │ streams... Empirical results │ │
│ │ │ show that structural │ │
│ │ │ filtration improves │ │
│ │ │ directional signal clarity in │ │
│ │ │ correlation and regime-based │ │
│ │ │ metrics │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 6 │ Building Low-Latency Order │ This paper proposes a simple │ 0.80 │
│ │ Books with Hybrid │ linear data structure for │ │
│ │ Binary-Linear ... │ tracking the order book and a │ │
│ │ https://ieeexplore.ieee.org/d │ hybrid binary-linear search │ │
│ │ ocument/10296447/ │ algorithm to maintain the top │ │
│ │ │ bid and ask │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 7 │ Order Book Reconstruction - │ Index reusing... Regional │ 0.75 │
│ │ dxFeed KB │ events... Event flags │ │
│ │ https://kb.dxfeed.com/en/data │ applicable to Order event... │ │
│ │ -model/dxfeed-order-book/orde │ Snapshots... Transaction │ │
│ │ r-book-reconstruction.html │ model... dxFeed market data │ │
│ │ │ feeds (real-time, delayed or │ │
│ │ │ historical) allow clients to │ │
│ │ │ reconstruct order books, price │ │
│ │ │ level aggregations, and │ │
│ │ │ aggregations by Market Maker │ │
│ │ │ or a data provider. │ │
├─────┼───────────────────────────────┼────────────────────────────────┼───────┤
│ 8 │ GitHub - │ This Limit Order Book is │ 0.70 │
│ │ brprojects/Limit-Order-Book │ developed in C++ from scratch │ │
│ │ https://github.com/brprojects │ and able to handle over │ │
│ │ /Limit-Order-Book │ 1,400,000 TPS (transactions │ │
│ │ │ per second), including Market, │ │
│ │ │ Limit, Stop and Stop Limit │ │
│ │ │ orders. │ │
└─────┴───────────────────────────────┴────────────────────────────────┴───────┘
Gaps
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Category ┃ Topic ┃ Detail ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ source_not_found │ Proprietary FPGA-based │ Actual FPGA hardware │
│ │ order book indexing schemes │ implementations used by │
│ │ │ firms like Virtu, Jane │
│ │ │ Street, or Citadel for │
│ │ │ on-chip order book indexing │
│ │ │ are not publicly │
│ │ │ documented. MIT project │
│ │ │ proposal references FPGA │
│ │ │ LOB but lacks │
│ │ │ implementation details. │
├──────────────────┼─────────────────────────────┼─────────────────────────────┤
│ source_not_found │ Exact data structures used │ No public disclosure exists │
│ │ by specific named HFT firms │ for the specific indexing │
│ │ │ implementations of major │
│ │ │ HFT firms (e.g., Virtu, Two │
│ │ │ Sigma, Jump Trading). All │
│ │ │ evidence is from │
│ │ │ practitioners sharing │
│ │ │ general principles or │
│ │ │ academic reconstructions. │
├──────────────────┼─────────────────────────────┼─────────────────────────────┤
│ scope_exceeded │ Co-location-specific memory │ NUMA-aware memory │
│ │ topology optimization for │ allocation and CPU affinity │
│ │ LOB │ strategies for LOB │
│ │ │ processes in co-located │
│ │ │ environments are referenced │
│ │ │ but not detailed in │
│ │ │ available sources. │
├──────────────────┼─────────────────────────────┼─────────────────────────────┤
│ source_not_found │ Crypto-specific LOB │ While one Medium article │
│ │ indexing differences vs │ covers crypto HFT system │
│ │ equity markets │ design, it does not detail │
│ │ │ how LOB indexing strategies │
│ │ │ differ for 24/7 crypto │
│ │ │ markets with different tick │
│ │ │ structures. │
└──────────────────┴─────────────────────────────┴─────────────────────────────┘
Discovery Events
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Suggested ┃ ┃ ┃
┃ Type ┃ Researcher ┃ Query ┃ Reason ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ related_research │ arxiv │ FPGA order book │ The MIT HFT │
│ │ │ matching engine │ Accelerator paper │
│ │ │ hardware │ and FPGA │
│ │ │ implementation │ references │
│ │ │ nanosecond │ suggest │
│ │ │ latency │ significant │
│ │ │ │ unpublished work │
│ │ │ │ on │
│ │ │ │ hardware-accelera │
│ │ │ │ ted LOB indexing │
│ │ │ │ that would │
│ │ │ │ directly answer │
│ │ │ │ the proprietary │
│ │ │ │ indexing question │
├──────────────────┼───────────────────┼───────────────────┼───────────────────┤
│ related_research │ arxiv │ limit order book │ Cache-oblivious │
│ │ │ data structure │ structures like │
│ │ │ cache-oblivious │ van Emde Boas │
│ │ │ van Emde Boas │ trees are │
│ │ │ tree HFT │ theoretically │
│ │ │ │ optimal for LOB │
│ │ │ │ operations but │
│ │ │ │ not mentioned in │
│ │ │ │ sources; academic │
│ │ │ │ literature may │
│ │ │ │ document their │
│ │ │ │ use │
├──────────────────┼───────────────────┼───────────────────┼───────────────────┤
│ new_source │ database │ Island ECN Levine │ The Island ECN │
│ │ │ order book ISAM │ B-tree/ISAM │
│ │ │ indexing original │ reference is │
│ │ │ documentation │ cited secondhand; │
│ │ │ 1996 │ primary │
│ │ │ │ documentation │
│ │ │ │ would provide │
│ │ │ │ authoritative │
│ │ │ │ details on the │
│ │ │ │ original │
│ │ │ │ production │
│ │ │ │ indexing strategy │
├──────────────────┼───────────────────┼───────────────────┼───────────────────┤
│ related_research │ arxiv │ order book │ L3 order-by-order │
│ │ │ reconstruction L3 │ reconstruction │
│ │ │ tick data index │ requires │
│ │ │ compression high │ per-order │
│ │ │ frequency │ indexing by │
│ │ │ │ order_id which │
│ │ │ │ has different │
│ │ │ │ data structure │
│ │ │ │ requirements than │
│ │ │ │ L2 price-level │
│ │ │ │ indexing │
└──────────────────┴───────────────────┴───────────────────┴───────────────────┘
Open Questions
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Priority ┃ Question ┃ Context ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ high │ Do modern HFT firms use │ Sources confirm cache-friendly │
│ │ NUMA-aware memory allocation │ arrays dominate in production, │
│ │ strategies specifically tuned │ but NUMA effects in │
│ │ for order book price-level │ multi-socket co-located servers │
│ │ index structures, and how does │ are not addressed │
│ │ this interact with CPU cache │ │
│ │ topology? │ │
├──────────┼─────────────────────────────────┼─────────────────────────────────┤
│ high │ How do HFT firms handle the │ dxFeed documentation describes │
│ │ transition from snapshot-based │ snapshot and transaction models │
│ │ full order book state to │ separately; the handoff between │
│ │ incremental delta updates in │ these modes in production │
│ │ their indexing layer without │ indexing is not detailed │
│ │ introducing consistency gaps? │ │
├──────────┼─────────────────────────────────┼─────────────────────────────────┤
│ medium │ What is the practical │ HftBacktest documents both │
│ │ throughput and latency tradeoff │ structures but does not provide │
│ │ between ROIVectorMarketDepth │ comparative benchmarks for edge │
│ │ and FusedHashMapMarketDepth │ cases like flash crashes where │
│ │ implementations under real │ price moves outside the ROI │
│ │ market conditions with large │ │
│ │ price spikes? │ │
├──────────┼─────────────────────────────────┼─────────────────────────────────┤
│ medium │ Does structural LOB filtration │ The filtration paper shows │
│ │ (by order lifetime or update │ improved OBI signal quality but │
│ │ count) as proposed in the 2025 │ acknowledges limited gains in │
│ │ arxiv paper degrade order book │ causal excitation; │
│ │ reconstruction accuracy under │ accuracy-speed tradeoff for │
│ │ normal market conditions │ indexing filtered vs raw │
│ │ compared to raw feeds? │ streams is unresolved │
├──────────┼─────────────────────────────────┼─────────────────────────────────┤
│ low │ How do exchanges like LMAX, │ The electronictradinghub │
│ │ Tokyo Stock Exchange, and NSE │ article cites these exchanges │
│ │ India differ in their │ as modern evidence but does not │
│ │ recommended order book │ detail their specific │
│ │ reconstruction protocols, and │ reconstruction protocol │
│ │ do these differences force │ differences │
│ │ different indexing strategies │ │
│ │ on client-side HFT systems? │ │
└──────────┴─────────────────────────────────┴─────────────────────────────────┘
╭───────────────────────────────── Confidence ─────────────────────────────────╮
│ Overall: 0.72 │
│ Corroborating sources: 8 │
│ Source authority: medium │
│ Contradiction detected: False │
│ Query specificity match: 0.65 │
│ Budget status: spent │
│ Recency: current │
╰──────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────── Cost ────────────────────────────────────╮
│ Tokens: 70892 │
│ Iterations: 3 │
│ Wall time: 97.77s │
│ Model: claude-sonnet-4-6 │
╰──────────────────────────────────────────────────────────────────────────────╯
trace_id: f4c43973-7cac-4193-a249-cbb1302de4f7
Reference in a new issue View git blame Copy permalink