Large-file / extracted-netlist mode

Design for opening post-layout and other extracted netlists (tens to hundreds of MB) in the editor without OOM or multi-second hangs, while keeping full analysis for normal schematic-scale decks.

Status: Implemented (size gate, defs-only index, thinned diagnostics, settings). Lazy include materialization and Tree-sitter incremental reuse remain follow-ups.
Related: Limitations, Architecture, Principles, Include and library resolution, LSP features, Design.

1. Problem

spice-lsp today eagerly:

1. Holds the full buffer as a String
2. Fully re-parses with Tree-sitter (no incremental old_tree reuse yet)
3. Builds an Index that records every instance plus hierarchical outline children
4. Loads and indexes .include / .lib targets for definition resolution

That matches interactive schematic netlists (thousands to tens of thousands of lines). It does not match extracted dumps (DSPF/SPEF-style or flattened SPICE) where a single open file can be ~300 MB and almost entirely instance lines.

Full per-instance symbol resolution at that scale is not feasible with the current model: source + dense owned symbol/outline tables amplify memory to multiple times the file size, and LSP payloads (outline, diagnostics) would overwhelm the client even if the server survived.

Performance targets in Design (~50k lines / <100 ms semantic) and Principles (typical <5k lines) already describe a different operating region.

2. Lessons from Astral’s ty

ty (Astral’s Python type checker / LSP; engine in astral-sh/ruff) is built for large projects: millions of lines across many modules, with millisecond incremental updates after edits. Public summary: language server docs, announcement.

2.1 What they do well

2.2 What we should not copy wholesale

2.3 Problem-shape caveat

ty’s “large” usually means many moderate files with fine-grained edits. A single ~300 MB netlist is a different stressor (they already see pain on ~28 MB dense stubs). Steal policy (lazy, layered, scoped diagnostics, drop heavy IR); do not assume Salsa alone makes full instance indexing viable.

3. Goals and non-goals

Goals

Non-goals (this design)

• Full per-device symbol tables for extracted dumps
• Find-all-references across millions of instances
• Connectivity / net-graph analysis on extracted top-levels
• Formatter rewrites of 100+ MB buffers
• Adopting Salsa in the first implementation slice

4. Proposed modes

Optional third lever later: spiceLsp.analysisMode = "auto" | "full" | "extracted" so users can force either side of the gate.

4.1 Threshold

Start with a simple gate, e.g.:

• text.len() >= N bytes (suggested starting point: 16–32 MiB), and/or
• line count ≥ 200k

Exact numbers should be tuned with a fixture and RSS measurements; document the defaults in settings and Limitations when implemented.

4.2 Feature matrix in extracted mode

5. Architecture sketch

Keep the existing crate split; add a policy layer in analysis rather than a second parser.

didOpen / didChange
  → choose AnalysisProfile { Full, Extracted }
  → parse / classify lines (shared)
  → build_index(profile)
       Full:      today’s symbols + outline
       Extracted: definitions (+ optional sparse model refs), thin outline
  → includes
       Full:      eager resolve (current)
       Extracted: stub graph; materialize IncludedFile defs on demand
  → publish diagnostics / serve navigation

5.1 Index changes (spice-parser)

• Extend build_index (or wrap it) with a profile flag:
  • Skip SymbolKind::Instance insertion
  • Skip pushing instances into document_symbols children
  • Optionally still record model/subckt name references from instance lines into the references map without storing an instance Symbol per line (supports unknown-model + goto without GB-scale vectors)
• Prefer interned or borrowed name keys where practical later; first slice can keep String keys if instance rows are omitted

5.2 Include graph (spice-parser / LSP)

• In extracted mode, do not retain full text + full Index for every include by default
• Materialize definition maps when resolving go-to-definition or unknown-model
• Cap concurrent materialized includes; depth cap already exists (DEFAULT_MAX_INCLUDE_DEPTH)

5.3 LSP backend (spice-lsp)

• Gate on didOpen / after large didChange (rare for extracted files)
• Avoid cloning entire workspace buffers into analyze snapshots when possible
• Prefer pull diagnostics later; until then, publish a small diagnostic set in extracted mode
• Log once: spice-lsp: extracted analysis mode (N bytes) — instance indexing disabled
• Cancel or spawn_blocking long analyzes so the server stays responsive

5.4 Easy wins independent of mode

Worth doing even for schematic files (aligned with ty’s “don’t redo work”):

6. Comparison: ty patterns → spice-lsp actions

7. Implementation slices

Public docs: LSP features, Limitations, VS Code README.

8. Testing

Do not check multi‑hundred‑MB binaries into the repo.

9. Open questions

1. Default threshold — bytes vs lines vs both; different defaults for .spf/.net associations?
2. Model refs without instance symbols — enough for unknown-model quality, or accept weaker checks in extracted mode?
3. Editor still holds 300 MB — server-side wins don’t fix VS Code memory; document that opening such files is inherently heavy
4. Partial / viewport analysis — analyze only visible ranges later? Powerful but much more complex; not in the first slices
5. Salsa later? — revisit if multi-file schematic workspaces (many includes, frequent edits) become the bottleneck after extracted mode lands

10. Success criteria

When implemented, an extracted ~100–300 MB netlist should:

1. Open without process kill / multi‑GB RSS from instance indexes
2. Show a usable structural outline (subcircuits/models/params)
3. Support go-to-definition for models/subcircuits with lazy includes
4. Leave schematic-scale files behaviorally unchanged
5. Document the mode and limits in Limitations