Replace LLM Wiki Pipelines With a Pure Python Compiler
A developer has engineered a deterministic, pure Python compiler to replace large language model-driven pipelines for personal knowledge management, eliminating token costs and stochastic variability. Responding to recent architectural trends popularized by Andrej Karpathy agent-based wiki models, the project reimagines note compilation as a mechanical, reproducible process rather than a probabilistic one. The system transforms unstructured markdown folders into a cross-referenced, linted wiki using only the Python standard library, requiring zero embeddings, external APIs, or network calls. The pipeline operates through four distinct stages. A regex metadata extractor handles inconsistent source formatting by prioritizing headers, capitalized lines, or filenames, ensuring robustness against malformed input. A graph builder subsequently maps lexical connections between files. Initially hindered by a quadratic pairwise regex approach, the developer resolved performance bottlenecks by implementing a word-indexed phrase matcher, reducing processing time for a 5000-file corpus from over 100 seconds to under 500 milliseconds. A section-aware rewriter then regenerates structured markdown while preserving developer annotations stored beneath dedicated headings. Finally, a structural linter scans the output for broken links and identifies orphaned pages, preventing silent data degradation. Cross-platform benchmarking conducted on Linux and Windows environments validated the system determinism, yielding identical connectivity metrics across all scales. At 1000 files, the full compilation pipeline completes in approximately 1.8 seconds, scaling to 12.4 seconds at 5000 files. Analysis revealed that the linter constitutes the primary performance bottleneck, consuming over half the runtime due to sequential disk I/O operations, with Windows environments exhibiting slower throughput largely attributed to system-level file scanning. Despite these hardware variables, the algorithmic output remained strictly reproducible. The architecture acknowledges inherent limitations, primarily its reliance on exact lexical matching rather than semantic understanding. Notes discussing identical concepts without shared terminology will not be cross-referenced, highlighting the boundary between deterministic parsing and contextual reasoning. The developer positions the compiler not as a replacement for AI, but as an optimal tool for the mechanical majority of knowledge structuring, leaving semantic synthesis to probabilistic models when necessary. Open-sourced with a comprehensive 17-test suite, the project demonstrates that local-first knowledge management can be achieved without external dependencies or computational overhead. The initiative underscores a growing industry preference for transparent, reproducible architectures in personal data tooling, offering a lightweight alternative to resource-intensive agent frameworks. The complete implementation, benchmarking harness, and test documentation are available for independent verification and adaptation.
