HyperAIHyperAI

Command Palette

Search for a command to run...

LumoSQL Enhances SQLite with Security, Privacy, and Pluggable Backends for Next-Gen Data Storage

LumoSQL is a modification—not a fork—of the widely used SQLite embedded database library. It is currently in Phase II of development and aims to enhance SQLite with advanced security, privacy, performance, and measurement features. The project is maintained using the Fossil version control system, and the official source is available at lumosql.org. The GitHub repository is a read-only mirror.LumoSQL introduces several key innovations:Pluggable Backends: LumoSQL supports interchangeable key-value storage engines. The current implementation includes three backends in version 0.4, with LMDB being a prominent example. This allows SQLite to leverage alternative storage engines while maintaining compatibility and performance benefits.Per-Row Encryption and Corruption Detection: LumoSQL adds modern encryption, including Attribute-Based Encryption (ABE), with support for encryption and checksums on a per-row basis. This enables fine-grained security and immediate detection of data corruption at the row level, improving both integrity and performance for search and comparison operations.Benchmarking and Performance Analysis: The project uses the Not-forking tool to track upstream changes and support consistent benchmarking across various systems. The Build and Benchmark System enables side-by-side comparisons of different SQLite and backend versions, with results stored in an SQLite database for analysis.Not-Forking Approach: LumoSQL does not fork any upstream projects, including SQLite. Instead, it uses the Not-forking tool to automatically track and integrate changes from multiple upstream sources, such as SQLite, LMDB, and Oracle Berkeley DB. This approach ensures long-term maintainability and collaboration. Open and Liberal Licensing: LumoSQL is distributed under the permissive MIT license. It is supported by the NLNet Foundation and runs on x86, ARM-32, and RISC-V architectures across various Linux and BSD operating systems.Targeted Use Case: While SQLite is used by billions of users in systems like Android, Firefox, and iOS, its conservative update policy limits the adoption of new features. LumoSQL serves as a testbed for improvements that may eventually be considered by the main SQLite project, such as enhanced security, performance, and data integrity.Build and Development Environment: The build system requires the not-forking tool (a Perl script), Fossil (preferred for development), and standard Unix tools. On Debian/Ubuntu, users need to enable source repositories and install build dependencies. On Fedora, a set of common development packages is required. The system caches all source code to minimize download and build time.Quickstart Guide: Developers can begin by cloning the repository with fossil clone https://lumosql.org/src/lumosql, then running make what to view available configurations. Benchmarking can be initiated with make benchmark SQLITE_VERSIONS='3.35.5', producing results in a local benchmarks.sqlite database. The benchmark-filter.tcl script helps analyze and compare test runs.Performance Observations: Initial benchmarks show that native SQLite 3.35.5 outperforms LMDB-based versions in most cases, but performance differences shift with data size, storage type, and specific workloads. Further testing reveals that LMDB can outperform SQLite on slower storage media, especially with larger datasets.Historical Context: The project was inspired by Howard Chu’s 2013 proof-of-concept port of SQLite to LMDB. LumoSQL has since revived and extended this work, exploring new storage paradigms and performance trade-offs. The project continues to evolve through collaboration and experimentation.LumoSQL is not a replacement for SQLite but a demonstration of what could be possible in the future. It emphasizes modularity, security, and performance, with the long-term goal of influencing the direction of the original SQLite project.

Related Links