IRCNF

SQLite Is the Most Deployed Database in the World — and Almost Nobody Talks About It

Share:
SQLite Is the Most Deployed Database in the World — and Almost Nobody Talks About It

The databases that get attention are the ones you operate: PostgreSQL clusters, MySQL replicas, MongoDB shards, cloud-managed services from AWS and Google. The database that's in more places than all of them combined runs silently in the background, requires no administrator, consumes less than 1 MB of disk space for its library, and is a single C source file that any developer can read in an afternoon.

SQLite is everywhere. It's the storage engine inside every iOS and Android application that needs structured data. It's embedded in every desktop browser — Chrome, Firefox, Safari, and Edge all use SQLite for history, bookmarks, and cached data. It's the database powering the flight management software on Airbus and Boeing aircraft. It's inside televisions, cars, routers, and medical devices. The SQLite project estimates over a trillion deployments in active use — a number so large it's difficult to verify, but also difficult to dispute given how many software stacks it underpins.

The Design That Makes It Different

Every other SQL database you've likely used is a client-server system: the database runs as a separate process, you connect to it over a socket (local or network), send queries, and receive results. SQLite is serverless — the entire database engine is a library that runs inside your application process. There's no server to start, no configuration file to manage, no user permissions system, no network port to open. You link against the library, and you have a full SQL database.

The database itself is a single file on the filesystem. This is both its greatest strength and its key limitation. Moving a SQLite database is a file copy. Backing it up is a file copy. Inspecting it requires no special tools — SQLite Browser or the sqlite3 CLI will open any database file on any operating system. When your application is done, there's nothing to shut down.

The limitation is concurrency: SQLite uses file-level locking, which means only one writer can modify the database at a time. Multiple concurrent readers are fine; multiple concurrent writers serialize. For applications with heavy write concurrency — high-traffic web servers handling simultaneous requests — this constraint matters. For the vast majority of applications, including most mobile apps, desktop tools, and moderate-traffic web services, it doesn't.

Why SQLite Is Reliable Enough to Fly On

The SQLite codebase is one of the most thoroughly tested pieces of software in existence. The project maintains approximately 600 times more test code than production code — the library itself is around 150,000 lines of C; the test suite is around 90 million lines, including a complete SQL fuzzer, TH3 (a commercial test harness for rigorous testing against edge cases), and exhaustive corruption recovery tests.

The reliability standard is unusual for open-source software. D. Richard Hipp, SQLite's creator, has explicitly designed it to be suitable for aviation safety-critical systems, and Airbus and others have certified it accordingly. The database provides ACID guarantees — Atomicity, Consistency, Isolation, Durability — through a write-ahead log system. A crash in the middle of a transaction leaves the database in a known-good state; the WAL ensures either the full transaction is committed or none of it is, even across unexpected power loss.

The Surprising Performance Floor

SQLite's serverless architecture means it avoids the overhead that client-server databases incur: no serialization of data to send over a socket, no context switching between client and server processes, no query planning across a network boundary. For workloads where queries are simple and the database is small to medium sized, SQLite's performance is often comparable to PostgreSQL and sometimes faster.

The SQLite team ran benchmarks in 2022 showing that for a read-heavy workload (35 reads per write), SQLite on a commodity NVMe SSD processed 48,000 transactions per second compared to PostgreSQL's 7,000 — largely because SQLite's in-process architecture eliminates the inter-process communication overhead. For write-heavy workloads, the picture is more nuanced; PostgreSQL's concurrent write handling gives it advantages in high-contention scenarios.

The practical implication: many applications that default to PostgreSQL because it's "production-grade" would perform well and be operationally simpler with SQLite. Cloudflare's D1 distributed SQLite service and Turso's libSQL (a fork of SQLite with replication support) are building infrastructure around the bet that SQLite's simplicity has been undervalued in server-side applications.

Modern SQLite: Not the 2005 Version

SQLite has evolved significantly. The JSON support added in 2015 and extended through 2022 allows querying and indexing nested JSON data with full SQL semantics — competitive with document databases for many use cases. The Full-Text Search 5 (FTS5) extension provides BM25-ranked full-text search with prefix queries, without requiring a separate search index. The R*Tree extension handles geospatial range queries. Virtual tables allow SQLite to query external data sources (CSV files, remote APIs) with SQL syntax.

The most recent addition is the vector similarity search extension — sqlite-vss, built on Facebook's FAISS library — which adds approximate nearest-neighbor search to SQLite. This means a single SQLite file can store documents, their vector embeddings, and perform semantic similarity search, making it a viable embedded RAG (retrieval-augmented generation) database for local AI applications. The combination of structured queries, full-text search, and vector search in a single zero-configuration database is compelling for applications that previously required three separate systems.

When Not to Use SQLite

The honest answer to "should I use SQLite?" requires acknowledging its genuine constraints. High-concurrency write workloads — think a web application handling thousands of simultaneous transactions — need the multi-writer concurrency that PostgreSQL provides. Datasets substantially larger than available RAM benefit from PostgreSQL's buffer pool management and query planner, which is optimized for large data scenarios. Applications requiring fine-grained access control (row-level security, column-level permissions) need a server-based database.

But these are constraints that affect a minority of deployments. The typical CRUD web application, the mobile app, the desktop tool, the embedded system, and the data pipeline script are all well-served by SQLite's simplicity. The advice that SQLite "doesn't scale" or "isn't production-ready" has been obsolete for a decade — it scales vertically to very large sizes and has been powering production applications reliably since the early 2000s. The gap between SQLite's actual capabilities and its reputation among developers who haven't used it recently is substantial, and worth correcting.

Share:
SQLite Is the Most Deployed Database in the World — and Almost Nobody Talks About It | IRCNF | IRCNF - Intelligent Reliable Custom Next-gen Frameworks