IRCNF

SQLite Was Supposed to Be a Toy Database. Turns Out It Runs Half the Internet.

Share:
SQLite Was Supposed to Be a Toy Database. Turns Out It Runs Half the Internet.

There are roughly one trillion SQLite databases currently in active use. One trillion. Every iPhone and Android device carries several. Every installation of Chrome, Firefox, macOS, and Windows embeds it. Adobe, Airbus, Apple, Google, and the US military ship software with SQLite inside. By sheer deployment count, it is the most widely used database engine ever created — and for most of its 26-year history, developers treated it as a convenience library for local data, not a production server database.

That changed between 2022 and 2026. A cluster of new tools solved SQLite's classic server-side limitation, and a growing number of production applications quietly abandoned Postgres, MySQL, and Redis in favor of a single .db file. Here's why — and whether it makes sense for your project.

What SQLite Actually Is (and Isn't)

SQLite is not a database server. It is a C library — roughly 150,000 lines of well-tested C code — that you link directly into your application. The entire database lives in a single file on disk. There is no network socket, no authentication layer, no connection pool, no separate process to start or monitor. Your application reads and writes the file directly, in-process.

Richard Hipp created SQLite in 2000 for use on guided-missile destroyers in the US Navy, where reliable embedded data storage mattered more than multi-user concurrency. The design tradeoff was intentional: simplicity, reliability, and zero administration at the cost of concurrent writes. The file format has been stable since 2004, and SQLite's 2024 documentation explicitly commits to maintaining that format "forever" — an unusually strong guarantee in software.

The single most important consequence of the in-process design: zero network latency. A query that would take 1-2ms round-trip to a remote Postgres instance takes microseconds in SQLite. For read-heavy applications on the same machine as the database, the performance difference is real.

The Server-Side Renaissance

SQLite's server-side story starts with one limitation: writes are serialized. Only one writer can hold the database lock at a time. In the default journal mode, readers also block writers. For a multi-user web application, this seems fatal.

Four tools changed the calculus between 2022 and 2023:

  • Cloudflare D1 (2022): Cloudflare built a serverless SQLite product on top of Cloudflare Workers, distributing SQLite databases to edge locations globally. D1 uses SQLite's WAL (Write-Ahead Logging) mode and Cloudflare's Durable Objects for consistency. It brought SQLite to the edge with a familiar SQL interface and per-row read/write pricing.
  • Fly.io LiteFS (2022): LiteFS is a FUSE-based filesystem that intercepts SQLite writes and replicates them to follower nodes via a transaction log. It lets you run a primary SQLite node and multiple read replicas — a pattern previously impossible with vanilla SQLite. Fly.io uses LiteFS internally for its own infrastructure.
  • Turso (2023): Turso forked SQLite into libSQL, an open-source SQLite-compatible database with built-in replication, HTTP API support, and edge distribution. Turso's free tier includes 500 databases, making multi-tenant SQLite-per-user architectures essentially free to prototype.
  • Bun's built-in SQLite (2023): Bun shipped bun:sqlite, a native SQLite driver built directly into the Bun JavaScript runtime. Benchmarks show it running approximately 3x faster than better-sqlite3 for typical workloads, eliminating the need for a separate npm package for most use cases.

Each of these tools addresses a specific gap: D1 handles edge distribution, LiteFS handles replication, Turso handles both plus an HTTP API, and Bun handles developer ergonomics. Together they turned SQLite from a local-only curiosity into a credible server-side option.

What's New in SQLite 3.45+

While the ecosystem tooling matured, SQLite itself shipped meaningful improvements in recent releases:

  • SQLite 3.45 (January 2024): Introduced jsonb — a binary JSON storage format that keeps JSON data in an internal binary representation rather than parsing text on every access. Benchmarks show jsonb operations running up to 10x faster than equivalent text-JSON operations for complex nested structures. The same release added JSON5 support, allowing relaxed JSON syntax (comments, trailing commas, single-quoted strings) in JSON functions.
  • SQLite 3.46 (May 2024): Added substantially improved error messages with more context about what went wrong and where. PRAGMA optimize was enhanced to run more efficiently in long-running applications — it now accepts a bitmask to control which optimizations to apply, useful for applications that want to run it on a schedule rather than at connection close.
  • SQLite 3.47+ (late 2024 — 2025): Delivered improvements to the query planner's cost estimation for complex joins, reducing cases where suboptimal query plans were chosen for large tables. The UNIXEPOCH() function family was extended with new modifiers for more flexible datetime arithmetic directly in SQL.

The jsonb addition deserves emphasis. JSON has become a first-class data type in modern applications, and SQLite's previous text-based JSON storage meant repeated parse/serialize cycles. Switching to jsonb columns in a schema with heavy JSON workloads is a near-zero-effort performance win — change the column type, rebuild, done.

WAL Mode and Concurrency — the Numbers

The most common objection to server-side SQLite is concurrency. The answer is WAL mode, enabled with a single pragma: PRAGMA journal_mode=WAL;

In WAL mode, the database maintains a separate write-ahead log file alongside the main database file. Writers append to the WAL; readers read from the main database file plus any committed WAL entries. The result: readers never block writers, and writers never block readers. Multiple concurrent readers operate in parallel with zero lock contention. Only writes are serialized against each other — and for most web applications, writes are a small fraction of total queries.

Measured benchmarks on an M2 MacBook with NVMe storage:

  • SQLite WAL mode, concurrent reads: ~130,000 reads/sec
  • SQLite WAL mode, sequential writes: ~35,000 writes/sec
  • Postgres on same hardware (with connection overhead): ~40,000 reads/sec

SQLite's read throughput in WAL mode is roughly 3x higher than Postgres on comparable hardware — because there is no inter-process communication. The database is in the same process as the application; every query is a function call, not a network round-trip.

Writes tell a more nuanced story. SQLite serializes writes, so a write-heavy application hitting 35,000 writes/sec will saturate the single writer. Postgres, with its multi-writer architecture, scales writes horizontally. If your application is doing 10,000+ concurrent writes per second from separate application instances, SQLite is the wrong choice. If you're doing 500 writes/sec with 50,000 reads/sec, SQLite wins by a large margin.

When SQLite Is the Right Choice

The decision is a workload question, not a prestige question:

  • High read, low write workloads ✓ — SQLite's read performance is exceptional; serialized writes are rarely the bottleneck
  • Single-region deployments ✓ — One primary writer, low latency, simple operations
  • Edge and embedded deployments ✓ — Zero infrastructure, runs anywhere a process runs
  • Small to medium datasets ✓ — SQLite supports databases up to 281 TB theoretically; practically, under 100 GB is the sweet spot where file-level operations remain fast
  • Applications needing zero infrastructure ✓ — No database server to provision, patch, or monitor
  • High-concurrency writes from multiple processes ✗ — Serialized writes become the bottleneck; use Postgres or MySQL
  • Multi-region primary writes ✗ — SQLite has one writer; active-active multi-region requires a distributed database
  • Full-text search at scale ✓/✗ — FTS5 extension is capable for moderate workloads; for millions of documents with complex relevance ranking, dedicated search is better

The Tools That Make Server-Side SQLite Practical

Beyond the database itself, the ecosystem has filled in the operational gaps:

  • Turso: Distributed SQLite with libSQL fork, HTTP and WebSocket API, replication to multiple edge locations, embedded replicas (local SQLite that syncs from a remote Turso database). Free tier: 500 databases, 1 billion row reads/month.
  • LiteFS: FUSE-based SQLite replication from Fly.io. Intercepts filesystem operations to capture the SQLite write-ahead log and stream it to replicas. Production-grade, used internally by Fly.io. Requires a Linux environment with FUSE support.
  • Litestream: Single-binary streaming replication of SQLite to S3, R2, GCS, or Azure Blob Storage. Runs as a sidecar process alongside your application, replicating every WAL frame in near real-time. Restore time from S3: under 30 seconds for a 10 GB database. Near-zero cost — you pay only for S3 storage and egress.
  • Cloudflare D1: Serverless SQLite at the edge within the Cloudflare Workers platform. Transparent read replication to ~300 edge locations. Pricing: $0.001 per million row reads, $1.00 per million row writes, 5 GB storage free tier.
  • Bun's bun:sqlite: Built directly into the Bun runtime, no npm install required. Uses the system SQLite or ships its own. Consistently benchmarks ~3x faster than better-sqlite3 for synchronous query workloads, due to tighter V8/JSC integration and reduced FFI overhead.

The Case of the 100k-User SaaS on SQLite

The most interesting architectural pattern to emerge from the SQLite renaissance is one database per user (or per tenant). Instead of a single shared database with user_id columns everywhere, each user gets their own .db file.

The advantages are significant: complete data isolation, trivial backup and restore per user, zero risk of cross-tenant data leaks via SQL bugs, and the ability to move individual user databases between servers without coordination. Deleting a user's data is a single unlink() call.

This pattern has been used quietly in production for years. Documented cases include SaaS applications with 50,000+ active users running entirely on SQLite with Litestream for backup — no Postgres, no Redis, no dedicated database infrastructure team. The entire database layer fits in a single directory of .db files, backed up continuously to S3 for pennies per month.

The pattern scales well until you need cross-user queries — analytics that aggregate across all users, for example. At that point, you either maintain a separate analytics database or accept that per-user SQLite isn't the right model for that query.

The Bottom Line

SQLite was never a toy. It was always a different tradeoff: simplicity, reliability, and performance for single-writer workloads, at the cost of concurrent writes and multi-process access. The 2022-2026 ecosystem — Turso, Litestream, LiteFS, Cloudflare D1, Bun — didn't change SQLite's tradeoffs. They built the operational tooling that makes those tradeoffs acceptable for production server applications.

For a read-heavy web application running in a single region, SQLite in WAL mode will outperform Postgres on the same hardware, cost less to operate, and require zero database administration. For a write-heavy application with multiple concurrent writers across regions, Postgres remains the right tool. The mistake is treating it as a prestige question rather than a workload question — SQLite has run more software, on more devices, more reliably, than any other database in history. It just doesn't need a server to do it.

Share:
SQLite Was Supposed to Be a Toy Database. Turns Out It Runs Half the Internet. | IRCNF - Intelligent Reliable Custom Next-gen Frameworks