plus/minus epsilon

For obvious reasons, I recently got interested in how to build websites that are widely accessible but also resistant to censorship. Naturally, my first instinct was to run off and come up with my own blue-sky designs of the most resilient, censorship-resistant website in the world. But censorship is not new and I realized it would be smart to learn from the past: in particular, The Pirate Bay and WikiLeaks, which both continue to operate even under immense pressure to shutdown.

Recently, a project I was working on required a cache that would ideally be capable of storing tens of millions of entries, reaching into the multiple-terabyte range. At this scale, using a single spinning disk is significantly cheaper than storing everything in memory, which Redis or Memcached would require. Another classic KV store, LevelDB, also ends up being impractical because it writes the value for each key to disk multiple times. If your values are small, LevelDB can be really really performant, but in my case the values are moderately sized (tens of kilobytes) and writing them to disk multiple times quickly becomes user-visible. And finally, I expected that storing this many entries on-disk as individual files in a folder would cause issues with inode exhaustion, and ultimately, the ACID guarantees that a filesystem provides have never been super clear to me.