Skip to content

Instantly share code, notes, and snippets.

@mikewaters

mikewaters/Sync-Database-Notes-20250101.md

Created January 1, 2025 19:45
Show Gist options
  • Save mikewaters/74f23c3665cca76349d5f89d3c631afe to your computer and use it in GitHub Desktop.
Save mikewaters/74f23c3665cca76349d5f89d3c631afe to your computer and use it in GitHub Desktop.
Download ZIP
Snapshot of local first-adjacent technologies for data synchronization research
Raw
Sync-Database-Notes-20250101.md
tags
active

Sync Database Notes

Non-exhaustive

Sync Platforms

Workloads that provide a synchronization mechanism between a local and remote state store.

Typically, some service sits in front of the primary data store, consuming its changes via CDC/Redo log/WAL etc. That service distributes changes to clients, who have an local store of some kind (sqlite, indexeddb, custom) with a matching or compatible schema. The client and service communicate via a network protocol (http, ws, etc) and the service may provide conflict resolution (crdt/ot or something else). The clients can either speak the protocol directly, and/or can use an SDK.

Most of these only support “web tech” SDKs, where the clients are presumed to be browser applications, but this pendulum is swinging to WASM.

Evaluation Criteria

  - SDK/client languages and runtimes

  - Backend type

  - Source database support

  - Replica persistence type

  - Protocol wire type

  - Synchronization protocol, reactivity

  - Licensing and hosting

  - e2e encryption

  - local-first, or offline supported

  - WASM

  - How can one use it with backend runtimes and have that be reactive on the frontend?

  #### refined criteria

  1. capabilities

  2. audience

  3. architecture

  4. ???

Actyx

Actyx Local-First Cooperation based on Event Sourcing

Actyx is a decentralized event database, streaming and processing engine that allows you to easily build ++local-first cooperative++ apps

… durable event stream storage in peer-to-peer network using ++libp2p++ and ++ipfs-embed++

  • mobile fe: android; fe: win/mac/lin desktop
  • backend: Rust
  • runtime: docker
  • protocol: protobuf
  • arch: pure p2p, event sourcing
  • doesn't depend on any central components (servers, databases, or cloud)
  • Small pieces of code — they are called local-twins — contain the logic. Any app on any device can summon any twin at any time. Actyx automatically synchronizes the twins within the local network.
  • Code

AnySync

Any Sync (Anytype) the everything app for those who celebrate trust & autonomy

An open-source protocol designed to create high-performance, local-first, peer-to-peer, end-to-end encrypted applications that facilitate seamless collaboration among multiple users and devices

  • Local first
  • Protocol: CRDT DAGs, local p2p via mDNS
  • Security: e2e, creators’ controlled keys; CR is on-device;  Ed25519 algorithm for signing
  • File storage in IPLD
  • component arch: four types of nodes: sync, file, consensus and coordinator nodes
  • Code

Convex

Convex the sync platform that replaces your backend and client state management ($ hosted option) BaaS

  • hosted open-source BaaS
  • Hosted; They do not recommend running this yourself (wtf??)

This repository is the wild west open source version

  • Official React/Typescript SDK, as well as wide runtime support (python, rust, swift, etc)
  • Convex backend is a combination of rust crates and typescript modules, exposes an HTTP API for clients.
  • Not sure if the non-TS runtimes support all the Functions stuff
  • Local database, optimistic concurrency, Table and Document objects
  • Functions:

  • Functions run on the backend and are written in JavaScript (or TypeScript). They are automatically available as APIs accessed through client libraries.

  • Queries (reactive, realtime), Mutations (transactional), Actions (web service calls etc)

  • Philosophy: The Software-defined Database
  • Docs, Code

DriftDB

DriftDB real-time data backend that runs on the edge

DriftDB is a real-time data backend that runs on the edge … a keyed, ordered broadcast stream with replays and compaction.

DriftDB is useful any time you wish you could connect directly from one browser to another, or from a non-web-connected backend (e.g. background task) to a browser.

DriftDB is intended for use cases where a relatively small number of clients need to share some state over a relatively short period of time. For example, it could be used to build a shared whiteboard, or act as a signaling server for WebRTC. Room state is currently persisted for 24 hours after the last write.

  • hosting in Cloudflare DO or Jamsocket

  • js/react client

  • API Server (Rust)

  • Worker impl in Rust using Cloudflare Durable Object persistence

  • API is websocket or http protocol

  • Not updated recently, unsure of status

  • Docs, Code

DXOS

DXOS build real-time, collaborative apps which run entirely on the client, and communicate peer-to-peer, without servers

  • Pivoted SDK effort to realtime collaborative editor product for now (12/2024)
  • TS/React client SDK, on pause for now; not ready for use
  • ECHO: realtime database with transparent data replication and conflict resolution

ECHO (The Eventually Consistent Hierarchical Object store) is a peer-to-peer graph database written in TypeScript.

  • HALO: keychain for identiity/auth
  • MESH: decentralized p2p protocol via WebRTC
  • Code, Interesting Architectures

Electric Clojure

Electric Clojure full-stack differential dataflow for UI TBD

Electric is a new way to build rich, interactive web products that simply have too much interactivity, realtime streaming, and too rich network connections to be able to write all the frontend/backend network plumbing by hand. With Electric, you can compose your client and server expressions directly (i.e. in the same function), and the Electric compiler macros will infer at compile time the implied frontend/backend boundary and generate the corresponding full-stack app.

Evolu

Evolu Local-First Platform Designed for Privacy, Ease of Use, and No Vendor Lock-In

Evolu Server is a simple message buffer and storage generic (the same) for all Evolu apps. Only UserId, NodeId, and messages timestamps are visible. Everything else is encrypted.

  • Typescript sdk
  • Ships Typescript backend, or BYO
  • Local SQLite with CRDT-based protocol to backend server which hosts a sqlite that only stores metadata
  • e2e encrypted
  • Docs, Code

Fireproof

Fireproof Realtime database. Runs anywhere ($ hosted option)

Fireproof uses immutable data and distributed protocols to offer a new kind of database that:

  • can be embedded in any page or app, with a flexible data ownership model
  • can be hosted on any cloud
  • uses cryptographically verifiable protocols (what plants crave)

… a decentralized realtime database that stores documents using prolly-trees

  • JS/TS client SDK
  • reactive using WebRTC
  • eventually consistent
  •  encrypted, immutable files for storage
  • Sync via ledger, using content-addressable archive (CAR) files in remote storage backends (including IPFS, PartyKit, S3)
  • CRDT for conflict resolution
  • Docs, Code, Interesting Architectures

Instant

Instant a modern Firebase … giving your frontend a real-time database ($ hosted option)

  • Hosting provider
  • Web client SDKs only (js, react etc)
  • Replication from local IndexedDB to postgres via a Clojure backend
  • Docs

Jazz

Jazz an open-source toolkit for building apps with distributed state* *($ hosted option)

Kinto

Kinto generic JSON document store with sharing and synchronisation capabilities from Mozilla

Kinto is a minimalist JSON storage service with synchronisation and sharing abilities. It is meant to be easy to use and easy to self-host.

... used at Mozilla for Firefox Sync

  • Ships JS client
  • Local store is json documents.
  • HTTP api
  • persists into postgres using python backend
  • Docs

Liveblocks

Liveblocks platform for adding collaborative editing, comments, and notifications into your application ($ hosted option)

  • Typescript
  • Mostly a components vendor that uses their APIs - commercial
  • Storage: Liveblocks Storage is a realtime sync engine designed for multiplayer creative tools such as Figma, Pitch, and Spline. LiveList, LiveMap, and LiveObject conflict-free data types can be used to build all sorts of multiplayer tools.
  • Yjs: Liveblocks Yjs is a realtime sync engine designed for building collaborative text editors such as Notion and Google Docs.
  • Docs, Code

Livestore

Livestore from jschickling, closed EA atm in Expo

OctoBase

OctoBase light-weight, scalable, data engine written in Rust

  • AFFiNE data layer
  • CRDT via y-octo, compatible with blocksuite from AFFiNE
  • Storage layer-agnostic CRDT data and object store (sqlite, postgres, s3 wip)
  • Collaboration protocols (websocket, webrtc, libp2p wip)
  • js, java, swift clients (“bindings”)
  • it is immature and may have some lock-in.

PartyKit

PartyKit open source deployment platform for AI agents, multiplayer and local-first apps, games and websites from Cloudflare ($ hosted option)

  • Bought by Cloudflare, hosted in DO
  • web tech
  • Backend is a wasm module hosted in a durable object, in whatever wasm source lang desired (ie not python probably), implementing your “Party” biz logic
  • Can run an yjs server on the DO, opening up the possibility for a python app on the client side (ypy)

Phoenix live view

Phoenix live view TBD

- Examples
  - <https://github.com/thisistonydang/liveview-svelte-pwa>

Pocketbase

Pocketbase Open Source realtime backend in 1 file

Open Source realtime backend in 1 file with database, auth, files, and admin

- js and dart client sdks
- pen source Go backend that includes … embedded SQLite, reactivity, rest api
- local embedded SQLite+WAL. Suggest using Litestream for replication if needed, and suggest fly.io for hosting

Prisma Pulse

Prisma Pulse distribute change events to your application at scale, enabling ($ commercial only)

  • From Prisma typescript ORM vendor, who is offering a managed Postgres (EA)
  • Works with BYO databases (postgres-based)
  • Not an open source product

Redwood

Redwood a highly-configurable, distributed, realtime database and application server

A highly-configurable, distributed, realtime database that manages a state tree shared among many peers.

Replicache

Replicache - client-side sync framework for arbitrary backends ($ hosted option)

Replicache is a client-side sync framework for building realtime, collaborative web apps with zero-latency user interfaces. It works with most backend stacks.

  • Hosting + open interface and some open components, license required for use
  • Typescript SDK
  • Supports arbitrary backends, as long as they use a database that supports snapshot isolation. Ships a default Typescript backend, and defines an interface for building custom backends.
  • Replicache sync protocol
  • Replica data is persisted in the browser/client via XXXXX.
  • “Mutators”, applied optimistically (locally) and then authoritatively (remotely), transaction is completed locally when the remote acks the write.
  • “Subscriptions” at the query level, with reactivity in the client.
  • “Poke” interface for push/pull streaming
  • Requires a license, but is free for non-commercial. Closed source client, ships with default implementations of interfaces they’ve defined which you can use to BYO
  • Docs

RxDB

RxDB Store data locally to build high performance realtime applications that sync data with the backend and even work when offline ($ licensed functionality)

  • javascript sdk

  • uses a custom replication protocol that can support arbitrary backends, like hosted postgres from supabase extension

  • has free edition but advanced features require a license

  • good for react native and mobile dev

Scale

Scale A framework for building high-performance plugin systems into any application, powered by WebAssembly.

  • found them through a pycon columbia 2024 session
  • guest plugin runtimes: Go, rust, ts
  • host app runtimes: go, ts
  • Audience: embedding behavior into an application using wasm
  • Code

ShareDB

ShareDB Realtime database backend based on Operational Transformation (OT)

  • JS client SDK and backend
  • can persist to DBs like Mongo/Postgres
  • Protocol is OT-based
  • Designed for DerbyJS MVC framework

SKDB

SKDB an embedded SQL database that stays in sync ($ hosted option)

  • TS client, reactive, conflict resolution, React client
  • Defines their own language sklang
  • sqlite is in there somewhere, it is opaque
  • Cloud offering in the works (2024-Dec)
  • Docs, Code

SQLSync

SQLSync a collaborative offline-first wrapper around SQLite prototype

SQLSync is a collaborative offline-first wrapper around SQLite

currently usable for prototypes

  • js/ts client using sqlsync-worker, “reducer” in rust (wasm); ships React lib
  • Reactive query subscriptions
  • Rust (wasm) backend controlling persistence (demo in Cloudflare)
  • active development, very interesting maybe later.
  • Eventually consistent replicas using optimistic r/w

Supabase

Supabase an open source Firebase alternative ($ hosted option)

hosted postgres; has a Realtime product that appears to be open source, and it has a python SDK as well as js/ts. Realtime is Elixir, just like Electric

TinyBase

TinyBase reactive data store for local-first apps

Triplit

Triplit an open-source database that syncs data between server and browser in real-time ($ hosted option)

  • js/ts SDK
  • has an open source server impl with pluggable storage engines like sqlite and leveldb.
  • Protocol is http
  • Ships default JS impl
  • Cloud hosting relationship with Railway
  • offline mode, rather than localfirst; clients do optimistic writes and are considered a cache

Verdant

Verdant a framework and philosophy for small, sustainable, human web apps

Source-to-Source Replication Architectures

These components might be (an in some cases are) used as part of a data layer for a Sync Platform.

Yjs or crdt backend persistence solutions

Y-Sweet

Realtime Cache and Proxy Components

ReadySet

ReadySet transparent cache (postgres, mysql)

- a transparent database cache for Postgres & MySQL that gives you the performance and scalability of an in-memory key-value store without requiring that you rewrite your app or manually handle cache invalidation
  • it keeps cached query results in sync with your database automatically by utilizing your database’s replication stream. It is wire-compatible with Postgres and MySQL and can be used along with your current ORM or database client.
  • rust
  • docker, linux binary
  • Integrates with Supabase via an extension
  • https://github.com/readysettech/readyset

Source-to-Local Replicas

To local embedded replicas (PGLite, SQLite/LibSQL)

Electric SQL

Electric SQL + PGLite/SQlite syncs little subsets of your Postgres data into local apps and services ($ hosted option)

  • TS and Elixir client SDKs
  • Backend in Elixir, fronting a Postgres instance
  • Electric Protocol over HTTP, sync and stream
  • Local database is SQLite in wasm; PGLite is supported but only for local reads atm (Dec-2024)
  • Docs, Code

Turso

Turso LibSQL ($ hosted option)

  • Wide client runtime support
  • LibSQL client-side
  • LibSQL Server (sqld) source SQLite
  • Edge hosted option
  • Docs, Code (libsql)

Powersync

Powersync a service and set of client SDKs that keeps backend databases in sync with on-device embedded SQLite databases ($ hosted option)

  • Hosted
  • js, React, flutter SDKs
  • postgres db source
  • typescript backend uses WAL, depends on mongodb internally
  • local replica is sqlite (maybe wasm?)
  • Docs, Code

SQLedge

SQLedge Replicate postgres to SQLite on the edge (updated fork pgreplsql)

- state: alpha, no commits since 2023
- written in Go

SQLedge uses Postgres logical replication to stream the changes in a source Postgres database to a SQLite database that can run on the edge. SQLedge serves reads from its local SQLite database, and forwards writes to the upstream Postgres server that it's replicating from

Pasted 2024-12-05-15-29-42.png

LibSQL Ecosystem

  • Turso
  • AstroDB fully-managed SQL database designed for the Astro ecosystem

Source-to-$X Connectors

Sequin

Sequin Postgres CDC to Sink ($ hosted offering, paywall features)

Sequin is a tool for change data capture (CDC) in Postgres. Sequin makes it easy to stream Postgres rows and changes to streaming platforms and queues (e.g. Kafka and SQS). You can backfill existing rows and stream new changes in real-time.

Sequin even supports native sinks (HTTP GET and webhooks), so you can get started without any other infrastructure.

SQLite Disaster Recovery

Litestream

Litestream Fully-replicated database with no pain and little cost

Litestream is a streaming replication tool for SQLite databases. It runs as a separate background process and continuously copies write-ahead log pages from disk to one or more replicas

  • Consumes WAL, writes to an S3-compatible object storage such as Tigris.
  • Code

Multi-master SQLite

cr-sqlite

Vlcn / cr-sqlite Convergent, Replicated SQLite. Multi-writer and CRDT support for SQLite ✅ how-to

- SQLite extension

node, deno, browser (wasm), python, c, rust; CR at the table layer (“CRR”), future to allow partial sync]

Moribund. Developer went full time with offline replication vendor. Interesting project. Devil in details.

simonw TIL

CR-SQLite is a ++run-time loadable extension(opens in a new tab)++ for ++SQLite(opens in a new tab)++ and ++libSQL(opens in a new tab)++. It allows merging different SQLite databases together that have taken independent writes.

 cr-sqlite adds multi-master replication and partition tolerance to SQLite via conflict free replicated data types (++CRDTs(opens in a new tab)++) and/or causally ordered event logs.

cr-sqlite is distributed as a ++run time loadable extension(opens in a new tab)++ for SQLite and can be used with any language that has SQLite bindings.

++cr-sqlite++ is network agnostic. You can use ++crsql_changes++ to pull changes from one database, ship them over any protocol you like, then use ++crsql_changes++ on the other end to apply the changes. In any case, a WebSocket implementation is provided for convenience ++here++ and an overview of writing your own network code ++here++.

IceFireDB

IceFireDB a DB storage and retrieval protocol built for web3.0

  • SQLite, SQLProxy, NoSQL, RedisProxy, and PubSub impls
  • SQLite impl: IceFireDB SQLite is a decentralized SQLite database designed to facilitate the construction of a global distributed database system. It allows users to write data to IceFireDB using the MySQL protocol. The data is stored in an SQLite database and automatically synchronized among nodes through P2P networking.
  • https://github.com/IceFireDB/IceFireDB

Expo SQLite

Expo SQLite

- <https://github.com/expo/expo/tree/main/packages/expo-sqlite>

Single-master SQLite

LiteFS

LiteFS FUSE-based file system for replicating SQLite databases across a cluster of machines from fly.io

  • Docs, Code, fly.io project
  • distributed file system that transparently replicates SQLite databases
  • Fly.io recommends Turso for hosting

FUSE file system: intercepts file system calls to record transactions.

Leader election: currently implemented by ++Consul++ using sessions

HTTP server: provides an API for replica nodes to receive changes.

rqlite

rqlite lightweight, user-friendly, distributed relational database built on SQLite

Go server impl, wide client lang support. Uses Raft for cluster consensus; each node can recv writes, but they will be forwarded to the leader synchronously to perform the actual write and then wait for consensus (latency considerations); reads might be serviceable by followers, given the consistency settings. sqlite is just a storage engine. protocol is HTTP. FAQ, sqlalchemy engine

All writes go to the Raft leader (rqlite makes sure this happens transparently if you don't initially contact the Leader node)

rqlite is a distributed relational database that combines the simplicity of SQLite with the robustness of a fault-tolerant, highly available system. It's developer-friendly, its operation is straightforward, and it's designed for reliability with minimal complexity.

dqlite

dqlite a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices

  • Canonical project (source), another raft consensus sqlite cluster

ChiselStore

ChiselStore - ChiselStore is an embeddable, distributed SQLite for Rust, powered by Little Raft.

ha-sqlite

ha-sqlite - High-availability sqlite database service based on raft.

raft-sqlite

raft-sqlite - Raft backend using SQLite.

ReSqlite

ReSqlite - ReSqlite is an extension of Sqlite that aims to add basic replication functionality to Sqlite database.

tqlite

tqlite - Distributed SQL database with replication, fault-tolerance, tunable consistency and leader election.

Multi-master Postgres

pgEdge

pgEdge distributed PostgreSQL, optimized for the network edge ($ hosted option)

  • uses spock, not sure how the replica works
  • Code

BDR

EDB BDR PostgreSQL extension providing multi-master replication and data distribution with advanced conflict management ($ commercial?)

- Commercial product, but seem to be a few docker containers that use it on github

BDR is a PostgreSQL extension providing multi-master replication and data distribution with advanced conflict management, data-loss protection, and throughput up to 5X faster than native logical replication, and enables distributed Postgres clusters with high availability up to five 9s.

PGD

EDB PGD multi-master replication and data distribution with advanced conflict management ($ commercial?) fork of pglogical

PGD provides loosely coupled, multimaster logical replication using a mesh topology. 

pglogical

pglogical extension provides logical streaming replication for PostgreSQL, using a publish/subscribe model

- Based on BDR

While pglogical is actively maintained, EnterpriseDB (which acquired 2ndQuadrant in 2020) focuses new feature development on a descendant of pglogical: ++Postgres Distributed++

Postgres Streaming

pgstream

replicates schema changes as well as data Go binary that monitors WAL, support buffering WAL to kafka for scaling dist by xata.io, a serverless postgres vendor pgstream and wal2json to emit webhooks on CDC events: https://xata.io/blog/postgres-webhooks-with-pgstream

Debezium

java+kafka stack

HA Postgres

PostgreSQL HA documentation

Citus

Citus Distributed PostgreSQL as an extension

TiDB

TiDB open-source, cloud-native, distributed SQL database ($ hosted option)

is this one lost??

Patroni

Patroni A template for PostgreSQL High Availability with Etcd, Consul, ZooKeeper, or Kubernetes

Embedded replicated databases (in-browser/wasm)

PGLite

PGLite Run a full Postgres database locally in WASM with reactivity and live sync from ElectricSQL

  • Postgres in WASM
  • Can integrate with ElectricSQL for reads, writes being worked on
  • IndexedDB and OPFS storage
  • Code

Embedded replicated databases (in-process)

PoloDB

PoloDB is an embedded document database.

Pouch

Pouch very cool, couchdb implementation for the browser. awesome sync, but targets web runtime. Supposedly can replicate with other similar impls like Couchbase

SurrealDB

SurrealDB Build real-time apps faster with the world's most powerful multi-model database

Bolt DB

Bolt DB (forks)

chDB

chDB a fast in-process SQL OLAP Engine powered by ClickHouse.

  • You can use it when you want to get the power of ClickHouse in a programming language without needing to connect to a ClickHouse server.
  • client runtimes: python, go, rust, nodejs, bun
  • https://clickhouse.com/docs/en/chdb

Standalone replicated databases

… that are within the sync ecosystem

Cockroach

Cockroach cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement ($ hosted option) new mandatory telemetry for free version

CockroachDB is a multi-master distributed database. It uses a distributed consensus protocol (an extended version of Raft) to maintain consistency across multiple nodes, where any node can accept both read and write operations.

  • True multi-master architecture
    • Every node can handle reads and writes
    • No primary/secondary distinction
    • Automatic load balancing
  • Consistency Model
    • Serializable isolation level
    • Strong consistency (CP in CAP theorem)
    • ACID transactions across nodes
  • Replication
    • Automatic sharding
    • Range-based data distribution
    • Configurable replication factor
  • Conflict Resolution
    • Timestamp-based conflict resolution
    • Uses hybrid logical clocks
    • Automatic handling of concurrent writes

AntidoteDB

AntidoteDB A planet scale, highly available, transactional database built on CRDT technology ($ hosted option)

  • AntidoteDB is a highly available geo-replicated key-value database. AntidoteDB provides features that help programmers to write correct applications while having the same performance and horizontal scalability as AP/NoSQL databases. Furthermore, AntidoteDB operations are based on the principle of synchronization-free execution by using Conflict-free replicated datatypes (CRDTs).
  • Code

Fauna

Fauna true serverless database ($ commercial only)

true serverless database that combines document flexibility with relational power, automatic scaling, and zero operational overhead.

Firebase

Firebase (Google) ($ commercial only)

CouchDB

CouchDB and Couchbase

Defra DB

Defra DB Peer-to-Peer Edge Database

Chotki

Chotki An LSM database turned a CRDT database

Clickhouse

Clickhouse TBD

- [clickhouse-local](https://clickhouse.com/docs/en/chdb/guides/clickhouse-local) cli with embedded db

FoundationDB

FoundationDB (Apple)

TigerBeetle

TigerBeetle The financial transactions database designed for mission critical safety and performance.

  • TigerBeetle is an Online Transaction Processing (OLTP) database built for safety and performance. It is not a general purpose database like PostgreSQL or MySQL. Instead, TigerBeetle works alongside your general purpose database, which we refer to as an Online General Purpose (OLGP) database.
  • Clustered replicas
  • https://docs.tigerbeetle.com/coding/system-architecture/
  • [[Interesting Architectures]]

greptimedb

greptimedb An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported ($ hosted option)

  • GreptimeDB is an open-source unified & cost-effective time-series database for Metrics, Logs, and Events (also Traces in plan).
  • Built for k8s, Single-master

While multiple nodes can handle reads, write operations are coordinated through primary nodes.

vector

Kuzu

Kuzu DB graph database ✅

For ETL workloads

Artie

Artie

Mycelial

Mycelial

  • new, and used for ETL workloads

Local and Source Databases

Hosted/Edge SQLite

Cloudflare D1

  • Cloudflare D1

Turso

Hosted Postgres

  • Supabase Postgres (and others) ($ commercial)
  • Neon hosted postgres ($ commercial)
  • Railway ($ commercial)
  • Fly Postgres ($ commercial)

In-process and/or in-browser (wasm) Databases

SQlite

  • SQlite and wa-sqlite

Duckdb

Duckdb DuckDB is an analytical in-process SQL database management system ($ hosting option)

Realtime Libraries and Components

Realtime UI Components

Sync Merge and Conflict Management

  • Yjs A CRDT framework with a powerful abstraction of shared data
  • Automerge a library of data structures for building collaborative applications
  • Loro Make your JSON data collaborative and version-controlled with CRDTs
    • Rust, JS (via WASM), and Swift
    •  rich text CRDT
    • hierarchy data with movable tree algo
  • Diamond Types The world's fastest CRDT. WIP.
    • rust, js (wasm)
  • impl of the Diamond Types sequence crdt
  • Various Dart implementations of CRDTs and SQL databases
  • Watermelon reactive database framework
    • iOS, Android, Windows, web, and Node.js; React/React-Native but framework-agnostic; reactive via RxJS
    • Local sqlite
    • Watermelon sync protocol, pull and push
    • BYO backend and source db, existing impls in TS (firemelon), Elixir, Laravel (ew)

Messaging protocols or frameworks

Peer to Peer/Decentralized Systems

Gun

Gun An open source cybersecurity protocol for syncing decentralized graph data

GUN is an ++ecosystem++ of tools that let you build ++community run++ and ++encrypted applications++ - like an Open Source Firebase or a Decentralized Dropbox. The ++Internet Archive++ and ++100s of other apps++ run GUN in-production.
Multiplayer by default with realtime p2p state synchronization!

  • Graph data lets you use key/value, tables, documents, videos, & more!

  • Local-first, offline, and decentralized with end-to-end encryption.

  • e2e encryption

Holepunch

Holepunch Pears Holepunch is a platform for creating apps that don’t use any servers whatsoever

Pear by Holepunch is a combined Peer-to-Peer (P2P) Runtime, Development & Deployment tool.

OrbitDB

OrbitDB Peer-to-Peer Databases for the Decentralized Web

  • too much tuna: Interesting Architectures IPFS based

WASM

Needs much more work

SQL.js

SQL.js A javascript library to run SQLite on the web.

SQLite, wasm, WebRTC

SQLite, wasm, WebRTC (PoC)

jawsm

jawsm JavaScript to WASM compiler (experimental)

WasmEdge

WasmEdge Runtime

  • For compiled languages (e.g., C and Rust), WasmEdge WebAssembly provides a safe, secure, isolated, and containerized runtime as opposed to Native Client (NaCl).
  • For interpreted or managed languages (e.g., JavaScript and Python), WasmEdge WebAssembly provides a secure, fast, lightweight, and containerized runtime instead of Docker + guest OS + native interpreter.
  • WASM VM
  • A cloud native WebAssembly runtime. A secure, lightweight, portable and high-performance alternative to Linux containers.
  • Lightweight database clients in the WasmEdge Runtime:
  • https://github.com/WasmEdge/wasmedge-db-examples

Communities, Projects, Resources

Articles

Graveyard

  • RemoteStorage old AF

  • RhizomeDB DEAD

  • Statecraft dead?

  • synQL not updated in a while

  • CRStore js only, built on cr-sqlite, not much movement lately. one guy

    Conflict-free replicated store.

inbox

algorithms like egwalker have access to all the information they need to do that. We store character-by-character editing traces from all users. And we store when all changes happened (in causal order, like a git DAG). This is far more information than git has. So it should be very possible to build a CRDT which uses this information to detects & mark conflict ranges when branches are merged. Then we can allow users to manually resolve conflicts.

Lies I was Told About Collaborative Editing, Part 1: Algorithms for offline editing / Moment devlog

Lies I was told about collab editing, Part 1: Algorithms for offline editing | Hacker News

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment