Ursula in Tonbo

Last updated 2 November 2026In Databases

Read this when you need to know what serves your databases and streams in a region: how Tonbo runs Ursula, the open-source durable streams server that holds every database and every stream.

The shape of a region

Each region that offers databases and streams runs one Ursula deployment on an Amazon EKS cluster, behind Tonbo’s gateway:

                      your host
                          |  HTTPS
                          v
   Network Load Balancer, TLS with an ACM certificate
                          |
                          v
   Envoy Gateway: a route for the databases host name
                          |
                          v
   Tonbo's streams gateway, 2 replicas: checks the token,
   finds the leader of the database's Raft group
          |               |               |
          v               v               v
   +-------------+ +-------------+ +-------------+
   |  voter 1    | |  voter 2    | |  voter 3    |
   |  zone a     | |  zone b     | |  zone c     |
   |  256 groups | |  256 groups | |  256 groups |
   |  Raft log   | |  Raft log   | |  Raft log   |
   +------+------+ +------+------+ +------+------+
          |               |               |
          +---------------+---------------+
                          |  background flush
                          v
                 S3, in the same region

A database is one stream, and so is each stream you create. Ursula hashes the stream’s path to one of the 256 groups, and one core of each voter serves that group, so databases spread across groups, and groups’ leaders spread across voters.

The gateway

HTTPS ends at a Network Load Balancer with a certificate from AWS Certificate Manager. Envoy Gateway routes the databases host name, through an HTTPRoute, to Tonbo’s streams gateway, a Tonbo service built on Ursula’s gateway library and run as two replicas on different nodes. For every request it:

1. Resolves the bearer token: an access token, a JWT that Tonbo signs (RFC 9068), or an API key, which it resolves with Tonbo’s identity service by introspection and caches for up to 30 seconds.
2. Checks that the token’s account may use the database, and refuses a write that adds data with 402 when the account cannot pay for it. Reads are never refused for billing.
3. Forwards the request to the leader of the database’s Raft group, which it learns per group, and sends it again to the new leader when a voter answers with a redirect.

API keys and tokens says which tokens to use.

The quorum and leadership

Each group replicates with Raft, through the OpenRaft library. The group’s leader orders its writes, appends each to its Raft log and replicates it to the other two voters in parallel. The write is acknowledged once two of the three hold it in their logs and it is applied:

  gateway --> leader of group g (voter 2)
                 |  append to its Raft log
                 +--------------------------+
                 v                          v
           voter 1: log it            voter 3: log it
                 |                          |
                 +---- the first answer ----+
                       makes 2 of 3
                 |
                 v
           apply to the stream, answer 200 with the offset

A voter that is not its group’s leader answers a write with a redirect to the leader, which the gateway follows. A read that must see the newest writes, such as the one an idle synced() makes, goes to the leader, which first confirms with a majority that it still leads.

Maintenance restarts

An upgrade replaces the voters one at a time, with a StatefulSet’s rolling update: Kubernetes replaces one voter and moves to the next only once the new one passes its readiness probe, Ursula’s /__ursula/ready. A stopping voter hands its leaderships to the other two, stops its groups, syncs its Raft log and records a clean shutdown, all within 20 seconds. Appends to a group whose leader moves wait a few seconds for the new one, and the extension sends them again until they are acknowledged. An upgrade that fails is not rolled back automatically: the voters not yet replaced keep running, and an operator decides.

walFsync: never

Tonbo runs Ursula with walFsync: never, which is also Ursula’s default for groups of three voters. A voter acknowledges an append once it is in the operating system’s page cache, without waiting for its disk; always would add an fsync to every acknowledgement, which costs write latency. Votes, each group’s log state and membership changes are always synced, and a graceful shutdown syncs everything.

Guarantees states when a commit is durable in a reader’s terms.