> Fetch the index of every Tonbo docs page as Markdown at https://tonbo.dev/llms.txt

# Ursula in Tonbo

Last updated 29 October 2026

Read this when you need to know what serves your databases and streams in a region: how Tonbo runs [Ursula](https://ursula.tonbo.io/docs), the open-source durable streams server that holds every database and every stream.

## The shape of a region

Each region that offers databases and streams runs [one Ursula deployment](https://ursula.tonbo.io/docs/deploy-cluster/#on-aws-eks) on an [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) cluster, behind Tonbo’s gateway:

```text
                      your host
                          |  HTTPS
                          v
   Network Load Balancer, TLS with an ACM certificate
                          |
                          v
   Envoy Gateway: a route for the databases host name
                          |
                          v
   Tonbo's streams gateway, 2 replicas: checks the token,
   finds the leader of the database's Raft group
          |               |               |
          v               v               v
   +-------------+ +-------------+ +-------------+
   |  voter 1    | |  voter 2    | |  voter 3    |
   |  zone a     | |  zone b     | |  zone c     |
   |  256 groups | |  256 groups | |  256 groups |
   |  Raft log   | |  Raft log   | |  Raft log   |
   +------+------+ +------+------+ +------+------+
          |               |               |
          +---------------+---------------+
                          |  background flush
                          v
                 S3, in the same region
```

| Part | In each region |
| --- | --- |
| Ursula | Release 0.11.0, the image pinned by digest |
| Voters | 3, one in each of three [availability zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html), each alone on its own node (arm64, 4 vCPUs, 32 GiB) |
| Placement | Required [anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) on the zone and a [topology spread](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) of at most one, so no two voters share a zone |
| Raft groups | 256, each with a replica on all three voters, and its own leader |
| Raft log | On each voter’s own 100 GiB [gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html) volume |
| Cold tier | S3 in the same region, shared by the three voters |

A database is one stream, and so is each stream you create. Ursula [hashes the stream’s path to one of the 256 groups](https://github.com/tonbo-io/ursula/blob/v0.11.0/docs/architecture/thread-per-core-multiraft.md#stream-placement), and one core of each voter serves that group, so databases spread across groups, and groups’ leaders spread across voters.

## The gateway

HTTPS ends at a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) with a certificate from [AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html). [Envoy Gateway](https://gateway.envoyproxy.io/) routes the databases host name, through an HTTPRoute, to Tonbo’s streams gateway, a Tonbo service built on Ursula’s gateway library and run as two replicas on different nodes. For every request it:

1. Resolves the bearer token: an access token, a JWT that Tonbo signs ([RFC 9068](https://www.rfc-editor.org/rfc/rfc9068.html)), or an API key, which it resolves with Tonbo’s identity service by [introspection](https://www.rfc-editor.org/rfc/rfc7662.html) and caches for up to 30 seconds.
2. Checks that the token’s account may use the database, and refuses a write that adds data with `402` when the account cannot pay for it. Reads are never refused for billing.
3. Forwards the request to [the leader of the database’s Raft group](https://ursula.tonbo.io/docs/examples/sqlite-vfs/#what-the-cluster-needs), which it learns per group, and sends it again to the new leader when a voter answers with a redirect.

[API keys and tokens](https://tonbo.dev/docs/platform/api-keys.md) says which tokens to use.

## The quorum and leadership

Each group replicates with [Raft](https://raft.github.io/), through the [OpenRaft](https://docs.rs/openraft/0.10.0-alpha.37/openraft/) library. The group’s leader orders its writes, appends each to its Raft log and replicates it to the other two voters in parallel. The write is [acknowledged once two of the three hold it](https://ursula.tonbo.io/docs/architecture/overview/#write-path) in their logs and it is applied:

```text
  gateway --> leader of group g (voter 2)
                 |  append to its Raft log
                 +--------------------------+
                 v                          v
           voter 1: log it            voter 3: log it
                 |                          |
                 +---- the first answer ----+
                       makes 2 of 3
                 |
                 v
           apply to the stream, answer 200 with the offset
```

| Voters running | Writes |
| --- | --- |
| 3 | Every group writes |
| 2, such as during a restart or a zone outage | Every group writes: two voters are a majority. Groups the lost voter led elect a new leader first |
| 1 | No group writes until a second voter returns; commits wait on your host |

A voter that is not its group’s leader answers a write with a redirect to the leader, which the gateway follows. A read that must see the newest writes, such as the one an idle `synced()` makes, [goes to the leader](https://ursula.tonbo.io/docs/concepts/durability-and-consistency/#consistency), which first confirms with a majority that it still leads.

## Maintenance restarts

An upgrade replaces the voters one at a time, with a StatefulSet’s [rolling update](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#rolling-updates): Kubernetes replaces one voter and moves to the next only once the new one passes its [readiness probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/), Ursula’s `/__ursula/ready`. A stopping voter [hands its leaderships to the other two](https://ursula.tonbo.io/docs/operations/#graceful-voter-shutdown), stops its groups, syncs its Raft log and records a clean shutdown, all within 20 seconds. Appends to a group whose leader moves wait a few seconds for the new one, and the extension sends them again until they are acknowledged. An upgrade that fails is not rolled back automatically: the voters not yet replaced keep running, and an operator decides.

## walFsync: never

Tonbo runs Ursula with `walFsync: never`, which is also [Ursula’s default](https://ursula.tonbo.io/docs/configuration/#raft) for groups of three voters. A voter [acknowledges an append once it is in the operating system’s page cache](https://github.com/tonbo-io/ursula/blob/v0.11.0/docs/architecture/raft-wal.md#durability-contract), without waiting for its disk; `always` would add an `fsync` to every acknowledgement, which costs write latency. Votes, each group’s log state and membership changes are always synced, and a graceful shutdown syncs everything.

| Failure | Acknowledged writes |
| --- | --- |
| A voter’s process crashes, or restarts | Kept: the page cache survives the process |
| One voter’s host crashes or loses power | Kept: the other two hold them. The voter [keeps what it can verify](https://github.com/tonbo-io/ursula/blob/v0.11.0/docs/architecture/raft-wal.md#recovery-gate) and catches up from the leader before it votes again |
| One voter loses its disk | Kept: the voter rejoins empty and catches up |
| Two voters’ hosts, in two zones, crash at the same moment | The latest may be lost. The groups they hold stop until [an operator accepts the loss](https://ursula.tonbo.io/docs/operations/#recovering-after-a-host-crash), and the streams get a new incarnation |

[Guarantees](https://tonbo.dev/docs/databases/how-it-works.md#commit) states when a commit is durable in a reader’s terms.