Use this when your jobs report progress that a web page shows live, and you keep that history for about 90 days: how to lay out the streams, keep a page live with no key in it, and what it takes and costs.
The example: 500 jobs a day, each reporting 2,000 progress events of 300 bytes, kept 90 days, read live by one web UI and afterwards by one archiver. Your numbers will differ, and the reasoning carries over.
One stream a day
A stream per job is the simplest to read, but kept 90 days it means 45,000 streams, past the account’s 10,000 (see Limits): at 500 jobs a day, streams per job fit only if each expires within 20 days.
There is no retention by time. A stream keeps everything until it expires, you delete it, or you advance its retention to a snapshot. So keep about 90 days with one stream per day, named by its date in UTC and created with an expiry 90 days out, which deletes the day once it is that old:
A single stream kept for all time could drop its old history only through a snapshot and retention, which suits a log you can summarize, not an audit log you keep word for word (see Snapshots and retention). One stream a day drops a whole day on its own instead, when the day expires. Each event names its job, so a reader picks out the jobs it shows, and the account holds about 90 such streams at a time.
Days, jobs and producers
A job appends to the stream of the day, in UTC, on which it started, so its events stay together: a job that runs past midnight keeps appending to the day before. Close a day’s stream once every job that started that day has ended, so its readers know it is complete (see Close):
A scheduled job creates tomorrow’s stream ahead of time, say an hour before midnight UTC, with a key that acts as you or as an Agent and is not narrowed, since a narrowed key never creates a stream (see Narrow a key). It fixes the expiry by the day, 90 days after its midnight, rather than by when the job runs, and gives the stream’s name as the idempotency key, so a retry repeats the same creation and answers the same stream:
Through the API, a retry sends the same expires_at and Idempotency-Key. A retry that names another expiry answers 409, and so does a creation under another key once the stream exists: the stream is there, and the job carries on (see Troubleshooting).
Each job is one producer of its day’s stream: its id as Producer-Id, and its count of appends as Producer-Seq. A stream takes up to 4,096 producers (see Limits), so a day’s 500 jobs, and those that ran past midnight, fit. A job that restarts without its count moves to the next Producer-Epoch from sequence 0: keep its epoch with the job’s own state, and add one at each restart (see Append).
Write in batches
Appended one at a time, 1,000,000 events a day are about 11.6 appends a second on average, and more at peaks, against the account’s 50 requests a second or more in the region, which every read shares. A writer that gathers a job’s events into one JSON array per append, as many as fit in one write of 10 KiB, sends far fewer requests and writes (see Batch messages): 34 events of 300 bytes come to 10,200 bytes, and 10,235 with the array’s brackets and commas, right at the 10,240 bytes of one write, so a little more in an append makes it two writes: leave room.
A job that reports a few events a second does not fill a batch quickly. A writer that flushes on a timer sends what it has when the timer fires: with a one-second timer and about 3 events a second, each append carries about 3, and the month costs several times what full batches do (see What it costs). Flush when a batch is full or the timer fires, whichever comes first: the timer bounds how late the page sees an event, and the batch bounds the cost. Send each batch with producer headers, so a batch sent again applies once.
Live progress to a web UI
A page should never hold an API key. A page could tail the day’s stream itself with a short-lived token your server mints for it (see A token for a web page), but every page would then hold a tail open, and the day’s stream holds every job, so its token would reach other jobs’ progress too. Here, your server holds the key, and one tail on the day’s stream serves every page:
Your server keeps one Server-Sent Events tail open on the stream, with its key, and passes each event on to the pages that watch its job, over its own connection to them:
When the tail drops, which a proxy may do to a quiet connection since the stream sends no heartbeats, it opens it again from the last streamNextOffset it saw, and misses nothing (see Tail).
A page that connects late, or reconnects, gets what it missed from a read, the one below with an offset in place of -1, then the live events:
What the stream hands your server is an offset for each batch, not each message. A tail sends each batch of appends as an event: data, then an event: control whose streamNextOffset follows the batch, and a read answers all its messages with one Stream-Next-Offset after them. So give each page, with each batch, the offset after it, and number the events inside each message yourself, such as with the job’s id and its count. A page that reconnects names the last offset it was given; your server reads from there and sends that batch again, and the page drops the events whose numbers it has already shown.
Catching up from the start of a day reads every job of that day: here about 300 MB, at most 8 MiB an answer. Keep the offset where each job began, the Stream-Offset its first append answered, and catch a page up on one job from there: the read then skips what came before the job.
One tail per stream on your server, not one per viewer, so the account’s open tails bound your servers, not the people watching: 16 or more in a region, since each of the region’s two gateways counts them on its own (see What “at least” means). If the progress may be public, publish the stream instead: a page then reads and tails it with no token at all (see Public streams).
Python, Go and other languages
These pages show curl, and there is no Tonbo example in Python or Go yet. From any language, you write against the HTTP these pages document: the producer headers and their count, the parsing of Server-Sent Events, and reopening a tail from its last offset.
The archiver
A day’s stream deletes itself at its expiry. To keep the log longer, or in a system of your own outside Tonbo, an archiver reads each day’s stream once it is closed, from the start until an answer says Stream-Up-To-Date: true, and writes what it read there (see Read).
What it takes
This example
What the account has
Streams
90, one a day, and any not yet deleted after their expiry
10,000
Open tails
One for the UI’s server, and one more if the archiver tails
16 or more in the region
Requests a second
About 11.6 appends one by one, about 3.9 on a one-second timer, or about 0.3 in full batches, plus reads
500 jobs a day, each appending 2,000 progress events of 300 bytes to the day’s stream, which expires 90 days after it is created, so the account keeps 90 days of events. One design appends each event alone; one flushes a one-second timer while a job reports about 3 events a second; one sends full batches of 34 events, as many as fit in 10 KiB.
Case
Writes
Writes cost
Storage
Month
After the free $5
One append per event
30,000,030
$45.00
$1.26
$46.26
$41.26
About 3 events per append, on a one-second timer
10,005,030
$15.01
$1.26
$16.26
$11.26
34 events per append
885,030
$1.33
$1.26
$2.58
$0.00
$46.26 one by one, $16.26 on the timer and $2.58 in full batches: the stored events cost the same each time, and what an append holds sets the writes and the requests.