governance-server
A JSON-RPC service that hands mining pools getblocktemplate templates built by the CUSF enforcer, accepts their submitblock calls, records every template and submission in SQLite, and shows it all on a dashboard with a JSON twin for every page.
Architecture
One process, one shared template cache. Pools never talk to the enforcer directly, and the service never asks bitcoind for a template: if the enforcer is down or its template is too old, pools get an error instead of a template that would drop the sidechain commitments.
flowchart LR pools["Pools and anyone checking
no login"] ops["Operators
browser / scripts"] prom["Prometheus"] subgraph gs["governance-server"] rpc["rpc.rs + clients.rs
JSON-RPC 1.0 + 2.0
limits per client IP"] cache["TemplateCache
watch channel
last 64 templates"] trig["triggers.rs
BMM poll + ZMQ"] dec["decode.rs + policy.rs
coinbase decode
NoopAnnotator"] side["sidechains.rs
snapshot every 30 s"] net["network.rs
network numbers every 15 s"] store["store.rs
one writer, read-only reader"] dash["dashboard
HTML pages + /api JSON"] end enfrpc["Enforcer JSON-RPC
:8122"] enfgrpc["Enforcer gRPC
:50051"] btc["bitcoind
RPC + ZMQ"] db[("SQLite")] pools -->|"getblocktemplate
submitblock"| rpc ops --> dash prom -->|"/metrics"| dash rpc -->|"wait_for_change"| cache trig -->|"request_refresh"| cache cache <-->|"GBT 2.0 + long-poll"| enfrpc cache --> dec rpc -->|"submitblock"| enfrpc rpc -.->|"also_submit"| btc enfgrpc -->|"GetSeenBmmRequests"| trig btc -->|"ZMQ sequence"| trig enfgrpc --> side btc -->|"headers, mempool"| net net --> dash cache -->|"TemplateSeen"| store rpc -->|"Submission"| store side -->|"Snapshot"| store store --> db db --> dash cache --> dash
Solid arrows are always on. The dashed bitcoind submit only runs when bitcoind.also_submit = true. The dashboard, /api and /metrics can be moved to a separate admin_listen port so pools never see them.
Upstream clients
upstream/enforcer.rs speaks JSON-RPC 2.0 through jsonrpsee and always asks for {"rules":["segwit"],"capabilities":["coinbasetxn","longpoll"]}. upstream/grpc.rs is a tonic client generated from the enforcer's own protos. upstream/bitcoind.rs is used for chain info, headers and the optional parallel submit.
Event bus
Request handlers and background tasks never touch the database. They send TemplateSeen, Submission and SidechainSnapshot events on a channel; a single writer task batches them. A slow disk cannot stall a pool's request.
Shutdown
Ctrl-C cancels one token. Listeners stop accepting, parked long-polls are answered with the current template, the writer flushes, and the process exits. The e2e test checks the long-poll part.
Config
TOML file plus GOVSRV_* env overrides (__ for nesting). Unknown keys are errors. Startup refuses an invalid dashboard password hash, zero limits, and an upstream long-poll timeout of 30 s or less. governance-server check validates without starting.
A pool calls getblocktemplate
- Parse.
rpc.rsaccepts a JSON-RPC 1.0 or 2.0 envelope, or a batch, and answers in the same version. It is written directly on axum because jsonrpsee's server rejects 1.0 requests. - Identify the client. There is no login. The client is the connecting IP address: one IPv4 address, or one IPv6 /64. Behind a proxy listed in
trusted_proxies, it is the rightmostX-Forwarded-Forentry the proxy added; the header is ignored from anyone else, so a client cannot choose its own identity. - Proposal mode.
mode: "proposal"is passed straight to the enforcer. - Long-poll or not. With a
longpollid, the request parks on the cache's watch channel until the template changes or 30 s pass. A client may hold 8 long-polls at once (4096 across everyone). Without alongpollid, the call counts against the client's rate limit (60/min, burst 10). Either limit answers error-32005. Batches over 16 requests are refused. - Serve. The cached template is returned with the service's own
longpollid(id:seq). If the cache has not been confirmed by the enforcer withinmax_template_age_s(65 s), the pool gets error-9instead.
Getting a new BMM bid into the template fast
The enforcer's own long-poll only wakes when a new block arrives (cusf-enforcer-mempool, lib/server.rs lines 909–921). A sidechain BMM bid that lands in the mempool mid-block would sit unseen until the 30 s long-poll timeout. Every second of that is a chance to mine a block without the bid, which loses the sidechain its block.
So the cache refreshes on four triggers: startup, the upstream long-poll returning, a change in the enforcer's GetSeenBmmRequests list (polled every second over gRPC), and a block-connect from bitcoind's ZMQ sequence feed. Any trigger fetches a fresh template. If its content is unchanged, the cache just renews its timestamp; if it changed, parked pool long-polls wake at once.
sequenceDiagram participant U as Sidechain user participant B as bitcoind participant E as Enforcer participant T as triggers.rs participant C as TemplateCache participant P as Pool (parked long-poll) U->>B: M8 bid tx enters mempool B->>E: mempool sync T->>E: GetSeenBmmRequests (every 1 s) E-->>T: new bid in list T->>C: request_refresh(Bmm) C->>E: getblocktemplate E-->>C: template with M7 accept in coinbase C->>C: decode, annotate, fold, publish C-->>P: long-poll wakes with new template
A pool calls submitblock
- Limit. 30 submissions per minute per client, burst 10, checked before the block is decoded.
- Match. The block's coinbase is compared with the last 64 templates.
exactmeans the coinbase sidechain outputs and the tx list match;commitmentsmeans the sidechain outputs match but the pool changed the tx list. No match is still submitted and recorded. - Submit. Sent to the enforcer, and in parallel to bitcoind when
also_submitis on. - Combine. The enforcer's answer goes back to the pool, with one exception: if bitcoind already accepted the block, the enforcer then reports
duplicate, and the pool is told it was accepted. Both raw answers are stored. - Record. A
Submissionevent with the client address, template id, match kind, block hash, height, both results and latency.
What the decoder recognises
decode.rs classifies every coinbase output and every template transaction. Encodings were taken from the enforcer source, not the BIP text, and the tests run on coinbases captured from a real enforcer.
| Message | Meaning | Where | Tag bytes |
|---|---|---|---|
| M1 | Propose a sidechain | coinbase OP_RETURN | D5E0C4AF |
| M2 | Ack a proposal | coinbase OP_RETURN | D6E1C5DF |
| M3 | Propose a withdrawal bundle | coinbase OP_RETURN | D45AA943 |
| M4 | Bundle votes (subversions 00–03) | coinbase OP_RETURN | D77D1776 |
| M7 | BMM accept (h* commitment) | coinbase OP_RETURN | D1617368 |
| M8 | BMM request / bid | mempool tx, output 0 | 00BF00 |
| Treasury | Sidechain deposit UTXO (CTIP) | tx output | OP_NOP5/8 01 <slot> 51 |
The decoded result feeds the TemplateAnnotator hook in policy.rs. Today it is NoopAnnotator, which accepts every template unchanged. The future policy feed will plug in here and can add a governance field or reject a template.
SQLite: fold repeats, share transaction lists
The enforcer returns the same template many times. Storing each copy would be wasteful, so templates are folded: a repeat with the same content key only moves last_seen, curtime_last and poll_count. The key covers prevhash, version, bits, height, the full coinbase (so a new M7 or M2 is a new row) and the txid list.
templates
One row per distinct template: header fields, fees, weight, commitment summary (e.g. m2:1 m7:1), decoded coinbase outputs as JSON, and what triggered the first fetch. Kept 90 days.
txsets + template_txs
Content-addressed transaction lists. Two templates that differ only in the coinbase share one list. Kept 3 days.
submissions
Every submitblock: client address, matched template, match kind, block hash, enforcer and bitcoind results, latency.
sidechain_snapshots
Periodic sidechain state from gRPC, kept 30 days. Per-client usage is in memory only (at most 4096 clients), not stored.
One writer task flushes every 10 s and prunes every hour. The dashboard reads through a separate read-only connection pool.
Dashboard and metrics
Server-rendered askama pages sharing one stylesheet in the same style as this document, with a small script for auto-refresh and the light/dark switch. Every data page has a JSON twin under /api, so anything on screen can be scripted. This document is served at /architecture inside the dashboard layout.
| Page | JSON twin | Shows |
|---|---|---|
| / | /api/mining | Mining overview, public: network numbers, ways to mine, the live pool list, eCash facts, resources |
| /dashboard | /api/overview | Network numbers (also at /api/network), current template, upstream health, recent clients with addresses shortened (IPv4 /24, IPv6 /48) |
| /templates | /api/templates | Folded template history with commitment summaries |
| /templates/{id} | /api/templates/{id} | Decoded coinbase, transactions, and a diff against the previous template |
| /sidechains | /api/sidechains | Each slot: status, ack progress, CTIP, withdrawal bundles and votes, recent BMM, live bids |
| /submissions | /api/submissions | Every block submitted, by shortened client address, with match kind and results |
| /catalog | /api/catalog | API catalog: every endpoint with parameters, the server's current limits, example curl calls and JSON-RPC error codes |
| /architecture | — | This page, compiled into the binary |
Prometheus metrics on /metrics:
govsrv_gbt_calls_total govsrv_template_changes_total govsrv_rate_limited_total govsrv_template_height govsrv_long_poll_waiters govsrv_template_tx_count govsrv_submit_results_total govsrv_upstream_errors_total govsrv_auth_failures_total govsrv_upstream_latency_seconds
Modules
| Path | Lines | Responsibility |
|---|---|---|
| src/main.rs | 238 | run, check, hash-password; wires tasks and shutdown |
| src/config.rs | 287 | TOML + env loading and validation |
| src/upstream/ | 660 | Enforcer JSON-RPC, enforcer gRPC (tonic), bitcoind clients |
| src/template/ | 634 | Cache, refresh loop, triggers, dedup key, template view |
| src/rpc.rs | 447 | Downstream JSON-RPC, long-poll, rate limits, submit matching |
| src/decode.rs | 462 | Coinbase and transaction classification |
| src/policy.rs | 53 | TemplateAnnotator trait and NoopAnnotator |
| src/clients.rs | 384 | Client identity, rate limiters, long-poll slots, usage table |
| src/auth.rs | 117 | Argon2 basic auth for the optional dashboard login |
| src/network.rs | 547 | Network numbers from bitcoind; eCash 288-block retarget estimate |
| src/registry.rs | 150 | Public pool list from the pool registry |
| src/sidechains.rs | 149 | Sidechain snapshot task |
| src/store.rs | 576 | SQLite writer, reader, folding, pruning |
| src/dashboard/ | 2,067 | Pages, JSON twins, formatting, API catalog, mining overview |
| src/metrics.rs | 179 | Prometheus registry |
| tests/regtest.rs | 691 | End-to-end regtest test |
| proto/ | — | Enforcer .proto files, copied unmodified |
| docs/upstream.md | — | Enforcer interfaces and encodings, with source links |
Main crates: axum 0.8, jsonrpsee 0.26 (client only), tonic 0.14, bitcoin 0.32, sqlx 0.8 (SQLite), askama 0.14, figment, governor, argon2, zeromq, prometheus.
Findings
New BMM bids would wait up to 30 s. Handled with the BMM poll and ZMQ triggers.
-txindex on bitcoind.Without it the enforcer did not pick up an M8 bid until restarted; with it, under 500 ms. Seen on Core v29.2 only, so marked TODO(verify). If it reproduces on a supported Core, it is an upstream bug that costs BMM blocks.
curtime can be below mintime.A block stamped with curtime is rejected as time-too-old. Pools must use max(curtime, mintime); documented in the README.
coinbasetxn, and the enforcer's JSON-RPC needs content-type: application/json. Don't run it with --gbt-cache-lifetime-s: a cached template hides new bids.GOVSRV_CONFIG broke config loading.The env prefix treated it as a config key. It is now ignored by the loader.
duplicate for a good block with also_submit.bitcoind stored the block first, then the enforcer said duplicate. The pool is now told it was accepted.
Changes from the brief
- Downstream server on axum, not jsonrpsee. jsonrpsee's server rejects JSON-RPC 1.0, which the brief requires accepting. jsonrpsee is still the upstream client.
- Decoder landed one commit early, in the template-cache commit, because the policy hook needs decoded coinbases.
- Your choices applied: trigger on SeenBmm poll + ZMQ; tonic + prost for gRPC;
coinbasevaluepassed through as the enforcer returns it. - Not built, as asked: the policy feed. Only the no-op hook exists.
Testing
Unit tests (20)
Coinbase decoding against real captured fixtures (M1 proposal, M2 ack, M7 BMM accept), template folding, shared and changed tx lists, pruning, JSON-RPC envelopes, submit result combining, config parsing, per-client limits, IPv6 grouping and X-Forwarded-For handling.
End-to-end regtest
Starts bitcoind and the enforcer, runs the service, and mines every block through it with a small pool client that swaps the payout output the way a pool would. It proposes and activates a sidechain, places a BMM bid and checks it reaches a template, submits blocks, checks the batch limit and the shortened client addresses, and checks shutdown answers parked long-polls.
REGTEST_BITCOIND=/path/to/bitcoind \ REGTEST_ENFORCER=/path/to/bip300301_enforcer \ cargo test --test regtest -- --nocapture
Skipped unless both paths are set. scripts/regtest.sh up|down|cli starts the same stack by hand; scripts/capture_fixture.py captures new coinbase fixtures.
Open items
Before relying on this
- Run the e2e test on a supported Core (30.4, 31.2 or 32.1). It has only run against v29.2 with the enforcer's version check skipped; downloading the patched release was blocked by a permission check. This also settles the
-txindexquestion. - Licence. The enforcer repo has no licence file, and its
.protofiles are copied intoproto/. - Withdrawal-bundle ID byte order. Shown as the enforcer returns it, consistent with the coinbase votes; not yet compared with how explorers show it.
- Set
trusted_proxiesto your reverse proxy's address in production. Otherwise every client behind the proxy shares one set of limits.
Commits
| 3e8b734 | docs: upstream enforcer interfaces (RPC, gRPC, encodings) |
| 7389cdc | upstream clients: enforcer JSON-RPC, enforcer gRPC (tonic), bitcoind |
| 7d70a1d | template cache + downstream JSON-RPC |
| 48ec412 | persistence: SQLite store with a single writer and a read-only reader |
| 3dc5edc | decode: real enforcer coinbase fixtures and capture script |
| 7ca1edb | dashboard: overview, templates, template detail, submissions |
| dafb3b9 | sidechain view: slots, ack progress, ctip, bundles, recent BMM, live bids |
| 4c4a3e1 | tests, metrics, docs: regtest end-to-end test and README |
| 8980457 | docs: architecture overview page |
| 82fbb73 | rpc: open access with per-client limits, no pool logins |
The build stages and the changes after them. For anything newer, see git log.