Operations
How Eden runs day to day: the keepers and their cadence, monitoring, what it costs, and the lessons from running the full stack on a live Robinhood Chain fork.
1. What runs
| Component | Hosting | Notes |
|---|---|---|
| Ops service (primary) | Managed container + volume | Runs every recurring protocol job |
| Ops service (standby) | Second container, own volume | Takes the leader lease if the primary stops |
| Postgres | Managed | Leader lease, replicated ops state, indexer database |
| Indexer + read API | Managed container | Ponder → Postgres; optional second instance behind a health-checked router |
| Keepers | Managed container | ROOT-LP sweep, treasury yield, graduation retries, scoring epoch pipeline |
| Web app and docs | Static / edge hosting | The app reads the indexer and two RPC providers |
| RPC | Two paid providers + the public endpoint | Ordered failover in every component; the public node is used for reads only |
2. Keeper jobs and cadence
| Job | What it does | Cadence |
|---|---|---|
| Fees | Sweep curve and pool fees and distribute them through the router | Every 6 h, only for tokens whose accrued fee is worth at least 10× the sweep gas (with an idle cap) |
| ROOT buyback | Buy ROOT with the buyback lane in small equal slices (dollar-cost averaging) | A new batch every 3 h, spread evenly across the following 3 h |
| Quality proofs and reward pokes | Prove thesis grades from finalised epochs; advance reward buckets only where a proof or flush is pending | Every 15 min |
| Finalise | Finalise epoch roots after the 72 h challenge window; record token scores | Hourly |
| Scoring epoch | Quality review → human blend → publish the signed root | Daily |
| Systematic re-score | Re-fetch project data | Every 6–12 h |
| Holder snapshots | Post one Merkle root per token and asset | Daily |
| Index inflow | Move the index lane's stablecoin into the index vault | Hourly, above a minimum |
| Index epoch | Deploy idle stablecoin into the basket within the epoch cap | Every 12 h, minimum $25 |
| Index price pushes | Push constituent prices | On a 0.25 % move or hourly |
| Oracle monitor | Chainlink age, launch-burn quote | Every minute |
| Launch discovery | Pick up new launches | Every 30 min |
| ROOT-LP sweep / treasury yield | Weekly keepers | Friday 17:00 / 15:00 UTC |
| Cost report | Gas ledger and incident log snapshot | Daily |
Every keeper write is simulated first and never broadcast if the simulation reverts. Every broadcast hash is persisted before its receipt is awaited, so a crash never double-sends.
3. High availability and keys
- Leader lease. Primary and standby share one lease row in Postgres. Only the holder runs jobs, funds roles or sends alerts, and it re-checks the lease (15 s fence) before every broadcast. State and artifacts are replicated every 5 / 30 s. Measured takeover: 39 s on the demo, 66 s on a mainnet fork. The primary takes the lease back two minutes after it returns.
- Per-duty keys. Fees (sweeps, distribute, buybacks, LP sweeps) and quality (proofs, pokes, finalise) use separate keys from holder posting, attestation and index pricing. Each key holds a small ETH balance.
- RPC failover. Ordered provider lists in ops, keepers, indexer and app; a probe checks every endpoint each minute.
- Deploy order. The standby is deployed first and must report healthy before the primary is touched; a pre-deploy import check aborts a broken image.
4. Monitoring and alerts
| Signal | Threshold | Route |
|---|---|---|
| Keeper transaction failures | 3 consecutive | Critical |
| Job overdue | Next run + max(20 min, 3 × cadence) | Warning |
| Role ETH balance | < 0.3 ETH keeper-class, < 0.03 others | Warning; critical at a third of that |
| Chainlink ETH/USD and stock feed age | ≥ 50 % of max age | Warning; critical at 90 % |
| Launch-burn quote reverts | Any | Critical |
| Indexer lag | > 120 s | Critical |
| RPC down | 3 min | Warning; critical at 10 min |
| Safe-queued change executable | ETA passed, still pending | Warning |
| Zap holds a balance | Non-zero at block end | Warning |
| Ops heartbeat | Two missed periods (dead-man check) | Critical |
Critical alerts go to a Telegram chat with phone notifications; warnings go to a team channel. Alerts are de-duplicated and rate-limited, and critical ones are never dropped. The ops service also serves live cost and health views.
5. What it costs
Gas on Robinhood Chain is cheap (L2 base fee at its 0.02 gwei floor). The model below is measured from the demo's per-transaction gas ledger (same contracts, same keeper cadence, 64 live tokens traded around the clock by bots) and priced at mainnet gas prices and ETH ≈ $2,460. Trades are user-paid; keeper cost scales with the number of live tokens, not with volume.
5.1 Protocol gas per month
| Tier | Live tokens | Scores / day | Low | Typical | High |
|---|---|---|---|---|---|
| Launch month | 50 | 20 | $369 | $371 | $3,746 |
| Demo today | 64 | 20 | $461 | $464 | $4,678 |
| Steady | 250 | 100 | $1,706 | $1,715 | $17,238 |
| Growth | 1,000 | 400 | $6,718 | $6,753 | $67,830 |
"High" assumes 10× today's L2 gas price and L1 data price.
| Job (demo today, typical) | Txs / day | $ / month |
|---|---|---|
| Fee sweeps, distribution, buybacks | 516 | $390 |
| Index price pushes | 442 | $24 |
| Scorer reward pokes and flushes | 86 | $18 |
| Holder snapshots | 109 | $16 |
| Quality proofs (modelled, one per score) | 20 | $5 |
| Finalise | 47 | $5 |
| Everything else | — | $7 |
5.2 Infrastructure per month
| Item | $ / month |
|---|---|
| Protocol gas (launch month, typical) | ≈ 370 |
| Two RPC providers | 50–250 |
| Containers + Postgres (ops, standby, indexer) | 45–120 |
| Second indexer + database (optional) | 25–60 |
| Static hosting, health checks, alerts | 0–20 |
| Total | ≈ 465–820 |
5.3 One-time launch
| Part | Transactions | Gas | Cost at mainnet prices |
|---|---|---|---|
| Genesis ceremony (deploy, Safe hand-off and wiring) | 199 | 129 M | ≈ $6.40 (high $64) |
| Stock pair approvals (Safe batches) | 20 | 44 M | ≈ $2.20 |
5.4 Further savings (not needed financially)
Batching sweeps and distribution for many tokens in one transaction, one holder root per epoch for all tokens, and batched quality proofs would each need a contract change. They matter more for key exposure and RPC load at the growth tier than for cost.
6. Design lessons from the live demo
The full stack ran on a hosted fork of Robinhood Chain with real role keys, a 2-of-3 Safe, the production deploy scripts and bots trading every token. The incidents that carry over to mainnet shaped the design:
| Lesson | Design response |
|---|---|
| One keeper process with one key is a single point of failure | Hot standby with a leader lease; per-duty keys |
| An indexer can drift or freeze silently | Lag watchdog with automatic restart; lag in the health endpoint; receipt-first UI; optional second indexer behind a router |
| A ceremony step can be skipped (a stock-pair approval batch was missed once) | The ceremony has explicit steps for pairs and the zap; a role-and-sink preflight and a parameter-parity check (≈ 2,700 values compared against the demo) are go/no-go gates |
| Timelocks delay fixes: stock feeds waited 48 h after launches opened | Genesis runs two days before launches open, so every timelocked pair setting is live at opening |
| Alerts that go nowhere are not alerts | Telegram routing with dedupe, rate limits and a test nonce; heartbeat dead-man check |
| Public RPC endpoints rate-limit and keep only minutes of state | Two paid providers with failover; a dedicated endpoint for the ceremony |
| Oracle-priced writes can run out of gas at inclusion | Gas padding for every oracle-priced write in the app, ops and keepers |
| A key can leak through an operator console | Secrets in a sealed store; variable names only in listings; rotation by Safe transaction is rehearsed |
| On-chain text is immutable | The app masks personal identifiers in launch text |
| The operator's machine should not be in the path | Keepers on managed hosting; the ceremony runs from a clean dedicated machine |
Demo-only issues (the forked node's memory limits and restarts, state-file reloads, rotating RPC tokens, price mirrors, test funding) do not exist on the real sequencer and are not carried over.