fix(sync-service): terminate serves whose clients stop accepting data

[robacourt]

Summary

Defense-in-depth follow-up to #4708: reap shape response serves whose clients stop accepting data. Response bodies are now written to the socket in ≤256 KiB pieces (the encoder batch cap; only oversized single items are split), and a per-serve watchdog terminates the handler when a single piece fails to complete within ELECTRIC_STALLED_SERVE_TIMEOUT (default 60s; 0 disables). A terminated client reconnects and resumes from its last offset.

The production incident (2026-07-01)

Same incident as #4708: a customer node OOM'd after ~400 serves to stalled clients accumulated over an hour, each pinning its in-flight response chunk (~3.9 GB total, live references, GC-immune). Two properties made the population unbounded:

1. Stalled serves are invisible — they never complete, so they emit no spans and increment no request counters. The node looked near-idle while dying; the population showed up only as a process-count rise tracking the memory curve.
2. Nothing reaps them. The 30s TCP send_timeout only fires when a single write is fully blocked for its whole window. In the incident, ~400 stalled serves produced exactly 2 send-timeout errors in an hour — trickling clients (or a proxy buffering for vanished ones) evade it indefinitely. conn_max_requests and idle timeouts apply between requests, never mid-response. There is no total-response deadline anywhere in the stack.

#4708 bounded what each stalled serve pins (~19.6 MB → <1 MB). This PR bounds how long one can exist, which also caps the population — and adds the observability that was missing: every reap logs a warning with the shape handle.

Why this design (two alternatives were disproven by measurement)

1. Time each write call — the natural first cut (watchdog around each Plug.Conn.chunk) was killed by our own guard test: a single write of a ~256 KiB body element to a slow-but-live client legitimately takes element_size / drain_rate — longer than any fixed deadline at some rate. Duration-per-write semantics are rate-coupled and over-reap healthy clients.

2. Poll socket statistics for progress — measuring send_oct/send_pend via :inet.getstat looked like true progress tracking. A probe disproved it: send_oct counts bytes at driver enqueue (it jumped to the full send size instantly), and send_pend stayed frozen while a test client steadily drained — the driver only moves data into the kernel in coarse bursts, so both statistics are blind to a trickle. (This also retroactively explains why the incident's serves survived send_timeout.)

3. Bounded pieces + per-piece deadline (this PR) — once writes are bounded, a completed write is itself the progress signal: after the transport buffers fill, each piece completes only when the client actually drains. One honest caveat, measured rather than assumed: write completion is quantized by the OS — a blocked write may only complete after the kernel send buffer frees a large fraction of its capacity (measured ~580 KB / ~29s at a 20 KB/s drain on macOS; Linux signals writability much more finely). So the documented contract is: a healthy client must drain roughly one OS send buffer per timeout window — on the order of 10 KB/s worst-case at the 60s default, trivial for any live client, unmeetable for a dead one.

Side benefit: the bounded write unit cuts the response data queued in a stalled socket's driver buffer from one body element (~256 KiB) to ~16 KiB, tightening #4708's per-connection bound further.

Implementation notes

• The watchdog is a small companion process per chunked serve, armed only while a write is in flight — serves idling between body elements (live long-poll holds, SSE waiting for changes) are never at risk, regardless of keepalive configuration.
• It monitors the handler and is stopped in an after block, so it cannot outlive the serve; the kill is Process.exit(:kill) since a wedged handler is blocked in prim_inet and cannot process messages.
• Config follows the existing tweaks plumbing (ELECTRIC_STALLED_SERVE_TIMEOUT → tweaks → StackConfig), runtime-tunable per stack.

Test plan

• New integration test: 3 live long-pollers on never-read sockets + a large transaction; every handler is terminated within the stall timeout (+ slack). RED before the fix.
• Guard test (caught design Working with Postgres WAL format from Elixir #1): a slow-but-draining client whose serve spans many timeout windows is never reaped and keeps receiving data.
• Stable across 3 seeds; storage suites, low_privilege_router_test, api_test, and all of test/integration/ green (including fix(sync-service): bound memory pinned by serves to slow or stalled clients #4708's memory-budget test). Full router_test including :slow-tagged tests green (98 tests) as of 877abf8 — an earlier revision broke two large-payload router tests in CI (see comments); the original claim here overstated the sweep by excluding :slow.
• mix compile --warnings-as-errors clean.

---

Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.05%. Comparing base (5a298c6) to head (d618f73).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4711   +/-   ##
=======================================
  Coverage   60.04%   60.05%           
=======================================
  Files         397      397           
  Lines       43766    43766           
  Branches    12587    12588    +1     
=======================================
+ Hits        26280    26282    +2     
+ Misses      17405    17403    -2     
  Partials       81       81           

Flag	Coverage Δ
packages/agents	72.64% <ø> (ø)
packages/agents-mcp	77.70% <ø> (ø)
packages/agents-mobile	80.67% <ø> (ø)
packages/agents-runtime	83.72% <ø> (-0.02%)	⬇️
packages/agents-server	75.67% <ø> (+0.05%)	⬆️
packages/agents-server-ui	8.32% <ø> (ø)
packages/electric-ax	51.06% <ø> (ø)
packages/experimental	87.73% <ø> (ø)
packages/react-hooks	86.48% <ø> (ø)
packages/start	82.83% <ø> (ø)
packages/typescript-client	91.89% <ø> (ø)
packages/y-electric	56.05% <ø> (ø)
typescript	60.05% <ø> (+<0.01%)	⬆️
unit-tests	60.05% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude Code Review

Summary

Incremental review (iteration 8). The only change since iter-7 is commit d618f737b, which adds a dedicated unit-test module for ServeWatchdog and a sweep_interval_ms start_link option so the tests can run sub-second. This closes the gap I noted last round — the watchdog's contract was previously exercised only through the full-stack integration tests. The tests are correct and well-targeted; no new issues. The two Important items from iter-7 are behavior-level and remain open (this commit was test-only).

What's Working Well

• The unit tests cover the watchdog's contract directly and at the right level. The five behavioural cases — reap past deadline (with the warning asserted), leave a within-deadline write, leave a finished write, let a new write reset the deadline, and prune a handler that died on its own — map one-to-one onto the state machine in keep_entry?/5, plus the fail-open path (write_started to a stack with no running instance returns :ok). That last case is a genuine correctness check: because name/1 yields a {:via, Registry, …} tuple, GenServer.cast swallows the unregistered-name error and returns :ok, so serving degrades to unguarded rather than crashing the handler — exactly the moduledoc's claim, now pinned by a test.
• The victim/1 helper is correctly designed for the mechanism under test. Registering from inside the spawned process is necessary (the cast captures self() as the watched pid), and using spawn + Process.monitor rather than spawn_link is deliberate: an untrappable Process.exit(:kill) on a linked victim would propagate to the test process. Monitoring side-steps that while still letting assert_receive {:DOWN, …, :killed} verify the exit reason.
• The sweep_interval_ms hook is the minimal, non-invasive change. It threads through start_link → init → state → schedule_sweep/1 with Keyword.get(opts, :sweep_interval_ms, @sweep_interval_ms), so the production default (1_000) is unchanged and only tests override it. No test-only conditionals leak into the hot path.
• async: true with per-test stack isolation. with_stack_id_from_test + the stack-scoped registry name means the started watchdog and the "no server" case can't collide across concurrent test modules — conforms to the project's async-test conventions.

Issues Found

Critical (Must Fix)

None.

Important (Should Fix)

None new. Two items from iter-7 remain open (both behavior-level, untouched by this test-only commit) — restated once, not re-litigated:

• ServeWatchdog.init/1 still omits Electric.Telemetry.Sentry.set_tags_context(stack_id: …) (serve_watchdog.ex:65-70), which the convention lists as required for every GenServer init/1 and which the sibling AsyncDeleter sets. Low impact, one line.
• No message-queue-depth telemetry on the per-stack watchdog. The "any new per-request-adjacent GenServer must be evaluated for contention at scale" pitfall still applies; a mailbox-depth metric would make a future regression visible rather than silent.

Suggestions (Nice to Have)

• The refute_receive victims (within deadline, finished, new write replaces) are left in Process.sleep(:infinity) and are unlinked, so they outlive the test as orphaned idle processes. This is the correct trade-off given the :kill-propagation reason above, and it's a fixed, tiny count (one run of one module), so no cleanup is warranted — just noting it's intentional, not an oversight, per @alco's earlier "don't over-add test cleanup" guidance.

Issue Conformance

No linked issue (references incident #4708) — nice-to-have per convention. PR description remains accurate; changeset present and correctly scoped (@core/sync-service patch). Codecov reports all modified lines covered, consistent with adding direct unit coverage for a previously integration-only module.

Previous Review Status

• Addressed since iter-7: the iter-7 observation that ServeWatchdog was covered only via integration tests is now resolved with a focused unit-test module ✅.
• Still open: Sentry.set_tags_context in init/1; queue-depth telemetry (both Important, both untouched by a test-only commit).
• No previously raised finding regressed; the sweep_interval_ms addition preserves the production default.

Reviewed statically — I did not run the suite (needs local Docker). Codecov and the author report green. Please confirm CI is green before merge.

---

Review iteration: 8 | 2026-07-15

[robacourt]

Review feedback addressed in 877abf8 — and correcting the test-evidence claim in the description.

The two RouterTest failures were real and caused by this branch, exactly as diagnosed: 16 KiB pieces turned multi-MB single elements (which the encoder byte cap cannot split) into 1,300+ chunk/2 calls, and the Plug.Test adapter's append-per-chunk made body readback quadratic. I missed it locally because can sync large binaries is @tag slow (excluded from my sweep) and the other test is fast enough on my machine to sneak under Task.await's 5s — the "full sweep green" claim was wrong for the slow-tagged test. The description is updated.

Fix: piece size raised to 256 KiB, matching the encoder's batch byte cap, with a fast path that writes in-bound elements as-is (no flattening — restores pre-watchdog zero-copy for the common case). Only oversized single items are split (20 MB element → ~80 writes, not ~1,300). This doesn't weaken the watchdog: measured write-completion granularity is quantized by the OS at up to a kernel send buffer (~600 KB) of drain, which dwarfs either piece size — the progress contract ("drain roughly one OS send buffer per timeout window") is unchanged.

Verified on the branch: both flagged tests pass, full router_test including :slow green (98 tests), both reaping tests + the #4708 memory-budget test green, --warnings-as-errors clean.

(Agreed the ShapeLogCollectorTest FlushTracker timeout looks like unrelated flakiness — happy to look at it separately if it recurs.)

[alco]

The :kill exit signal destroys the process without executing another instruction. Every ill effect below falls out of that one property: cleanup and accounting that the handler was going to do in code silently never happens.

1. Admission-control permit leak

ServeShapePlug.check_admission stashes the acquired permit in the process dictionary; the only* release site is the after clause in call/2 (serve_shape_plug.ex:94–107). The module doc explains after was chosen deliberately over register_before_send — correct for exceptions and halts, but after does not run on :kill.

Each watchdog reap permanently leaks an :existing admission permit for the lifetime of the admission-control state. Once enough permits leak to reach the configured limit, subsequent existing-shape requests are rejected with 503.

Cheapest fix: pass the held {stack_id, kind} to the watchdog at start and have the watchdog call AdmissionControl.release/2 after the kill. There is a caveat here: blindly releasing from the watchdog after calling Process.exit/2 could undercount if the handler completes and releases its permit before the asynchronous exit takes effect. Permit ownership should be transferred atomically, or the watchdog should monitor the handler and release only when it observes the watchdog-induced :killed exit while the handler still owns the permit.

2. Observability: reaps are invisible everywhere except one log line

The kill skips every emission the handler would have made on the way out:

• emit_shape_telemetry never runs, so [:electric, :plug, :serve_shape] and [:electric, :shape, :response_size] are never emitted — reaped serves stay permanently
  invisible to request metrics, exactly the "stalled serves are invisible" pathology this PR's own
  description complains about, now one layer up.
• The root Plug_shape_get span and the in-flight stream_chunk span are never ended; their
  entries orphan in the OTel span ETS table until the SDK's default sweeper drops them (strategy
  drop, TTL 30 min, sweep every 10 min)

The watchdog's Logger.warning is therefore the sole explicit signal a reap ever happened.

Scope caveat worth documenting

These kill semantics — and the PR's fd/memory-release guarantees — hold for Bandit HTTP/1 only, where the plug runs in the socket-owning process. Under Bandit h2c the kill hits a stream process, freeing its memory but not the fd; the socket-writing connection process is unguarded. In embedded mode (api_plug_opts → Phoenix.Sync) under Cowboy, chunk/2 is fire-and-forget messaging to the connection process, so writes "complete" instantly and the watchdog is silently inert — and when it does kill, it's killing a host-application process, skipping the host's own after/instrumentation, not just Electric's.

[alco]

I wonder how this watchdog would help in a situation where a proxy is buffering for a vanished client.

[robacourt]

Good challenge — the compressed claim was misleading and is reworded in b20b420. Precisely: while a proxy's buffer absorbs writes, the serve completes and releases everything (handler, pinned data, fd) — nothing is held server-side, so there is nothing to reap. The watchdog matters at the point the proxy stops accepting (buffer cap / backpressure for the vanished client): writes stop completing and the deadline runs. So it helps exactly when server-side resources are actually being held, and is correctly inert before that.

[alco]

How is this materially different from "The TCP send timeout only catcing a fully blocked write"?

[robacourt]

Materially different in two ways, one semantic and one empirical (reworded in b20b420). Semantic: the send timeout detects a driver-level send with zero kernel acceptance for its entire window; the watchdog times full completion of a bounded application-level write — a minimum-throughput floor rather than a not-perfectly-frozen check. A connection that dribbles a few bytes resets the former forever but cannot indefinitely satisfy the latter. Empirical: in the incident this addresses, hundreds of stalled serves produced exactly two send-timeout errors in an hour; and in a local probe a send blocked against a full kernel buffer survived ~29s of continuous client draining at 20KB/s without the timeout firing.

[alco]

Claude loves to add cleanup code to tests but it's usually not needed. The test process will terminate after the end of the test and with it all of the resources directly attributable to it.

[robacourt]

Removed in 859b16b — you're right, the sockets are owned by the test process and close with it.

[alco]

Since we explicitly use :kill as the exit reason, matching here on :killed would be a nice sanity check of the mechanism working as intended.

[robacourt]

Done in 859b16b — the DOWN assertions now match :killed specifically.

[alco]

This comment can be removed without any loss of information.

[robacourt]

Gone in 7a54d42 (the surrounding block was rewritten as part of the module extraction).

[alco]

The Watchdog should live in a standalone module. It's way more complicated for a bunch of private functions.

[robacourt]

Extracted in 7a54d42 as Electric.Shapes.Api.ServeWatchdog — combined with the GenServer suggestion below, so the module is the server rather than a bag of private functions.

[alco]

Starting a new unlinked task for every write is expensive. it basically doubles the number of processes responsible for request handling.

Wouldn't a single persistent gen server be a good fit for the job?

[robacourt]

One correction and then agreement. Correction: it was one task per chunked response (spawned once in send_stream), not per write — but that still doubles the processes involved in every in-flight serve, so the substance stands. Implemented in 7a54d42 as a per-stack supervised GenServer: handlers cast a registration around each bounded write, and a 1s sweep kills entries past deadline and prunes handlers that died on their own. Casts mean no synchronous coupling on the serve path (volume is bounded by socket throughput / 256 KiB — a node pushing 1 GB/s generates ~4k casts/s), casts to a stack without a running instance drop silently so serving fails open, and a restart self-heals since in-flight serves re-register on their next write — supervision semantics the per-serve process could not offer. Reap precision becomes deadline + sweep interval, noise against the 60s default.

[robacourt]

@alco ready for re-review. All seven comments addressed, one commit per point (each thread has a reply with its SHA):

• GenServer: watchdog is now a standalone, per-stack, supervised Electric.Shapes.Api.ServeWatchdog (7a54d42) — cast-based registration around each bounded write, 1s sweep, fail-open when absent, self-healing on restart. This replaced the per-response task and subsumed the module-extraction and comment-removal points. One correction noted in-thread: the old design was one task per chunked response, not per write — your conclusion stood regardless.
• Tests: DOWN assertions now match :killed specifically; redundant socket cleanup removed (859b16b). Added serve_watchdog_test.exs (d618f73) — six unit tests covering the kill/no-kill/deregister/re-register/prune/fail-open contract in under a second, so the behavior you were reviewing from prose is now executable.
• Moduledoc (b20b420): reworded to answer your two questions directly — the send-timeout distinction (throughput floor on completion of a bounded application write, vs zero-kernel-acceptance-for-a-whole-window at driver level) and the buffering-proxy case (while a proxy absorbs, the serve completes and holds nothing; the deadline runs once it stops accepting). Fuller detail with the measured numbers is in the thread replies.

CI: full router suite including :slow, api, storage, and all integration tests green locally; the branch run is on this push.

[alco]

@robacourt Thanks for addressing my comments. You have resolved all the issues pointed out in-line but my top-level review comment remains unanswered. Below, the last point in favour of using the timer approach shows a mechanism that can be used to keep track of which resources to release when a request handler needs to be killed.

---

The move to a single supervised process is directionally right, but the cast transport gives it the wrong load profile: two casts per bounded write means the watchdog's message rate is proportional to throughput, with a single consumer. The "~4k casts/s at 1 GB/s" estimate assumes every write is at the 256 KiB cap. That holds for snapshot serving, but live-mode deliveries are small: a fanout wakeup of ~20k long-pollers at 2-3 body elements each is a burst of ~100k casts to one process, per transaction.

And queue lag here isn't just a latency problem, it's a correctness one. FIFO protects an entry only up to the enqueue time of the sweep message: a sweep enqueued at T that processes at T+L kills any handler whose write_finished cast was enqueued inside (T, T+L), i.e. handlers whose writes did complete. Under Bandit HTTP/1 keepalive that pid may by then be serving a different request on the same connection. The more backed up the watchdog gets, the more likely it is to kill a healthy handler.

Proposal: use the timer wheel instead

For "enforce a deadline on a blocked operation" the BEAM already has a purpose-built primitive. Arm a BIF timer around each bounded write and cancel it on completion:

defp guarded_chunk(conn, data, {killer, shape_handle, timeout}) do
  ref = :erlang.start_timer(timeout, killer, {self(), shape_handle})
  Process.put(:watchdog_write_ref, ref)
  result = Plug.Conn.chunk(conn, data)
  :erlang.cancel_timer(ref, async: true, info: false)
  result
end

The killer process then hears about a write only when it overruns its deadline:

def handle_info({:timeout, ref, {pid, shape_handle}}, state) do
  with {:dictionary, dict} <- Process.info(pid, :dictionary),
       ^ref <- dict[:watchdog_write_ref] do
    Logger.warning("Terminating stalled shape response serve", shape_handle: shape_handle)
    Process.exit(pid, :kill)
  end

  {:noreply, state}
end

Why this shape:

• Message rate ∝ stalls, not writes. A cancelled timer never sends anything, so in steady state the killer's inbox is empty. Stalls are the rare event this PR exists for (~400/hour in the incident); a single GenServer is exactly the right consumer when its load scales with failures rather than throughput. No sweep loop either: timers fire at the actual deadline instead of with up-to-1s slack.
• start_timer/cancel_timer are per-scheduler O(1) BIFs (since the OTP 18 timer redesign), running on the same scheduler that's doing the socket write, with no shared lock and no serialization point. This churn pattern (arm, block briefly, cancel) is what every receive ... after and every gen_server:call timeout already does at request rate in any BEAM system.
• The misdirected-kill race closes. The pd-stamped ref means the killer only kills a handler still inside that write: a write_finished-raced kill (or one landing on the next keepalive request) becomes a no-op instead of a wrong kill. Process.info/2 on a dead pid returns nil, so handler-died-on-its-own needs no pruning at all.
• Fail-open is preserved. A timer whose destination is dead or unregistered drops its message silently, so serving degrades to unguarded exactly like the casts do. (start_timer takes a pid or plain atom, not a via-tuple, so the handler resolves the killer's pid via ProcessRegistry, per serve or per write if you want self-healing after a killer restart; either way it's one ETS read.)
• The kill-side cleanup from my review has a natural home. The timer payload can carry whatever the reap accounting needs (admission permit kind, span context), so the killer can release the permit and emit the reap telemetry after observing the :killed DOWN. The items from the review body (permit leak, invisible reaps, adapter scoping) still need addressing regardless of transport.

Points of comparison

I also considered an ETS table (write_concurrency, insert/delete around each write, 1s sweeper). It's viable, but it pays a per-write cost for information that's only interesting in the failure case, needs Process.alive? pruning and table-lifecycle care, and, measured below, it doesn't actually scale across writers: all handlers contend on the same table. The casts do worst of all, and the benchmark reproduces the backlog in miniature: the producers finish sending and the GenServer then spends another ~2s draining the queue they left behind.

Each "write" below is one arm/disarm pair, on 16 schedulers, OTP 28:

Design	1 proc	16 procs, aggregate
start_timer + cancel_timer	578 ns/write	71 ns/write (14.2M writes/s)
ETS insert+delete (write_concurrency)	382 ns/write	358 ns/write (2.8M writes/s)
2 casts to one GenServer (this PR)	n/a	2,926 ns/write (0.34M writes/s, incl. 2s queue drain)

Timers scale near-linearly because each scheduler owns its own timer service; ETS flatlines under contention; the cast design is bounded by the single consumer. For headroom context: the worst realistic burst (~100k writes in one fanout wakeup) costs ~200k timer BIF calls ≈ 7 ms of aggregate scheduler time, riding alongside socket writes that each cost one to two orders of magnitude more. Usual microbenchmark caveats apply (tight loop, cache-hot), but the ordering and scaling shapes are the point.

Benchmark script

defmodule WD do
  use GenServer
  def init(s), do: {:ok, s}
  def handle_cast({:start, pid, dl}, s), do: {:noreply, Map.put(s, pid, dl)}
  def handle_cast({:finish, pid}, s), do: {:noreply, Map.delete(s, pid)}
  def handle_call(:sync, _f, s), do: {:reply, :ok, s}
end

defmodule Bench do
  @timeout_ms 60_000

  def timer_loop(0), do: :ok

  def timer_loop(n) do
    ref = :erlang.start_timer(@timeout_ms, self(), :w)
    :erlang.cancel_timer(ref, async: true, info: false)
    timer_loop(n - 1)
  end

  def ets_loop(0, _tab), do: :ok

  def ets_loop(n, tab) do
    :ets.insert(tab, {self(), 123_456_789, :handle})
    :ets.delete(tab, self())
    ets_loop(n - 1, tab)
  end

  def cast_loop(0, _gs), do: :ok

  def cast_loop(n, gs) do
    GenServer.cast(gs, {:start, self(), 123_456_789})
    GenServer.cast(gs, {:finish, self()})
    cast_loop(n - 1, gs)
  end

  def par(procs, fun) do
    1..procs
    |> Enum.map(fn _ -> Task.async(fun) end)
    |> Enum.each(&Task.await(&1, 120_000))
  end

  def report(label, us, writes) do
    ns_per = Float.round(us * 1000 / writes, 0)
    mps = Float.round(writes / us, 2)
    IO.puts("#{String.pad_trailing(label, 42)} #{ns_per} ns/write  (#{mps}M writes/s)")
  end
end

n = 500_000
schedulers = System.schedulers_online()
IO.puts("schedulers_online: #{schedulers}, #{n} writes/proc\n")

{t, _} = :timer.tc(fn -> Bench.timer_loop(n) end)
Bench.report("timer arm+cancel, 1 proc:", t, n)

{t, _} = :timer.tc(fn -> Bench.par(schedulers, fn -> Bench.timer_loop(n) end) end)
Bench.report("timer arm+cancel, #{schedulers} procs:", t, n * schedulers)

tab = :ets.new(:wd, [:public, :set, write_concurrency: true])
{t, _} = :timer.tc(fn -> Bench.ets_loop(n, tab) end)
Bench.report("ets insert+delete, 1 proc:", t, n)

{t, _} = :timer.tc(fn -> Bench.par(schedulers, fn -> Bench.ets_loop(n, tab) end) end)
Bench.report("ets insert+delete, #{schedulers} procs:", t, n * schedulers)

cast_n = 100_000
{:ok, gs} = GenServer.start(WD, %{})

{t_send, _} = :timer.tc(fn -> Bench.par(schedulers, fn -> Bench.cast_loop(cast_n, gs) end) end)
{t_drain, _} = :timer.tc(fn -> GenServer.call(gs, :sync, :infinity) end)

total = cast_n * schedulers
Bench.report("2 casts sent, #{schedulers} procs:", t_send, total)
IO.puts("  ...but single GenServer needed another #{div(t_drain, 1000)} ms to drain its queue")
Bench.report("cast design true throughput:", t_send + t_drain, total)

schedulers_online: 16, 500000 writes/proc

timer arm+cancel, 1 proc:                  578.0 ns/write  (1.73M writes/s)
timer arm+cancel, 16 procs:                71.0 ns/write  (14.15M writes/s)
ets insert+delete, 1 proc:                 382.0 ns/write  (2.62M writes/s)
ets insert+delete, 16 procs:               358.0 ns/write  (2.79M writes/s)
2 casts sent, 16 procs:                    1666.0 ns/write  (0.6M writes/s)
  ...but single GenServer needed another 2015 ms to drain its queue
cast design true throughput:               2926.0 ns/write  (0.34M writes/s)