The Evicted-Stream Retention

The sync/ Hub Population settled where the hubs come from — one per SynchronizationStream, two per cross-hub subscription — and left the question that decides #3432 open: are those streams garbage, and if so what holds them. This page answers it for one named mechanism, and the evidence is a controlled in-process experiment rather than a heap dump.

🚨 MEASURED AND RE-SCOPED, 2026-09-10 — read this before acting on the page below. A live census of a production replica classified all 475 parked streams: 474 are MeshNodeReference streams holding a lease of 1 (the leases != 0 branch, which is the correct one — released with their mesh-node cache entry), and exactly 1 is the unleased LayoutAreaReference shape §3 predicts is "the volume case". The parked count was also FLAT (474/475/475) while the replica's sync/ hub population grew by 120 in eight minutes, so this mechanism is real but is neither the bulk nor the growth of #3432. The growth was pinned elsewhere, by experiment — see The Read Path Minted a Hub Per Read. 🚨 And do not apply §9's remedy to ReclaimIfUnheld's predicate: the eviction parks a HEALTHY stream precisely because an undeclared reader may still be attached, so disposing on "no lease was ever declared" cuts live readers off silently.

🚨 NARROWED 2026-09-15 by #1174. The eviction below no longer runs on every change-feed event: a versioned Updated — one per write — keeps the mirror and records the version it announced (see Live Mirrors and the Change Feed). The retention described here is unchanged as a mechanism, but it is now reached only by a delete, a recreate or the version-less recycle broadcast. EvictedUnleasedStreamRetentionTest drives that last shape for exactly that reason, and its third arm pins that the per-write shape leaves one stream, not one per write.

The verdict is RETENTION, not a leak. Nothing is created and forgotten. The streams are parked deliberately, by code that says so, and then the one routine that could release them structurally declines to — for a reason that is correct in general and wrong for the majority of call sites.

1. The two lines

Workspace.EvictForPath runs on every change-feed event and parks every cached remote stream whose owner matches the changed path:

// src/MeshWeaver.Data/Workspace.cs — EvictForPath
_evictedRemoteStreams[removed.Value] = 0;
ReclaimIfUnheld(removed.Value);

and ReclaimIfUnheld opens with:

// src/MeshWeaver.Data/Workspace.cs — ReclaimIfUnheld (the claim, since #5087 — see §10)
var claim = -Interlocked.Increment(ref _reclaimClaims);   // a fresh negative token
if (!_remoteStreamLeases.TryUpdate(stream, claim, 0))
    return;

Read that condition carefully. A stream nobody ever leased has no entry in _remoteStreamLeases at all, so the compare-exchange from 0 fails and the method returns having disposed nothing. "A holder is still declared" (count above zero) and "no holder was ever declared" (no entry) take the same branch, and only the first of those is a reason to keep the stream.

The stream is now in _evictedRemoteStreams, an instance ConcurrentDictionary on the singleton workspace, holding its client sync/ hub and (through the owner's mirror) its owner-side twin. The next caller misses the cache, builds a fresh stream, and the next change event parks that one the same way.

🚨 The same file already handles this case correctly two hundred lines away. DiscardFaultedRemoteStream parks a stream, checks _remoteStreamLeases.ContainsKey explicitly, and disposes on the spot when no holder was ever declared. Two sites, one situation, opposite outcomes — the second delegates to a helper that cannot distinguish "held" from "never declared".

2. Why the parking is deliberate — and why that is not the whole story

EvictForPath carries an explicit instruction not to dispose unconditionally:

Do NOT unconditionally dispose the evicted stream — an undeclared reader (e.g. a MeshDataSource reduce callback that handed the stream on) may still be attached and needs to keep receiving updates until it drops on its own.

That is sound. An undeclared reader is invisible, so the workspace genuinely cannot know. The _remoteStreamLeases note (the #1324 fix) states the intended remedy in as many words:

A stream NOBODY leased is never in this registry and keeps the old conservative parking — undeclared holders … are unaffected. Opting a call site in is one line and is what makes its streams reclaimable.

So the design is: leasing is opt-in, and the un-opted-in remainder is knowingly retained. The finding here is that the un-opted-in remainder includes the portal's highest-volume path, and that nothing ever drains it.

3. Nothing else collects them

There is exactly one other drain, Workspace.DetachRemoteStreams, and it is reachable from exactly one call site — MeshNodeStreamHandle.DetachUpstreams() — which always passes new MeshNodeReference(). It matches parked streams on Equals(parked.Reference, reference).

For the second row the only remaining disposal is Workspace.Dispose() — i.e. process exit.

And LayoutAreaReference is the volume case. LayoutExtensions.GetControlStream — the call every rendered layout area goes through — is hub.GetWorkspace().GetRemoteStream(address, new LayoutAreaReference(area) { Id = id }): the public GetRemoteStream, which caches the stream and takes no lease. EvictForPath matches on the owner alone and ignores the reference, so a change event on that owner parks the layout area's stream too. When the Blazor circuit later ends, its subscription drops — and the stream itself is disposed by nobody.

The other unleased openers are MeshOperations (also LayoutAreaReference), and the MeshDataSource / SyncedQueryDataSource reduce callbacks (MeshNodeReference, so at least reachable by the idle release).

4. The controlled experiment

test/MeshWeaver.Data.Test/EvictedUnleasedStreamRetentionTest.cs drives the identical sequence twice — resolve a remote stream for one (owner, reference, identity) triple, fire one owner-path change-feed event, five times — and counts the live sync/ hub population the way #3432 counts it: HostedHubsCollection.Hubs filtered to Address.Type == SynchronizationAddress.AddressType.

The two arms differ in exactly one thing: whether a lease is taken.

DIAG unleased: opened=5 closed=0                   clientSyncHubs=+5
DIAG leased:   opened=5 closed=0 realLeases=5/5    clientSyncHubs=+0     (run 1)
DIAG leased:   opened=5 closed=0 realLeases=5/5    clientSyncHubs=+1     (run 2)

Five change events on ONE cache key: the unleased arm ends with five live client sync/ hubs where the cache's own documented invariant is "one stream per key, never two competing live subscriptions". The leased arm ends at 0 or 1 — the spread is whether the final stream's eviction had landed when the count was taken, and 1 is the denominator: the one current mirror for that key. The unleased arm's 5 is flat growth with the number of change events; the leased arm does not grow at all.

🚨 The leased arm is the positive control, and it earned its keep. The first version of this test measured reclamation by counting UnsubscribeRequest at the owner, and the control arm failed — closed=0 in both arms. The instrument was wrong, not the mechanism: disposal in this teardown ordering does not deliver an UnsubscribeRequest the owner's rule chain sees, so that counter cannot distinguish "not reclaimed" from "reclaimed quietly". Had the control been omitted, the unleased arm's closed=0 would have been reported as proof of a retention it does not actually measure. realLeases=5/5 is in the output for the same reason — AcquireRemoteStreamUnchecked returns Disposable.Empty when the stream it resolved is not usable, and a control arm that silently leased nothing would have looked like "leasing does not help".

5. A second, independent root on the same population

JsonSynchronizationStream registers the stream's owner-protocol subscription twice:

reduced.RegisterForDisposal(observeSubscription);
// Belt-and-suspenders: dispose the subscription when the HUB tears down too (idempotent).
hub.RegisterForDisposal(observeSubscription);

hub here is the outer, long-lived hub, not the stream's own. MessageHub.RegisterForDisposal adds to a plain Rx CompositeDisposable, and CompositeDisposable.Add never prunes — disposing a child does not remove it from the composite. So the outer hub accumulates one strong reference per remote stream it has ever opened, and the closure captures reduced, the stream. For a stream that is never disposed, that subscription is never disposed either, so the closure stays live and the stream — with its sync/ hub still attached — is rooted a second time.

Line 575 alone already ties the subscription to the stream's own lifetime. The hub. line adds no guarantee and one monotonic root.

The same monotonic-composite shape feeds DataExtensions per message (SubscribeRequest, DataChangeRequest, PatchDataRequest, UpdateUnifiedReferenceRequest), where the same file elsewhere uses the correct .TakeUntil(hub.DisposalCompleted) + self-disposing SingleAssignmentDisposable pattern.

6. What was ruled OUT

Both were plausible and both are closed, so nobody needs to re-run them:

• No static collection anywhere in src/ holds a hub or a stream. A full audit found zero static fields of type IMessageHub, MessageHub, ISynchronizationStream, IWorkspace or IMeshService. Every hub and stream registry is already an instance field on a mesh-scoped singleton. (Unrelated static state does exist and is catalogued in No Static State — none of it can reach a hub.)
• No timer roots a hub. Every periodic subscription that could — the per-hub stale-callback scanner, the sync-stream heartbeat, the MeshDataSource persistence sampler, the kernel idle disconnect, the NodeType debounce — holds a WeakReference and self-disposes on the tick after the target is collected. Each carries a comment naming the TimerQueue → PeriodicTimer → MessageHub chain it was written to break.

7. The denominator

_remoteStreamCache is keyed (Owner, Reference, Identity) and its own comment states the invariant: "one stream per key, never two competing live subscriptions." So:

correct live client streams  =  |distinct (Owner, Reference, Identity) triples in use|
correct live sync/ hubs      =  2 × that          (client + owner-side twin)
excess                       =  2 × |_evictedRemoteStreams|

Everything in _evictedRemoteStreams is above the denominator by construction — it is out of the cache, so no future caller can adopt it, and it is not the current stream for any key.

In the experiment the denominator is 1 and the measurement is 1 per change event. For the production figure the denominator is unknown, and that is the honest remaining gap: §4 establishes the mechanism and its unboundedness, not that it accounts for all 6 925.

8. 🚨 The one measurement that closes #3432 — and it is not the referrer walk

#3432's plan calls for a ClrMD referrer walk, which needs the dump that (measured, 2026-09-06) freezes the replica for 106 s and restarts it. §7 makes a far cheaper reading decisive, because the excess is a field on one object:

On the singleton Workspace, read _evictedRemoteStreams.Count and _remoteStreamCache.Count.

Two field reads on a single instance — no referrer walk, no full-heap traversal, no type histogram.

Both counts are also derivable without a dump from existing Debug logging, which already emits one line per open ("opened remote stream for as "), one per eviction, and one per reclaim ("disposed superseded remote stream … no declared holder remains"): opened − reclaimed is the retained count.

9. The fix, and why it is not in this change

The remedy is the one the source already names — opt the unleased call sites into the lease — tying it to the consumer's subscription:

// LayoutExtensions.GetControlStream, in shape
Observable.Create<object?>(observer =>
{
    var (stream, lease) = workspace.AcquireRemoteStreamUnchecked<JsonElement, LayoutAreaReference>(
        address, new LayoutAreaReference(area) { Id = id });
    return new CompositeDisposable(stream.GetControlStream(area).Subscribe(observer), lease);
});

It is safe in the way that matters: ReclaimIfUnheld disposes only a stream that is also evicted, so a stream still in the cache is never touched, and once every consumer of a given path leases, no undeclared reader remains to be surprised.

It is nevertheless a separate change, for reasons that are about blast radius rather than effort: GetControlStream is public and changes the lifetime of every layout area's stream; resolving the stream inside Observable.Create moves resolution from call time to subscribe time; the consumers most affected are in-mesh layout areas that no dotnet build and no CI job ever type-checks; and AcquireRemoteStreamUnchecked is internal to MeshWeaver.Data with no InternalsVisibleTo for MeshWeaver.Layout. Deleting the redundant hub.RegisterForDisposal of §5 is smaller but lands on the same hot path.

Neither belongs in a change whose purpose was to establish the cause.

10. A lease and a reclaim are ONE decision (#5087)

The lease registry only works if "no holder remains" cannot become false between the moment the reclaim decides it and the moment the stream is disposed. It could. ReclaimIfUnheld used to READ the count, then remove the entry and dispose:

if (!_remoteStreamLeases.TryGetValue(stream, out var leases) || leases != 0) return;
if (!_evictedRemoteStreams.TryRemove(stream, out _)) return;
_remoteStreamLeases.TryRemove(stream, out _);   // drops whatever count is there NOW
stream.Dispose();

A writer that resolved the mirror before its eviction could take its lease inside that window. Its count went 0 → 1 after the reclaim had read 0; its post-lease IsUsable re-check passed, because nothing was disposed yet; then the reclaim removed the writer's entry and disposed the mirror under it. The writer's base read completed EMPTY — RequireBaseState's "Update aborted: this hub's mirror ended without ever carrying the node's state", the terminal #5087 was filed for (LogIncidentControlPlane's Comment write on Admin/_LogIncident/…).

The fix is a claim token, not a lock. The reclaim CLAIMS the stream with a compare-exchange of its lease count from 0 to a fresh negative token (-Interlocked.Increment(ref _reclaimClaims)), and a lease is taken only by a compare-exchange from a non-negative count. Both decisions are over the SAME stream's lease state, so exactly one wins:

Why a token per claim and not a shared -1. Ownership can be handed out and back INSIDE a claim: the mesh-node cache's idle release calls DetachRemoteStreams (which takes the parking entry and cancels a negative reclaim claim, while preserving positive holder counts) and, when it loses its own zero-subscriber race, hands the stream back with ParkRemoteStreams. That re-park is a NEW ownership. With a shared sentinel the reclaim could not tell it from its own claim, would take the new parking entry and dispose a stream a consumer had just re-attached to. So after taking the parking entry the reclaim re-reads the slot: its own token means no handover happened and it disposes; anything else means its claim was superseded, and it puts the parking entry back and leaves the stream alone. After that re-read the stream is out of _evictedRemoteStreams, so no detach can reach it before the dispose.

The token is reset only AFTER Dispose, so a lease attempted later finds a dead stream, fails its re-check and resolves a fresh one. The state is keyed weakly by its stream, so it cannot retain a disposed stream after ownership leaves the workspace. A claim that loses _evictedRemoteStreams.TryRemove to another owner hands the slot back (token → 0, only if still its own).

test/MeshWeaver.Data.Test/ReclaimLeaseAtomicityTest.cs drives both windows deterministically through the ReclaimClaimed seam, which runs at the instant the claim has landed:

• the race — writer B leases there. On the pre-fix code B's lease was granted and the mirror was disposed under it (leaseBGranted=True mirrorUsable=False); now the lease is refused and a fresh acquire returns a new, live mirror;
• the handover — the idle release detaches and re-parks there. With a shared sentinel the re-parked mirror was disposed; with the token it survives;
• the control — a lease taken before the last release keeps the evicted mirror alive until that lease, too, is released, so the refusal is not overbroad.

Detach must preserve existing holders. Removing the lease entry at detach erased every outstanding lease even when the cache lost its idle-release race and re-parked the stream. A caller that had already resolved that same mirror could then take a new lease, starting the count at one. Releasing either the old or the new lease consumed that single count and disposed the mirror under the other reader, completing its observation silently. The lease state now belongs to the stream through the handover; detach cancels only a negative reclaim token. Both release orders are pinned by real-mesh tests in ReclaimLeaseAtomicityTest, including completion of the original observer only after the final holder releases. This proves the lifetime defect; it does not by itself prove the cause of a particular production heartbeat freeze.

A reclaim must also match the ownership it observed before claiming. A parking check alone is a snapshot: detach/re-park can land after that check but before the zero-count CAS. Each count is held in an immutable snapshot, and detach replaces the snapshot even when its count is already zero. The reclaim CAS matches the exact snapshot captured before reading the parking set, so a claim begun against old ownership fails after handover. The pre-claim regression first failed on the holder-preserving repair, then passed with this ownership check; it also verifies that a new holder can release the mirror under the current ownership, so the check does not disable reclamation.

What this does NOT cover: the other two producers of the same message (a ReplaySubject that never carried a value, and the write path's Where(Value is not null) filter — see Write Verdict Totality) produce byte-identical text, so a recurrence of the line alone does not say which producer fired.

Related

• The sync/ Hub Population — the 1:1 stream↔hub identity and the idle sweep that traffic keeps re-arming
• Stream Liveness and the Hub Reference — #3321 step 3, which reclaimed the Dead half
• Portal Heap Is Hubs — the five dumps and the measurement hazard
• Live Mirrors and the Change Feed — why the eviction is unconditional and must stay so
• No Static State — the rule §6 checked against