Fix the beat rate collapsing with the size of the place

alloverse/allonet2#46 · six commits · reconstructed from the pull request and its commits
a2d7fa8f...7697b777 · 7 files, +333 −21

A place with about a hundred entities was sending one changeset every 200 ms, where the coalesce delay allows one every 20 ms. The cause was a lookup that decoded the whole place to answer a question about one entity, inside a loop over every entity, on every commit. This change makes those lookups decode one component, shares one state diff between clients that are in step, and closes a crash that the new lookup exposed.

Prologue

A beat that slows down as the place grows

The place server batches changes into a beat: it waits for the 20 ms coalesce delay, commits what arrived, and sends every client a changeset. The beat self-clocks. Its period is the coalesce delay plus however long the commit and broadcast take, so the server never queues beats behind each other.

That design has a sharp edge. If something inside the beat costs more as the place grows, nothing looks slow: there is no slow request and no timeout. The beat simply comes around less often, and every client sees the place update at a lower rate.

That is what a knackeriet-scale place showed: 107 entities and about five clients, minting one changeset per ~200 ms instead of per 20 ms.1 The previous week it had been fine.

  1. Observed on alloplace v3.5.1. The orphan sweep (Chapter 2) landed in v3.4.8 on 2026-08-24, three days before the report; v3.4.7 has no sweep. ↩︎
Chapter 1

Measure before blaming

The obvious suspect was the per-client diff: every beat, the server diffs the whole place against each client's acknowledged revision, and that is the one cost that visibly grows with both the place and the client count.

It was not the culprit. Timing each stage of the beat at 5 clients and 110 entities put the diff at 0.9 ms of a 10.7 ms beat. Almost all of the rest was sweepOrphans, the pass that removes entities whose parent is gone: 8.6 ms of a 9.0 ms beat.2

The reason is in the component store. ComponentLists has a typed list subscript, components[T.self], that hands back every component of one type, decoded. It decodes on every access and caches nothing 1. Reaching through it for one entity, as in components[Relationships.self][eid], decodes the whole place's worth of that type to answer one question. The change documents that cost on the subscript itself, and in the architecture notes where the next reader will look 2.

  1. Per-stage timings from a local harness in a release build: a place populated to scale, five clients acking at 20 ms, one of them walking. Full numbers in the appendix. ↩︎
PlaceContents.swiftSources/allonet2/PlaceContents.swift
@@ -166,10 +166,30 @@ public extension Component
⋯
169+ /// Every component of a type, decoded. Decoding is done on each access and is not cached, so
170+ /// this costs one decode per component in the place: reach for it when you want the whole
171+ /// list, and for `[componentType, of:]` when you want one entity's.
169172 public subscript<T>(componentType: T.Type) -> [EntityID: T] where T : Component
170173 {
171174 return (decodedLists[componentType.componentTypeId] ?? [:]) 1as! [EntityID: T]
172175 }
⋯
architecture.mddocs/architecture.md
@@ -26,7 +26,15 @@ links. Swift tools 6.1, language mode 5, platforms macOS 15 / iOS 18 / visionOS
2626 history (`PlaceContents+Changes.swift`); per client it diffs from that client's
2727 `ackdRevision` (`PlaceServer/PlaceServer+ECS.swift`). A client that can't apply a delta
2828 acks revision 0, which requests a full resync (`AlloClient.swift`,
29− `didReceivePlaceChangeSet`).
29+ `didReceivePlaceChangeSet`). The diff is over the whole place, so it is computed once per
30+ distinct acked revision and shared by every client sitting on it.
31+- The beat runs on the main actor and self-clocks: its period is the 20 ms coalesce *plus*
32+ however long it takes, so anything superlinear in the size of the place surfaces as a
33+ collapsed beat rate rather than as one slow function. `ComponentLists`' typed list subscript
34+ (`components[T.self]`) decodes every component of that type on every access and caches
35+ nothing — reaching through it for one entity, inside a loop over entities, is what made
36+ `sweepOrphans` quadratic and cost 9 ms a beat at 110 entities. Use
37+ `components[T.self, of: eid]` for a single lookup, and 2hoist the list out of any loop.
3038- A beat commits only its net effect on committed state, so a write of the value already there
3139 — or one walked away and back by two requests inside the same coalescing window — leaves
3240 `revision` alone (`PlaceServer+ECS.swift`): a spent revision costs every client a delta and
Chapter 2

One pass over Relationships

sweepOrphans runs on every commit. It walked every entity and, for each one, asked the list subscript for that entity's Relationships 1. Each ask decoded every Relationships component in the place, so one sweep decoded the whole list N times: quadratic in the size of the place, on every beat.

The sweep now iterates the Relationships list once 2 and keeps the entities whose parent is gone. The result is the same, because an entity with no Relationships can never be an orphan. At 110 entities the sweep went from 9.7 ms to 0.15 ms a beat.

PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -356,9 +361,10 @@ extension PlaceServer
356361 private func sweepOrphans()
357362 {
358363 let present = Set(place.current.entities.keys)
359− let orphans = place.current.entities.keys.filter {
360− if let parent = place.current.components1[Relationships.self][$0]?.parent { return !present.contains(parent) }
361− return false
364+ // One pass over Relationships, not one per entity: this runs on every commit, and reaching
365+ // through the list subscript per entity made it quadratic in the size of the place.
366+ let orphans = place.current.components[Relationships.self]2.compactMap {
367+ present.contains($0.key) && !present.contains($0.value.parent) ? $0.key : nil
362368 }
363369 guard !orphans.isEmpty else { return }
364370
Chapter 3

Asking about one entity

The sweep was the worst case of a general pattern, so the fix is a general accessor. components[T.self, of: eid] looks up the one stored component 1 and decodes only that 2.

Unlike the list subscript, it does not trap when a payload fails to decode. It logs the entity and the type and reads the component as absent: one corrupt component must not take down every client in the place along with it. Chapter 6 explains why a failed decode here now means a broken invariant rather than bad input.

The tests pin the accessor to exactly what the list subscript answers 3, including for a component type that the entity doesn't have.

PlaceContents.swiftSources/allonet2/PlaceContents.swift
@@ -166,10 +166,30 @@ public extension Component
⋯
176+ /// One entity's component of a type, or nil if it hasn't got one. Decodes that one component;
177+ /// `[componentType][entityId]` decodes the whole place's worth to answer the same question.
178+ ///
179+ /// Every stored component decodes as the type it names — the place refuses one that doesn't at
180+ /// ingress (`AnyComponent.isWellFormed`) — so a decode failure here is a broken invariant, not
181+ /// absence. It is logged with the entity and the type, and read as nil: a component corrupted
182+ /// by a bug must not take down every client in the place along with it.
183+ public subscript<T>(componentType: T.Type, of entityId: EntityID) -> T? where T : Component
184+ {
185+ guard let stored = 1lists[componentType.componentTypeId]?[entityId] else { return nil }
186+ guard let decoded = 2stored.decoded(as: T.self) else
187+ {
188+ Self.logger.error("Entity \(entityId) holds a \(T.componentTypeId) that does not decode as one — reading it as absent. A malformed component reached storage, which ingress should have refused.")
189+ return nil
190+ }
191+ return decoded
192+ }
173193 public subscript(componentTypeID: ComponentTypeID) -> [EntityID: AnyComponent]?
174194 {
175195 return lists[componentTypeID]
PlaceContents.swiftSources/allonet2/PlaceContents.swift
@@ -180,6 +200,8 @@ public struct ComponentLists
180200 {
181201 lists = [:]
182202 }
203+
204+ private static let logger = Logger(labelSuffix: "place.contents")
183205
184206 internal let lists: Dictionary<ComponentTypeID, [EntityID: AnyComponent]>
185207 internal var decodedLists : LazyMap<ComponentTypeID, [EntityID: AnyComponent], [EntityID: any Component]>
ComponentAccessTests.swiftTests/allonet2Tests/ComponentAccessTests.swiftadded
@@ -0,0 +1,79 @@
1+import Testing
2+import Foundation
3+@testable import allonet2
4+
5+/// `components[T.self, of: eid]` answers exactly what `components[T.self][eid]` does, without
6+/// decoding every component of the type to get there.
7+@MainActor
8+@Suite struct ComponentAccessTests
9+{
10+ private func contents(_ changes: [PlaceChange]) throws -> PlaceContents
11+ {
12+ Transform.register()
13+ Relationships.register()
14+ let empty = PlaceContents(logger: testLogger)
15+ return try #require(empty.applyChangeSet(PlaceChangeSet(changes: changes, fromRevision: 0, toRevision: 1)))
16+ }
17+
18+ @Test func 3matchesTheListSubscript() throws
19+ {
20+ let owner = UUID()
21+ let a = EntityData(id: "a", ownerClientId: owner)
22+ let b = EntityData(id: "b", ownerClientId: owner)
23+ let world = try contents([
24+ .entityAdded(a), .entityAdded(b),
25+ .componentAdded(a.id, AnyComponent(Transform(translation: [1, 2, 3]))),
26+ .componentAdded(b.id, AnyComponent(Relationships(parent: a.id))),
27+ ])
28+
29+ #expect(world.components[Transform.self, of: a.id] == world.components[Transform.self][a.id])
30+ #expect(world.components[Transform.self, of: a.id]?.matrix.translation == [1, 2, 3])
31+ #expect(world.components[Relationships.self, of: b.id]?.parent == a.id)
32+ #expect(world.components[Transform.self, of: b.id] == nil, "b has no Transform")
33+ #expect(world.components[Relationships.self, of: "nobody"] == nil, "and nobody has anything")
34+ }
35+
36+ /// The list subscript reads a type that isn't in the place at all as an empty list; reaching
37+ /// for one entity's must agree rather than trap.
38+ @Test func anAbsentComponentTypeIsNil() throws
39+ {
40+ let world = try contents([.entityAdded(EntityData(id: "a", ownerClientId: UUID()))])
41+ #expect(world.components[Transform.self, of: "a"] == nil)
42+ #expect(world.components[Transform.self].isEmpty)
43+ }
44+
45+ /// Nothing in the wire format ties a component's payload to the type id it travels under, so
46+ /// the check that they agree is what lets everything downstream assume they do.
⋯
Chapter 4

The simulation asks one at a time too

The same pattern was all over the simulation. The movement tick decoded every Transform in the place to read one per client 1. Every step of every ancestor walk in the grab path, through transformToWorld, resolveActuation and isEntity, decoded a whole list to look up one parent.

Each of those reads now names the entity it wants 2. These changes are mechanical, but they matter: the movement tick went from 13 ms to under 1 ms at 110 entities, and it shares the main actor with the beat that it feeds.

PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -148,7 +153,6 @@ extension PlaceServer
148153
149154 private func stepMovement(dt: Float) async -> Bool
150155 {
151− 1let transforms = place.current.components[Transform.self]
152156 var changes: [PlaceChange] = []
153157 for client in clients.values
154158 {
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -161,7 +165,7 @@ extension PlaceServer
161165 // Integrate from our own last simulated transform: several ticks can run before the
162166 // heartbeat commits them, and re-reading committed state would make each of those
163167 // ticks start from the same base, so all but the last displacement is overwritten.
164− let transform = client.simulatedTransform ?? transforms[avatarId]
168+ let transform = client.simulatedTransform ?? place.current.2components[Transform.self, of: avatarId]
165169 else { continue }
166170
167171 guard let moved = MovementSimulation.step(transform: transform, velocity: &client.velocity, direction: direction, dt: dt)
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -199,7 +203,7 @@ extension PlaceServer
199203 return nil
200204 }
201205 let contents = place.current
202− guard let grabbable = contents.components[Grabbable.self][grab.entity] else {
206+ guard let grabbable = contents.components[Grabbable.self, of: grab.entity] else {
203207 client.logger.warning("Ignoring grab of \(grab.entity): not Grabbable")
204208 return nil
205209 }
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -224,7 +228,7 @@ extension PlaceServer
224228 // fresh intent can re-arm the grab between a queued removal and its apply — the
225229 // queue-time stopGrabbing hook alone can't prevent the poisoned update.
226230 guard !pendingRemovals.contains(actuated),
227− let actuatedTransform = contents.components[Transform.self][actuated] else {
231+ let actuatedTransform = contents.components[Transform.self, of: actuated] else {
228232 client.grabBase = nil
229233 client.grabSimulated = nil
230234 return nil
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -236,7 +240,7 @@ extension PlaceServer
236240 }
237241
238242 let parentToWorld: simd_float4x4
239− if let parent = contents.components[Relationships.self][actuated]?.parent
243+ if let parent = contents.components[Relationships.self, of: actuated]?.parent
240244 {
241245 guard let composed = contents.transformToWorld(of: parent, overrides: overrides) else { return nil }
242246 parentToWorld = composed
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -263,7 +267,7 @@ extension PlaceServer
263267 switch actuateOn
264268 {
265269 case .entity: return (eid, .identity)
266− case .parent: target = contents.components[Relationships.self][eid]?.parent
270+ case .parent: target = contents.components[Relationships.self, of: eid]?.parent
267271 case .ancestor(let ancestor): target = ancestor
268272 }
269273 guard let target else { return nil }
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -275,8 +279,8 @@ extension PlaceServer
275279 // Like transformToWorld, a missing Transform rejects the path rather than
276280 // silently posing the node at its parent's origin.
277281 guard visited.insert(current).inserted,
278− let parent = contents.components[Relationships.self][current]?.parent,
279− let local = overrides[current]?.matrix ?? contents.components[Transform.self][current]?.matrix
282+ let parent = contents.components[Relationships.self, of: current]?.parent,
283+ let local = overrides[current]?.matrix ?? contents.components[Transform.self, of: current]?.matrix
280284 else { return nil }
281285 composed = local * composed
282286 current = parent
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -291,7 +295,7 @@ extension PlaceServer
291295 while let id = current, visited.insert(id).inserted
292296 {
293297 if id == ancestorId { return true }
294− current = contents.components[Relationships.self][id]?.parent
298+ current = contents.components[Relationships.self, of: id]?.parent
295299 }
296300 return false
297301 }
GrabSimulation.swiftSources/allonet2/Simulation/GrabSimulation.swift
@@ -99,10 +99,10 @@ public extension PlaceContents
9999 while let id = current
100100 {
101101 guard visited.insert(id).inserted,
102− let local = overrides[id]?.matrix ?? components[Transform.self][id]?.matrix
102+ let local = overrides[id]?.matrix ?? components[Transform.self, of: id]?.matrix
103103 else { return nil }
104104 transform = local * transform
105− current = components[Relationships.self][id]?.parent
105+ current = components[Relationships.self, of: id]?.parent
106106 }
107107 return transform
108108 }
Chapter 5

One diff per acknowledged revision

With the sweep fixed, the per-client diff became the largest remaining cost: 1.5 ms of a 2.0 ms beat at 5 clients and 110 entities.

The diff is over the whole place, against the revision a client has acknowledged. Every client that is keeping up has acknowledged the same revision, so each of them was paying for the same computation. The beat now keeps one diff per distinct acknowledged revision 1 and computes it only when no client before has asked for that revision 2.

The wire format is unchanged: each client still receives a changeset stamped from the revision it acknowledged. A client that has acknowledged nothing, or has fallen out of history, is caught up from revision 0, the empty place. The tests check both sides of that: clients in step get the same changeset, and a laggard is still caught up from its own revision.

PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -71,10 +71,15 @@ extension PlaceServer
7171 assert(success) // bug if this doesn't succeed
7272 sweepOrphans()
7373 }
74+ // One diff per distinct acked revision, not per client: the diff is over the whole place,
75+ // and clients in step ask the same question. Revision 0 is the empty place, which is also
76+ // what a client that has acked nothing, or has fallen out of history, is caught up from.
77+ 1var deltas: [StateRevision: PlaceChangeSet] = [:]
7478 for client in clients.values {
75− let lastContents = client.ackdRevision.flatMap { place.getHistory(at: $0) } ?? PlaceContents(logger: logger)
76− let changeSet = place.current.changeSet(from: lastContents)
77−
79+ let acked = client.ackdRevision ?? 0
80+ let changeSet = deltas[acked]
81+ 2?? place.current.changeSet(from: place.getHistory(at: acked) ?? PlaceContents(logger: logger))
82+ deltas[acked] = changeSet
7883 client.session.send(placeChangeSet: changeSet)
7984 }
8085 }
StateBroadcastTests.swiftTests/allonet2Tests/StateBroadcastTests.swiftadded
@@ -0,0 +1,173 @@
1+import Testing
2+import Foundation
3+import PotentCBOR
4+@testable import allonet2
5+
6+/// What each client is told every beat: one diff per distinct acked revision, and a client that
7+/// has fallen behind still gets a changeset that catches it all the way up.
8+@MainActor
9+@Suite struct StateBroadcastTests
10+{
11+ private func makeServer() -> PlaceServer
12+ {
13+ PlaceServer(name: "Test Place",
14+ options: TransportConnectionOptions(routing: .direct),
15+ alloAppAuthToken: "apptoken", requiresAuthentication: false)
16+ }
17+
18+ private func makeClient(on server: PlaceServer, app: Bool = false) async throws -> ConnectedClient
19+ {
20+ let status = ConnectionStatus()
21+ let transport = MockTransport(with: TransportConnectionOptions(routing: .direct), status: status)
22+ let client = ConnectedClient(session: AlloSession(side: .server, transport: transport), status: status)
23+ client.identity = Identity(expectation: app ? .app : .existingUser, displayName: "C",
24+ emailAddress: "", authenticationToken: "apptoken")
25+ transport.clientId = client.cid
26+ if app { try await server.authenticate(identity: client.identity!, from: client, in: client.logger) }
27+ client.announced = true
28+ server.clients[client.cid] = client
29+ return client
30+ }
31+
32+ /// The changesets this client was actually sent, decoded off the wire.
33+ private func received(by client: ConnectedClient) throws -> [PlaceChangeSet]
34+ {
35+ let transport = client.session.transport as! MockTransport
36+ return try transport.sentMessages
37+ .filter { $0.channel == .intentWorldState }
38+ .map { try CBORDecoder().decode(PlaceChangeSet.self, from: $0.data) }
39+ }
40+
41+ @Test func clientsAtTheSameRevisionGetTheSameChangeset() async throws
42+ {
43+ let server = makeServer()
44+ let app = try await makeClient(on: server, app: true)
45+ let a = try await makeClient(on: server)
46+ let b = try await makeClient(on: server)
47+ let ent = try await server.createEntity(from: EntityDescription(components: [Opacity(opacity: 0.5)]), for: app)
48+ await server.heartbeat.awaitNextSync()
49+
50+ for client in [app, a, b] { client.ackdRevision = server.place.current.revision }
51+ try await server.changeEntity(eid: ent.id, addOrChange: [AnyComponent(Opacity(opacity: 0.75))], remove: [], for: app)
52+ await server.heartbeat.awaitNextSync()
53+
54+ let last = try [a, b].map { try #require(received(by: $0).last) }
55+ #expect(last[0] == last[1], "one diff, shared by every client that acked the same revision")
56+ #expect(last[0].changes == [.componentUpdated(ent.id, AnyComponent(Opacity(opacity: 0.75)))])
57+ }
58+
59+ /// The memo is keyed on the acked revision, so a laggard must not be handed the changeset
60+ /// computed for the clients that are in step.
61+ @Test func aLaggardIsCaughtUpFromItsOwnRevision() async throws
62+ {
63+ let server = makeServer()
64+ let app = try await makeClient(on: server, app: true)
65+ let uptodate = try await makeClient(on: server)
66+ let laggard = try await makeClient(on: server)
67+ let ent = try await server.createEntity(from: EntityDescription(components: [Opacity(opacity: 0.5)]), for: app)
68+ await server.heartbeat.awaitNextSync()
69+
70+ let behind = server.place.current.revision
71+ for client in [app, uptodate, laggard] { client.ackdRevision = behind }
72+
73+ try await server.changeEntity(eid: ent.id, addOrChange: [AnyComponent(Billboard(blendFactor: 0.25))], remove: [], for: app)
74+ await server.heartbeat.awaitNextSync()
75+ // Only one of them acks the beat it just got; the other stays where it was.
76+ uptodate.ackdRevision = server.place.current.revision
77+
78+ try await server.changeEntity(eid: ent.id, addOrChange: [AnyComponent(Opacity(opacity: 0.75))], remove: [], for: app)
79+ await server.heartbeat.awaitNextSync()
80+
81+ let fresh = try #require(received(by: uptodate).last)
82+ let stale = try #require(received(by: laggard).last)
83+ #expect(fresh.fromRevision == server.place.current.revision - 1)
84+ #expect(fresh.changes == [.componentUpdated(ent.id, AnyComponent(Opacity(opacity: 0.75)))])
85+ #expect(stale.fromRevision == behind, "the laggard is caught up from where it actually is")
86+ #expect(stale.changes.count == 2, "both revisions it missed, in one changeset")
87+ }
88+
89+ /// `AnyComponent` decodes its type id and its payload independently, so a peer can send a
90+ /// payload that isn't the type it claims. Stored, every simulation that reads it back can only
91+ /// treat it as absent — the avatar silently stops moving — so it is refused where it arrives.
⋯
StateBroadcastTests.swiftTests/allonet2Tests/StateBroadcastTests.swiftadded
@@ -0,0 +1,173 @@
⋯
123+ @Test func everyClientsChangesetAppliesOntoWhatItAcked() async throws
124+ {
125+ let server = makeServer()
126+ let app = try await makeClient(on: server, app: true)
127+ let a = try await makeClient(on: server)
128+ let b = try await makeClient(on: server)
129+ let ent = try await server.createEntity(from: EntityDescription(components: [Opacity(opacity: 0.5)]), for: app)
130+ await server.heartbeat.awaitNextSync()
131+ for client in [app, a, b] { client.ackdRevision = server.place.current.revision }
132+
133+ // b keeps acking; a is left one beat behind throughout.
134+ for step in 0..<4
135+ {
136+ switch step
137+ {
138+ case 0: try await server.changeEntity(eid: ent.id, addOrChange: [], remove: [Opacity.componentTypeId], for: app)
139+ case 1: try await server.changeEntity(eid: ent.id, addOrChange: [AnyComponent(Opacity(opacity: 0.25))], remove: [], for: app)
140+ case 2: _ = try await server.createEntity(from: EntityDescription(components: [Billboard(blendFactor: 1)]), for: app)
141+ default: try await server.changeEntity(eid: ent.id, addOrChange: [AnyComponent(Opacity(opacity: 0.9))], remove: [], for: app)
142+ }
143+ await server.heartbeat.awaitNextSync()
144+ b.ackdRevision = server.place.current.revision
145+ app.ackdRevision = server.place.current.revision
146+ }
147+
148+ for client in [a, b]
149+ {
150+ // Through a real PlaceState, which keeps the history a changeset from an older
151+ // revision is applied onto - that is the whole repair path for a client that lags.
152+ let mirror = PlaceState(logger: testLogger)
153+ for changeSet in try received(by: client)
154+ {
155+ #expect(mirror.applyChangeSet(changeSet),
156+ "changeset \(changeSet.fromRevision)->\(changeSet.toRevision) didn't apply")
157+ }
158+ #expect(mirror.current.changeSet(from: server.place.current).changes.isEmpty,
159+ "every client converges on the place as it is")
160+ }
161+ }
162+
163+ /// An EntityDescription as it arrives off the wire, where components are type-erased and
164+ /// nothing has checked them. The public initialiser takes concrete Components, which a peer
165+ /// is under no obligation to have sent.
166+ static func wireDescription(components: [AnyComponent] = [], children: [EntityDescription] = []) throws -> EntityDescription
167+ {
168+ struct Wire: Encodable { let components: [AnyComponent]; let children: [EntityDescription] }
169+ return try CBORDecoder().decode(
170+ EntityDescription.self,
171+ from: try CBOREncoder().encode(Wire(components: components, children: children)))
172+ }
173+}
Chapter 6

Refusing a component that lies about its type

AnyComponent decodes its type id and its payload independently. Nothing ties the two together, so a peer can send an Opacity payload labelled Transform, and the place will store it. Reading it back through the list subscript, which force-tries the decode, traps the whole server: reproduced on main as a SIGTRAP from a single changeEntity request.

The place now checks at ingress, where the entity is still in hand to name in the error. isWellFormed decodes the payload as the type it names 1. Both the create path 2 and the change path refuse a component that fails, with an invalidRequest error. Components of types this binary doesn't know stay opaque and pass, because the place forwards them without ever decoding them.

That is what turns a decode failure in the new accessor into a broken invariant rather than untrusted input, and it is why the accessor logs one instead of staying silent.

PlaceContents+Codable.swiftSources/allonet2/PlaceContents+Codable.swift
@@ -93,7 +93,19 @@ public struct AnyComponent: Codable, Equatable
9393 {
9494 return CustomComponent(typeId: componentTypeId, fields: anyValue)
9595 }
96−
96+ /// Whether the payload decodes as the type it names. What the place checks before storing a
97+ /// component off the wire: nothing in `AnyComponent`'s own decoding compares the two, and
98+ /// `decoded()` force-tries, so a mismatch traps whoever reads it back instead of failing where
99+ /// it arrived.
100+ ///
101+ /// A type this binary doesn't have is well-formed by definition: the place stores and forwards
102+ /// components it cannot decode, and never looks inside them.
103+ public var isWellFormed: Bool
104+ {
105+ 1guard let type = ComponentRegistry.shared.component(for: componentTypeId) else { return true }
106+ return (try? AnyValueDecoder.default.decode(type, from: anyValue)) != nil
107+ }
108+
97109 // The type-erased content, available whether the concrete type is available or not
98110 public var anyValue: AnyValue
99111 public var componentTypeId: String
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -303,9 +307,10 @@ extension PlaceServer
303307 // commits or one created here; otherwise the place would hold a child nothing can render.
304308 var willExist = projectedEntities
305309 for case .entityAdded(let e) in changes { willExist.insert(e.id) }
306− for change in changes
310+ 2for case .componentAdded(let eid, let comp) in changes
307311 {
308− guard case .componentAdded(let eid, let comp) = change, let rel = try relationship(in: comp) else { continue }
312+ try reject(malformed: comp, on: eid)
313+ guard let rel = try relationship(in: comp) else { continue }
309314 guard willExist.contains(rel.parent) else {
310315 throw AlloverseError(code: PlaceErrorCode.notFound, description: "Can't parent entity \(eid) to \(rel.parent): no such entity")
311316 }
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -427,6 +433,17 @@ extension PlaceServer
427433 return index
428434 }
429435
436+ /// Refuse a component whose payload doesn't decode as the type it names. Nothing in the wire
437+ /// format ties the two together, and stored, such a component is indistinguishable from an
438+ /// absent one to every simulation that reads it back — so it is turned away here, where the
439+ /// entity it was meant for is still in hand to name.
440+ private func reject(malformed comp: AnyComponent, on eid: EntityID) throws(AlloverseError)
441+ {
442+ guard !comp.isWellFormed else { return }
443+ throw AlloverseError(code: PlaceErrorCode.invalidRequest,
444+ description: "Malformed \(comp.componentTypeId) payload for entity \(eid)")
445+ }
446+
430447 /// The `Relationships` in `comp`, or nil when it is another component. Throws rather than traps
431448 /// on a malformed payload: this runs on untrusted create/change input, and `decodedIfAvailable`
432449 /// force-tries the decode.
PlaceServer+ECS.swiftSources/allonet2/PlaceServer/PlaceServer+ECS.swift
@@ -503,6 +520,7 @@ extension PlaceServer
503520 // cycle the tree - and a cycle corrupts every transformToWorld through it.
504521 for comp in addOrChange
505522 {
523+ try reject(malformed: comp, on: eid)
506524 guard let rel = try relationship(in: comp) else { continue }
507525 guard rel.parent != eid, !descendants(of: eid, using: projectedChildIndex()).contains(rel.parent) else {
508526 throw AlloverseError(code: PlaceErrorCode.invalidRequest, description: "Parenting \(eid) to \(rel.parent) would make a cycle")
ComponentAccessTests.swiftTests/allonet2Tests/ComponentAccessTests.swiftadded
@@ -0,0 +1,79 @@
⋯
47+ @Test func aPayloadThatDoesntMatchItsTypeIdIsNotWellFormed() throws
48+ {
49+ Opacity.register()
50+ Transform.register()
51+ #expect(AnyComponent(Transform()).isWellFormed)
52+ #expect(Self.mislabelled(Opacity(opacity: 0.5), as: Transform.self).isWellFormed == false)
53+
54+ var unknown = AnyComponent(Opacity(opacity: 0.5))
55+ unknown.componentTypeId = "NotCompiledIntoThisBinary"
56+ #expect(unknown.isWellFormed, "the place forwards components it can't decode without looking inside them")
57+ }
58+
59+ /// Reachable only through a bug, since ingress refuses it — but if one is ever stored, reading
60+ /// it must not trap, and must not pass for absence in the log either.
61+ @Test func aMalformedStoredComponentReadsAsAbsent() throws
62+ {
63+ let ent = EntityData(id: "a", ownerClientId: UUID())
64+ let world = try contents([
65+ .entityAdded(ent),
66+ .componentAdded(ent.id, Self.mislabelled(Opacity(opacity: 0.5), as: Transform.self)),
67+ ])
68+ #expect(world.components[Transform.self, of: ent.id] == nil)
69+ }
70+
71+ /// A component carrying one type's payload under another's type id: what a peer can put on the
72+ /// wire, since `AnyComponent` decodes the id and the payload independently.
73+ static func mislabelled(_ component: some Component, as type: (some Component).Type) -> AnyComponent
74+ {
75+ var comp = AnyComponent(component)
76+ comp.componentTypeId = type.componentTypeId
77+ return comp
78+ }
79+}
StateBroadcastTests.swiftTests/allonet2Tests/StateBroadcastTests.swiftadded
@@ -0,0 +1,173 @@
⋯
92+ @Test func aComponentWhosePayloadIsNotItsTypeIsRefused() async throws
93+ {
94+ let server = makeServer()
95+ let app = try await makeClient(on: server, app: true)
96+ let ent = try await server.createEntity(from: EntityDescription(components: [Opacity(opacity: 0.5)]), for: app)
97+ await server.heartbeat.awaitNextSync()
98+ let revision = server.place.current.revision
99+ let bad = ComponentAccessTests.mislabelled(Opacity(opacity: 0.5), as: Transform.self)
100+
101+ let changeError = await #expect(throws: AlloverseError.self) {
102+ try await server.changeEntity(eid: ent.id, addOrChange: [bad], remove: [], for: app)
103+ }
104+ #expect(changeError?.description.contains(Transform.componentTypeId) == true, "names the type")
105+ #expect(changeError?.description.contains(ent.id) == true, "and the entity it was meant for")
106+
107+ await #expect(throws: AlloverseError.self, "and on the way in as a new entity too") {
108+ _ = try await server.createEntity(from: try Self.wireDescription(components: [bad]), for: app)
109+ }
110+ // A child's components are as untrusted as its parent's.
111+ await #expect(throws: AlloverseError.self) {
112+ _ = try await server.createEntity(
113+ from: try Self.wireDescription(children: [Self.wireDescription(components: [bad])]), for: app)
114+ }
115+
116+ await server.heartbeat.awaitNextSync()
117+ #expect(server.place.current.revision == revision, "nothing malformed was committed")
118+ #expect(server.place.current.components[Transform.componentTypeId]?[ent.id]?.isWellFormed != false)
119+ }
120+
121+ /// A removal and a re-add are the cases the shared diff could get wrong for one client and
122+ /// not another, so pin that every client's changeset applies cleanly onto what it acked.
⋯
Epilogue

What this leaves alone

  • Not reproduced over a real transport. The harness drives PlaceServer in process over a mock transport. That is where the cost lives, since all of it is main-actor work, but it does not exercise libdatachannel.
  • Voice was not measured. Forwarded voice frames never touch the main actor, so they cannot starve the beat. That conclusion comes from reading the receive path, not from a measurement.
  • The bigger fast path is not built. A client that has just acknowledged could be sent the beat's own committed changeset, restamped, instead of a fresh diff. At 0.28 ms a beat, the diff isn't worth that correctness risk yet.
  • No CI guard on the complexity. A timing assertion would be flaky. The tests pin behaviour; the performance claim rests on the harness numbers in the appendix.

Open question

The diff becomes the dominant cost again past roughly 300 entities or 20 clients. Should the restamped-changeset fast path be built before places reach that size, or only when one does?

Appendix

The numbers

Beat rate counted off the wire in the local harness, release build on an M-series Mac. Before is origin/main, after is this branch, run back to back.

World Clients Before After
106 entities, 35 relationships 1 30.0 beats/s 43.3 beats/s
110 entities, 35 relationships 5 28.9 42.2
125 entities, 35 relationships 20 27.3 42.6
110 entities, 84 relationships (widget-nested) 5 22.3 43.0
305 entities, 240 relationships 5 7.5 40.6

The shape matters more than the absolute numbers. Before, the beat rate fell as the world grew. After, it is flat at about 41 to 43 beats a second across every world shape, so it is bounded by the coalesce delay and scheduling rather than by the size of the place. The reported 200 ms is what the 305-entity case looks like on a slower host.

Per-stage timings, 5 clients and 110 entities
beat            10.7 ms  ->  2.0 ms
  sweep          9.7 ms  ->  0.15 ms
  diff           0.9 ms  ->  0.28 ms
movement tick   13.1 ms  ->  < 1 ms

305 entities
beat            45.1 ms  ->  3.7 ms