Rigorously tested for production use.
0 server-side failures and 0 N/A across the tests.
These tests were designed to probe the throughput ceiling of a single-core service under real network conditions. The actual ceiling was not reached. Both instances of physical capacity limits encountered during testing occurred at the client and network equipment level; the Igropyr server itself was never genuinely saturated.
| Driven from home fibre | At approximately 16,000 small packets per second downstream, the network line's buffer is exhausted. The resulting packet loss corrupts the load generator's accounting, invalidating measurements beyond this threshold. |
| Driven from a second cloud host | Using the same private network, the single-threaded load generator failed to saturate the target. Throughput rose with each additional client process, indicating the generator itself was the bottleneck. |
Environment: A single Igropyr process running on one core of a 2 vCPU / 2 GB Lightsail instance in Paris, driven over a WireGuard tunnel from a European home broadband client.
| Concurrency | rps | p50 | p95 | p99 | Failed |
|---|---|---|---|---|---|
| 100 | 13,812 | 7 ms | 8 ms | 12 ms | 0 |
| 250 | 13,780 | 18 ms | 20 ms | 29 ms | 0 |
| 500 | 13,847 | 35 ms | 40 ms | 54 ms | 0 |
| 1,000 | 13,652 | 71 ms | 81 ms | 112 ms | 0 |
| 2,000 | 12,940 | 72 ms | 372 ms | 1,255 ms | 48 |
| 1,000, re-run | 13,661 | 71 ms | 81 ms | 117 ms | 0 |
| The plateau itself | Throughput neither collapses nor oscillates as concurrency increases tenfold. It remains strictly between 13,6K and 13,8K RPS, with less than 1.5% variance across four rungs. Concurrency beyond c=100 translates linearly to queueing delay rather than additional throughput. |
| The re-run | The c=1,000 baseline was executed twice in the same session — once before pushing the host into the c=2,000 network failure region, and once after. The second pass yielded 13,661 RPS (p50: 71 ms, p99: 117 ms) compared to the first pass's 13,652 RPS (p50: 71 ms, p99: 112 ms). The deviation is 0.07% in throughput, with an identical median latency and zero failures in both instances. Forcing the system past its network limit resulted in zero residual state or performance degradation. |
| The 48 failures | These occurred at the client, not the server. Packet capture (PCAP) verifies the server continued responding at its baseline rate, executing 6 to 7 retransmissions for each unacknowledged reply. The failure occurred entirely in the intermediate network layer. Igropyr successfully reclaimed the stalled connections via the configured 30-second read timeout. All 48 connections stalled simultaneously as a single network drop event, rather than degrading sequentially. |
At c=2,000, the test pushes approximately 16,000 small packets per second down a domestic broadband line. This burst rate overruns the client router's downstream buffer. The network equipment preceding the server fails first; subsequent load increases would only measure the router's degradation limit.
This identical network boundary appears on a different, longer routing path. Due to the higher round-trip time (RTT), greater concurrency is required to reach the same packet rate threshold. Consequently, the buffer failure occurs at c=8,000 rather than c=2,000.
| Concurrency | rps | p50 | p99 | Failed |
|---|---|---|---|---|
| 100 | 562 | 176 ms | 180 ms | 0 |
| 500 | 2,808 | 176 ms | 183 ms | 0 |
| 1,000 | 5,625 | 175 ms | 186 ms | 0 |
| 2,000 | 11,165 | 176 ms | 196 ms | 0 |
| 4,000 | 13,210 | 244 ms | 1,462 ms | 0 |
| 8,000 | 12,601 | 245 ms | 5,088 ms | 483 |
| 4,000, re-run | 13,214 | 244 ms | 1,454 ms | 0 |
These tests were designed to answer three structural questions: whether an intercontinental mesh link remains stable over extended periods; whether the server exhibits memory leaks or performance degradation under sustained load; and whether it successfully reforms the mesh topology while under high concurrency and strict memory constraints. To test the latter, the mesh link was deliberately severed every 20 seconds throughout the entire duration of the run.
Environment: A single Igropyr process on a 2 vCPU / 1 GB Lightsail instance in Hong Kong, driven at a constant 3,000 concurrent connections, concurrently running a full application stack (five service processes, MySQL, Nginx, and Redis). This left approximately 140–150 MB of available physical memory for the Igropyr process.
The 20-hour segment tabulated below details 231 of those rounds, encompassing 91,286,309 requests with 0 failures and 0 N/A results.
| Hour | Rounds | Requests | rps | Mean p50 | Max | Failed |
|---|---|---|---|---|---|---|
| 1 | 10 | 4,083,760 | 1,360.97 | 2,206.6 ms | 8,216 ms | 0 |
| 2 | 12 | 4,991,871 | 1,386.36 | 2,169.2 ms | 7,914 ms | 0 |
| 3 | 12 | 4,967,187 | 1,379.66 | 2,170.4 ms | 7,831 ms | 0 |
| 4 | 12 | 4,810,819 | 1,335.94 | 2,247.5 ms | 8,187 ms | 0 |
| 5 | 12 | 4,457,237 | 1,237.69 | 2,466.0 ms | 11,723 ms | 0 |
| 6 | 11 | 4,184,149 | 1,267.75 | 2,367.5 ms | 11,159 ms | 0 |
| 7 | 12 | 4,745,417 | 1,317.73 | 2,251.6 ms | 8,494 ms | 0 |
| 8 | 12 | 4,787,486 | 1,329.53 | 2,240.3 ms | 8,034 ms | 0 |
| 9 | 12 | 4,767,255 | 1,323.84 | 2,243.2 ms | 8,108 ms | 0 |
| 10 | 11 | 4,358,580 | 1,320.22 | 2,248.6 ms | 8,077 ms | 0 |
| 11 | 12 | 4,771,915 | 1,325.18 | 2,243.0 ms | 8,116 ms | 0 |
| 12 | 12 | 4,729,870 | 1,313.63 | 2,265.0 ms | 8,411 ms | 0 |
| 13 | 12 | 4,711,750 | 1,308.46 | 2,279.2 ms | 8,020 ms | 0 |
| 14 | 12 | 4,680,554 | 1,299.64 | 2,299.1 ms | 8,214 ms | 0 |
| 15 | 11 | 4,282,685 | 1,297.52 | 2,306.0 ms | 8,659 ms | 0 |
| 16 | 12 | 4,670,329 | 1,297.04 | 2,281.8 ms | 8,235 ms | 0 |
| 17 | 12 | 4,736,843 | 1,315.33 | 2,258.2 ms | 8,300 ms | 0 |
| 18 | 12 | 4,713,668 | 1,309.17 | 2,257.8 ms | 8,058 ms | 0 |
| 19 | 12 | 4,705,040 | 1,306.48 | 2,258.8 ms | 9,214 ms | 0 |
| 20 | 8 | 3,129,894 | 1,303.36 | 2,265.6 ms | 8,013 ms | 0 |
| Noise | The standard deviation of the hourly throughput means is 10.2 RPS (0.78%), with a total range spanning from 1,297.04 to 1,329.53 RPS (2.5%). |
| Throughput drift | The mean throughput of the first seven hours was 1,319.80 RPS, compared to 1,304.08 RPS for the final seven hours. This represents a −1.19% deviation (approximately 1.5σ), confirming negligible performance degradation over the sustained run. |
| P50 drift | Median latency shifted from 2,252.99 ms to 2,275.33 ms (+0.99%). Per Little's Law, which mathematically ties latency to throughput at a fixed concurrency, a 1.19% throughput drop will inherently manifest as a proportional latency increase. This is an expected artifact of the throughput variance rather than an independent metric of degradation. |
| MAX latency | Maximum latency remained flat at approximately 8,200 ms across the undisturbed fourteen hours, exhibiting no upward trend. As the sole genuinely independent variable among the three, its absolute stability confirms that queue depths remained strictly bounded without progressive pileups. |
| Hours 5 and 6 (external interference) | The performance anomaly during these hours was caused by the monitoring instrument, not the server. A newly deployed metric collector exhausted the host's physical memory, pushing the operating system into swap. Physical memory swap-outs (swap_pgout) surged by 13,050 pages in Hour 5 and 5,465 pages in Hour 6, while remaining at exactly zero for all other hours. Upon the cessation of swap activity, the system's throughput recovered to its baseline entirely unaided from Hour 7 onward. |
This test evaluates the application's Resident Set Size (RSS) trajectory to mathematically rule out long-term memory leakage. Measurements were sampled at one-minute intervals across two consecutive 24-hour node lifetimes. The datasets were linearly regressed separately to isolate variables, as the two runs utilized different builds and operated at different connection counts.
| Zero-leak regression | During the latter 24-hour run (sustained at c=3,094), the linear regression of memory consumption yielded a slope of −0.021 ± 0.022 MB/h. This value is statistically indistinguishable from zero. An independent hourly-median regression corroborates this flatline, yielding an aligned slope of −0.016 ± 0.033 MB/h. |
| Measurement sensitivity and bounds | The 2σ detection floor over the 24-hour window is strictly bounded at 0.044 MB/h, equating to a maximum theoretical unmeasured drift of 1.1 MB across the entire lifetime. Normalized against the 278 million requests served within that window, this establishes a measurement sensitivity of 0.004 bytes per request. The measured slope sits strictly below this sensitivity threshold. Any theoretical memory accumulation is physically bounded to less than 4 millibytes per request. |
| State initialization vs. memory leakage | The slope trajectory across the combined lifecycles progressed in three distinct phases: +1.28 MB/h → +0.075 MB/h → −0.021 MB/h. Memory exhibited a significantly positive slope for the first 4.8 hours, flattened, and then turned slightly negative. This deceleration curve characterizes a bounded internal data structure reaching its operational capacity (filling). An actual memory leak would exhibit continuous, non-terminating positive growth. |
| Garbage collection behind the high-water marks | Hourly memory troughs exhibited no monotonic upward trend in either lifetime. Furthermore, in every hour but one per run, memory samples dipped below that specific segment's 10th percentile (p10). This confirms that the underlying garbage collection mechanism is consistently and fully reclaiming memory behind the high-water marks. |
procs − mon-chain − conns)| Zero-drift distribution | The residual mode across the entire observation window was exactly 35, accounting for 76.4% of all samples. The mode for the first third of the run was 35, and the mode for the final third was 35, confirming an absolute distribution shift of 0. |
| Clean exit state | The final sample taken immediately prior to the node's natural exit calculated as 3052 − 16 − 3001 = 35. This perfectly matches the baseline residual, confirming a mathematically clean teardown upon shutdown. |
| Conclusion | Zero unreleased actors were observed. Based on the dataset, the theoretical upper bound for an actor leak is exceptionally constrained: ≤ 1 unreleased actor per 828,000 connection lifecycles (1.2 × 10⁻⁶ actors per connection), and ≤ 1 unreleased actor per 266,375,577 requests (3.8 × 10⁻⁹ actors per request). |
These tests were designed to observe system behavior and identify failure thresholds under a sudden, severe concurrency burst. It investigates three parameters: the throughput and failure rates under burst conditions, the viability of mesh reformation during severe memory starvation, and the secondary errors induced by extreme OS paging.
The run intentionally forces the server into physical swap from the outset by requesting 15,000 concurrent connections on a host with under 460 MB of available memory. As in previous tests, the node is configured to deliberately sever and redial its mesh link every 20 seconds.
The service sustained the 15,000 concurrent connections for approximately eight minutes. Throughput metrics for this period are tabulated below:
| Round | Window (Z) | Mean rps | Notes |
|---|---|---|---|
| 7 | 00:34:09–00:39:18 | 10,903 | full 300 s; p50 966 / p99 5,673 / max 66,579 ms |
| 8 | 00:39:23–00:44:06 | 11,290 | 283 s, 3,194,323 completed; cut off at 00:44:05, the exact second the OS OOM killer fired |
| Time | pid | Process RSS | Free | Pages in / min | Pages out / min | Major faults / min | Event |
|---|---|---|---|---|---|---|---|
| 00:35 | 10827 | 168 MB | 21 MB | +5.4k | +9.1k | +1.3k | 15,000 connections just established |
| 00:36 | 10827 | 166 MB | 23 MB | +39.3k | +41.0k | +13.5k | thrashing begins |
| 00:37 | 10827 | 163 MB | 34 MB | +51.2k | +44.1k | +21.7k | |
| 00:38 | 10827 | 167 MB | 38 MB | +57.9k | +45.0k | +23.3k | |
| 00:39 | 10827 | 178 MB | 20 MB | +42.5k | +40.4k | +18.7k | ab round 7 ends 00:39:18 (10,903 rps) |
| 00:40 | 10827 | 153 MB | 71 MB | +12.9k | +11.5k | +6.1k | subsides |
| 00:41 | 10827 | 162 MB | 39 MB | +7.2k | +18.5k | +2.6k | 00:41:21 swap_pager: I/O error - pageout failed — twice, error 5 |
| 00:42 | 10827 | 156 MB | 62 MB | +48.1k | +47.5k | +17.9k | thrashing resumes |
| 00:43 | 10827 | 159 MB | 34 MB | +55.7k | +43.9k | +21.9k | |
| 00:44 | — | 0, killed | 234 MB | +33.3k up to death | +28.7k | counter gone with it | 00:44:05 pid 10827 killed: failed to reclaim memory; sockstat killed in the same second; 00:44:47 cannot allocate bio |
| 00:45 | 16504 | 151 MB | 71 MB | +13.0k | +27.4k | 2.5k, new process | 00:44:08 supervisor restart; all 15,000 re-established within a minute |
| What ran out | The application process was not the constraint; the Resident Set Size (RSS) remained stable between 165–178 MB throughout the run. The constraint was the operating system kernel: socket allocation pushed wired (unpageable) memory to 203 MB, with network buffers consuming an additional 50 MB. The Scheme heap was never exhausted. |
| Mesh, during the thrashing | Despite severe major faulting, the scheduled control-plane churn continued uninterrupted. The node executed 86 link disconnections and 86 successful reconnections. |
| Then the kernel killed it | The OS OOM (Out of Memory) killer terminated the process strictly because it was the largest memory consumer. Up to the exact millisecond of termination, the application recorded zero internal faults, zero failed client responses, and zero failed mesh reformations. It was functioning correctly until the OS reclaimed the memory. (The OOM killer fired twice during the run; the second execution terminated a background sampling process.) |
| After the kill | The supervisor restarted the engine three seconds later, at 00:44:08. Within a minute it had resumed the full 15,000-connection load and immediately re-entered the paging state. |
1. Cost per connection
per connection 9.8 KB = Scheme 6.9 + kernel structures 2.2 + in-flight mbuf 0.7
13.5 KB = peak, including memory not yet collected
fixed baseline ~356 MB = kernel 230 + Scheme baseline 106 + userland 20
Maximum Connections N ≈ (total memory − 356 MB − reclaim margin) ÷ 9.8 KB2. Pre-OOM alarm metrics — a pageout I/O error event appeared in the system logs approximately three minutes before the OOM kill, whereas process RSS and free RAM metrics fluctuated ambiguously and provided no deterministic warning. Therefore, pageout I/O error is the definitive threshold metric for automated load shedding.
3. Extrapolated capacity for a 1 GB host
| Steady state | Approximately 30,000 concurrent connections with zero paging. This is an empirical measurement, not a projection: a subsequent test maintained ≥30,000 connections successfully throughout a 23-hour window on a 1 GB host. |
| Burst limit | Approximately 50,000 concurrent connections. At this volume, the system survives strictly by relying on the kernel pager (derived via the capacity formula above, not physically tested for long-duration stability). |
This test evaluates the mesh topology's ability to reconstruct itself under extreme disruption and high latency.
Environment: A three-node mesh (Paris, Hong Kong, and a local Mac) connected via a cloud-to-cloud WireGuard tunnel. Throughout the 17-hour test window, the Hong Kong node concurrently served a baseline load of 3,000 keep-alive connections to ensure the mesh was not tested in an idle state.
| Hard kills (Mac node) | The node received 1,036 SIGKILL commands at randomized intervals (p10: 34 s, p50: 82 s, p90: 133 s, max: 149 s). Strict randomization ensures no phase-locking occurs with the framework's 3-second base reconnect timer, preventing flattered metrics. |
| Deliberate drops (Paris node) | Executed 233 deliberate link drops configured to redial immediately. |
| Event parity | The centralized observer log recorded 1,038 peer down events against 1,040 peer up events. The absence of orphaned down events confirms 100% link recovery. |
| Hard-kill recovery time (N=1,037) | p50: 4,803 ms | p90: 9,126 ms | p99: 9,521 ms | max: 10,960 ms. This includes the external crash-loop supervisor's ~5-second restart penalty before any dialling commences. It represents the absolute upper bound of system downtime, not the framework's internal negotiation speed. |
| Soft-drop recovery time (N=233) | p50: 1,520 ms. The p99 tail lands at 3,210 ms, which strictly reflects instances where the immediate redial missed and fell back to the attempt-0 delay timer (3 s ± 25%). |
| Accounting invariant | 57,062 quiescent-state integrity checks were performed during the run, yielding zero anomalies. A minor subset of 49 checks required a single 250 ms retry before the system reached full quiescence. |
| Scope limits | This soak test asserts the quiescent-state accounting invariant under heavy load. It does not verify continuous one-sided reachability or strict event ordering across the distributed network. |
Because the distributed soak test relies on well-behaved nodes that do not transmit malicious states, a localized single-process fixture is used to enforce strict event ordering and validate adversarial inputs.
| Identity and replacement | Validates that when a same-named node reconnects with a higher generation number, the old connection is cleanly closed. The observer guarantees a strictly ordered node-down followed by a node-up event. Monitors spanning the two incarnations correctly answer oppositely for each. |
| Poison and late delivery | A message failing delivery three times is safely quarantined rather than crashing the node, allowing for enumeration and redelivery. Late mdown events still accurately reach their watchers, and dialing mechanics survive even if the registrar process is killed. |
| Close paths and write gates | Exercises six distinct TLS teardown paths (owner death, normal exit, double close, sealing, re-arm failure) via dedicated cells. The write gate ensures that two concurrent writers never interleave their payloads. |