A lossy link does not care about your protocol
I spent an afternoon convinced I had a bandwidth problem. Same server, same evening, same laptop: one transport moved twenty-one megabits, the other managed four and a half. The obvious conclusion is that someone downstream is shaping one of them, and the obvious conclusion was wrong.
What I had not measured was loss. Thirty pings, a hundred and ninety-eight milliseconds of round trip, and thirteen percent of them never came back. Jitter, meanwhile, was one millisecond. The link is not unstable in the sense people usually mean — it is remarkably steady, and it is quietly throwing away one packet in eight.
Two answers to the same problem
A congestion-controlled sender treats loss as a message: slow down, you are the reason the queue is full. At thirteen percent loss and a two-hundred millisecond round trip, that conversation is expensive. Every loss costs a round trip to discover and another to repair, so a single stream settles somewhere around five megabits and stays there, politely.
A sender that ignores loss entirely and transmits at a declared rate does much better here — not because it is smarter, but because it refuses to have the conversation. The repair happens above the congestion controller, in the application, and the cost shows up as packets the network silently dropped instead of a stalled window.
| Transport | One stream | Four streams |
|---|---|---|
| Congestion-controlled | 4.7 Mbit/s | 10.7 Mbit/s (2.3×) |
| Fixed-rate | 20.9 Mbit/s | 22.0 Mbit/s (1.1×) |
The four-stream column is the interesting one. If the ceiling were a shaper, adding streams would not help. It helped by exactly the factor you would expect from four independent congestion windows, which tells me the limit is per-flow window, not available bandwidth. That is a useful distinction: one is a wall, the other is a door you can only open with more handles.
Measurement first, architecture second. I have never once regretted running the ping before redesigning the thing.
What I changed
Nothing about the protocol — the protocol is fine. I stopped assuming that “slow” means “shaped”, started keeping the two transports side by side instead of picking a winner, and stopped testing throughput with a single large download, which is the one access pattern that hides everything else.
If you take one thing from this: when a link is slow, measure loss and jitter before you measure anything else. Throughput is the last number to look at, and it is the only one everybody looks at first.