Close connections that have more packets in flight than a configurable limit.
Note, this is me commandeering the pending changes from !2815 from @jrouviere
poll() marks a reset stream as finished and removes it if the local
side is also finished, but recv_body() returned the same StreamReset
error without doing either.
A reset consumed by recv_body() stops the transport stream from being
readable, so poll() never handles it. If the application had already
finished its side of the stream, the entry was never removed. If it
had not, a later recv_body() call found the transport stream finished
and queued an Event::Finished after the reset.
Move the reset handling from poll() into a helper and call it from
recv_body() as well.
RecvBuf::write() discards a buffer whose data is entirely below the
read offset, including its FIN flag. This happens when a retransmission
of data that the application already read also carries the stream's
FIN for the first time, for example when the sender appends the FIN to
lost data that it retransmits.
The FIN is never delivered, but the sender considers it acknowledged,
so the stream stays open on the receiver until the connection closes.
Instead of discarding such a buffer when it carries the first FIN,
replace it with an empty buffer at the final size. The application
then reads the FIN without receiving duplicate data, and the queued
buffer does not keep the duplicate payload alive. Duplicate FINs are
still discarded, and draining streams do not become readable.
RecvBuf::reset() notifies the application of a reset by writing an
empty FIN buffer at the final size. When a FIN had already set the
final size, write() discarded that buffer, because nothing else needs
to be kept from an empty buffer once the final size is known.
A reset that replaced unread data or an unread FIN therefore completed
the stream without making it readable. The application never saw the
reset, and HTTP/3 kept the request entry until the connection closed.
Insert the empty buffer directly instead, unless the application has
already read the FIN or shut down the read side. Since write() no
longer checks the final size against the flow control limit, reset()
now does so before replacing the receive state.
WaitingStreams previously stored Abortable<WaitForStream> values in a
FuturesUnordered with per-direction abort maps. Each wait existed only
to register a channel waker and return readiness to H3Driver.
Channel endpoints now remain in StreamCtx with one shared StreamWake per
stream. RecvState owns RecvDelivery in Open and FinPending, pairing the
app-bound sender with its capacity-wait flag. mark_fin_pending moves the
whole payload, preserving waits. Closed drops both without manual wait
cancellation. Capacity polling retains Pending waits and releases its
readiness permit before application reads. Closed delivery cannot wait.
The outbound receiver remains independent because queued FIN can drop
it before transport writes finish. An outbound waiting flag identifies
the receiver to re-poll, and stale ready IDs are ignored after cleanup.
ReadyStreams encapsulates coalescing: the queue owns Arc<StreamWake>
notifications, and dequeue clears their AtomicBool before returning a
stream ID. Channel polling no longer participates in queue
acknowledgment. Direct polls preserve queued notifications; dequeue
rearms wakes even without a subsequent channel poll. Wake metadata owns
no stream endpoints.
STOP is probed before H3 frame validation, except for queued
PeerStreamError. STOP drains ordinary buffered frames, preserves
PeerStreamError, and redispatches it to close both directions. Errors
sent later through outstanding permits are rejected when the receiver is
dropped.
App-bound readiness is serviced before outbound data. Ready-queue
draining retains Tokio cooperative budgeting. Regressions verify FIN
wait preservation, automatic wait removal on closure, stale wakes,
cooperative yields, wake coalescing, and deferred cleanup. Remaining
lifecycle invariant checks have runtime paths and debug assertions.
The cooperative-yield regression bounds its post-yield receive with a
deadline. A lost readiness notification fails both channel-direction
cases instead of leaving the test suite waiting indefinitely.
Peer-reset handling remains idempotent when the body reader and HTTP/3
polling both report a reset. Unit coverage checks repeated notifications
after open or FIN-pending delivery and verifies that capacity waits are
dropped without closing the response channel. Strict cancellation checks
apply only to local reset and STOP operations.
FinPending remains distinct from Closed when the inbound application
channel is full. Outbound FIN, peer STOP, and local reset now retain the
context until FIN reaches the app channel or its receiver is abandoned.
Successful FIN delivery closes the app-bound sender. Receiver
abandonment closes pending delivery without sending another read
cancellation. A single derived fully_closed decision requires closed
receive and send states. Private read outcomes distinguish normal
completion, FIN delivery, blocking, reset, and forced cleanup on receive
errors.
StreamClosed and idle connection cleanup wait for enqueueing FIN or
receiver abandonment; the application need not consume the queued FIN.
Separate coalesced-readiness and receiver-abandonment regressions share
a real-transport fixture with the three delayed-write-closure cases. The
client regression requires an explicit FIN before disconnection.
RecvState records an open receive direction, pending peer FIN, or closed
application delivery. SendState records transport write completion. Both
states remain explicit while waiting futures temporarily own channels.
Read, write, and cleanup decisions distinguish these states. The
refactor preserves the existing cleanup policy so the pending-FIN fix
remains independently reviewable. Bodyless client requests start with a
closed send state.
The STOP matrix names each request and response case and checks exact
receive states. Scoped guidance forbids overlapping completion
predicates while allowing one derived cleanup decision.
SendBuf stores PeerStopSendingState instead of an optional error code
and a separate reporting flag. NotReceived is the default; NotReported
retains a received error; Reported retains its code for later write
errors. Local shutdown remains independent. Pure presence checks use
is_stopped, while notification transitions match the states
exhaustively.
PeerStopSendingState::report performs the idempotent notification
transition and returns StreamStopped when a peer error exists.
check_not_stopped retains notification state.
cap_and_report_stop_sending reports STOP_SENDING only when capacity or a
write is returned to the application. Internal writability checks still
use the non-reporting cap. The stream map collects or disarms stopped
streams after reporting.
Retransmission and emission only require absence of STOP_SENDING,
regardless of reporting state. Regressions cover non-reporting capacity
checks, repeated reporting, duplicate frames, and capacity without STOP.
Stream::state() distinguishes incomplete streams, completion with
pending application delivery, and streams ready for collection. The
classification is derived from receive buffers, acknowledgements, and
STOP notifications rather than stored as another lifecycle flag.
Collection sites only accept Collectable, so future states default to
retaining the stream. The completion regression covers pending STOP
errors and receive resets.
Scoped guidance favors exhaustive decisions where new variants could
need different behavior, simple safe-default predicates, and one derived
helper for repeated decisions.
`H3Driver::upstream_read_ready()` assumes that `StreamCtx::queued_frame`
is empty when downstream channel readiness is delivered. A violation
previously overwrote buffered output, including when the channel closed.
The invariant now has a debug assertion and an `InternalError` return
path in release builds. Existing queued output remains intact when the
check fails.
The regression cases cover both received data and channel closure. They
catch the expected assertion in debug builds and verify the runtime
error when debug assertions are disabled. Both configurations inspect
the retained frame after the failed operation.
The migration helper previously waited synchronously for response FIN
after draining the currently available UDP flight. Polling Done did not
receive more packets or yield to the server tasks, so a split response
could stall before migration began.
HTTP/3 polling now processes one event per call and returns Done to its
caller. The async response loop retains header observation, awaits more
flights on Done, and yields cooperatively while draining buffered
events. A deadline bounds response waits. Reset and read errors fail
explicitly.
The regression delivers HEADERS, DATA, and FIN in separate in-memory
flights and explicitly checks Done between flights. Single-event polling
keeps those checks finite instead of spinning on an incomplete response.
send_pkt_to_server() returns the size of the peer response to the
injected MAX_STREAM_DATA frame. That response is an ACK whose delay is
derived from elapsed wall-clock time and encoded as a QUIC varint. The
fixed 39-byte expectation therefore requires ACK generation within about
512 microseconds. Unoptimized QEMU can exceed that interval and require
a two-byte delay encoding.
The test only needs the MAX_STREAM_DATA update to be processed before it
checks the resulting body capacity and STREAM_DATA_BLOCKED suppression.
Checking successful packet exchange preserves that coverage without
constraining a valid ACK encoding.
A peer reset in the middle of a DATA frame can be observed by both
HTTP/3 polling and the body reader. The second notification reaches an
already closed receive direction, so requiring an open direction is an
invalid invariant.
Peer reset handling now records the error and closes application
delivery without asserting that the direction was open. This preserves
existing release behavior while allowing the same valid reset sequence
in debug builds.
The regression withholds the payload of a DATA frame before resetting
the stream. It verifies that the reset code is recorded, body delivery
closes, and the response channel and connection remain open.
Some deployments constrain transmission outside quiche but still need
QUIC recovery. Add an opt-in controller with a fixed congestion window
that does not adapt to ACKs or loss.
The controller preserves ACK processing, RTT estimation, loss detection
and bytes-in-flight accounting. Its window defaults to usize::MAX, and
callers can set a finite byte window through the Rust, C and
tokio-quiche configuration APIs. The setting has no effect unless the
controller is enabled and selected.
Restore the configured window in controller callbacks and after PMTU
updates. This prevents shared recovery code from changing the policy
permanently.
The controller is behind the non-default
congestion_window_unchecked_available feature in quiche and
tokio-quiche. With the default window, callers must enforce their own
transmission limits.
A reset received while waiting for the rest of a DATA frame makes the
stream readable without any payload left to read. poll() returned
Event::Data in this case, and a following recv_body() call returned
Done with the QUIC stream finished. Applications that check
stream_finished() at that point, such as tokio-quiche, treated the
truncated body as complete before seeing the reset.
When a stream in the DATA state is finished, which includes every
reset stream, poll() now does a zero-length read to detect the reset
and returns Event::Reset directly. It also marks the HTTP/3 stream as
finished when returning the reset, so that a later recv_body() call
does not queue an Event::Finished as well.
A FIN in the middle of a DATA frame still produces Event::Finished.
The new test for that case documents the existing behavior.
Fixes#2277.
Co-authored-by: Alessandro Ghedini <alessandro@cloudflare.com>
This covers additional items from #2492 that are not implemented in
merged #2697.
* The MAX_STREAM_OVERHEAD constant we used to guard the path that writes
STREAM frames was too low; STREAM frame headers can be up to 19 bytes.
However, using the maximum overhead is also problematic, since it would
prevent us from writing the more common, smaller frames. Instead, remove
the guard and check the STREAM frame actual header size.
* If there is not enough space for the STREAM frame header, we incorrectly
removed the stream from flushable. This can permanently strand the stream.
* Preserve the legacy recovery threshold independently of STREAM header
sizing.
Follow up on #2766 and #2784:
* document the safety invariant for removing a stopped stream from its intrusive tree;
* avoid panicking if the stream has already been removed when its stop is reported;
* include the receive-closure and stream-closure hooks in the HTTP/3 driver guide.
Expose an opt-in DSCP handle for server connections. Packets inherit the UDP socket's TOS/TClass until the handle selects a value; then apply per-packet IPv4/IPv6 marking, including GSO. Stateless replies inherit the socket default.
Test connection isolation, explicit zero, IPv4-mapped and IPv6 packets, and stateless socket-default inheritance.
Completed-stream collection is centralized in
`StreamMap::try_collect()`. ACK and `RESET_STREAM` behavior remains
unchanged, including retention of readable and peer-stopped streams
until their application-facing state is observed.
Responses can terminate with RESET_STREAM or FIN before response HEADERS
while the request send direction remains open. Final stream cleanup occurs
only after both directions finish.
A shared receive-termination hook handles both cases. Client state still
waiting for response headers is cleared, while the RESET_STREAM and FIN
event paths otherwise retain their existing behavior. StreamCtx remains
until the request writer closes.
Coverage includes RESET_STREAM and FIN with an open request body.
`cwnd_available()` now consistently reports the available congestion
window. Bypassing that window for a PTO probe is specific to packet
generation, where the packet epoch is known.
Pending keyed probe epochs are attempted before normal packet selection.
The packet-generation path also preserves close-frame priority when another
packet number space has a pending probe.
Co-authored-by: Antonio Vicente <avicente@cloudflare.com>
libc 0.2.190 made the dyld image functions, such as
`_dyld_image_count`, available only on macOS. As a result `backtrace`
0.3.76, which the workspace depends on through `foundations` and
`wirefilter-engine`, fails to build on iOS and breaks the `quiche_ios`
CI jobs.
Cargo cannot exclude a single version, so cap the requirement below
0.2.190 rather than pinning an exact version. This is less restrictive
for downstream users of the published crates. The cap can be lifted
once libc releases a fix.
See https://github.com/rust-lang/libc/issues/5601.
STOP_SENDING can arrive while a response is idle and its outbound
receiver is parked. Waiting for another application frame to observe
the stop leaves the response channel open and delays transport cleanup.
The driver probes stream capacity when no H3 send is attempted, records
the stop code, and closes active or parked response receivers. Late
channel completions cannot restore canceled output, but peer stream
errors returned by parked receivers still shut down the read side.
Pending stops are handled before exposing new response senders and
remain available to existing contexts across receive-side completion.
Finished streams without a context report pending stops so transport
cleanup can proceed. The tests cover idle responses, reordered request
frames, terminal stream errors, and stream-credit recovery.
IoSlice::new() created a shared borrow, which were later used as a receive buffer for recvmmsg. This violates Rust aliasing/mutability rules and could produce undefined behaviour.
Opening a higher-numbered stream implicitly opens lower streams of the
same type, even when no Stream object exists. Accept peer frames for
those local streams while rejecting frames for streams not yet opened.
Apply this check to STOP_SENDING, STREAM, RESET_STREAM, and
MAX_STREAM_DATA, and cover the behavior with transport tests.
Track unreported STOP_SENDING errors independently of writable queue
membership. A stopped stream can be popped from the queue before the
application tries to send and still needs to return StreamStopped.
Keep stopped streams visible to writable polling when connection send
capacity is zero. Collect completed streams after reporting the error.
Also collect completed streams when the receive side finishes without
an application read, such as after stream_shutdown(Read). The send
side is only complete once its FIN is acknowledged, so a queued FIN is
still delivered first.
Co-Authored-By: vendemiat <vicky@geeks.net.np>
SendBuf::is_complete() used acknowledged byte ranges as a proxy for FIN acknowledgement. A zero-length FIN contributes no byte range, so an ACK for an earlier data-only frame is indistinguishable from an ACK for the FIN.
Fixes#2525.
Prevent additions to HTTP/3 driver settings from breaking downstream
crates. Update integration tests to configure settings through a default
value because external crates can no longer use struct literals.
A zero-length non-fin STREAM frame advances the peer's largest received
offset for the stream (RFC 9000 Section 19.8). The receive path charges
connection-level flow control from that offset, but RecvBuf::write()
discarded empty non-fin buffers without recording the new high-water
mark, so the gap up to the frame's offset was charged a second time
when the in-flight data covering it arrived.
The sender produces exactly such frames: send_single encoded a STREAM
frame even when emit() returned zero bytes without fin. That happens
when packet packing leaves exactly header-sized space, which requires
the header to exceed MAX_STREAM_OVERHEAD — a two-byte stream id varint
plus an eight-byte offset varint — so it surfaces on long-running bulk
streams past 1 GiB of transfer.
Between two quiche endpoints the receiver-side inflation accumulates
silently until a legal, contiguous frame appears to exceed MAX_DATA and
the receiver closes the connection with FLOW_CONTROL_ERROR, while the
sender has consumed exactly the credit it was granted. Observed in
long-running bulk transfers over a lossy path at 1-3 spurious
connection kills per hour; instrumented captures showed rx_data
exceeding the sum of per-stream high-water marks by exact whole-frame
multiples.
Fix both halves:
- RecvBuf::write() advances the high-water mark for empty non-fin
buffers before the early return, keeping the flow-control charge and
the stream state in lockstep. This is the load-bearing fix and also
defends against any other sender emitting such frames.
- send_single skips zero-length non-fin frames entirely. The bytes such
a frame references were already charged to tx_data when buffered, so
this is defensive/efficiency behavior: the frame is pure overhead
plus a consistency hazard for receivers. The skip is excluded from
the on_app_limited() transition via stream_data_skipped, since data
is pending and the sender is size-limited on that packet, not
application-limited.
Tests:
- empty_stream_frame expectations updated to the consistent semantics;
the old sequence also accepted a final size below the largest
received offset, which RFC 9000 Section 4.5 forbids and which no real
quiche sender can produce (fin_off >= off_front).
- New flow_control_empty_stream_frame_not_double_counted pipe test
injects an empty forward frame and verifies the charge lands exactly
once; it fails on the previous code with a spurious Error::FlowControl.
- New zero_length_stream_frame_not_sent test seeds a stream's send
state at 2^30 via a #[cfg(test)] seam and sweeps output-buffer sizes
across the header-only window; without the sender fix it fails at
exactly the cap that leaves header-sized space, and it verifies the
skipped data is delivered afterwards.
* Keep draining stream's read offset in sync with empty STREAM frames
An empty non-fin STREAM frame now advances RecvBuf::len, but on a
draining stream the early return skipped the `off = len` update the
buffered path performs. Connection already counts the frame's offset
delta as consumed for a draining stream, so a later RESET_STREAM
credited `final_size - off` a second time.
Also drop the redundant flushable-queue bookkeeping from the sender-side
skip (a stream reaching it is still flushable), restore off_back()'s
misplaced doc comment, and trim the new comments.
* Keep rotating incremental streams when a STREAM frame is skipped
Dropping the whole flushable-queue update from the header-only skip
left the skipped stream at the head of the queue. Since packet space is
also bounded by the congestion window, the same size can recur on every
send() until an ACK arrives, so an incremental stream with a long header
would block other same-urgency incremental streams whose data does fit.
Restore the rotation (the not-flushable branch stays removed: a stream
reaching the skip still has data queued) and add a regression test.
Fixes#2696.
Pick up upstream fixes for undefined behavior in RBTree iterators and
races when mutating atomic links. Quiche uses both for stream priority
queues, so update to the release containing these safety fixes.
Version 0.10 changes the intrusive_adapter! link-field separator. Update
the stream priority adapters to use the new syntax while preserving
their existing behavior.
gregor-cf