Comments

edited

I tried to turn on TCP Fast Open upon TLS 1.3 0-RTT in an attempt to reduce connection startup latency even more. (Regular socket IO with TCP_FASTOPEN_CONNECT on kernel 4.11+.)

OpenSSL will flush the socket after ClientHello regardless of presence of early data. Linux's implementation of TCP Fast Open will only send the first write as the SYN packet. A second write (early data in this case) has to wait for the TCP 1-RTT handshake. Thus OpenSSL's 0-RTT and Linux's TCP Fast Open defeats each other.

(Selected is the early data.)

The setup: Nginx running on TLS 1.3 draft 21 reverse proxying to an HTTP server at localhost:10080. The loopback interface is set to have 1500 MTU and 100 ms delay on TLS ports. The client is ./openssl s_client -sess_in /tmp/s -alpn h2,http/1.1 -early_data /tmp/http.txt -connect 127.0.0.43:10443. The early data is GET / HTTP/1.0\r\n\r\n. Tested with e44480c.

The TLS draft doesn't seem to be concerned about the transport layer or require ClientHello and early data be sent as separate packets. I don't really know. Or is there some other reason for the flush after ClientHello?

4.1.2. Client Hello
[...] After sending the ClientHello message, the client waits for a ServerHello or HelloRetryRequest message. If early data is in use, the client may transmit early application data (Section 2.3) while waiting for the next handshake message.

I was able to start a connection with both Client Hello, Application Data in a single SYN packet by hacking the state machine a little bit to skip the flush.

This comment has been minimized.

I should note that I've read #3906 before reporting this issue and the tests were done with TCP_NODELAY all along. In the scenario of TCP_FASTOPEN even enabling TCP_NODELAY would not avoid the 1 RTT before the early data packet is sent. LWN has noted this issue about TCP Fast Open:

This assumes that the client's initial request is small enough to fit inside a single TCP segment. This is true for most requests, but not all. Whether it might be technically possible to handle larger requests—for example, by transmitting multiple segments from the client before receiving the server's ACK—remains an open question. [1]

I guess what I'm asking for OpenSSL is that can changes be made to allow this usage of TCP Fast Open by optionally not flushing immediately after ClientHello (e.g. if TCP_NODELAY is detected on this socket).

This comment has been minimized.

So reading how TCP Fast Open works, it seems to require that you
use sendto() for the first packet instead of write(), and that
that sendto() should contain all the data you want to send in that
first packet. How do you currently collect that data?

This comment has been minimized.

@kroeckx This is not portable but Linux kernel provides TCP_FASTOPEN_CONNECT which allows regular sockets to use Fast Open without sendto() calls. See torvalds/linux@19f6d3f. The use case here is to minimize the latency of a TLS tunnel between two endpoints both controlled by the user (thus with updated Linux kernel). I haven't looked into how this can be implemented with sendto() on older Linux or other platforms.

This comment has been minimized.

The SYN packet is sent immediately on the first write(). The kernel TCP stack doesn't buffer for Fast Open. I just tested that both disabling TCP_NODELAY and turning on TCP_CORK suffered the 1-RTT TCP handshake.

We'd like the first bit of early_data and the ClientHello to go in the
same TCP packet if at all possible to enable things like TCP Fast Open.
Also, if you're only going to send one block of early data then you also
don't need to worry about TCP_NODELAY.
Fixesopenssl#4783

We'd like the first bit of early_data and the ClientHello to go in the
same TCP packet if at all possible to enable things like TCP Fast Open.
Also, if you're only going to send one block of early data then you also
don't need to worry about TCP_NODELAY.
Fixesopenssl#4783