Thanks for implementing this! We should have more cancellable Deferreds. This is looking pretty good. However, some improvements are needed:

Please write tests for cancelling after the response download has started, but hasn't finished. E.g. your download is too slow so you decide to stop it. To make sure you cover all cases maybe mention the covered state machine states in the cancellation tests, and make sure they match the full set of tests.

Needs a news file.

You should add a mention and/or example to the web client howto about the fact that you can cancel requests.

If request was cancelled, and any bytes at all have been written (or for completeness probably always is easier), rather than using loseConnection, abortConnection should be used. Given that you may be dealing with uncooperative server, and/or you're giving up and don't care about writing any more, having the connection close immediately is what you want, as opposed to waiting until all write buffers are cleared, which may take an arbitrarily long amount of time.

Cancelling when in TRANSMITTING eventually results in loseConnection(); this should really be abortConnection(). I believe the relevant code is ebRequestWriting in HTTP11ClientProtocol.request.

For completeness sake it might be worth testing that cancellation works during the connection process, i.e. that HTTPConnectionPool.getConnection results are cancellable... but that just devolves to endpoint cancellation. So not sure quite what to do here. Probably nothing.

The auto-retry code in _RetryingHTTP11ClientProtocol._shouldRetry should be modified so it does not retry if the reason for failure was cancellation.

File a ticket for making the result of FileBodyProducer.startProducing cancellable.

test_cancelDuringBodyProduction calls HTTP11ClientProtocol.connectionLost - several of the other new cancel tests don't, though. is there something special about this one?

test_cancelDuringChunkedBodyProduction seems pretty similar to test_cancelDuringBodyProduction. Maybe the repetition could be factored out into a helper.

I agree that cancellation before connection setup is basically just endpoint cancellation. Maybe there should be a test for HTTPConnectionPool.getConnection that exercises this cancellation code path, though? Seems like it should be easy to write. If you get stuck, I wouldn't be opposed to seeing this done in a follow-up ticket.

twisted/web/test/test_agent.py

I see the test docstring follows the local convention of using "should", but maybe fixing the existing docstrings is simple enough that we can make it all follow the modern style instead?

test_onlyRetryIdempotentMethods and test_onlyRetryIfNoResponseReceived should be split up into multiple tests. Can you file a ticket for that?

twisted/web/client.py

The change to _shouldRetry is incompletely tested. There is no test coverage for a _WrapperException without a CancelledError

That's all. Please address to your satisfaction and then merge. Thanks again to everyone who contributed!