ASF

Project Documentation

Legend

External Link

Opens in a new window

Introduction

By default HttpClient is configured to provide maximum reliability and standards
compliance rather than raw performance. There are several configuration options and
optimization techniques which can significantly improve the performance of HttpClient.
This document outlines various techniques to achieve maximum HttpClient performance.

Contents

Reuse the HttpClient instance

Generally it is recommended to have a single instance of HttpClient per communication
component or even per application. However, if the application makes use of HttpClient
only very infrequently, and keeping an idle instance of HttpClient in memory is not warranted,
it is highly recommended to explicitly
shut down the multithreaded connection manager prior to disposing
the HttpClient instance. This will ensure proper closure of all HTTP connections in the
connection pool.

Connection persistence

HttpClient always does its best to reuse connections. Connection persistence is enabled
by default and requires no configuration. Under some situations this can lead to leaked
connections and therefore lost resources. The easiest way to disable connection persistence
is to provide or extend a connection manager that force-closes connections
upon release in the
releaseConnection method.

Concurrent execution of HTTP methods

If the application logic allows for execution of multiple HTTP requests concurrently
(e.g. multiple requests against various sites, or multiple requests representing
different user identities), the use of a dedicated thread per HTTP session can result in a
significant performance gain. HttpClient is fully thread-safe when used with a thread-safe
connection manager such as
MultiThreadedHttpConnectionManager. Please note that each respective thread of execution
must have a local instance of HttpMethod and can have a local instance of HttpState or/and
HostConfiguration to represent a specific host configuration and conversational state. At the
same time the HttpClient instance and connection manager should be shared among all threads
for maximum efficiency.

Request/Response entity streaming

HttpClient is capable of efficient request/response body streaming. Large entities may be submitted
or received without being buffered in memory. This is especially critical if multiple HTTP
methods may be executed concurrently. While there are convenience methods to deal with entities such as
strings or byte arrays, their use is discouraged. Unless used carefully they can easily lead to
out of memory conditions, since they imply buffering of the complete entity in memory.

Response streaming: It is recommended to consume the HTTP response body as a stream of
bytes/characters using HttpMethod#getResponseBodyAsStream method. The use of HttpMethod#getResponseBody and
HttpMethod#getResponseBodyAsString are strongly discouraged.

Request streaming: The main difficulty encountered when streaming request bodies is that
some entity enclosing methods need to be retried due to an authentication failure or an I/O failure.
Obviously non-buffered entities cannot be reread and resubmitted. The recommended approach is to create a custom
RequestEntity capable of
reconstructing the underlying input stream.

Expect-continue handshake

The purpose of the HTTP 100 (Continue) status is to allow a client sending a request entity to
determine if the target server is willing to accept the request (based on the
request headers) before the client sends the request entity. It is highly inefficient for the client
to send the request entity if the server will reject the request without looking at the body.
Authentication failures are the most common reason for the request to be rejected based on the request
headers alone. Therefore, use of the 'Expect-continue' handshake is especially recommended with
those target servers that require HTTP authentication. For proxied requests caution
must be taken as older HTTP/1.0 proxies may be unable to correctly handle the 'Expect-continue'
handshake.

Stale connection check

HTTP specification permits both the client and the server to terminate a persistent (keep-alive)
connection at any time without notice to the counterpart, thus rendering the connection invalid
or stale. By default HttpClient performs a check, just prior to executing a request, to determine if the
active connection is stale. The cost of this operation is about 15-30 ms, depending on the JRE used.
Disabling stale connection check may result in slight performance improvement, especially for small
payload responses, at the risk of getting an I/O error when executing a request over a connection
that has been closed at the server side.

Cookie processing

If an application, such as web spider, does not need to maintain conversational state with the target
server, a small performance gain can made by disabling cookie processing. For details
on cookie processing please to the HttpClient Cookies Guide.