Edge PAAS Endpoints

Traffic Throttling Mechanisms

In website performance optimisation it is easy to focus on improving the speed of the experience of
a single visitor but it is just as important to consider the speed and availability of your website as
the number of concurrent users grows.

Whether your website has generally high traffic or experiences occasionally spikes of very high
traffic, the problem is the same - if your infrastructure cannot handle the load, the experience of
every visitor will suffer. The site may become slow for everyone or it may even become completely
unavailable.

Typically you will want to invest in the necessary infrastructure changes so that you can give every
visitor the best experience under high load conditions but this can take time. If you need an interim
solution you may want to consider traffic throttling

What is throttling

Throttling, at least in the context of this article, is concerned with limiting traffic coming from
legitimate visitors as opposed to dealing with denial-of-service attacks.

The idea is, that as the number of concurrent visitors grows, instead of the experience of all visitors
degrading, you give the best experience to a portion of the visitors, and for the remaining visitors you
provide a ‘Sorry, please try again soon’ message. You may have seen something similar if you’ve ever
tried to purchase event tickets online on the day the tickets are released.

This way, you are able to provide a quality service to the maximum number of visitors that your current
infrastructure can handle, and as those visitors complete their transactions and leave, you can allow
the next batch of waiting visitors access to your site.

It is true that the visitors that fall into the ‘try again soon’ bucket may be annoyed and choose not
to wait and not come back but given that there are many other visitors still having a great experience,
for many site owners this is a reasonable alternative to upsetting all visitors.

It is also possible for the ‘try again soon’ experience to offer simple features like subscribing
to a mailing list, or browsing a static catalogue of products.

Sample image showing a user when throttling engaged:

Choosing what to throttle

Typically, the threshold for throttling is directly connected to the number of concurrent HTTP requests
that your website infrastructure can handle at a good level of performance. As such, the simplest mechanism
for throttling is to limit the number of incoming HTTP requests. Unfortunately because any given web
page load will involve multiple requests for HTML, stylesheet, script, and image resources, it is
possible that request throttling could engage in the middle of a visitor’s page load and result in
broken images, an unstyled page, or broken behaviour.

The layer of infrastructure used to implement throttling would typically be a load balancer or reverse-proxy
positioned between the user-agents and the origin web servers. At this layer, the next simplest throttling
mechanism would be to throttle by connection. This gives a slightly better chance of ensuring that a
single visitor will not engage throttling in the middle of their page load but still isn’t perfect
because browsers today will open several connections for a single visitor to download multiple resources
in parallel.

Additionally, if the idle browser connection times-out while the visitor decides whether to proceed,
it is possible that the next connection they make (eg to add a product to their cart) will be subject
to throttling.

One more possible throttling mechanism worth mentioning is to throttle by IP address. Often visitors
inside an enterprise environment or using a cellular network will be proxied through one or more
common IP addresses. This can lead to either allowing or throttling a large portion of visitors just
by allowing or throttling a single IP address and is not a great option.

Throttling by cookie

To give your visitors the most consistent experience, albeit reducing the correlation to origin
requests, a very effective option is to throttle by cookie.

By giving each unique visitor a cookie and choosing which cookies will be allowed and which cookies
will be throttled you can ensure that a single allowed visitor will not be subjected to broken resources
on a page load, or become blocked midway through their transaction.

Using cookies however can require more capable infrastructure that is able to parse and manipulate
the HTTP data - something that many simple load balancers cannot do. A good reverse-proxy, like
Varnish Cache for example, is a good tool for this
job.

Server state versus client state

If you decide to using cookies for throttling, there is another decision to be made about where to
keep track of which cookies are throttled and which are allowed.

A simple mechanism is to specify in the cookie value whether to throttle or allow. This has very low
resource requirements on your infrastructure (which is already under load at this point) but does
carry the risk that an intelligent visitor may manipulate the cookie value to grant them access when
they otherwise would be throttled. This can be mitigated by encrypting the cookie but comes at the cost
of CPU time on your infrastructure.

The other downside this simple approach is that it is difficult to know just how many visitors are
allowed or throttled at any given time and means you will probably end up using a percentage threshold
to throttle visitors instead of an absolute number.

The alternative is to track the state of a given visitor cookie on your infrastructure. To do this
well will typically require a shared, fast, and highly-available data store accessible by all your
servers responsible for implementing the throttling. You may prefer to invest the time and money
required for this setup on your primary website infrastructure instead.

However, if you can do server-side state tracking for throttling this can enable features like:

Throttling to an absolute number of visitors instead of a ratio.

Providing a guide to throttled visitors about how long the wait may be.

Allowing the throttled visitors waiting longest to access the site first when traffic subsides.

Ultimately, whatever mechanism you implement, throttling should be a short term solution until you
can improve your website to handle larger numbers of concurrent visitors.