Search This Blog

Server-sent events with RxJava and SseEmitter

Spring framework 4.2 GA is almost released, let's look at some new features it provides. The one that got my attention is a simple new class SseEmitter - an abstraction over sever-sent events easily used in Spring MVC controllers. SSE is a technology that allows you to stream data from server to the browser within one HTTP connection in one direction. It sounds like a subset of what websockets can do. However since it's a much simpler protocol, it may be used when full-duplex is not necessary, e.g. pushing stock price changes in real-time or showing progress of long-running process. This is going to be our example.

Imagine we have a virtual coin miner with the following API:

public interface CoinMiner {
BigDecimal mine() {
//...
}
}

Every time we call mine() we have to wait few seconds and we get around 1 coin in return (on average). If we want to mine multiple coins, we have to call this method multiple times:

This works, we can request /mine/10 and mine() method will be executed 10 times. So far so good. But mining is a CPU intensive task, it would be beneficial to spread computations over multiple cores. Additionally even with parallelization our API endpoint is quite slow and we have to patiently wait until all work is done without any progress notifications. Let's fix parallelism first - however since parallel streams give you no control over underlying thread pool, let's go for explicit ExecutorService:

However due to lazy nature of streams in Java 8, that tasks will be executed sequentially! If you don't grok laziness of streams yet, always read them from bottom to top: we ask to join some future so stream goes up and calls mineAsync() just once (lazy!), passing it to join(). When this join() finishes, it goes up again asking for another Future. By using collect() we force all mineAsync() executions, starting all asynchronous computations. Later on we wait for each and every one of them.

Introducing SseEmitter

Now it's time to be more reactive (there, I said it). Controller can return an instance of SseEmitter. Once we return from a handler method, container thread is released and can serve more upcoming requests. But the connection is not closed and the client keeps waiting! What we should do is keep a reference of SseEmitter instance and call its send() and complete methods later, from a different thread. For example we can start a long-running process and keep send()-ing progress from arbitrary threads. Once the process is done, we complete() the SseEmitter and finally the HTTP connection is closed (at least logically, remember about Keep-alive). In example below we have a bunch of CompletableFutures and when each completes, we simply send 1 to the client (notifyProgress()). When all futures are done we complete the stream (thenRun(sseEmitter::complete)), closing the connection:

We will learn later how to interpret such response on the client side. For the time being let's clean up the design a little bit.

Introducing RxJava with Observable progress

Code above works, but looks quite messy. What we actually have is a series of events, each representing progress of computation. Computation finally finishes, so the stream should also signal end. Sounds exactly like Observable! We start by refactoring CoinMiner in order to return Observable<BigDecimal:

Every time an event appears in Observable returned from mineMany(), we just mined that many coins. When all futures are done, we complete the stream as well. This doesn't look much better yet on the implementation side, but look how clean it is from the controller's perspective:

After calling coinMiner.mineMany() we simply subscribe to events. Turns out Observable and SseEmitter methods match 1:1. What happens here is pretty self-explanatory: start asynchronous computation and every time the background computation signals any progress, forward it to the client. OK, let's go back to the implementation. It looks messy because we mix CompletableFuture and Observable. I already described how to convert CompletableFuture into an Observable with just one element. Here is a recap, including rx.Single abstraction found since RxJava 1.0.13 (not used here):

RxJava has a built-in operator for merging multiple Observables into one, it doesn't matter that each of our underlying Observables emit just one event.

Deep-dive into RxJava operators

Let's use the power of RxJava to improve our streaming a little bit.

scan()

Currently every time we mine one coin, we send(1) event to the client. This means that every client has to track how many coins it already received in order to calculate total calculated amount. Would be nice if server was always sending total amount rather than deltas. However we don't want to change the implementation. Turns out it's pretty straightforward with Observable.scan() operator:

Sampling with sample()

It might be the case that our back-end service produces way too many progress updates then we can consume. We don't want to flood client with irrelevant updates and saturate bandwidth. Sending an update at most twice a second sounds reasonable. Luckily RxJava has a built-in operator for that as well:

sample() will periodically look at the underlying stream and emit the most recent item only, discarding intermediate ones. Fortunately we aggregate items on-the-fly with scan() so we don't loose any updates.

window() - constant emit intervals

There is one catch though. sample() will not emit the same item twice if nothing new appeared within selected 500 milliseconds. It's fine, but remember we are pushing these updates over the TCP/IP connection. It's a good idea to periodically send an update to the client, even if nothing happened in the meantime - just to keep the connection alive, sort of a ping. There are probably many ways of achieving this requirement, e.g. involving timeout() operator. I chose grouping all events every 500 ms using window() operator:

This one is tricky. First we group all progress updates in 500 millisecond windows. Then we calculate total (similar to scan()) of coins mined within this time period using reduce. If no coins were mined in that period, we simply return ZERO. We use scan() in the end to aggregate sub-totals of every window. We no longer need sample() since window() makes sure an event is emitted every 500 ms.

Client-side

There is a lot of examples of SSE usage in JavaScript, so just to give you a quick solution calling our controller:

Comments

Have you tried RxNetty? If so, how do you think it compares to SseEmitter? I've been thinking of using that on a new project to do something very similar to above, so this sort of design has been on my mind a lot lately.

Yes, I was experimenting with RxNetty: http://www.nurkiewicz.com/2014/12/accessing-meetups-streaming-api-with.html (client side). It depends on your current setup. If you already use Spring MVC, SseEmitter looks cool. Byt I *guess* Netty is more scalable.

Watch out for timeouts though! For example, asyncTimeout on Tomcat is 10s by default (according to its docs, though it was 30s for my Spring Boot app). It basically means that the server will forcibly timeout and terminate the connection.

I wonder which is better: removing the timeout (setting asyncTimeout=0) and letting the connection run for minutes, or implementing reconnect on the client. What do you think?

Yup, I got beaten by that as well, see end of this article: http://www.nurkiewicz.com/2011/03/tenfold-increase-in-server-throughput.html. But not only timeouts are an issue, even worse Tomcat was reusing request/response objects back then, making debugging extremely hard.