Search This Blog

Reactive emoji tracker with WebClient and Reactor: consuming SSE

In this article we will learn how to consume infinite SSE (server-sent events) stream with Spring's WebClient and Project Reactor. WebClient is a new HTTP client in Spring 5, entirely asynchronous and natively supporting Flux and Mono types. You can technically open thousands of concurrent HTTP connections with just a handful of threads. In standard RestTemplate one HTTP connection always needs at least one thread.

As an example, let's connect to this cute little site called emojitracker.com. It shows emojis being used in real-time on Twitter. Looks quite cool! All credits go to Matthew Rothenberg, the creator of that site. It's very dynamic so there obviously has to be some push mechanism underneath. I wore my hacker glasses and after hours of penetration testing, I discovered the following URL in Chrome DevTools: http://emojitrack-gostreamer.herokuapp.com/subscribe/eps. If you connect to it, you'll get a fast stream of emoji counters:

Dozens of data points per second, ready to be consumed via convenient SSE stream. Each event represent the number of emojis that appeared on Twitter since last event. For example {"1F604":1,"267B":2} means "😄" once and "♻" twice. We would like to read this stream in Java efficiently and make something useful out of it. Well, maybe not useful (it's emojis after all) but at least fun. Consuming SSE stream with WebClient is pretty simple:

There's no JSON parsing, Spring does it's magic for us! At this point we have a stream of Map<String, Integer> instances, not raw ServerSentEvent classes. Two caveats. First of all we need flatMap(e -> Mono.justOrEmpty(e.data())) rather than just a simple map(ServerSentEvent::data) because ServerSentEvent.data() sometimes returns null. Secondly .map(x -> (Map<String, Integer>)x) needs to be used as opposed to simple .cast(Map.class) because of type erasure. Alright, our stream is a bit too complex right now. Rather than having three-dimensional data (event contains map, map contains entries, entries contain count) we'd like to have a single event per each emoji appearence. Easy!

With just few lines of code we transformed one event: {"1F604":1,"267B":2} into three: "1F604", "267B", "267B". I was feeling a bit guilty at this point, reverse-engineering the emojitracker.com. Then I discovered that the source code of the website is on GitHub and the API is documented. Moreover, there is already an endpoint that sends individual emojis, as opposed to aggregated JSON maps: