Architecture: Onsite Notification Service

16 April 2017 2 min read

We want to make sure every Bukalapak user received the message they need. So, we develop a notification platform. It’s similar to Facebook notification, personalized notification for every user. This is the onsite notification.

Requirements

Support multiple types of notification

User personalized notification

Accommodate 10 million Bukalapak users

Independent service

Architecture: bit by bit

Database

MySQL. Simple reason, most of our engineers had the most experience in using MySQL. This will alleviate the scaling pain in the long run. We considered other bleeding edge databases like MongoDB, Cassandra, and RethinkDB, but for the sake of stability and maintainability, we chose to be conservative this time.

Database Design

Only 1 table for this service. These are the fields.

No.

Field

Type

Description

1

id

SERIAL

your standard row identification

2

user_id

INT

reference to user, indexed

3

type

VARCHAR(10)

to differentiate kind of notifications, indexed

4

data

TEXT

notification payload

5

read_at

DATETIME

time of read, indexed

6

created_at

DATETIME

standard timestamp, indexed, just in case

We chose to as denormalize as possible to improve performance and easier future maintenance. We considered splitting into 2 tables to reduce redundancy but joining table reducing performance by quite a number. If in the future we want to scale horizontally by parting the table, one table is easier to split. We used SERIAL() for id field to accommodate more than billions of notifications. Using standard `INT` will make us reach the ceiling in a short time. In fact, we have reached 1/3 road to that 2 billion integer ceiling after 4 months of the release. We need new SSD.

data field will be JSON serialized string. The rest of the field are indexed except the data field as we swore not to query based on the data field.

Application

We want the notification platform to be independent in order to reduce the load of our main monolithic service and make teams move swifter. Yes, this is a step into micro-services architecture.

On to the application, we used Rails because this starts as an experimental feature, we want to develop the app rapidly. Secondly, we tried other lower level language like Go, but after some calculation, we believe choosing Go will not give us a significant performance boost. After all, this application is an API only application with no templating, no authentication, and no complex logic, simple insertion and simple read based on user_id. But, of course, if we need more performance and efficiency, we would happy to try other options. Rust, maybe, since we have rust prophet in our engineering team.

Background Worker

To make other services could insert notification, we use RabbitMQ as the queue service. Thus, we only need to implement worker(background application) that listen to the queue. We decided not to make API for inserting notification because by using queue service, we are not worried to be overwhelmed by tremendous amount of request. If the request is greater than our throughput, the rest of the notification will safely be stored in the queue and will be consumed eventually. Like every good things in this life, it’s best consumed with moderation.

While on the other hand if the same case happens with API, either we need to scale up the application server or the performance will degrade. Though we will implement insertion API for other services that need synchronize request.

Result

All API endpoint got less than 20 ms in average response time. Moreover, the most used endpoint, the one that counts the unread notification, is averaging in 3 ms. We clearly did not expect to be this fast, since ruby is only “fast enough”! The service store about 40 GiB worth of notifications every month and there is no sign of slowing the performance. The only time that slows down the response time is when we insert a lot of(millions) notifications in short time. Phew.

This notification platform is only available in desktop, for now. The mobile web, Android, and iOS will be released in the near future. Let us know if you have any insight!