Redis module in Kotlin/Native

Posted on 2020-03-11
15 mins to read

Those of you who have some experience with Redis may know that it is not just a simple cache or a plain key-value store, but actually a data structures server, supporting different kinds of values.
Out of the box it supports binary-safe strings, sets, lists, hashes, bit arrays, streams and HyperLogLogs.
Redis also provides a simple C API for custom data structures, called native types.
Some of the popular community-supported native types are Bloom filters, graphs, JSON objects, and tensors.

Let’s use Kotlin/Native to implement a simple data structure for parentheses expression validation just because we can.
But first, I want to say thanks to Artyom Degtyarev from JetBrains, who helped a lot with Kotlin/Native in Kotlin Slack.

Redis 101

The best way to start with Redis, is, probably, The Little Redis Book by Karl Seguin.
It is absolutely free and takes about an hour to read (Karl mentioned in his blog that the book was written in only two days).

But the bare minimum required to understand the rest of the article is that the idea of Redis is storing different data structures and providing access to them by key:

Redis persistence

Redis persistence is an advanced topic and not every regular Redis user needs to dig in it.
But as an author of a native type you need to understand it, as every native type needs to support the persistence.

Snapshotting is the simplest persistence mode.
It produces a point-in-time snapshot of the whole Redist dataset.
Snapshots can be taken with SAVE or BGSAVE commands, or configured to be taken periodically or after some predefined number of changes, whatever occurs first.
Snapshots produce a binary file called dump.rdb in Redis’s data directory.

AOF is more cunning: every time a change is performed, that operation is logged into the append-only file appendonly.aof in the data directory.
Operations are logged in the same format used by Redis, so the AOF can be just “replayed” to reconstruct the whole dataset.
The problem is that the AOF grows as changes are performed.
So Redis supports an interesting feature: it is able to rebuild the AOF in the background without downtime upon the execution of the BGREWRITEAOF command or periodically.
As a result of AOF rewriting it will contain the shortest sequence of commands needed to rebuild the current dataset in memory.

Redis modules

Redis modules make it possible to extend Redis functionality by implementing new functions and data structures.
Redis modules are dynamic libraries (.so files), that can be loaded into Redis at startup or using the MODULE LOAD command without downtime.
Redis exports its API for the module authors in the form of a single C header file called redismodule.h.

Parentheses expression validation problem

Every opening parenthesis should come before the corresponding closing parenthesis.

The classic approach to this problem is using a stack:

Declare an empty stack.

Traverse the expression from left to right.

Push every opening parenthesis on the top of the stack.

For every closing bracket, check the topmost stack element:

If it’s a matching bracket, simply drop both.

If it’s not a matching bracket, push the closing bracket on the top of the stack.

If the expression is valid,​ then the stack will be empty once the input string finishes.

One way to implement a stack is a singly linked list.
Singly linked lists contain nodes that have a data field and a pointer to the next node in line of nodes.
The last node will point to nothing, thereby marking the end of the list:

Let’s finally write some code.
Here is a Kotlin class for the single stack node for the parenthesis expression validation problem:

The implementation is neither perfect nor safe, but it’s just an example.
There is a test for the Brackets class on my GitLab, take a look.
Also, note that we used only pure Kotlin code here (for both the domain logic and the tests), without any platform-specific dependencies.
This code could be shared across JVM, JS and Native targets if needed, and that’s a cool feature of Kotlin Multiplatform!

C Interop

Before being able to interact with Redis via bindings to its C API, we need to configure a C interop with its redismodule.h.
As the whole Redis API is defined in that single header, let’s just copy it from their GitHub to the src/nativeInterop/cinterop/redismodule.h.
Next step is to define a src/nativeInterop/cinterop/redismodule.def file describing what things to include into the binding:

headers = redismodule.h
---
# Custom declarations

Here we simply want to create bindings for the contents of redismodule.h plus a few custom declarations.

Custom declarations

Redis relies heavily on macros in redismodule.h: all the API functions are exported using a macro REDISMODULE_API_FUNC.
This results in functions like RedisModule_CreateCommand, used to provide a callback for custom command, to be seen by Kotlin/Native as a nullable global variable:

RedisModuleWrapper_CreateCommand and Brackets_EmitAOF will be seen by Kotlin/Native as a regular functions.

Module initialization

Now, having the domain objects defined and the C interop configured the next thing to do is to actually create a Redis module.
Every Redis module needs to expose a RedisModule_OnLoad function.
Redis will call it upon loading the module, this is the place where you tell the Redis what your module is.
Let’s define it:

initRedisModule is a wrapper around RedisModule_Init, provided by Redis.
Its parameters include module context, module name, module version, and target Redis API version.
We’ll use "brackets.kn" as a module name and integer "1" as a module version, defined in a global constant BRACKETS_KN_VERSION.
REDISMODULE_APIVER_1 is provided by Redis in redismodule.h.

Exporting a native type

The implementation of some kind of new data structure and commands operating on the new data structure.
We’ve done the Redis-agnostic part in the Brackets class.

A set of callbacks that handle: RDB saving, RDB loading, AOF rewriting, releasing of a value associated with a key and some other, optional, events.

A 9 character name that is unique to each module native data type.

An encoding version used to persist into RDB files a module-specific data version so that a module will be able to load older representations from RDB files.

A very easy to understand but complete example of native type implementation is available inside the Redis distribution in the /modules/hellotype.c file.
Actually, our stack is the same singly linked list as in this file.

To register a new native type into the Redis core, the module needs to declare a global variable that will hold a reference to the data type.
The API to register the data type will return a data type reference that will be stored in the global variable.
That global variable will be used later to check the types of the values in commands operating on that native data type.

Calling the RedisModule_CreateDataType function via a wrapper to register a native type.
Returning false as a guard here results in module registration failure upper in the stack, in RedisModule_OnLoad.

Check the number of arguments.
brackets.kn.push is called with two arguments — a key and a bracket, so the total number of arguments will be three (the first one will be the command itself).
Calling RedisModule_WrongArity here will result in an error telling the user about the wrong number of arguments.

This actually should not happen, but…

Extracting the bracket character from the third argument (argv[2]).
memScoped is needed for alloc<ULongVar>, but that value is not used, it is only needed for the RedisModule_StringPtrLen call.

Validating the input.
Only brackets are allowed.

Opening the key for writing so that it is possible to call other APIs with the key handle as an argument to perform operations on the key.
Don’t forget to call RedisModule_CloseKey.
Yeah, better wrap that with try one day…

Querying the key type.
If there is no value associated with that key, REDISMODULE_KEYTYPE_EMPTY will be returned.

Fail with REDISMODULE_ERRORMSG_WRONGTYPE message if there is a value associated with that key and it is not empty or of our type.

Create a new Brackets value, push the bracket into it, and store the value in the dataset.
The value is wrapped in a StableRef so that Kotlin/Native runtime will maintain a stable address for it.
dispose must be called on that StableRef instance when it’s not needed anymore allowing Kotlin/Native’s GC to collect the object.

bracketsKnPrint and bracketsKnValid are similar to the bracketsKnPush: they open the key, check the type and call .toString() or .valid on the Brackets value.
I won’t provide the code here, as this article became really big.

Now, let’s take a look at the utility functions bracketsRdbLoad, bracketsRdbSave, bracketsAofRewrite and bracketsFree.
They have nothing to do with our problem, but they are required by Redis.

Utility functions

bracketsRdbLoad and bracketsRdbSave callbacks are required by Redis to support RDB persistence.
Developers are free to use any kind of encoding for their types.
The only limit is imagination and the set of available API functions:

Let’s use RedisModule_SaveStringBuffer / RedisModule_LoadStringBuffer to persist our stack as a simple string.
Redis will call bracketsRdbSave with a pointer to the RedisModuleIO structure, used for RBD operations, and a pointer to the memory location with our data.
As you saw in the previous section the values will be stored using Kotlin/Native’s StableRef, a class used to provide a way to create a stable handle to any Kotlin object.
So, in bracketsRdbSave we cast the value to StableRef<Brackets>, then, if it’s not empty, convert it to a string using Brackets#toString function, and save it.
memScoped is needed to obtain a short-lived pointer to the null-terminated string to pass to the RedisModule_SaveStringBuffer.
Note that this may be unsafe if RedisModule_SaveStringBuffer store that pointer for later use, but it seems to use it immediately, so we’re good.

In bracketsRdbLoad we’ll do the opposite: read the null-terminated string from the RDB file and recreate Brackets by pushing the brackets one by one.
The result is wrapped into a StableRef and the pointer returned.

Testing

You’ve already seen a few links to the source code for this article, but to be clear: madhead-playgrounds/redis on GitLab or madhead/kn-redis if you prefer GitHub.
Clone or fork, or just give it a star.
If you want to get your hands dirty, follow the instructions in the README, you’ll need Docker Compose.
I’ve tried to configure things so that you only need to build the code and start the container, the modules will be loaded automagically.

Let’s tail the logs of the Redis container in a separate console and see what happens upon the execution of some commands:

Let’s also check the AOF:

Seems good.
The dataset is recreated correctly after the restart with both RDB and AOF.

Congratulations, we’ve done!
Thank you for reading to the end of the article, I hope you found it informative.