Hi, I like the idea of your library and I’m currently working on a couple of data pipelines I want to tackle with Clojure. I was wondering whether your library could help me, but I am a bit worried about performance since we will have to retrieve millions of rows from at least 7-8 tables, perform joins, filtering, and such.

Have you already used it for something similar?

If I try it, are you willing to get some feedback about it and in case it’s needed some help?

hi, thanks for your interest.
Walkable is quit new and I’ve been focus on designing api and implementing feature. Performance was not taken into account, even async api is not done yet. Joins are done in memory.

Walkable’s main aim was to produce the whole data for a client app to consume in one go, hence the data should not be too big.

For your case, your concern about performance is reasonable. My guess is that you need to write (or generate, walkable queries are plain data structure ) many small queries with pagination (offset, limit) then throw the results to a core.async channel, a message queue or onyx.

If you think walkable still brings some value to your task, we can discuss on #walkable slack channel or via email. PM me in that case.