After investigating a performance analysis by somebody from Intel we learned that there is still a significant overhead per each IPC call. In the analysis the person made miner-fs combine multiple insert transactions together and then send it over as a single big transaction. This was noticeably faster than making many individual IPC requests.

The problem with this is that if one of the many insert queries fail, they all fail: not good.

We’re now experimenting with a private API that allows you to pass n individual insert transactions, and get n errors back, using one IPC call.

@Alexander Larsson: One reason is because (in this API) each query is independent of the other queries. Query 7 and 8 should not stop when query 6 has an error. (In this API) each ‘query’ is a transaction by itself (a ‘query’ can with whitespace as delimiter have multiple INSERTs). It’s wrong when (with this API) somebody would make a query 7 depend on a query 6. He should merge query 6 and 7 together with a space as 6, instead (that’s a transaction).

Your proposal is of course still possible if we allow to just continue sending queries (transactions, if you prefer that word), and then at the end return index plus error of all that got fed. But we’d still need to return a array of index+error at the commit API, just like the current API (there would be no benefit, except that now you need to collect all queries and pass them as a big array, whereas when the API would be split perhaps client developers could have less memory use … hmm).

Returning the errors per query immediately would require executing the query. That would mean that it’s again blocking per query (not good).

Anyway, yes, the API isn’t final and might change. It’s not intended to really be public either atm.

We don’t think that shared memory with message queues is a good idea for data that is short lived, like queries. But you are welcome to try to adapt Adrien’s test software to also include the IPC-mechanism that you have in mind, then publicize your findings. Also note that a query result (or the data to insert) can be large; a shared-memory with message-queues solution would require us to buffer all that in memory (per message) until the other side has received it. With FD passing you can elegantly stream it over, which allows you to keep memory consumption low and horizontal (= important).