Explore Website

Accessing Bitcoin's ZeroMQ interface

Introduction

In this tutorial, we will be taking a closer look at bitcoin's ZeroMQ messaging interface. This interface is useful for developing applications which might require data related to block and transaction events from a Bitcoin core node. Some applications which include block explorers, wallets and reporting dash boards to name just a few.

Background

ZeroMQ is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.

The Bitcoin Core daemon can be configured to act as a trusted "border router", implementing the [bitcoin wire protocol](/bitcoin wire protocol/) and relay, making consensus decisions, maintaining the local blockchain database, broadcasting locally generated transactions into the network, and providing a queryable RPC interface to interact on a polled basis for requesting blockchain related data. However, there exists only a limited service to notify external software of events like the arrival of new blocks or transactions.

The ZeroMQ facility implements a notification interface through a set of specific notifiers. Currently there are notifiers that publish blocks and transactions. This read-only facility requires only the connection of a corresponding ZeroMQ subscriber port in receiving software; it is not authenticated nor is there any two-way protocol involvement. Therefore, subscribers should validate the received data since it may be out of date, incomplete or even invalid.

ZeroMQ sockets are self-connecting and self-healing; that is, connections made between two endpoints will be automatically restored after an outage, and either end may be freely started or stopped in any order.

Because ZeroMQ is message oriented, subscribers receive transactions and blocks all-at-once and do not need to implement any sort of buffering or reassembly.

What you'll need

You will need access to a bitcoin node. We suggest executed against a node configured in regtest mode so that we can have the freedom of playing with various scenarios without having to loose real money. You can however execute these against either the testnet or mainnet configurations.

Note:
If you don't currently have access to a bitcoin development environment set up, dont' worry, we have your back! We've setup a web based mechanism which provisions your very own private session that includes these tools and comes preconfigured with a bitcoin node in regtest mode. https://bitcoindev.network/bitcoin-cli-sandbox/
Alternatively, we have also provided a simple docker container configured in regtest mode that you can install for testing purposes.

By default, the ZeroMQ feature is automatically compiled in if the necessary prerequisites are found. To disable, use --disable-zmq during the configure step of building bitcoind:

gr0kchain bitcoindev $ ./configure --disable-zmq ....

Let's get started

Before we dive into things, let's have a brief taste of what we can expect to learn from this tutorial.

In the use case demonstrated above, we have executed a script which listens for one of four configurable events from our Bitcoin Core node. The script prints out the transaction identifiers as they are received by our node.

Bitcoin's ZMQ uses a publish/subscribe (pubsub) design pattern, where the publisher in our case is our bitcoin node which published messages based on events. The subscriber on the other hand, includes any application looking to utilise it by subscribing to these events. There are currently 4 different publish (PUB) notification topics we can expose via bitcoind which notify us when ever a new block or transaction is validated by the node.

zmqpubhashtx : Publishes transaction hashes

zmqpubhashblock : Publishes block hashes

zmqpubrawblock : Publishes raw block information

zmqpubrawtx : Publishes raw transaction information

Enabling zmq

To enable these features, you will need to update your bitcoin.conf file. We will start by reviewing the --help flag from our bitcoind command.

Here we can see the four message types together with information on how to enable debugging for various components including zmq.

Note:
When developing, it is useful to always keep debug configured to 1 in your bitcoin.conf file.

Each of our PUB topics can be configured using an tcp socket on the configured address. This address is where Bitcoin Core will be publishing messages to the various topics a subscribed might be interested in.

We can call the getzmqnotifications remote procedure call (RPC) method to see if any of the zmq services are currently running.

Writing our client

There are many ways in which you can write clients that can connect to zmq servers. For simplicity, we will be using javascript, but feel free to search for libraries related to your language of choice.

Running this should result in what appears to be a terminal in a waiting state. Our script is waiting to be notified of incoming transactions. Since we're using a node configured in regtest mode, we need to manually generate some blocks in a separate terminal session.

Assuming all went well, you should have received output which indicates that you have successfully subscribed and received your first notification, good job!

The information returned however seems to be loaded into a Buffer for both our topic and message. Each of the pub topics will return a binary representation of the data we have subscribed to whenever the node fires an event. We can decode this by updating our message handler as follows.

Note
We will cover transactions in more detail in future tutorials, for more information on these, checkout Raw Transaction Format. Some useful libraries to decode these on the client include bitcoin-js or bcoin. There are many more, so search for libraries which suit your needs.

The Bitcoin Core RPC interface exposes a useful method called decoderawtransacton which we can used for decoding our raw transaction hex string by passing it as the first parameter.

Here we are calling the decodeRawTransaction method and passing it the hex encoded string of our rawtx message we received from the zmq interface. We then proceed by printing out the message by encoding it using JSON.stringify with some additional parameters for making it legible on screen.

Note
Ensure to update the config parameters to match your local environment.

Happy hacking fellow bitcoiner!

Conclusion

In this tutorial, we had a closer look at the zmq interface exposed by a Bitcoin Core node. We configured our node to expose 4 kinds of messages including, hashblock, hashtx, rawblock and rawtx. We subscribed to these and processed the raw data on receipt of new blocks and transactions.

Explore Website

About Bitcoin Developer Network

The Bitcoin Developer Network (BDN) is a community-driven project looking at educating the generation of Bitcoin developers. We invite developers, authors, editors, proofreaders, enthusiasts, subject matter experts and technical writers to get in touch so that we may reduce the barrier to entry in developing solutions at various layers of this technology.

Bitcoin Developer Network Newsletter

Enter your email address to subscribe to this blog and receive notifications of new posts by email.