An important step forwards for performance and scalability
After two months of intensive development and testing, we’re proud to release the latest alpha of MultiChain, with a completely rewritten in-node wallet. This new wallet transforms the performance and scalability of creating, receiving and storing transactions in MultiChain.
Before we get into the details, let me provide some context. When we began developing MultiChain, we made the decision to use Bitcoin Core, the standard node for the public bitcoin network, as a starting point. In programming terms, this means that MultiChain is a “fork” of the bitcoin software. Our primary reasoning was that bitcoin was (and continues to be) the highest valued and most battle-tested cryptocurrency ecosystem, by quite some way.
On the plus side, this decision helped us get to market quickly, compared to coding up a blockchain node from scratch. Despite the many differences between public and private blockchains, they share a large amount of technical common ground, including the peer-to-peer protocol, transaction and block structure, digital signature creation and verification, consensus rules, key management, and the need for a node API. Forking from Bitcoin Core allowed us to leverage its maturity and focus on what MultiChain adds to blockchains – configurability, permissioning and native asset support. As a result, we were able to release the first alpha in June 2015, just 6 months after starting development.
However, alongside these benefits, we also had to accept the fact that some aspects of Bitcoin Core are poorly architected. While they work just fine at small scales, their performance degrades dramatically as usage grows. With the public bitcoin network still restricted to a few transactions per second, this won’t be an issue for most Bitcoin Core users for a long time. But with private blockchains aiming for hundreds or thousands of transactions per second, we knew that, sooner or later, these bottlenecks would need to be removed.
Bitcoin Core’s wallet
The “wallet” within Bitcoin Core was always the most crucial of these pain points. Its job is to store the transactions which are of particular relevance to the node, because they involve a blockchain address which it owns or a “watch-only” address whose activity it is tracking. For example, every transaction which sends funds to or from a node must be stored in that node’s wallet. And every time a node creates a transaction, it must search for one or more “unspent outputs” of previous wallet transactions which the new transaction will spend.
So what’s wrong with the wallet we inherited from Bitcoin Core? Actually, three things:
- All wallet transactions are held in memory. This causes slow startup times and rapidly increasing memory usage.
- Many operations perform an inefficient “full scan” of every transaction in the wallet, whether old or new.
- Every transaction in the wallet is stored in full, including any arbitrary “metadata” which has no meaning from the node’s perspective and is already stored in the blockchain on disk. This is very wasteful.
The consequence is that, with around 20,000 transactions stored, Bitcoin Core’s wallet slows down significantly. After 200,000 or so, it practically grinds to a halt. Even worse, since a MultiChain blockchain allows up to 8 MB of metadata per transaction (compared to bitcoin’s 80 bytes), the wallet’s memory requirements can balloon rapidly even with a small number of transactions.
It’s important to clarify that these shortcomings apply only to Bitcoin Core’s wallet, rather than its general transaction processing capacity. In other words, it can comfortably process and store millions (or even billions) of transactions which don’t relate to its own addresses, since these are held on disk rather than in memory. For example, many popular bitcoin exchanges and wallets use Bitcoin Core as-is, but store their own transactions externally rather than inside the node.
MultiChain’s new wallet
We could have made the same demand of MultiChain users, to store their own transactions outside of the node. However this didn’t feel like the right solution because it would greatly complicate the setup and maintenance for each of a chain’s participants. So instead, we bit the bullet and rewrote the wallet from the ground up.
How does the new wallet differ? If you have any experience with databases, the answers may be obvious:
- Rather than keeping the wallet transactions in memory, they are stored on disk in a suitable format, with transactions of interest retrieved when necessary.
- Instead of performing full wallet scans, the transactions are “indexed” in various ways to enable those which fulfill particular criteria to be rapidly located.
- Any piece of transaction metadata which is larger than 256 bytes is not stored in the wallet. Instead, the wallet contains a pointer to that metadata’s position in the blockchain itself.
In other words, we’ve rebuilt the in-node wallet to be properly database-driven (using LevelDB), rather than relying on a naïve in-memory structure that can’t be searched efficiently. Unsurprisingly, the difference (as measured on a 3.4 GHz Intel Core i7) is rather dramatic:
The graphs show that, once the old wallet contains 250,000 transactions, its send rate drops to 3 tx/sec and it adds 600 MB to the node’s memory usage. By contrast, the new wallet sustains over 100 tx/sec and only adds 90 MB. We stopped testing the old wallet at this point, but even with 6-8 million stored transactions, the new wallet continues to send over 100 tx/sec, and it tops out at around 250 MB of RAM used (due to database caching).
These tests were performed under realistic conditions, with multiple addresses and assets (and therefore many unspent transaction outputs) in the node’s wallet. In an idealized scenario (one address, one asset, few UTXOs), the sustained send rate was over 400 tx/s. Either way, as part of this rewrite, we have also properly abstracted all of the wallet’s functionality behind a clean internal interface. This will make it easy to support other database engines in future, for even greater robustness and speed.
To reiterate, all of these numbers refer to the rate at which a node can create, send and store transactions in its local wallet, rather than its throughput in terms of processing transactions created by others. For general network throughput, MultiChain can currently process 200 to 800 tx/sec, depending on the hardware it’s running on. (Be skeptical of any blockchain software promising numbers like 100,000 tx/sec on regular hardware, because the bottleneck is digital signature verification, which takes real time to perform. If nodes are not verifying individual transaction signatures, a blockchain cannot possibly be used across trust boundaries, making it no better than a regular distributed database.)
To finish, I’d like to mention the next major feature coming to MultiChain, which required this wallet rewrite. This feature, called streams, provides a high-level abstraction and API for general purpose data storage on a blockchain. You can think of a stream as a time-series or key-value database, with the added blockchain-related benefits of decentralization, digital signatures, timestamping and immutability. We know of many blockchain use cases that could use this functionality, and we’re already hard at work on building it. Watch this space.
Please post any comments on LinkedIn.
Technical addendum
Starting in MultiChain alpha 22, you can verify which version of the wallet is currently running by examining the walletdbversion field of the getinfo or getwalletinfo API calls. A value of 1 means the original Bitcoin Core wallet, and 2 means the new MultiChain wallet.
If you run the new version of MultiChain on an existing chain, it will not immediately switch to the new wallet. You can upgrade the wallet by stopping the node and then re-running multichaind with the parameters -walletdbversion=2 –rescan. You can downgrade similarly using –walletdbversion=1 –rescan.
By default, when you start a node on a new chain, it will automatically use the new wallet. You can change this by running multichaind for the first time with the parameter –walletdbversion=1.
With the new wallet, all MultiChain APIs work exactly the same way as before, with the exception of the old transaction querying APIs getreceivedbyaddress, listreceivedbyaddress and listtransactions (use listwallettransactions or listaddresstransactions instead). In addition, the new wallet does not support API calls and parameters relating to Bitcoin Core’s poorly implemented and soon-to-be-deprecated “accounts” mechanism, which was never properly supported by MultiChain. These calls are safely disabled with an error message.