MultiChain 1.0 beta 2 and 2.0 roadmap

Where we are today and where we’re going tomorrow

Today we’re delighted to release the second beta of MultiChain 1.0 for Linux, Windows and Mac (for now the Mac version requires compilation). This concludes the planned development of MultiChain 1.0 – with the exception of any bug fixes, the final release of MultiChain 1.0 over the summer will be unchanged.

This month also marks two years since the first alpha release of MultiChain in June 2015. As with any new product, we weren’t sure how the market would react, and knew there was only one way to find out – release a minimum viable product, meaning an initial version which provides significant value but is preliminary by design. Thankfully, unlike our first product CoinSpark, MultiChain received a strong and immediate positive response. This was accompanied by a tsunami of sensible feature requests, many of which we’ve now implemented. In parallel to the product’s development, usage has also grown remarkably by every measure. For example, the MultiChain website received under 3,000 visitors in July 2015, and now brings in ten times that number monthly.

MultiChain performance

Over the past two years we’ve invested a lot of effort in optimizing MultiChain, which was forked from Bitcoin Core, the reference implementation for the public bitcoin network. Below is a comparison of transaction throughput for a single-node setup using five versions of the product:

Average transactions per second, including API overhead and building, signing, mining and verifying transactions and blocks.
Tests performed using the ab HTTP server benchmarking tool sending two concurrent requests to the sendtoaddress API.
Server specifications: Intel Core i7-4770, 4 cores @ 3.4 MHz, 32 GB RAM, Seagate 2 TB 7200 RPM SATA, CentOS 6.4.

Naturally, the biggest jump came in alpha 22 when we transitioned to a database-driven wallet. But since that release, we’ve almost doubled MultiChain’s speed again. We hope we’ve demonstrated that bitcoin’s limit of 4 transactions per second is due to its particular network parameters, and has no relation to blockchains in general.

Of course, performance optimization is a never-ending task, and there’s no reason why MultiChain can’t reach 10,000 tx/sec on a 16-core processor with the appropriate architectural changes. However, based on conversations with our users and partners, it seems that few expect to need more than 1,000 tx/sec for the next few years. So we’re refocusing our development efforts on new features, which brings us nicely onto the subject of MultiChain 2.0.

MultiChain 2.0 overview

Version 2.0 of MultiChain will be the first to come in two editions – Community (open source) and Enterprise (commercial). I’m going to focus here on the free Community edition, since we’re only discussing the details of MultiChain Enterprise with our partners. In any event, the Community and Enterprise editions will be highly compatible, in that: (a) applications built on the Community edition will run without modification on MultiChain Enterprise, and (b) both editions will be able to connect and transact with each other on the same chain.

The three key areas of enhanced functionality in both editions of MultiChain 2.0 will be:

• Richer data model for streams, including JSON documents.
• Custom programmable transaction filters for on-chain validation.
• Seamless updating of a blockchain’s protocol and parameters.

Let’s turn to discuss each of these in detail.

Richer data model for streams

MultiChain streams were introduced in September 2016 and have proven extremely popular. As described in this post, streams provide a simple and natural abstraction for general purpose data storage, indexing and retrieval on a blockchain. A MultiChain blockchain can contain any number of named streams, each of which can either be open to all for writing, or writable only from certain addresses.

In MultiChain 1.0, each stream item has one or more publishers (who sign it), an optional key, a binary data payload up to 64 MB in size, and a timestamp (derived from the block in which it’s embedded). Each node can freely decide which streams to subscribe to, or can subscribe to all streams automatically. If a node is subscribed to a stream, it indexes that stream’s content in real time, allowing efficient retrieval by publisher, key, block, timestamp or position.

MultiChain 2.0 will enrich this streams functionality in a number of ways:

• JSON items. As well as binary data, stream items will support structured JSON objects, stored on the blockchain in an efficient serialization format such as UBJSON. Since the MultiChain API already uses JSON throughout, these JSON objects will be writable and readable in a natural and obvious way.
• Multiple keys. Stream items will support multiple keys, enabling a single piece of data to be indexed in multiple ways for retrieval using liststreamkeyitems. We’re constantly evaluating how much database functionality to include within MultiChain, and don’t expect to support indexing of the sub-elements within JSON stream items in version 2.0. Allowing multiple keys per stream item provides a reasonable workaround.
• Atomic writes of multiple items. MultiChain 1.0 allows a single transaction to write to multiple streams, but not to write multiple items to the same stream. MultiChain 2.0 will remove this restriction.
• JSON merging. Any ordered list of JSON objects can be naturally flattened or summarized to create a “merged” object. The merged object contains all the keys which appear in the individual objects, where the value corresponding to each key is taken from the last object in which that key appears. If you like, the merged object is the final state of a database row, whose columns are defined by the first object and extended or updated by later objects. MultiChain 2.0 will add APIs to easily and rapidly retrieve the merged object for the JSON items in a stream with a particular key or publisher.

These features are derived from common ways in which developers are currently using streams. In other words, we’re observing what many people are building on top of MultiChain at the application level, and bringing that functionality into MultiChain itself – a pattern that we intend to continue applying. Now that stream items will include type information, they can easily be extended in future to support other data formats such as XML, HDF5 and MIME-identified content. Not to mention the possibilities of transparent on-chain compression and encryption.

MultiChain 2.0 will also support JSON objects for raw transaction metadata (i.e. not stream items) as well as the metadata for asset issuance and stream creation events, instead of the text-only key/value pairs implemented in MultiChain 1.0. The listassets API will offer JSON merging across all of an asset’s issuance events, so that each issuance’s metadata can effectively update the asset’s final description.

Custom transaction filters

We’ve thought a lot about how to add custom programmable rules to MultiChain. While Ethereum’s “smart contract” paradigm is popular, it has a number of key shortcomings for high-throughput permissioned blockchains. First, smart contracts introduce a global dependency across the blockchain’s entire state, which drastically impairs concurrency and performance. Second, smart contracts cannot stop incorrect transactions from being embedded in a blockchain, but only prevent those transactions from updating the blockchain database’s state. While in the long term we expect an Ethereum-compatible virtual machine to be offered as a high-level abstraction within MultiChain, we don’t think it’s the right solution for low-level validation.

MultiChain 2.0 will introduce a different paradigm called transaction filters, which validate individual transactions without reference to any global state. We expect filters to be written in Javascript and executed within an embedded runtime engine such as v8, which is used in Google’s Chrome browser and the Node.js platform. Of course, we’ll need to ensure that filter code runs identically on every node in a blockchain, blocking any sources of non-determinism such as reading the time, using random numbers, accessing network or disk, or performing math operations that depend on the host server’s architecture. Creating a deterministic Javascript runtime environment is a challenge, but (without giving too much away) we believe it will be useful for several other MultiChain features in future.

Filters will be passed a JSON object describing an individual transaction, structured like the output of decoderawtransaction but with extra fields. For example, each transaction input in the JSON will include a structure describing the previous transaction output it spends, and each address will be accompanied by a list of permissions currently held by that address. A filter’s job is to return a Boolean value indicating whether the transaction is acceptable and if not, provide a textual error explaining why. MultiChain’s API will include commands for creating filters, testing them on previous or new transactions, and activating them subject to administrator consensus.

Unlike smart contracts, if a bug is discovered in the code for a filter, it can easily be replaced by a new version. Nonetheless, like all Turing-complete code, filters still run the risk of entering an infinite loop. This problem will be mitigated in two ways:

• Filters can only be installed and activated by the chain’s administrators, subject to consensus. This gives each administrator the opportunity to examine a filter’s code in depth before voting for it to be activated.
• All well-behaved nodes will validate new transactions using the active filters before forwarding them on to their peer nodes. As a result, if a transaction sends a filter into an infinite loop, the transaction should not propagate beyond the node which created it.

We expect one popular application for filters to be validating stream items. For example, a filter could ensure that certain fields in a stream’s JSON items contain numbers in a specific range. In MultiChain 1.0 this type of validation has to be done at the application level, either when writing stream items (if the source is trusted) or when reading them. By contrast, MultiChain 2.0 will enable these rules to be embedded within the blockchain itself, rather like check constraints in a relational database.

MultiChain 2.0 will include two additional features to make filters even more powerful. First, it will introduce user-defined permissions, which exist alongside the eight permissions defined by MultiChain. As with regular permissions, these will be granted to specific addresses by administrators (and in some cases, by users with activate privileges) and included alongside addresses in the JSON object passed to a filter. For example, a filter could ensure that only addresses with a particular user-defined permission can write certain types of data to a stream, or transact in a particular asset above a certain threshold.

Second, MultiChain 2.0 will support custom (binary or JSON) metadata within regular transaction outputs. This will enable any output to act as a general database row, “owned” by the address within. Filters will see any metadata within a transaction’s spent and created outputs as part of its JSON description. As a result, MultiChain will become a universal shared database engine, where a transaction’s validity is determined by a customizable function of the rows it creates and deletes. (If this sounds a little abstract, we’ll be sure to provide some concrete examples.)

Blockchain updating

Since blockchains are designed to run for many years, their characteristics might need to be changed over time. The current version of MultiChain already provides a fair degree of flexibility, allowing permissions changes (including of administrators and miners by consensus), new assets and streams to be created, and nodes to be seamlessly added or removed from the network. Nonetheless, in MultiChain 1.0 a blockchain’s basic parameters, such as the maximum block size and target confirmation time, are fixed when the chain is created and cannot be subsequently changed.

MultiChain 2.0 will add the ability to update a blockchain, allowing many (but not all) of its parameters to be modified while the chain continues to run. Like other important operations, updating a blockchain will require a customizable level of administrator consensus, where this level itself is a parameter that can be changed. Updates will come into effect from a certain block, and apply thereafter to every subsequent block until the next update.

Blockchain parameters that can be updated will include:

• Protocol version. This will enable a blockchain created with one version of MultiChain to be upgraded to support the features in a new version, such as JSON stream items or transaction filters. Indeed, the protocol version 10008 introduced in MultiChain 1.0 alpha 29 (and used in the beta) has already been future-proofed with undocumented support for this type of upgrade. Once a MultiChain 1.0 blockchain is upgraded to the 2.0 protocol, it will also gain access to the other parameter changes described here.
• Blockchain scaling. Blockchains that become popular may outgrow the initial values set for their target confirmation time or maximum transaction and block sizes. MultiChain 2.0 will allow these values to be increased or decreased as necessary.
• Permissioning model. MultiChain 2.0 will allow the updating of many parameters relating to permissioning and governance, including: (a) anyone-can-* parameters that control the ways in which a blockchain is open or closed, (b) admin-consensus-* parameters that determine the levels of administrator consensus required for certain operations, and (c) the mining-diversity parameter that controls the strictness of the round-robin consensus algorithm.

Once this updating functionality is implemented, there should be no reason why a blockchain created in MultiChain cannot run for many decades or more.

Looking ahead

We’ve already started work on MultiChain 2.0, and look forwards to delivering on this roadmap. No doubt other enhancements will be included as well. As with MultiChain 1.0, we’ll have alpha releases along the way, so that developers can use and learn new features as they are implemented (and of course, report any problems or shortcomings). Naturally, we’ll continue to maintain version 1.0 throughout this period, fixing any bugs that appear.

I’d like to finish by thanking our development team, led by Dr Michael Rozantsev, for their continued excellence and hard work. We see MultiChain as a straightforwards software engineering project, in which code quality and testing counts above all. It’s my privilege to work with people who can turn a complex product vision into stable working software with such remarkable efficiency and speed.

Please post any comments on LinkedIn.