Moving on from big blockchains

What we gain in flexibility by losing proof of work

When it comes to using blockchains for inter-enterprise coordination, there’s an elephant-sized problem in the room. In my view, nobody’s talking about this issue enough, whether due to denial or the need to keep the hype going. The problem, in a nutshell, is confidentiality.

To recap what I’ve explained previously, a blockchain allows a database to be shared between entities who do not fully trust each other, without requiring a central administrator. Instead, a blockchain-based database is based on a set of “nodes” which are owned by the participating entities. Nodes send transactions to each other in a peer-to-peer fashion, with each node independently verifying each transaction. Groups of transactions are then confirmed in “blocks” created by special nodes called “miners”. These blocks link to form a “blockchain” which acts as a unified transaction log, ensuring that all nodes reach consensus on the database’s state.

At this point, blockchains are a proven technology, both in public cryptocurrencies like bitcoin and their private equivalents. But they still suffer from a fundamental problem. Putting aside advanced cryptography (for now), blockchains reveal the content of every transaction to every participant. Why? Because in order to verify a transaction, every node has to see that transaction. This makes blockchains fundamentally different from centralized databases, in which transactions are only visible to their creators and the database administrator.

So if you’re considering a blockchain for a project, you should bear this simple principle in mind:

Blockchains are for shared databases in which everyone sees what everyone else is doing.

To be clear, seeing what someone is doing doesn’t necessarily mean that you know who is doing it. Blockchains represent identity using meaningless alphanumeric “addresses”, and most participants need not know to whom these belong. Nevertheless, a lot can be learned by analyzing the behavior of an address, and especially from how it transacts with other addresses. In formal terms, this means that blockchains provide pseudonymity rather than anonymity, because identities persist over time. In the case of bitcoin, several companies are already selling services which mine the “transaction graph” to reveal information about the owners of bitcoin addresses.

The bottom line is that blockchains are best suited for shared databases which are write-controlled but read-uncontrolled. Or, to put it more poetically, blockchains are transparency machines.

The economics of mining

Blockchains began with bitcoin – a digital, decentralized and censorship-proof form of money. One of bitcoin’s key design goals was allowing anybody to “mine” a block which confirms transactions, to prevent governments or banks controlling who can pay whom. In theory, open mining sounds democratic, but on its own it leads to dictatorship by stealth. Why? Because on the Internet it’s possible for one entity to use many different identities, a problem known as the Sybil attack. This means that someone could seize control of block mining, deciding unilaterally which transactions get confirmed, without anyone else even knowing it happened.

Bitcoin cleverly resolves this problem through proof of work. Bitcoin mining may be open, but it is also extremely difficult. In order to create a block, a miner must win a global race to solve a pointless and tricky computational problem, which consumes a lot of electricity (and therefore money). These days mining is performed by specially optimized hardware, but this doesn’t make it any cheaper, because the network regularly adjusts the problem’s difficulty to maintain a steady rate of 1 block every 10 minutes. This makes it hard for any single actor to seize control of the chain and, so far at least, the scheme has worked.

In exchange for the hard work and expense, the winning miner receives a reward, currently 25 newly-minted bitcoins per block (to halve during 2016). Miners also receive a little extra from the fees attached to transactions, although for now these play a minor role. And here are some shocking numbers: During 2015, bitcoin miners raked in $375 million in rewards and fees, in exchange for confirming 45 million transactions. That comes out to over $8 per transaction, even ignoring the fact that many of these weren’t genuine transfers of funds.

Who on earth is paying for all this? The answer is: bitcoin investors. For the most part, miners exchange their new bitcoins for regular currencies like dollars and yuan, because they need this money to pay for mining hardware and electricity. And what will happen if the investors stop coming? Well, the bitcoin price will crash, as miners are forced to dump their bitcoins at a significant loss. Indeed, looking at bitcoin’s price history, there have been several periods during which the price drifted gradually and undramatically downwards, because of the constant supply of bitcoins to be sold.

In the meantime, as the first and most prominent blockchain, bitcoin continues to attract an impressive level of incoming investment. Clearly there’s room for just a handful of such high-profile public blockchains, because the economics of open mining leads inevitably to consolidation. Any new blockchain secured by a low quantity of mining power will not be attractive to end users, because of its inherent insecurity. This will keep its currency value low, which will prevent it attracting additional miners. In other words, the virtuous circle underlying bitcoin’s explosive growth will be hard to repeat. In my view, the only likely exceptions will be newcomers such as Ethereum and Dash which offer a step change in terms of functionality. (I’m ignoring so-called merged mining as well as ideas like proof of stake, because they have not yet been proven to work at scale.)

As luck would have it, private blockchains avoid all of this trouble. Instead of open mining, private chains rely on a whitelist of permitted miners, with all blocks signed digitally by their miner of origin. This is combined with some form of distributed consensus scheme which prevents a small group of these miners from monopolizing the process. If you like, it’s democracy for the privileged, rather than democracy for all. Since private blockchains have no need for proof of work to enforce diversity, they also don’t have to incentivize miners with a financial reward. Instead, a private blockchain costs no more to run than a regular replicated database. The reward is simply the immediate and sufficient benefit of being able to make use of the chain.

With the economics of open mining out of the way, a universe of possibilities opens up. One organization can participate in thousands of blockchains, just like it accesses thousands of (internal or external) databases today. And globally there can be millions (or billions) of blockchains, all serving different purposes and sets of users. But if the world will be filled by so many blockchains, it’s safe to assume that each of them is going to be small.

From monolithic to small blockchains

What do I specifically mean by a “small blockchain”? I mean a blockchain whose scope is restricted to a narrow and specific purpose. This is the polar opposite of catch-all public blockchains like bitcoin and Ethereum, or even the permissioned global bank blockchain that some think is in the offing. It is, in fact, rather more like a regular database, but with a different model of sharing and trust.

Of course, there are many ways in which a blockchain’s scope can be restricted, so I’ll focus here on three simple examples: (a) per-order blockchains, (b) bilateral blockchains, and (c) notarization by hash.

Per-order blockchains

Let’s imagine a blockchain designed to manage the lifecycle of a single container of branded goods, manufactured in China and sold in the US. There can be a bewildering number of parties involved in this process, such as a retailer, agent, distributor, importer, shipping company, manufacturer, licensor and designer, as well as multiple subcontractors, shipping ports, banks, customs agencies and tax authorities. A large amount of information has to flow back and forth between these parties, leading to bureaucratic delays, errors and expenses. In theory, all this could be streamlined using a centralized database, but the question is: who will run it? Considering the gap in geography, culture and legal systems, it may not be easy to find someone that all the parties can trust.

Now, much has already been said about how blockchains can simplify coordination in supply chains. A blockchain can be used to record important documentation, digitally signed as appropriate, as well as enable the transfer of digital equivalents of key assets such as a bill of lading or letter of credit. However, putting all of this data on a monolithic blockchain can leak confidential information. For example, if two competing manufacturers use the same shipping company and bank, they could learn a lot about each other’s activities from transactions which involve those counterparties but are not their own.

One solution is to keep all the information relating to a single order in a blockchain which is dedicated just to that order. In this case, the confidentiality problem is much diminished. For example, two competing manufacturers will never participate in the same chain. At the beginning of the process, a new private blockchain can be set up, and connected to by all the participants. This blockchain makes the state of the order visible to all users in real time. And once it is safely delivered and paid for, the order’s blockchain can be decommissioned and archived away, only to be reopened in case of a dispute.

One issue with per-order blockchains is identity management. When using a blockchain for inter-enterprise coordination, each participant needs to know the real-world identity behind many of the other addresses used on the chain. Obtaining this mapping securely is a potentially inconvenient process, involving either direct exchange of information (by fax?) or a trusted administrator who provides it. But the good news is that there’s no need for this process to take place every time a new blockchain is set up. Instead, participants can have the same address on all the chains that they use. Alternatively, a separate long-running blockchain could be used purely for identity management, allowing each entity to securely distribute its address for each new chain.

Bilateral blockchains

Now let’s consider a blockchain which is used for the rapid settlement of exchanges of financial assets, such as government-backed currencies. This chain would involve at least three types of participants: (a) the trading parties which are performing the transactions, (b) the custodial bank which holds the currencies and issues on-chain tokens to represent them, (c) regulators and/or auditors who receive a read-only view of the activity taking place.

This is a perfectly natural application of blockchains, and already supported in full by off-the-shelf platforms such as MultiChain (our own). But again, the problem of confidentiality rears it head. If the trading parties are locked in intense competition, they can watch each other in order to infer:

• How much of each currency is held by each trader.
• Which currencies they actively trade in, with what frequency and quantity.
• Who else they trade with on the blockchain, and at what prices.

Even if we assume that the parties are not told who is using which address (or multiple addresses), it won’t take them long to work it out. Fierce competitors in a marketplace tend to know a lot about each other, and this prior knowledge can be correlated with patterns of blockchain transactions in order to learn more. For many financial use cases, the risk of this leakage is simply a deal-killer, because the efficiency gained is outweighed by the confidentiality lost.

Nonetheless blockchains can still provide some assistance in this scenario – namely, to record the flow of transactions and messages across each bilateral communications channel between trader and custodian. By combining signed transactions with signed commitments, the blockchain provides realtime reconciliation across this channel, ensuring there is no way in which the parties can differ over what was done and when. In addition, regulators and/or auditors could be granted read-only access to many or all of these pairwise blockchains, giving them a comprehensive view of the activity in a particular marketplace, without needing to explicitly request data from its participants.

Notarization by hash

As I hope is now clear, blockchains can be used to digitally sign, store and timestamp any important data, including text, documents, images and database entries. So long as the blockchain’s miners do not collude maliciously, the chain becomes an irreversible and incontrovertible audit trail for all of the information within. For example, all of the emails sent between the members of a group could be recorded on a blockchain, with each message signed by both the sender and receiver.

But once again we come up against the problem of confidentiality. In many cases, the two parties to a correspondence will not want its content to be visible to anyone else. Their sole purpose in using the blockchain is to prevent future disputes, so that they cannot disagree over what was said, by whom and when.

In this case the solution is simple. Instead of storing the full text of the messages within the blockchain, a “hash” (or digital fingerprint) of their content is embedded instead. A hash is based on a one-way function, which means a function whose output is easy to compute for a given input, but which is practically impossible to reverse. By collaboratively embedding and signing the hash of a message’s content in a blockchain, the parties are able to “lock down” that content in an auditable way, without revealing it to the other participants.

In parallel to embedding this hash, both correspondents store the full message content on their own systems. If a dispute arises in future, either party can reveal this content to an independent party, who can calculate its hash and confirm that this matches the hash on the chain. If so, there is no denying the correspondence that took place. Indeed, this same principle is already applied by many services to notarize documents on the public bitcoin blockchain. Doing so on private blockchains gives greater scalability, lower transaction costs, and hides the entire process from the outside world.

Zero knowledge proofs

So there we have it – three examples of how blockchains can be used, given the limitations posed by radical transparency. But before I finish, it’s important to mention some emerging cryptographic techniques. Sporting names like homomorphic encryption and zero-knowledge proofs, these promise to untie the gordian confidentiality knot. In the context of a blockchain, they offer a seemingly impossible separation of visibility and verification. A partially encrypted transaction can be embedded in a blockchain, along with a proof of its validity, without revealing the transaction’s contents. Every participant can then verify the proof, while still only seeing the transaction in encrypted form. And the unencrypted version is revealed on a need-to-know basis, presumably only to the transaction’s recipient.

Although there has been some real progress in this space, these technologies are yet to mature. It’s still not computationally feasible to generate and verify a proof regarding the validity of a blockchain transaction while keeping its contents fully private. Nonetheless let’s assume that, at some point in the future, this technical problem is solved. I still think we might have a psychological one. You see, in the current way of doing things, a CIO knows that her employer’s confidential data is protected by physical and organizational barriers. Data can only escape if someone is grossly negligent or deliberately commits a crime. But when it comes to advanced cryptography, the picture is rather different, with the CIO relying on advanced mathematics and the soundness of random number generators.

So even when the technology problem is solved, I think it could still take a long time to overcome the emotional barrier. In the meantime, where does this leave us? With the stark assumption that every participant in a blockchain sees everything else that is going on. While this assumption might restrict the sphere of feasible applications, it will also prevent time being wasted on projects that will never be moved to production. And as others have said before me, 2016 is the year to transition from thinking and talking about blockchains, into building some real applications.

Please post any comments on LinkedIn.