Problems when running MultiChain

Question

Hi!

I'm conducting some tests with MultiChain on an Azure VM (Standard D8s v3, 8 cores, 32 GB RAM). It's a two-node setup with default blockchain configuration. I ran the following command against the first node:
ab -A <node1_credentials> -p payload.json -n 1000000 -c 1 -s 300 <node1_url>

Content of the payload file is:
{ "method": "sendasset", "params": [<node2_address>, "testasset", 1] }​

ab results usually ends with something like this:

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      2
  98%      2
  99%      2
 100%  14045 (longest request)

Is it to be expected that some of the requests take a very long time to complete? Why? In the above example, it took MultiChain more than 14 seconds to send a response! In some tests, ab even timed out after five minutes waiting.

To confirm this was an issue with MultiChain and not ab, I tried to send some commands (getmultibalances, getinfo, getpeerinfo...) to both nodes with multichain-cli and had the same results, sometimes it's instantaneous other times it takes a lot of time to get an answer.

Also, in my latest test runs, the second node started to throw the following exception (transfering 2,000,000 assets, one at a time):

************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

So I deleted my chain, created a new one, launched the test again and got the same error. Any idea what could cause it?

Unfortunately, I cannot run multichaind with the -debug flag because the log files fill up my disk before the test ends.

Thanks for your time!

Comments

I've made further testing to find out why some requests are taking a very long time to complete and I think I've found the culprit: https://github.com/MultiChain/multichain/blob/master/src/rpc/rpcserver.cpp#L1370

For some reasons, getting the lock may take a long time and the longer you run the test, the longer it will be to get the lock.

Using ab to send 1,000,000 requests to a node, the long lock time started at 1.5s and by the end of the test it grew up to 30s. As noted by the ab result, less than 1% of the queries suffer this issue.

My lack of understanding of MultiChain architecture and source code prevents me to go any further in my investigation but I hope those information will be helpful to you in some way.

Thanks for your time!

Answer 1

Thanks for the question. There are two separate things going on here:

The slow API response times are likely to be due to the time taken by the node in processing a new block, during which the node's state is in transition. You should take a look at the size of the blocks you are generating (the listblocks API is a good place to start) and consider if you might use a shorter block time, to reduce the size of this delay.

The second issue is an out-of-memory error. Each time a transaction is sent to the second node, this inflates the number of unspent transaction outputs which that node must track, and at some point it runs out of memory. The general solution to this problem is to use the UTXO autocombining features on the second node, but this is only possible if the address you're using on the second node has send and receive permissions. There are various runtime parameters which control this, and you can also manual initiate it using the combineunspent API.

Comments

Thanks for your answer. I've made the following change: target-block-time = 2. With this setting, a brief test shows that the occasional slow queries mostly stay below 3 seconds with the slowest at 11 seconds. That should be more manageable.

---

Great - thanks for the update. I presume the load you're testing under is higher than the real-world load you'll be handling.

---

We don't yet know what our real-world load will be but we expect the biggest blockchains on our platform to receive a couple billions stream items per year. The distribution most likely will not be uniform and to handle the load we are planning to use a message queue in front of MultiChain but the faster MultiChain will be the better as we would like to provide as near real-time as possible analysis to our customers.

---

2 billion stream items in one year = average of 63 items/second, which is well within MultiChain's capabilities.

You should also think about the size of your blockchain. Even for nodes that are not subscribed to the stream, and for small stream items, you can expect transaction sizes of around 250 bytes. So 2 billion items is around 465 GB/year. Although this assumes each stream item is a separate transaction, which doesn't have to be the case.