Starting Multichain 2.0.3 results in Segmentation fault

Question

Hi

We have started running multichain 2.0.3 images within our kubernetes cluster (we have been running 1.x nodes for a couple of years now), and our new instances are crashing upon startup with the message below.

 

MultiChain 2.0.3 Daemon (Community Edition, latest protocol 20011)

Other nodes can connect to this node using:
multichaind asset-chain@172.16.42.75:6489

Node ready.

Segmentation fault (core dumped)

 

Do you know what would cause this error and where we should be looking for a root cause?

 

Thanks

 

Marty

Answer 1

Hi, the place to start is to run MultiChain with some more debug options, then take a look at the end of the debug.log file. Here are the command-line options to include:

-debug=mchn -debug=mcblock -debug=mcapi -debug=mcminer

Also, a couple of questions:

1. Are these 2.0.3 instances starting up with a blockchain directory created by a 1.0.x node, or are they fresh nodes that are creating their own blockchain directory?
2. What happens if you run the node (with the same blockchain directory) on a bare metal Linux operating system instead of in Kubernetes? This will help isolate the problem.

Comments

Did as you said, and here's the last part of the debug log which ends with the same error as above...

2019-11-17 15:17:48 mchn-miner: Running MultiChainMiner with 1 transactions in block (177 bytes)
2019-11-17 15:17:48 MultiChainMiner: Block Found - 00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d, prev: 009b602700fc408dbcec6353cde10a08cc75722a156a380681e898e9f0b096a7, height: 21, txs: 1
2019-11-17 15:17:48 mchn-miner: generated 0.00
2019-11-17 15:17:48 VerifyBlockMiner: Block: 21, Fork: 20, Chain: 20
2019-11-17 15:17:48 VerifyBlockMiner: Rolled back to block 20
2019-11-17 15:17:48 VerifyBlockMiner: Verifying block 21
2019-11-17 15:17:48 VerifyBlockMiner: New block 00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d (height 21)
2019-11-17 15:17:48 VerifyBlockMiner: Restoring chain, block 21
2019-11-17 15:17:48 VerifyBlockMiner: Restored on block 20
2019-11-17 15:17:48 mchn-block: New block index:   00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d, prev: 009b602700fc408dbcec6353cde10a08cc75722a156a380681e898e9f0b096a7, height: 21, mined-by-me: 21, can-mine: 256
2019-11-17 15:17:48 mchn: Connecting block 00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d (height 21) ...
2019-11-17 15:17:48 mchn: Checking block signature and miner permissions...
2019-11-17 15:17:48 mchn: Checking Block with 1 transactions
2019-11-17 15:17:48 mchn: Committing permission changes for block 21...
2019-11-17 15:17:48 UpdateTip:            new best=00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d  height=21  log2_work=12.459432  tx=35  date=2019-11-17 15:17:48 progress=1.000000  cache=0
2019-11-17 15:17:48 mchn-block: height: 21, miner: 1MiE71vmD9zmbFU3RGRM9inMAXu21QE1zj
2019-11-17 15:17:48 mchn: Replaying memory pool (0 new transactions, total 0)
2019-11-17 15:17:48 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-11-17 15:17:48 mchn: Wallet optimization after block
2019-11-17 15:17:48 mchn: Wallet optimization
2019-11-17 15:17:48 mchn: Asset Grouping. Assets: 1. Group Size: 1. Group Count: 1. Single-Asset Groups: 0.
2019-11-17 15:17:48 mchn: Asset Grouping. Assets: 1. Group Size: 1. Group Count: 1. Single-Asset Groups: 0.
2019-11-17 15:17:48 mchn-block: Chain activated:   00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d (height 21), can-mine: 256
2019-11-17 15:17:48 mchn-miner: Block successfully processed
2019-11-17 15:17:48 mchn-miner: delay:   12.137s, miner: 1MiE71vmD9zmbFU3RGRM9inMAXu21QE1zj, height: 21, gap:   11.153s, miners: (tot: 3, max: 3, pool: 1) In Pool Old
2019-11-17 15:18:01 mchn-miner: Running MultiChainMiner with 1 transactions in block (177 bytes)
2019-11-17 15:18:01 MultiChainMiner: Block Found - 00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb, prev: 00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d, height: 22, txs: 1
2019-11-17 15:18:01 mchn-miner: generated 0.00
2019-11-17 15:18:01 VerifyBlockMiner: Block: 22, Fork: 21, Chain: 21
2019-11-17 15:18:01 VerifyBlockMiner: Rolled back to block 21
2019-11-17 15:18:01 VerifyBlockMiner: Verifying block 22
2019-11-17 15:18:01 VerifyBlockMiner: New block 00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb (height 22)
2019-11-17 15:18:02 VerifyBlockMiner: Restoring chain, block 22
2019-11-17 15:18:02 VerifyBlockMiner: Restored on block 21
2019-11-17 15:18:02 mchn-block: New block index:   00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb, prev: 00f62e2d50c6e69e97dee136d6eccfeed762387752a356023b8e95fa749f561d, height: 22, mined-by-me: 22, can-mine: 256
2019-11-17 15:18:02 mchn: Connecting block 00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb (height 22) ...
2019-11-17 15:18:02 mchn: Checking block signature and miner permissions...
2019-11-17 15:18:02 mchn: Checking Block with 1 transactions
2019-11-17 15:18:02 mchn: Committing permission changes for block 22...
2019-11-17 15:18:02 UpdateTip:            new best=00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb  height=22  log2_work=12.523562  tx=36  date=2019-11-17 15:18:01 progress=1.000000  cache=0
2019-11-17 15:18:02 mchn-block: height: 22, miner: 1MiE71vmD9zmbFU3RGRM9inMAXu21QE1zj
2019-11-17 15:18:02 mchn: Replaying memory pool (0 new transactions, total 0)
2019-11-17 15:18:02 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-11-17 15:18:02 mchn: Wallet optimization after block
2019-11-17 15:18:02 mchn: Wallet optimization
2019-11-17 15:18:02 mchn: Asset Grouping. Assets: 1. Group Size: 1. Group Count: 1. Single-Asset Groups: 0.
2019-11-17 15:18:02 mchn: Asset Grouping. Assets: 1. Group Size: 1. Group Count: 1. Single-Asset Groups: 0.
2019-11-17 15:18:02 mchn-block: Chain activated:   00a371a97ca9ea8fdbc360f24fade59f92431aefe598cf0e192d6dd0b0ca16bb (height 22), can-mine: 256
2019-11-17 15:18:02 mchn-miner: Block successfully processed
2019-11-17 15:18:02 mchn-miner: delay:   11.556s, miner: 1MiE71vmD9zmbFU3RGRM9inMAXu21QE1zj, height: 22, gap:   11.374s, miners: (tot: 3, max: 3, pool: 1) In Pool Old


To answer your questions:

1. These are fresh 2.0.3 nodes
2. I didn't run the nodes bare metal as we don't do that and it would be a nightmare to try and replicate the container environment, but I did run the docker container locally which seems to run OK which is weird.  I need to check if the helm chart changes any runtime parameters that the docker-compose does not, but barring that, the only difference then would be the data directories, which on kubernetes are Azure Managed Disks which get attached via Persistent Volume Claims.  We have been running this setup in a production environment for over a year using multichain 1.0.6

---

OK, please let us know what you find. If it still doesn't help we should take this offline because there are a lot of things we will need to suggest trying, and you may prefer to give us access to an example setup instead.

---

I've tried to strip it right back to practically a default install and it still crashes.  Happy to take this offline (we have paid for support so I'll pick up this on email)

Thanks

Marty