Multichain stopped unexpectedly

Question

I have 15 multichain nodes running on a single machine all are different blockchain nodes. Everything was working fine, one of the blockchains unexpectedly stopped

The debug.log ends up as

2020-06-16 05:44:55 MultiChainMiner: Block Found - 0059a074411d355dc8702bb0e420423bdc027e471f6c08cfc457f573007fec13, prev: 0083650ff5e6b6f0b824cfd2a0adbbb85ad77c4aca185948632f00744c967bf4, height: 2351, txs: 1
2020-06-16 05:44:55 UpdateTip:            new best=0059a074411d355dc8702bb0e420423bdc027e471f6c08cfc457f573007fec13  height=2351  log2_work=19.199672  tx=2573  date=2020-06-16 05:44:55 progress=1.000000  cache=0
2020-06-16 05:44:56 ResendWalletTransactions()
2020-06-16 05:44:58 UpdateTip:            new best=00a5cb02de84bff5ccd67470f12b4de69a1c4777db8e71a589a4ab553a692ca9  height=2352  log2_work=19.200286  tx=2574  date=2020-06-16 05:42:14 progress=1.000000  cache=0
2020-06-16 05:45:02 UpdateTip:            new best=00dda12d0f171e312c2d29b106b5619356bce3e0accc18a14d07770c3f0b9d81  height=2353  log2_work=19.200899  tx=2575  date=2020-06-16 05:43:54 progress=1.000000  cache=0
2020-06-16 05:45:05 ResendWalletTransactions()
2020-06-16 05:45:17 MultiChainMiner: Block Found - 009fcec821bad6585aefc0675b528c48860c6e601fe4075ab450c223f078c1f8, prev: 00dda12d0f171e312c2d29b106b5619356bce3e0accc18a14d07770c3f0b9d81, height: 2354, txs: 3
2020-06-16 05:45:17 UpdateTip:            new best=009fcec821bad6585aefc0675b528c48860c6e601fe4075ab450c223f078c1f8  height=2354  log2_work=19.201511  tx=2578  date=2020-06-16 05:45:17 progress=1.000000  cache=0
2020-06-16 05:45:17 Accepted header for block 00fc219bf9105602246ad9d7f31982a8a878dc55088386faa08b95332b24994a found on alternative chain at height 2354
2020-06-16 05:45:17 wtxs: RollBackSubKeys: Couldn't find tx 000000080000000e6a4f8b77d9e228f4392516de0bc5d6717c99e6e800d83826, error: 15
2020-06-16 05:45:17 wtxs: RollBack: Error when rolling back soubkeys: 15
2020-06-16 05:45:17 wtxs: RollBack: Error: 15
2020-06-16 05:45:17 UpdateTip:            new best=00dda12d0f171e312c2d29b106b5619356bce3e0accc18a14d07770c3f0b9d81  height=2353  log2_work=19.200899  tx=2575  date=2020-06-16 05:43:54 progress=1.000000  cache=2
2020-06-16 05:45:17 UpdateTip:            new best=00fc219bf9105602246ad9d7f31982a8a878dc55088386faa08b95332b24994a  height=2354  log2_work=19.201511  tx=2578  date=2020-06-16 05:42:33 progress=1.000000  cache=0
2020-06-16 05:45:21 MultiChainMiner: Block Found - 009573dfa6052cf2efa3f3be7c8c7a21a1887d04fd9b23734be6c6d39f022637, prev: 00fc219bf9105602246ad9d7f31982a8a878dc55088386faa08b95332b24994a, height: 2355, txs: 1
2020-06-16 05:45:21 UpdateTip:            new best=009573dfa6052cf2efa3f3be7c8c7a21a1887d04fd9b23734be6c6d39f022637  height=2355  log2_work=19.202124  tx=2579  date=2020-06-16 05:45:21 progress=1.000000  cache=0
2020-06-16 08:15:08

 

no logs after this.

There is this line in the debug log which I am not able to understand also.

2020-06-16 05:45:17 wtxs: RollBackSubKeys: Couldn't find tx 000000080000000e6a4f8b77d9e228f4392516de0bc5d6717c99e6e800d83826, error: 15

2020-06-16 05:45:17 wtxs: RollBack: Error when rolling back soubkeys: 15
2020-06-16 05:45:17 wtxs: RollBack: Error: 15

checked the kernel and syslogs no sign of OOM kicking in, the memory usage of the machine is still under 60%

 

I was able to restart this node without any issues but it keeps on happening on with the other nodes. What can be the possible causes?

Comments

It wasn't clear from what you wrote above whether this is happening on one node or all nodes - please clarify. This will help us understand the problem.

---

it happened with one other node also, the same way no logs left and no signs of the os stopping the process.

---

Thanks for your reply - I'll ask the team about this and get back to you here.

Answer 1

The most likely explanation is that the node's wallet got corrupted due to some crash or hard shutdown. Versions 2.0.6+ of MultiChain handle this situation better. So our recommendation is to upgrade to the latest version, and run multichaind for the first time with the -rescan option. That will rebuild the wallet contents based on the blockchain.

Comments

Thanks for your help, this was resolved a couple of days later, the problem was with peers, all the peers were not able to communicate with each other properly there was no p2p connection between two of the peers of  same blockchain, the firewall was not configured properly to allow communication on the network port of the machines, which was causing problems in mining and ultimately freezing the blockchain.

Thanks again.

---

OK, thanks for the update.