The mining in a blockchain has unexpectedly stopped, I had a blockchain running for about a couple of months, the blockchain has three nodes all with mining permissions, and mining diversity set to 0.66.
The blockchain was working fine but today I noticed excess memory consumption by the blockchain, on closer inspection I found that the mining has stopped about a month ago, and the mempool is filled with about 4000 transactions.
All three nodes are still connected and broadcasting new transactions to each other but none of the transactions is getting mined, the last block was mined on 24th June 2020.
In getinfo api call, the mining paused parameter is false.
This is the debug log, which also shows something different happening after 24th june 2020
2020-06-24 23:32:30 ResendWalletTransactions()
2020-06-24 23:32:39 UpdateTip: new best=00930c22e713129df7458d6c5e20ca3de5c17ec4ff1ba0b175f489734c4f73b2 height=31160 log2_work=22.927454 tx=34372 date=2020-06-24 23:31:24 progress=1.000000 cache=0
2020-06-24 23:32:41 MultiChainMiner: Block Found - 002120d5c508d36fb47ebc1ebf5f63fa2a259288acc299834d41d2c5b0820b70, prev: 00930c22e713129df7458d6c5e20ca3de5c17ec4ff1ba0b175f489734c4f73b2, height: 31161, txs: 1
2020-06-24 23:32:41 UpdateTip: new best=002120d5c508d36fb47ebc1ebf5f63fa2a259288acc299834d41d2c5b0820b70 height=31161 log2_work=22.9275 tx=34373 date=2020-06-24 23:32:41 progress=1.000000 cache=0
2020-06-24 23:33:03 ResendWalletTransactions()
2020-06-26 16:18:21 CommitTransaction: 1f1ab826af80026512b92be1a9c515722405e2788e5cac536ab3b94a761e3a09, vin: 1, vout: 2
2020-06-26 16:25:02 CommitTransaction: b9f51c3f27fdb2039a5f6dbb3567206dbdb5d0dce0f5416fa0a1ce2b9330b05e, vin: 1, vout: 2
2020-06-26 16:25:12 CommitTransaction: 38083a29ced1dbf2db12f18241f8ef9ebf10d04252af0020f2b8b36df7d35323, vin: 1, vout: 2
2020-06-26 16:25:22 CommitTransaction: bd189c83a98aebd1e06013951fdd69f62ff899646b8b84be84b68b28e482748e, vin: 1, vout: 2
2020-06-26 16:25:52 CommitTransaction: 425b2ca954dc1ab436e891b2daafa7217034f89183915e531eb4013f161a02a8, vin: 1, vout: 2
2020-06-26 16:26:02 CommitTransaction: 961a228bd581f965ca6060486d49b234e14b921ac37aacc8ffc897478297f281, vin: 1, vout: 2
2020-06-26 16:26:12 CommitTransaction: 639bda696c2ae49fe476c7a4ad608610659a20fdad3e035e45be897b30270896, vin: 1, vout: 2
2020-06-26 16:26:22 CommitTransaction: b9308a0699474402195b9e6a39c446c304bf8cbd0a6e070f43a688941efaba8f, vin: 1, vout: 2
2020-06-26 16:27:52 CommitTransaction: eae40ed402ae2a5e6402b49abc6798031ecab43e7d094416e299cfcb785782d8, vin: 1, vout: 2
2020-06-26 16:28:02 CommitTransaction: da7d7ab2668fda67901283335e4db1c412e60dbdcd71bba33100471197364e28, vin: 1, vout: 2
2020-06-26 16:28:12 CommitTransaction: f6335a9cc5b70201217f4d094c481cbaae79e129146a4af59211690d01c70587, vin: 1, vout: 2
2020-06-26 16:28:22 CommitTransaction: 71ddc56bb11311be177d0bdcb3b7bcdfb8c2a00df7a9eb4dc41ae8a486c4d166, vin: 1, vout: 2
2020-06-26 16:28:32 CommitTransaction: 45d3ea3644fe15aff4a7f1597af769a9e15c0bfbef77d6ea9b23627ba840ca20, vin: 1, vout: 2
2020-06-26 16:30:22 CommitTransaction:
.........
2020-07-05 15:32:39 socket sending timeout: 61s
2020-07-05 15:35:15 mchn: Sending minimal parameter set to 10.0.2.218:47216
2020-07-05 15:35:15 receive version message: /MultiChain:0.2.0.10/: version 70002, blocks=31161, us=11.0.2.19:6011, peer=44
2020-07-05 15:35:15 mchn: Connection from 1LWFDzTX9GHwHdpx8zBLExZbwAevCGC8s3EuVS received on peer=44 in verack
2020-07-05 15:35:15 mchn: Connection from 1LWFDzTX9GHwHdpx8zBLExZbwAevCGC8s3EuVS received on peer=44 in verackack (10.0.2.218:47216)
2020-07-05 15:35:15 mchn: Parameter set from peer=44 verified
2020-07-05 15:35:16 mchn: Synced with node 44 on block 31161 - requesting mempool
2020-07-05 15:35:16 mchn: Sending minimal parameter set to 10.0.4.238:6011
2020-07-05 15:35:16 receive version message: /MultiChain:0.2.0.10/: version 70002, blocks=31160, us=11.0.2.19:58198, peer=45
2020-07-05 15:35:16 mchn: Connection from 149kFCMKAenKXZAcENGLD7FMB5BvuXt1JhHGSb received on peer=45 in verack
2020-07-05 15:35:16 mchn: Connection from 149kFCMKAenKXZAcENGLD7FMB5BvuXt1JhHGSb received on peer=45 in verackack (10.0.4.238:6011)
2020-07-05 15:35:16 mchn: Parameter set from peer=45 verified
2020-07-05 19:31:50 CommitTransaction: 0213d891653686ee1dc9ba5153077aa73c7734abe8cd90a424a3f5ce07b71051, vin: 1, vout: 2
2020-07-05 19:32:09 CommitTransaction: e617efe2efee075b85fdcd0ce6ac641a22e19e5e4f49cb8f64c7c2542f2a95d0, vin: 1, vout: 2
2020-07-05 19:35:08 CommitTransaction: 61ee58ce6c5fd3b1e2a2a8e858709d6196c5f832df7dc090d494f8f5517d6937, vin: 1, vout: 2
2020-07-06 04:55:41 CommitTransaction: 3a02331bea3751f9ccaf3853da3d0bbd45ae066b51885bd07a2518eedb032bdd, vin: 1, vout: 2
2020-07-06 05:11:08 CommitTransaction: cb20a200e4be2321faf58ebcfddcdd6d6de4745f0ae08cb7d4581c410e56dd43, vin: 1, vout: 2
2020-07-06 05:14:39 CommitTransaction: 1b695abd3ec00e89c84434486ab8f3542b5a7f1e4985a27fd48f2f2b87497364, vin: 1, vout: 2
2020-07-06 05:26:15 CommitTransaction: 4c732e38089e3006fe91556bb439033962bd3ed9bae38bfb45f151656de95d0c, vin: 1, vout: 2
2020-07-06 05:29:38 CommitTransaction: acfd840431885ea2b2c4744dbc5380e91a50890128e164ac2a5d9175ec25f5b8, vin: 1, vout: 2
2020-07-06 05:32:59 CommitTransaction: 405f1729041503fa7a6af5d86851de1fb80d60a05040e4f5b49a0551202d7023, vin: 1, vout: 2
2020-07-06 05:50:54 CommitTransaction: 34f0f285471d4b1e0ace1b42b94df6992ab7f2130b1e7ceb7a80d9018e826097, vin: 1, vout: 2
2020-07-06 06:06:15 CommitTransaction: 36b7c44a74619a98acbd1018179b26c8aa47b66edc03e6e76bf38ff316372d9d, vin: 1, vout: 2
2020-07-06 06:06:32 CommitTransaction: 257f061dc05453f11f1ce6116d29df8336893e220b006f4d64b7b9f0bc5c97be, vin: 1, vout: 2
2020-07-06 06:19:55 CommitTransaction: 71596557511585e33d0809f3182165232e7a4facb34c8a35cdf5e437000e1f98, vin: 1, vout: 2
2020-07-06 06:21:25 CommitTransaction: 424815aa076186c4fd8ca66412371ebb37657513858b85b6314dbfd52745834d, vin: 1, vout: 2
2020-07-06 06:24:13 CommitTransaction: bb770909698bf71d99f2e02a677bd58748878b7f395f119156cd7f7088a93b3e, vin: 1, vout: 2
2020-07-06 06:27:54 CommitTransaction: 080825c10bd21852a35f21c5246ef39e2e15ab5c4e969993cb37c398b5879cf8, vin: 1, vout: 2
2020-07-06 06:45:58 CommitTransaction: 0c0837454310870bda106667b26951d9663517697f3640eab1ce526a7f6f7437, vin: 1, vout: 2
2020-07-06 06:57:58 CommitTransaction: 1fc900bbb1b451d086d245390dc105f1e63462c0c1a7d94044d95be66b370c8f, vin: 1, vout: 2
2020-07-06 06:58:09 CommitTransaction: 240a1a82dcd2ec4b224854ddbfcaba036e1992df3d8543ef409df4908ab067b6, vin: 1, vout: 2
2020-07-06 08:41:09 CommitTransaction: 433eea579db4969491960d4856c1d7938f042d9330dc6766c77ba937cb622876, vin: 1, vout: 2
2020-07-06 08:41:29 CommitTransaction: e240a48e37eb9bbac01048bb39e10df98ae2bfb8e73e194f21604c75d3f2dde1, vin: 1, vout: 2
2020-07-06 08:45:55 CommitTransaction: 4ac900ac0137b28767e0ad708ef6e335b9e1b83b3e825390eb98ed7a8e5f527e, vin: 1, vout: 2
2020-07-06 08:47:44 CommitTransaction: 7b309d628c715f631d94fec1c50c5732f3bf4423c6d9615080865624d24c559e, vin: 1, vout: 2
2020-07-06 08:48:30 CommitTransaction: 7bd43c5e4ff5152a7195ba9d0e288a566c4e80041ed31074fa5613499e231192, vin: 1, vout: 2
2020-07-06 08:49:13 CommitTransaction: 411ec0d859f8e3183e3b54782f4214e2d4e26c6ea8806247f20932f864ed4570, vin: 1, vout: 2
2020-07-06 08:49:34 CommitTransaction: d81d15d5c67b3ecf190271121491871b5226278a40bf7cf485be4907d8393d8b, vin: 1, vout: 2
Did not post the full log as the logs are quite a lot. The logs end with commit transaction
Not able to find the exact cause of the problem.
Thanks
