Mining stopped on round robin when node revoked and new node granted

Question

Mining has stopped, after revoking one address and granting another and two of the nodes by accident got new IP. I noticed also that the logs contain a lot of timeout when connecting to IP, that are not listed in peers, and ports seem to be random. I have added nodes and removed nodes by ip.

In a nutshell, all 3 nodes have the same blocks, getpeerinfo shows the same nodes, permissions show the same addresses, but -1 for all sync headers/blocks and the mining wont resume.
 

Running listpermissions mine * true
This address 1bWg4k.. is *new* and has been granted with mine.
Mining for this address was revoked "1aXHYV..."

All three nodes have the same 3 addresses.

[
    {
        "address" : "1En1g5...",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Yjcr4...",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1bWg4k...",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    }
]

All nodes have the last 5 blocks.

NOTE:  Block 11185 address (miner) has been revoked.

[
    {
        "hash" : "0094e62c5a5625b2498afb12380169f5e6d4139f27358e6dff17a81ca0ebe8b5",
        "miner" : "1Yjcr4...",
        "confirmations" : 5,
        "height" : 11183,
        "time" : 1620402689,
        "txcount" : 8
    },
    {
        "hash" : "00b93e84a241b61f4594fc5c018ed0ca0a1abb356e2ab75a7f03b3efa4b58f16",
        "miner" : "1En1g5...",
        "confirmations" : 4,
        "height" : 11184,
        "time" : 1620402736,
        "txcount" : 1
    },
    {
        "hash" : "00bd0c65b5ae2536e4a949598b50b7f7768d5e493535c796c01ca970ee5b6396",
        "miner" : "1aXHYV...", <- this node has been revoked
        "confirmations" : 3,
        "height" : 11185,
        "time" : 1620406320,
        "txcount" : 2
    },
    {
        "hash" : "0079265b34d729fde68fa6711da1c65e863e234a427d2491dee099c3975eff32",
        "miner" : "1Yjcr4...",
        "confirmations" : 2,
        "height" : 11186,
        "time" : 1620406364,
        "txcount" : 1
    },
    {
        "hash" : "009bc6a7edaf92e49e53bc9c61ceaa603c88dc1f165376be285f05815f4ba606",
        "miner" : "1En1g5...",
        "confirmations" : 1,
        "height" : 11187,
        "time" : 1620755260,
        "txcount" : 2
    }
]

[
    {
        "id" : 6,
        "addr" : "172.31..:5000",
        "addrlocal" : "172.31.5.117:63070",
        "services" : "0000000000000001",
        "lastsend" : 1620899387,
        "lastrecv" : 1620899387,
        "bytessent" : 15965,
        "bytesrecv" : 12112,
        "conntime" : 1620898029,
        "pingtime" : 0.07813200,
        "version" : 70002,
        "subver" : "/MultiChain:0.1.0.9/",
        "handshakelocal" : "1Yjcr4...",
        "handshake" : "1En1g5...",
        "inbound" : false,
        "startingheight" : 11187,
        "banscore" : 0,
        "synced_headers" : 11187,
        "synced_blocks" : 11187,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 7,
        "addr" : "54.246..:50136",
        "addrlocal" : "52.215.211.244:5000",
        "services" : "0000000000000001",
        "lastsend" : 1620899389,
        "lastrecv" : 1620899389,
        "bytessent" : 9934,
        "bytesrecv" : 9935,
        "conntime" : 1620898030,
        "pingtime" : 0.06250000,
        "version" : 70002,
        "subver" : "/MultiChain:0.1.0.9/",
        "handshakelocal" : "1Yjcr4...",
        "handshake" : "1En1g5...",
        "inbound" : true,
        "startingheight" : 11187,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 8,
        "addr" : "34.254..:5000",
        "addrlocal" : "52.215..:63091",
        "services" : "0000000000000001",
        "lastsend" : 1620899391,
        "lastrecv" : 1620899391,
        "bytessent" : 14688,
        "bytesrecv" : 10461,
        "conntime" : 1620898093,
        "pingtime" : 0.09370900,
        "version" : 70002,
        "subver" : "/MultiChain:0.1.0.9/",
        "handshakelocal" : "1Yjcr4...",
        "handshake" : "1bWg4k...",
        "inbound" : false,
        "startingheight" : 11187,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 9,
        "addr" : "172.31..:62172",
        "addrlocal" : "172.31..:5000",
        "services" : "0000000000000001",
        "lastsend" : 1620899387,
        "lastrecv" : 1620899387,
        "bytessent" : 8924,
        "bytesrecv" : 10048,
        "conntime" : 1620898094,
        "pingtime" : 0.09370900,
        "version" : 70002,
        "subver" : "/MultiChain:0.1.0.9/",
        "handshakelocal" : "1Yjcr4...",
        "handshake" : "1bWg4k...",
        "inbound" : true,
        "startingheight" : 11187,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    }
]

 "anyone-can-connect" : false,
    "anyone-can-send" : false,
    "anyone-can-receive" : false,
    "anyone-can-receive-empty" : false,
    "anyone-can-create" : false,
    "anyone-can-issue" : false,
    "anyone-can-mine" : false,
    "anyone-can-activate" : false,
    "anyone-can-admin" : false,
    "support-miner-precheck" : true,
    "allow-p2sh-outputs" : true,
    "allow-multisig-outputs" : true,
    "setup-first-blocks" : 60,
    "mining-diversity" : 1.00000000,
    "admin-consensus-upgrade" : 0.50000000,
    "admin-consensus-admin" : 0.25000000,
    "admin-consensus-activate" : 0.25000000,
    "admin-consensus-mine" : 0.50000000,
    "admin-consensus-create" : 0.00000000,
    "admin-consensus-issue" : 0.00000000,
    "lock-admin-mine-rounds" : 10,
    "mining-requires-peers" : true,
    "mine-empty-rounds" : 0.20000000,

Answer 1

Setting mining-diversity=1.0 is a very strict policy that means any address that has mining permission has to be available to mine at all times. Can I recommend downloading a copy of MultiChain 2.1.2 and attaching it to your network, or upgrading one of your nodes to that version? You can then use the listminers API to see exactly what is going on in terms of who is expected to mine the next block;

Comments

perhaps the question would be:
How do I remove the two nodes here, add the other and change mining-diversity to a more usable value?
[
    {
        "address" : "1C8AAD...",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : null,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1aXHYV...",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11185,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1Yjcr4...",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11186,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1En1g5...",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 3,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    }
]

---

I'm afraid you cannot currently change the mining-diversity value on a blockchain, this is set as part of the original design.

As for resolving the blocked mining, the best thing to do is resurrect the nodes with the active mining addresses, and then revoke their mine permissions and ensure that revocation is confirmed before disconnecting them again.

If this is not possible, you will need to use the method in the answer below to roll back the chain (on all nodes) to the point where one block can still be mined by someone live, clear the mempool, then revoke the mining permissions for the dead addresses, and allow mining to proceed again. That will get those revocations confirmed in a block by the live miner address, and then things can proceed as normal.

https://www.multichain.com/qa/5386/mining-stop-possible-bug?show=5400#a5400

---

I will give this a go, but it strikes me as a good enhancement to make.

mining diversity, plans of mice and men and all that. One should be able to by consensus of admins, revoke and effectively kick a node off the mining pool and replace it with something else? It seems like a fair practice. Imagine if a partner left the chain for whatever reason and switched off their 20 nodes. How would the chain recover?