Mining stop - possible bug?

Question

How to replicate: using a single node

- create a new chain
- create a new stream
- subscribe to it
- publish something

When querying for the asset published, we can see that the item was confirmed

- revoke mining rights (e.g. no more miners)
- grant back mining rights

At this point, no more confirmations will take place. We can publish something and see that the confirmation will remain at 0. The previous item's confirmation number will either remain stable.

It seems that mining stopped when number of miners reached 0 and did not restart - is there a way to force a restart ? Is this a bug?

Answer 1

The set of permitted miners for a block is defined by the state of the blockchain at the end of the previous block. So if you have a block in which the last miner is revoked, then there is no way for any new blocks to be added without rolling back the chain. This is (intentionally) a little tricky but can be achieved by doing this on each node:

pause mining,incoming
setlastblock [hash or height of block before the one where the miner was revoked]
clearmempool
resume mining,incoming

Make sure you have run clearmempool on every node before running the resume command on any node, otherwise the mining revocation transaction can be redistributed again over the peer-to-peer network. Note that this will lose all transactions from after the block you rewound to, not just the one that caused the problem. If you need to be more selective, use the command below instead of clearmempool on every node:

setruntimeparam bantx [the txid in which miner was revoked]

Comments

I have three nodes.
I ran the commands on all three nodes, followed the sequence.
There were no other muldichaind processes running.
I have tried this 3 times.

When  I run the first node - listblocks is back to the original height.
I haven't resumed mining or incoming yet.

******************
If anyone else has this - I was restarting the node after clearmempool
But it turns out, the even if I resume without restarting nodes, the block height is not rolled back.
******************

after running listblocks -3 to check on each node, they all have the correct height which I want. When I resume, it goes back to its original height.

---

Someone from the dev team will take a look shortly.

---

You have to make sure that transaction which revokes last miner permission is removed completely and cannot reappear. The node which created this transaction will try to resend it, so setruntimeparam bantx step is absolutely necessary (at least on that node). For the same reason there is no danger in clearmempool command, sooner or later the nodes will resend transactions they sent.

This is what you have to do on all 3 nodes :

pause mining,incoming
setlastblock [hash or height of block before the one where the miner was revoked]
setruntimeparam bantx [the txid in which miner was revoked]
clearmempool
resume mining,incoming

Please note that the tx in question is banned only until you restart multichaind. To be 100% it will never reappear even after restart, you have to "double-spend" one of its inputs. To do this, while tx is still banned, run

getrawtransaction <txid> 1

Take one of the inputs in vin array and use it in any transaction, like this, for example:

createrawtransaction '[{"txid" : "...","vout" : ...}]' '{"any-address":0}' [] send

If you get an error like "Transaction was not signed properly" or API was successful  - the input is spent and problematic tx will never reappear.

---

I ran bantx and now all three nodes are at the correct height - but mining wont continue.

listminers

[
    {
        "address" : "1En1g",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11184,
        "chainstate" : "mining-permitted",
        "localstate" : "mining-paused"
    },
    {
        "address" : "1aXHY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11185,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1Yjcr4",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11186,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    }
]

listpermissions mine

[
    {
        "address" : "1En1g5",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Yjcr4",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1aXHY",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    }
]

---

getpeerinfo

[
    {
        "id" : 3,
        "addr" : "54.246...:55119", <- curious my port is 5000
        "addrlocal" : "52.215.211.244:5000",
        "services" : "0000000000000001",
        "lastsend" : 1621072825,
        "lastrecv" : 1621072825,
        "bytessent" : 406409,
        "bytesrecv" : 404862,
        "conntime" : 1621009275,
        "pingtime" : 0.062502,
        "version" : 70002,
        "subver" : "/MultiChain:0.2.0.12/",
        "handshakelocal" : "1Yjcr4",
        "handshake" : "1En1g",
        "inbound" : true,
        "encrypted" : false,
        "startingheight" : 11186,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 4,
        "addr" : "172.....:55120",
        "addrlocal" : "172.31.5.117:5000",
        "services" : "0000000000000001",
        "lastsend" : 1621072826,
        "lastrecv" : 1621072826,
        "bytessent" : 405522,
        "bytesrecv" : 405523,
        "conntime" : 1621009276,
        "pingtime" : 0.062502,
        "version" : 70002,
        "subver" : "/MultiChain:0.2.0.12/",
        "handshakelocal" : "1Yjcr4.",
        "handshake" : "1En1g5",
        "inbound" : true,
        "encrypted" : false,
        "startingheight" : 11186,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 5,
        "addr" : "172....:5000",
        "addrlocal" : "172.3...:53778",
        "services" : "0000000000000001",
        "lastsend" : 1621072821,
        "lastrecv" : 1621072821,
        "bytessent" : 405554,
        "bytesrecv" : 405553,
        "conntime" : 1621009277,
        "pingtime" : 0.078126,
        "version" : 70002,
        "subver" : "/MultiChain:0.2.0.12/",
        "handshakelocal" : "1Yjcr4.",
        "handshake" : "1aXHY.",
        "inbound" : false,
        "encrypted" : false,
        "startingheight" : 11186,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    },
    {
        "id" : 6,
        "addr" : "34.......:60041",
        "addrlocal" : "52...:5000",
        "services" : "0000000000000001",
        "lastsend" : 1621072821,
        "lastrecv" : 1621072821,
        "bytessent" : 404870,
        "bytesrecv" : 405553,
        "conntime" : 1621009278,
        "pingtime" : 0.078126,
        "version" : 70002,
        "subver" : "/MultiChain:0.2.0.12/",
        "handshakelocal" : "1Yjcr",
        "handshake" : "1aXHY",
        "inbound" : true,
        "encrypted" : false,
        "startingheight" : 11186,
        "banscore" : 0,
        "synced_headers" : -1,
        "synced_blocks" : -1,
        "inflight" : [
        ],
        "whitelisted" : false
    }
]

---

Did you run resume mining,incoming? It looks like on the first node (with address 1En1g), mining is still paused - see output of listminers.

---

Yes, I ran resume on all 3 nodes, but its not resuming.
I double spent the transaction and the one miner did the new block, the next node received the block, but the next miner is stuck.

This node 1En1g5 (has new block)
has this has next miner
{
        "address" : "1aXHY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11185,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },

But the node  1aXHY (hasn't received new block)
and has this as miner

{
        "address" : "1En1g5",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11184,
        "chainstate" : "mining-permitted",
        "localstate" : "waiting-new-transactions"
    },

---

You can see this post for the issue

https://www.multichain.com/qa/31991/mining-paused-listminers-odds-miningpaused-false-getinfo

---

It looks like a bug in listminers output - the info in getinfo is correct, node mining is not paused. We will fix it in the next version.

But now I see waiting-new-transactions. MultiChain node doesn't mine new blocks if there are no new transactions (see mine-empty-rounds blockchain parameter).

If the last transaction (excluding the banned) was sent more than 10 blocks ago, it is normal behavior, but if you think there were transactions after that, this may be an effect of "pause incoming"+clearmempool: mining node can miss some transactions and wait until other nodes will resend them - it may take time. To speed up the process, you can either call resendwallettransactions on all nodes or send some new transaction on the miner node.

---

The next node wouldnt respond to bantx and doublespend. it kept loading the old height. The other two nodes both show the new node and the new height

After a number of attempts multichaind asked for reindex...

Before this though the node still thought that the previous node wasn't done.

unchanged on 1En1g5, and the state still says
 {
        "address" : "1En1g5",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11184,
        "chainstate" : "mining-permitted",
        "localstate" : "waiting-new-transactions"
    },

while the other two nodes have moved on and both claim
 {
        "address" : "1aXHY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11185,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },


This node 1aXHYV thinks that 1En1g is waiting transactions, but the other two nodes have already moved on and waiting for 1aXHYV.

After reindex - I will see whats transpired, but my guess will be, I will have the old block height. What would be blocking this node from accepting the new block/height?

---

I think there is some misunderstanding here. lastmined is the height of the block which was mined by this specific address, not the chain height on this node. Block 11184 was mined by the address 1En1g5, block 11185 was mined by address 1aXHY and block 11186 was mined by address 1Yjcr4.

You should have identical information for "listblocks -3" on all 3 nodes - please check that hashes of the blocks are identical and "miner" field reflects information above.

If this is not a case, please post output of "listblocks -3" on all 3 nodes.

---

Here are the 3 outputs

[
    {
        "hash" : "0079265b34d729fde68fa6711da1c65e863e234a427d2491dee099c3975eff32",
        "miner" : "1Yjcr",
        "confirmations" : 3,
        "height" : 11186,
        "time" : 1620406364,
        "txcount" : 1
    },
    {
        "hash" : "008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621",
        "miner" : "1En1g",
        "confirmations" : 2,
        "height" : 11187,
        "time" : 1621075754,
        "txcount" : 1
    },
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYV",
        "confirmations" : 1,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    }
]

[
    {
        "hash" : "0079265b34d729fde68fa6711da1c65e863e234a427d2491dee099c3975eff32",
        "miner" : "1Yjcr",
        "confirmations" : 3,
        "height" : 11186,
        "time" : 1620406364,
        "txcount" : 1
    },
    {
        "hash" : "008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621",
        "miner" : "1En1g",
        "confirmations" : 2,
        "height" : 11187,
        "time" : 1621075754,
        "txcount" : 1
    },
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYV",
        "confirmations" : 1,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    }
]

[
    {
        "hash" : "0079265b34d729fde68fa6711da1c65e863e234a427d2491dee099c3975eff32",
        "miner" : "1Yjcr",
        "confirmations" : 3,
        "height" : 11186,
        "time" : 1620406364,
        "txcount" : 1
    },
    {
        "hash" : "008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621",
        "miner" : "1En1g",
        "confirmations" : 2,
        "height" : 11187,
        "time" : 1621075754,
        "txcount" : 1
    },
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYV",
        "confirmations" : 1,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    }
]

---

So, the output is identical. It looks like the mining stopped because there are no new transactions. It is normal behavior. It will be automatically resumed when there are new transactions. As I said before, the next node to mine is 1Yjcr, when it has new transaction, it should mine a block.

---

Sine then, I sent new transactions, set mineemptyrounds to 1 on each node and called resendwallettransactions - nothing yet

---

Please confirm, on the node with address 1Yjcr:

1. Output of getrawmempool is not empty
2. getblockcount returns 11188
3. Output of getpeerinfo is not empty
4. You don't see something that looks like an error in the end of debug.log

If all this is true, what is the output of listminers on this node?

---

Even though I ran a dozen transactions, getmempool is empty

{"method":"getrawmempool","params":[],"id":"53938413-1621098922","chain_name":"clicktochain"}

[
]

getblocount == 11188
get peerinfo - there are 4 peers (should there not be 3?)


debug.log
a couple of timeouts
pong peer=5 /MultiChain:0.2.0.12/: Timing mishap, 20a9aa854ed48495 expected, 20a9aa854ed48495 received, 8 bytes

ERROR: CheckBlock() : banned transaction: f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09
2021-05-15 14:48:01 ERROR: ProcessNewBlock : CheckBlock FAILED

VerifyBlockMiner: Rolled back to block 11186
2021-05-15 14:48:01 VerifyBlockMiner: Verifying block 11187
2021-05-15 14:48:01 VerifyBlockMiner: New block 008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621 (height 11187)
2021-05-15 14:48:01 VerifyBlockMiner: Restoring chain, block 11187
2021-05-15 14:48:01 VerifyBlockMiner: Restored on block 11186
2021-05-15 14:48:01 mchn: Connecting block 008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621 (height 11187) ...
2021-05-15 14:48:01 mchn: Checking block signature and miner permissions...
2021-05-15 14:48:01 mchn: Committing permission changes for block 11187...
2021-05-15 14:48:01 UpdateTip:            new best=008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621  height=11187  log2_work=21.449665  tx=48164  date=2021-05-15 10:49:14 progress=1.000000  cache=0
2021-05-15 14:48:01 mchn: Replaying memory pool (0 new transactions, total 0)
2021-05-15 14:48:01 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2021-05-15 14:48:01 Loaded 11188 blocks from external file in 2055658ms
2021-05-15 14:48:01 Reindexing finished

---

* Even though I ran a dozen transactions, getmempool is empty

OK, it is likely  to be a reason why block is not mined. Though pong error may be a result of some connectivity problem, it should not prevent node from mining the block.

Let's focus on "sending transactions". On the node with address 1Yjcr: can you please send any transaction and make sure it's appears in getrawmempool. If it is not there, please restart multichaind with -debug=mcapi, try again and post the end of debug log (with this transaction)

---

I stopped and restarted multichaind and got
: Error loading block database.

its the 3rd time I have to run reindex

---

After this reindex, it went to 11189 and listminers has a new address
1C8AA - I have not idea what this is. I only have three nodes.

[
    {
        "address" : "1C8AA",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : null,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1En1g",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXHY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1Yjcr",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 3,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11189,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    }
]

---

* has a new address 1C8AA - I have not idea what this is. I only have three nodes.

Any node may have several addresses. And these addresses may all have mining permissions, and all these addresses are involved in mining. You can run getaddresses to find out which of your nodes has this address.

Currently it is 1C8AA's turn to mine. Please make sure the node having this address is up. Actually, with mining-diversity=1 you need all nodes with addresses having mining permission to be up.

When you find this node, please make sure it has new transaction. The simplest way to do it is to send tx to yourself:

send 1C8AA 0

If this API returns txid, you should see it in the output of getrawmempool immediately after that. Then the node should mine the block 11190 and you should see this transaction confirmed and removed from getrawmempool's output .

If all your nodes are up and mineemptyrounds=1 on these node, you should see 4 more blocks 11191-11194. Then the mining  will stop again until new transactions are sent to the network.

---

This is the state of 1c8AA
 {
        "address" : "1C8AA",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : null,
        "chainstate" : "mining-permitted",
        "localstate" : "other-local-address-preferred"
    },

getmempoolinfo
{"method":"getmempoolinfo","params":[],"id":"49409161-1621160730","chain_name":"clicktochain"}

{
    "size" : 2,
    "bytes" : 7694
}

---

OK, maybe I see the problem here. Can you please post the output of these commands on the node with 1C8AA:

listpermissions mine,admin
getaddresses

---

[
    {
        "address" : "1C8AA",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1En1g",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Yjcr",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Yjcr",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1aXHYV",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1aXHYV",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295
    }
]

---

getaddresses has 16k addresses - so cant paste it here!!

---

From the previous outputs you posted it looks like your nodes share different addresses (3 addresses 1Yjcr,1aXHYV and 1En1g appear in the output of listminers with islocal:true, on the other hand in getpeerinfo they connect to each other). This may happen if (a) you created the nodes by copying entire directory or (b) you used one of the importaddress/importprivkey/importwallet APIs. It is OK, but it makes very difficult to find out what's going on on each of the nodes using existing APIs.

So, I propose the following solution (providing you want to get rid of the stuck miner 1C8AA) - this solution forces the node to mine block using specific address and we also remove mining permission from this address.

Restart node with 1C8AA with the following switches:

-gen=0 -mineblocksondemand

Please DON'T make -reindex - it only complicates the picture.

Then run the following commands on this node (if something is wrong, don't continue and let me know):

getblockcount - should return 11189

listminers - make sure all addresses except 1C8AA have  "chainstate" : "waiting-mining-diversity"

revoke 1C8AA mine - should return txid

setgenerate 1C8AA - should return block hash

getblockcount - should return at least 11190

getblock 11190 - should have "miner":1C8AA

listminers - for 1C8AA there should be "chainstate" : "no-mine-permissions"

listpermissions mine - should not have 1C8AA in the list

Restart node with 1C8AA normally (without -gen=0)

After that you should see at least 3 blocks 11191-11193 if other nodes are connected

---

The command setgenerate returned error (expected ?)

{"method":"setgenerate","params":["1C8AA"],"id":"26732688-1621177031","chain_name":"clicktochain"}

error code: -704
error message:
Couldn't find wallet address with mining permission

---

OK, this explains a lot. This node doesn't have private key for address 1C8AA. You can verify this with  

listaddresses 1C8AA

"ismine" is correct there, it should be false (if it is true, please ignore everything below and let me know)

This address is reported as "islocal":true probably because of the minor bug I mentioned in another thread. It should be "islocal":false and "localstate":null.

This explains, of course, why the mining is stuck. All you have to do is to find the private key for this address:

1. Find a node where  "listaddresses 1C8AA" returns "ismine":true and make sure this node is connected and have new transactions (getpeerinfo, getrawmempool)

or

2. If this is external address and you have its private key somewhere - you can use importprivkey API

If none of these options is available, you should use the "rollback" procedure similar to that we started this thread from - please let me know and I will give more precise instructions.

But in any case - please don't use -reindex - it rebuilds everything from the beginning ignoring all banned transactions, rolled back blocks, etc. BTW, I believe this address (1C8AA) is some old address restored by reindex.

---

This is on the primary node

{"method":"listaddresses","params":["1C8AA"],"id":"59302652-1621178508","chain_name":"clicktochain"}


[
    {
        "address" : "1C8AA",
        "ismine" : false
    }
]

the other two nodes have this
{"method":"listaddresses","params":["1C8AA"],"id":"24535355-1621178559","chain_name":"clicktochain"}

[
]

---

For the record - there has never been a fourth node - and never had there been another address given mine permissions.

I have never copied the folder either. :(

reindex was demanded by multichaind. I got this error
: Error loading block database.

and it would not start the node unless reindex was run.
Except as a backup - when the  node server could not be accessed, I moved the backup to another server.

---

I have reset to block 11188, one before this address was set in permissions.
I have not resumed, restarted anything - unless you advise.

cli listminers is now
 {
        "address" : "1Yjcr4",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : "disabled-by-gen-parameter"
    },
    {
        "address" : "1En1g",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXHY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
}

---

If it helps you can see the chain here.. Its not cached - actually live rpc - so a little slow.

https://clicktopurchase.com/BlockChain/Explorer

---

* For the record - there has never been a fourth node

OK, it is easy to create address using APIs like createkeypairs+importaddress. If this is a case, please try to find its private key - it is easier to call importprivkey than use the procedure below.

* never had there been another address given mine permissions.

It is strange, we've never heard about permissions appearing/disappearing without grant/revoke commands. Is there a chance it was an original miner address on the first node of the chain and its mining permission was revoked later? Anyway, we cannot resolve this issue without looking at the files in the blockchain directory.

Let's focus on removing this address from the list of miners:

1. Verify that last block is 11189 and it is mined by 1Yjcr:

getblockcount + getblock 11189

2. Store the first (and probably the only) transaction ID in the "tx" list of getblock 11189

3. On all 3 nodes:

pause mining,incoming
setlastblock 11188
setruntimeparam bantx [the txid from step 2]
clearmempool

4. On the node 1Yjcr:

revoke 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N mine

(1Yjcr is admin, so hopefuly it is enough)

5. On the node 1Yjcr:

listpermissions mine

make sure 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N disappeared

6. On the node 1Yjcr:

resume mining

7. On the node 1Yjcr:

wait for block 11189 (getblockcount)

8. resume mining,incoming

on all nodes

---

So, are you saying that 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N was granted mine permission in 11189?

Then I suggest to ban this transaction in step 3 too.

Then, to make this ban permanent, I suggest to "double-spend" one of its inputs as I suggested before, otherwise this address will pop up again in the next reindex...

---

I rolled back to 11188 and i don't have 1C8AA in list of miners.

I still have "disabled-by-gen-parameter" on 1Yjcr4

Not sure how I should continue.

---

This is because you started multichaind with -gen=0

To fix this, run

setgenerate true after step 5

continue with the steps 6-8 I mentions

when you see 11190+ on the node 1Yjcr - restart it normally (without -gen=0)

---

Done the steps above.

but is not moving past 11188


{
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : "disabled-by-gen-parameter"
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : true,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : "no-mine-permissions"
    }

---

You have to run

setgenerate true

on the node 1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD

This will return the hash of the new block 11189. You should see it immediately on the node 1Yjcr and after resume mining,incoming on other nodes

Then, when you see 11190+, you can restart 1Yjcr without -gen=0

---

Yes, Ive done that.
The 1aXH node says 11189 but the other two nodes are on 11188.
I have resumed, all three are now set
"incomingpaused" : false,
"miningpaused" : false,

last 2 blocks on the node
[
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHY",
        "confirmations" : 2,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    },
    {
        "hash" : "00141787b45d1bc2a512428691a965b2e12bb9ac57020e5ff5bb398fdc34072d",
        "miner" : "1Yjcr",
        "confirmations" : 1,
        "height" : 11189,
        "time" : 1621104673,
        "txcount" : 2  <- this has a permission change for 1C8AA
    }
]

Does this look like what you would expect?
[
    {
        "address" : "1C8AA",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : null,
        "chainstate" : "mining-permitted-pending-revoke",
        "localstate" : "disabled-by-gen-parameter"
    },
    {
        "address" : "1En1",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXH",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1Yjcr",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 3,
        "lastmined" : 11189,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    }
]

---

OK, now it is related to the concern I had before: (1Yjcr is admin, so hopefully it is enough) - apparently it is not enough (because you have two admins and both should revoke mining permission).

1. Restart 1Yjcr normally (without -gen=0 -mineblocksondemand) if you didn't do it already

2. Verify that last block is 11189 and it is mined by 1Yjcr:

getblockcount + getblock 11189

3. Store the first (and probably the only) transaction ID in the "tx" list of getblock 11189

4. On all 3 nodes:

pause mining,incoming
setlastblock 11188
setruntimeparam bantx [the txid from step 2]
clearmempool
resume incoming

5. On the node 1Yjcr AND 1aXH :

revoke 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N mine

You have now 2 txids

6. On the node 1Yjcr:

getrawmempool

The last 2 transactions from the list should be those from step 5

I don't care about txs before, even if one of them grants permission to 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N

7. On the node 1Yjcr:

listpermissions mine

make sure 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N disappeared

8. On the node 1Yjcr:

resume mining

9. On the node 1Yjcr:

wait for block 11189 (getblockcount)

10. resume mining on other nodes

---

I have tried this numerous times and the address doesn't disappear from miners
despite having no permission - everything then mines to block 11190 and then stops.
I will retry another time as it seems a little different.



 {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : true,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : "no-mine-permissions"
    }
{
    "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
    "islocal" : false,
    "permitted" : true,
    "diversitywaitblocks" : 0,
    "lastmined" : null,
    "chainstate" : "mining-permitted-pending-revoke",
    "localstate" : null
},

---

I've taken a look at your Explorer. Let's stop for a second and try to understand what the problem is. You have 3 normal miners: 1Yjcr, 1En1 and 1aXH. 1Yjcr and 1aXH are also admins. Until block 11188 everything was fine - they mined blocks in round robin fashion. Then one of the transactions granted mining permission to the address 1C8A. It is not important where this tx came from, the problem is - we don't have private key for this address, so we cannot mine blocks using this address and since mining-diversity=1, this address is absolutely necessary - the mining is stuck. The important fact is: while this granting transaction is not confirmed - it is harmless, because only confirmed mining grants count. But if it is confirmed - we are doomed.

So, we have two possible approaches to resolve the problem:
1. Somehow remove this transaction (by rolling chain back and banning this transaction)
2. While tx is still unconfirmed send "revoke" transactions AFTER that, effectively cancel this grant and confirm all these transactions in one block.

The first approach has several drawbacks - you have to make sure tx is properly banned, you have to "double-spent" its inputs, otherwise it can reappear after restart and there is a chance it reappears after -reindex (I suppose it is how it appeared in the first place)

So, I prefer the second approach - it is much cleaner and resolves the problem permanently and I see that you tried to implement it: I see 6 transactions changing permission of 1C8A - I suppose one of them is problematic grant and 5 - your revoke attempts. I think the reason you was not successful so far is: to revoke mining permission, you need all your admins to do this. And I see only transactions from  1Yjcr, not from 1aXH - this is a bit you are missing.

This approach requires transactions to be send in very precise order, so before we continue, can you please do the following:

1. getblockchainparams

Please send me a value of "admin-consensus-mine"

2. We should find the transaction(s) we want to cancel. To do this please run
getrawtransaction 5ae4263e791d32a918bf36cd89d05f42425dd40c69ff406bad7198a469d76eb1 1
getrawtransaction 43946ef41e9a4e7b53c91d85734a7244f63a5279ad9f357f9d9eb0df0a552c4f 1
getrawtransaction 0c35019f6a986854d64f687dbae6dfb8929b917c129f6b57ae89cfccc9d3f8fe 1
getrawtransaction e327497312f71cd9cb7b1583b2c132b9cdc76e20112c1d1207c805ee66af5103 1
getrawtransaction f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 1
getrawtransaction 16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9 1

And find out which of these transactions has "permissions" block with "mine":true and "endblock" : 4294967295

---

Sorry for delay.

This transaction has what you asked for.

getrawtransaction f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 1

                    "for" : null,
                    "connect" : true,
                    "send" : true,
                    "receive" : true,
                    "create" : false,
                    "issue" : false,
                    "mine" : true,
                    "admin" : false,
                    "activate" : false,
                    "startblock" : 0,
                    "endblock" : 4294967295,
                    "timestamp" : 1620755240

here is blockchainparams
{
    "chain-protocol" : "multichain",
    "chain-description" : "MultiChain clicktochain",
    "root-stream-name" : "root",
    "root-stream-open" : true,
    "chain-is-testnet" : false,
    "target-block-time" : 15,
    "maximum-block-size" : 8388608,
    "default-network-port" : 5000,
    "default-rpc-port" : 5001,
    "anyone-can-connect" : false,
    "anyone-can-send" : false,
    "anyone-can-receive" : false,
    "anyone-can-receive-empty" : false,
    "anyone-can-create" : false,
    "anyone-can-issue" : false,
    "anyone-can-mine" : false,
    "anyone-can-activate" : false,
    "anyone-can-admin" : false,
    "support-miner-precheck" : true,
    "allow-p2sh-outputs" : true,
    "allow-multisig-outputs" : true,
    "setup-first-blocks" : 60,
    "mining-diversity" : 1,
    "admin-consensus-upgrade" : 0.5,
    "admin-consensus-admin" : 0.25,
    "admin-consensus-activate" : 0.25,
    "admin-consensus-mine" : 0.5,
    "admin-consensus-create" : 0,
    "admin-consensus-issue" : 0,
    "lock-admin-mine-rounds" : 10,
    "mining-requires-peers" : true,
    "mine-empty-rounds" : 0.2,
    "mining-turnover" : 1,
    "first-block-reward" : -1,
    "initial-block-reward" : 0,
    "reward-halving-interval" : 52560000,
    "reward-spendable-delay" : 1,
    "minimum-per-output" : 0,
    "maximum-per-output" : 100000000000000,
    "minimum-relay-fee" : 0,
    "native-currency-multiple" : 100000000,
    "skip-pow-check" : false,
    "pow-minimum-bits" : 8,
    "target-adjust-freq" : -1,
    "allow-min-difficulty-blocks" : false,
    "only-accept-std-txs" : true,
    "max-std-tx-size" : 4194304,
    "max-std-op-returns-count" : 10,
    "max-std-op-return-size" : 2097152,
    "max-std-op-drops-count" : 5,
    "max-std-element-size" : 8192,
    "chain-name" : "clicktochain",
    "protocol-version" : 10008,
    "network-message-start" : "ffcfe9eb",
    "address-pubkeyhash-version" : "0052b2d4",
    "address-scripthash-version" : "0551ab3b",
    "private-key-version" : "80106254",
    "address-checksum-value" : "32afecac",
    "genesis-pubkey" : "02011158f7adf0a5a19414d087e55f5246230db0db25b883c7009bc40b5ecf8a12",
    "genesis-version" : 1,
    "genesis-timestamp" : 1506003982,
    "genesis-nbits" : 536936447,
    "genesis-nonce" : 38,
    "genesis-pubkey-hash" : "eacdaa7df376714f46032105fc565d3e12bfbb12",
    "genesis-hash" : "00cf333ee84f6c9376bb5b3806a927ce730f4fd1a58a1e56ffd83f533644f453",
    "chain-params-hash" : "2d75a75b9b69a2406a6d2bddc8570d19048e5dc1a90063eb59e21cc2e0b9396f"
}

---

It strange as it looks like one admin is enough with your admin-consensus-mine setting if you have two admins. Anyway, let's try to make clean sequence  - it is an overkill, but let's do things carefully. If something is wrong - please don't continue and let me know:

1. Restart 1Yjcr normally (without -gen=0 -mineblocksondemand) if you didn't do it already

2. On all 3 nodes:

pause mining,incoming
setlastblock 11188
resume incoming

3. On all 3 nodes (please do this after you made resume incoming on all 3 nodes)

resendwallettransactions

4. On 1Yjcr

getrawmempool

You should see f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09
in the list. I don't care about the rest.

listminers

You don't see 1C8A in the list or see it with "pending-mine-permissions" or "no-mine-permissions"
You see 1Yjcr with "chainstate" : "mining-permitted"

5. On 1Yjcr:

revokefrom 1Yjcr 1C8A mine

6. On 1aXH:

revokefrom 1aXH 1C8A mine

7. On 1Yjcr:

getrawmempool

You should see f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 AND two transactions from steps 5 and 6 in the list

8. On 1Yjcr:

resume mining

Wait for block 11189

listminers

You should see
1C8A with "chainstate" : "no-mine-permissions"
1En1 with "chainstate" : "mining-permitted"

9. 1En1

listminers

You should see
1C8A with "chainstate" : "no-mine-permissions"
1En1 with "chainstate" : "mining-permitted"

10. 1En1 and 1aXH

resume mining

wait for block 11192

---

Two nodes are not moving pass 11188, and the main node moves to 1190.
I do not see  f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 in getmempoolinfo at that step, its already moved to 1190

---

Which of the steps didn't work as expected?

Which node do you call main?

Which node is reflected on your Explorer page?

---

Step 7

7. On 1Yjcr:

getrawmempool

You should see f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 AND two transactions from steps 5 and 6 in the list

I am seeing this (before and after)

[
    "065450f612a1a6ac124c655c1bde44c34777c926d5dfd963e3ef82e49abe30d2",
    "2e77ce547c3bbacf9809a6642734c2460caa72f1dad6dc5eab324d27b5156c8c",
    "37067ea6003fa62cb024991c973e96cebadb1d48ebcbc104379ec1d60c7d4e3e",
    "48f2b582c235ca59fb81c84a02ccf05ae9c8ac499d595b8faa3078d000fa5040",
    "695cbc1e18cab71c2b8f916704ee1bcf5b24b44a3bf1045dd96a2e4b2637b173",
    "7b06593e80a50e17bea031ecd13c36fd35eed59264261426c196e3ca074ec159",
    "7c7d717cab6074441eb888369d2dd8335facf22f5ca6ad0d956bfb2ab258481d",
    "9a47ea5f05893e36f3638c0d4290e9b31b6e95e7770d6ff9f299a8e6640050fc",
    "a4ca2400f31a576cfe78ead40733d2daa2d089982ce6e860f3915d947e768c6b",
    "ac4a4eb7d2b16e5fdb8e3ae68080c2be97b80f47374a5211905d35cf384119cd",
    "b2a5270402f0dcfd648bc7410f781c9b9de1cc67bd1b4b0fecc75933adddbe34",
    "b76d94dc669cf824752679e0d1e99441c423aaee3ba06bf4e2439eff9a23bae0",
    "c1ca7546409f3bbd74084fefcdf30fc7af33ef779c17183c70dfd2ee6161376d",
    "ca05855c02fd8267b3f5e20cfd82836fe2cd8d2e4ae82c36acf5fc9d68e7a04b",
    "e2b349ce9677349c8aa08fe85634237b489274852ba95a4a79c6d7ee21e758c9",
    "f8cc8a57ca65860e7492cc805f98fe124b152181b9039f5cd957f09b839e017d"
]

step 8 - goes to 11190 - not 11189

---

Please, don't continue if one of the steps (like step 7) does not work as expected. We should find this transaction first - if we don't cancel it properly - it will pop up at some moment...

Please restart from Step 2 and make sure that you call
pause mining,incoming

and only

resume incoming

on all 3 nodes


If you don't see f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 on 1Yjcr

please check other nodes - it should be somewhere.

---

All nodes have the address marked as "chainstate" : "no-mine-permissions"


1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD

block 11190

The other two nodes are stuck on 11188


After I restarted all the nodes, to see what they think of this state.

1aXHY
thinks the next miner
{
    "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
    "islocal" : false,
    "permitted" : true,
    "diversitywaitblocks" : 0,
    "lastmined" : null,
    "chainstate" : "mining-permitted-pending-revoke",
    "localstate" : null
},


1Yjcr
thinks the next miner
 {
        "address" : "1aXHY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : "mining-requires-peers"
    },

1En1g5
thinks the next miner
 {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },

---

This time, I did a restart as you suggested.

When I called revokefrom - it crashed multichaind

---

This time, I see the revoke transaction in the list on 1Yjcr

and the revoke on 1aXH



Step 8: It moves to 11190 - not 11189

I have not continued..

Current State at this point

1Yjcr - resume mining

last two blocks
[
    {
        "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
        "miner" : "1Yjcr",
        "confirmations" : 2,
        "height" : 11189,
        "time" : 1621195165,
        "txcount" : 3
    },
    {
        "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
        "miner" : "1En1g",
        "confirmations" : 1,
        "height" : 11190,
        "time" : 1621195229,
        "txcount" : 5
    }
]

listminers

{
        "address" : "1aXH",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : "other-local-address-preferred"
    },
    {
        "address" : "1Yjcr",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11189,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1En1",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11190,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AA",
        "islocal" : true,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : "no-mine-permissions"
    }
]

---

Do you see f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09?

in getrawmempool on 1Yjcr?

Did you see it before "resume mining"?

---

Sorry - no i dont see it
[
    "065450f612a1a6ac124c655c1bde44c34777c926d5dfd963e3ef82e49abe30d2",
    "0b93e47ca2ccba5a02d9db30a168515264e386da4c1940aa51ed78f02a588ae3",
    "2e77ce547c3bbacf9809a6642734c2460caa72f1dad6dc5eab324d27b5156c8c",
    "37067ea6003fa62cb024991c973e96cebadb1d48ebcbc104379ec1d60c7d4e3e",
    "48f2b582c235ca59fb81c84a02ccf05ae9c8ac499d595b8faa3078d000fa5040",
    "6498a299f0116c5dd11398a4850e66dc5dabe2e56cafe0b598d2a3d946c453c0",
    "695cbc1e18cab71c2b8f916704ee1bcf5b24b44a3bf1045dd96a2e4b2637b173",
    "7b06593e80a50e17bea031ecd13c36fd35eed59264261426c196e3ca074ec159",
    "7c7d717cab6074441eb888369d2dd8335facf22f5ca6ad0d956bfb2ab258481d",
    "9a47ea5f05893e36f3638c0d4290e9b31b6e95e7770d6ff9f299a8e6640050fc",
    "a4ca2400f31a576cfe78ead40733d2daa2d089982ce6e860f3915d947e768c6b",
    "ac4a4eb7d2b16e5fdb8e3ae68080c2be97b80f47374a5211905d35cf384119cd",
    "b2a5270402f0dcfd648bc7410f781c9b9de1cc67bd1b4b0fecc75933adddbe34",
    "b76d94dc669cf824752679e0d1e99441c423aaee3ba06bf4e2439eff9a23bae0",
    "c1ca7546409f3bbd74084fefcdf30fc7af33ef779c17183c70dfd2ee6161376d",
    "ca05855c02fd8267b3f5e20cfd82836fe2cd8d2e4ae82c36acf5fc9d68e7a04b",
    "e2b349ce9677349c8aa08fe85634237b489274852ba95a4a79c6d7ee21e758c9",
    "f8cc8a57ca65860e7492cc805f98fe124b152181b9039f5cd957f09b839e017d"
]

---

yes - before mining

[
    "065450f612a1a6ac124c655c1bde44c34777c926d5dfd963e3ef82e49abe30d2",
    "0b93e47ca2ccba5a02d9db30a168515264e386da4c1940aa51ed78f02a588ae3",
    "0c35019f6a986854d64f687dbae6dfb8929b917c129f6b57ae89cfccc9d3f8fe",
    "16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9",
    "2e77ce547c3bbacf9809a6642734c2460caa72f1dad6dc5eab324d27b5156c8c",
    "37067ea6003fa62cb024991c973e96cebadb1d48ebcbc104379ec1d60c7d4e3e",
    "43946ef41e9a4e7b53c91d85734a7244f63a5279ad9f357f9d9eb0df0a552c4f",
    "48f2b582c235ca59fb81c84a02ccf05ae9c8ac499d595b8faa3078d000fa5040",
    "5ae4263e791d32a918bf36cd89d05f42425dd40c69ff406bad7198a469d76eb1",
    "6498a299f0116c5dd11398a4850e66dc5dabe2e56cafe0b598d2a3d946c453c0",
    "695cbc1e18cab71c2b8f916704ee1bcf5b24b44a3bf1045dd96a2e4b2637b173",
    "7b06593e80a50e17bea031ecd13c36fd35eed59264261426c196e3ca074ec159",
    "7c7d717cab6074441eb888369d2dd8335facf22f5ca6ad0d956bfb2ab258481d",
    "9a47ea5f05893e36f3638c0d4290e9b31b6e95e7770d6ff9f299a8e6640050fc",
    "a4ca2400f31a576cfe78ead40733d2daa2d089982ce6e860f3915d947e768c6b",
    "ac4a4eb7d2b16e5fdb8e3ae68080c2be97b80f47374a5211905d35cf384119cd",
    "b2a5270402f0dcfd648bc7410f781c9b9de1cc67bd1b4b0fecc75933adddbe34",
    "b76d94dc669cf824752679e0d1e99441c423aaee3ba06bf4e2439eff9a23bae0",
    "c1ca7546409f3bbd74084fefcdf30fc7af33ef779c17183c70dfd2ee6161376d",
    "ca05855c02fd8267b3f5e20cfd82836fe2cd8d2e4ae82c36acf5fc9d68e7a04b",
    "e2b349ce9677349c8aa08fe85634237b489274852ba95a4a79c6d7ee21e758c9",
    "e327497312f71cd9cb7b1583b2c132b9cdc76e20112c1d1207c805ee66af5103",
    "f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09",
    "f8cc8a57ca65860e7492cc805f98fe124b152181b9039f5cd957f09b839e017d"
]

---

So, there is no problem with Step 7? Did you see txs from steps 5 and 6 as well on Step 7?

Then what's wrong with step 8? 11190 instead of 11189? If yes, please post the output of listminers

---

listminers from 1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD
[
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : "other-local-address-preferred"
    },
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11189,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "startblock" : 0,
        "endblock" : 4294967295,
        "lastmined" : 11190,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : true,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "startblock" : 0,
        "endblock" : 0,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : "no-mine-permissions"
    }
]

listminers from  1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY

[
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : false,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : null
    }
]

listminers from 1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj

[
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 1,
        "lastmined" : 11187,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : "waiting-mining-diversity"
    },
    {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 2,
        "lastmined" : 11188,
        "chainstate" : "waiting-mining-diversity",
        "localstate" : null
    },
    {
        "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
        "islocal" : false,
        "permitted" : false,
        "diversitywaitblocks" : null,
        "lastmined" : null,
        "chainstate" : "no-mine-permissions",
        "localstate" : null
    }
]

---

It looks like you resumed mining not incoming on 1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj on step 2.

Anyway, it looks good now. But let's double check that everything is OK

On 1Yjc, please run
getblock 11189 and make sure

f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09

is there

If yes - just resume mining,incoming on all nodes....

---

This is the output from 11189
{
    "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
    "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
    "confirmations" : 2,
    "size" : 874,
    "height" : 11189,
    "version" : 3,
    "merkleroot" : "8a3f99a04fcefab645fdde3bc38a32f0eb3a0ed28d27f137532176e47ead79c0",
    "tx" : [
        "fabae173ac522a46de2cb7dcc0c28c9587037873bfeb6921fc88ae3a1f869cf7",
        "5ae4263e791d32a918bf36cd89d05f42425dd40c69ff406bad7198a469d76eb1",
        "43946ef41e9a4e7b53c91d85734a7244f63a5279ad9f357f9d9eb0df0a552c4f"
    ],
    "time" : 1621195165,
    "nonce" : 425,
    "bits" : "2000ffff",
    "difficulty" : 5.96046447753906e-8,
    "chainwork" : "00000000000000000000000000000000000000000000000000000000002bb600",
    "previousblockhash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
    "nextblockhash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402"
}

---

this is the output from 11190

{
    "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
    "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
    "confirmations" : 1,
    "size" : 1462,
    "height" : 11190,
    "version" : 3,
    "merkleroot" : "586d549a79d648352ebf1da81a4fe6130d6733f36aa2e0f4021dc1f447cd7680",
    "tx" : [
        "f50d17e3d335485aec6ca28c7bc2a27b7ae4a4327ddcd5d1090fff57b1047970",
        "0c35019f6a986854d64f687dbae6dfb8929b917c129f6b57ae89cfccc9d3f8fe",
        "e327497312f71cd9cb7b1583b2c132b9cdc76e20112c1d1207c805ee66af5103",
        "f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09",
        "16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9"
    ],
    "time" : 1621195229,
    "nonce" : 449,
    "bits" : "2000ffff",
    "difficulty" : 5.96046447753906e-8,
    "chainwork" : "00000000000000000000000000000000000000000000000000000000002bb700",
    "previousblockhash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b"
}

---

Well, it looks like we have another factor -  1En1 is local on node 1Yjc... Please check that it is true.

On 1Yjc:

listaddresses 1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD
listaddresses 1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj

both should return

ismine:true

If yes, it explains 11190 on step 8

If you confirm this, please check that 16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9 is one of the transactions from step 5 or 6. If you are not sure, please send me the output of

getrawtransaction 16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9 1

---

{"method":"getrawtransaction","params":["16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9",1],"id":"61870784-1621785047","chain_name":"clicktochain"}

{
    "hex" : "0100000001ca61552cf69379e70292fe9a8d266ad9528827d089b157e3b9487cf9a3580205000000006a473044022007067f9720aa8867b4635468abbfd0f5b9a84dc3996c2b57f6b922852dd7b0a60220388487ad13a9e1f26ef88166f486b6f3b75b2426c2682af0e825e00bc5adbede0121028ccce10d144e36898cf30a48279a5f02d99affadd3e9c56295aa5c08a2bd369cffffffff0300000000000000002f76a914524d2218d31ee88a57b897995785c42ea6bb4e3888ac1473706b70000100000000000000000000f558a1607500000000000000001976a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac00000000000000003b3873706b69000000002f76a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac1473706b700001000000000000fffffffff561956075756a00000000",
    "txid" : "16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9",
    "version" : 1,
    "locktime" : 0,
    "vin" : [
        {
            "txid" : "050258a3f97c48b9e357b189d0278852d96a268d9afe9202e77993f62c5561ca",
            "vout" : 0,
            "scriptSig" : {
                "asm" : "3044022007067f9720aa8867b4635468abbfd0f5b9a84dc3996c2b57f6b922852dd7b0a60220388487ad13a9e1f26ef88166f486b6f3b75b2426c2682af0e825e00bc5adbede01 028ccce10d144e36898cf30a48279a5f02d99affadd3e9c56295aa5c08a2bd369c",
                "hex" : "473044022007067f9720aa8867b4635468abbfd0f5b9a84dc3996c2b57f6b922852dd7b0a60220388487ad13a9e1f26ef88166f486b6f3b75b2426c2682af0e825e00bc5adbede0121028ccce10d144e36898cf30a48279a5f02d99affadd3e9c56295aa5c08a2bd369c"
            },
            "sequence" : 4294967295
        }
    ],
    "vout" : [
        {
            "value" : 0,
            "n" : 0,
            "scriptPubKey" : {
                "asm" : "OP_DUP OP_HASH160 524d2218d31ee88a57b897995785c42ea6bb4e38 OP_EQUALVERIFY OP_CHECKSIG 73706b70000100000000000000000000f558a160 OP_DROP",
                "hex" : "76a914524d2218d31ee88a57b897995785c42ea6bb4e3888ac1473706b70000100000000000000000000f558a16075",
                "reqSigs" : 1,
                "type" : "pubkeyhash",
                "addresses" : [
                    "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N"
                ]
            },
            "permissions" : [
                {
                    "for" : null,
                    "connect" : false,
                    "send" : false,
                    "receive" : false,
                    "create" : false,
                    "issue" : false,
                    "mine" : true,
                    "admin" : false,
                    "activate" : false,
                    "startblock" : 0,
                    "endblock" : 0,
                    "timestamp" : 1621186805
                }
            ]
        },
        {
            "value" : 0,
            "n" : 1,
            "scriptPubKey" : {
                "asm" : "OP_DUP OP_HASH160 f80728cb8ea3ac33b09da23e26b1cfa8321f6cf3 OP_EQUALVERIFY OP_CHECKSIG",
                "hex" : "76a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac",
                "reqSigs" : 1,
                "type" : "pubkeyhash",
                "addresses" : [
                    "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY"
                ]
            }
        },
        {
            "value" : 0,
            "n" : 2,
            "scriptPubKey" : {
                "asm" : "73706b69000000002f76a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac1473706b700001000000000000fffffffff561956075 OP_DROP OP_RETURN",
                "hex" : "3873706b69000000002f76a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac1473706b700001000000000000fffffffff561956075756a",
                "type" : "nulldata"
            },
            "inputcache" : [
                {
                    "vin" : 0,
                    "asm" : "OP_DUP OP_HASH160 f80728cb8ea3ac33b09da23e26b1cfa8321f6cf3 OP_EQUALVERIFY OP_CHECKSIG 73706b700001000000000000fffffffff5619560 OP_DROP",
                    "hex" : "76a914f80728cb8ea3ac33b09da23e26b1cfa8321f6cf388ac1473706b700001000000000000fffffffff561956075"
                }
            ]
        }
    ],
    "blockhash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
    "confirmations" : 1,
    "time" : 1621195229,
    "blocktime" : 1621195229
}

---

The listaddresses is not as you expected. 1En1 belongs to my other node.
[
    {
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "ismine" : true
    },
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "ismine" : false
    }
]

---

* The listaddresses is not as you expected. 1En1 belongs to my other node.

It is strange as block 11190 is mined by 1En1 and it can be done only by the node having private key of 1En1. So somewhere there is a node with  "ismine" : true for 1En1 and non-paused mining. According to what you say 1En1 doesn't see block 11190 it mined by itself!

Let's check one more thing - we don't have banned transactions
getruntimeparams

(check bantx is empty)



 

Anyway, I'm happy with what I currently see:

1. 16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9 is the transaction revoking mining permission from 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N
2. This transaction is confirmed AFTER the transaction we want to cancel f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09 in block 11190
3. On all 3 nodes you see 1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N with "no-mine-permissions"

So. Let's try to run

resume mining,incoming on all 3 nodes and see what happens

---

bantx is empty.

and this is the output for the addresses on the 1En1g address with mining false
There are only 3 nodes.I have never had a 4th.

{"method":"listaddresses","params":["1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj,1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD"],"id":"85931577-1621789410","chain_name":"clicktochain"}

[
    {
        "address" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "ismine" : true
    }
]

{"method":"getinfo","params":[],"id":"39729911-1621789426","chain_name":"clicktochain"}

{
    "version" : "2.1.2",
    "nodeversion" : 20102901,
    "edition" : "Community",
    "protocolversion" : 10008,
    "chainname" : "clicktochain",
    "description" : "MultiChain clicktochain",
    "protocol" : "multichain",
    "incomingpaused" : false,
    "miningpaused" : true,
}

---

And what is the output of listblocks -3 on 1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD?

---

Never mind, I see it in the one of the posts above.

Let's try

resume mining,incoming on all 3 nodes

---

All is resumed - but no mining.

1Yjcr4 - is on 11190
    "incomingpaused" : false,
    "miningpaused" : false,
1aXHY - is on 11188
    "incomingpaused" : false,
    "miningpaused" : false,
1En1g - is on 11188
      "incomingpaused" : false,
    "miningpaused" : false,

I also sent

send transaction 1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD 0

All three nodes have 1C8AA  with  "no-mine-permissions",


There  is a difference with the nodes  for listminers



1Yjcr
 thinks its   {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : true,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : "other-local-address-preferred"
    },


The other two nodes things its the other node
 which must mine
{
        "address" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11186,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },

---

{"method":"listblocks","params":["-3"],"id":"95528427-1621790069","chain_name":"clicktochain"}

[
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 3,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    },
    {
        "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
        "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "confirmations" : 2,
        "height" : 11189,
        "time" : 1621195165,
        "txcount" : 3
    },
    {
        "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 1,
        "height" : 11190,
        "time" : 1621195229,
        "txcount" : 5
    }
]

---

I hope I know what's wrong now. Let's try the following on nodes 1aXH and 1En1

reconsiderblock 00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b
reconsiderblock
00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

---

node 1aXH
moved to 11191 and now lists this as miner

 {
     "address" : "1C8AADEmeDXCirkNcsuHE6QBipdU7YwCev352N",
     "islocal" : false,
     "permitted" : true,
     "diversitywaitblocks" : 0,
     "lastmined" : null,
     "chainstate" : "mining-permitted-pending-revoke",
     "localstate" : null
 },

1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj moved to 11190
but lists this as miner

 {
        "address" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "islocal" : false,
        "permitted" : true,
        "diversitywaitblocks" : 0,
        "lastmined" : 11188,
        "chainstate" : "mining-permitted",
        "localstate" : null
    },

---

OK, let.s continue. Please run listblocks -4 on 1aXH

---

[
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 4,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    },
    {
        "hash" : "00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963",
        "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "confirmations" : 3,
        "height" : 11189,
        "time" : 1621185103,
        "txcount" : 2
    },
    {
        "hash" : "00fe5198704c6051f92c44168c7de06eaf936e00f62c27b61068ef90a98b3b56",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 2,
        "height" : 11190,
        "time" : 1621185339,
        "txcount" : 3
    },
    {
        "hash" : "008db70d580736377573e35fb1f0170b56c998c4aa65af00d1be0ca6bb5bb321",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 1,
        "height" : 11191,
        "time" : 1621185349,
        "txcount" : 2
    }
]

---

OK,  I see a problem

Only on 1aXH:

getblock 00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963

pause mining, incoming
setlastblock 11188
setruntimeparam bantx <first tx in the output of getblock above>
resume mining,incoming

listblocks -4

make sure the block with height 11189 has hash 00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b

---

{"method":"getblock","params":["00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963"],"id":"66806543-162179
8245","chain_name":"clicktochain"}

{
    "hash" : "00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963",
    "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
    "confirmations" : 3,
    "size" : 560,
    "height" : 11189,
    "version" : 3,
    "merkleroot" : "6046d372752848b0e3448f7cd351fcf663ff0a66e3614c6d294533dc7729d4de",
    "tx" : [
        "f91434b4daa6af64a1ef56eaeeb2d0420c96e7feecd247461b94317c67833cad",
        "8d6cbef072bff48cf98b750752638c1b7aa79ee6a99ead6d2a44efbbf3d844ef"
    ],
    "time" : 1621185103,
    "nonce" : 26,
    "bits" : "2000ffff",
    "difficulty" : 5.96046447753906e-8,
    "chainwork" : "00000000000000000000000000000000000000000000000000000000002bb600",
    "previousblockhash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
    "nextblockhash" : "00fe5198704c6051f92c44168c7de06eaf936e00f62c27b61068ef90a98b3b56"
}


I set bantx on f91434b4daa6af64a1ef56eaeeb2d0420c96e7feecd247461b94317c67833cad

---

After resume, the last 4 blocks - the height on this node is still 11188
[
    {
        "hash" : "00bd0c65b5ae2536e4a949598b50b7f7768d5e493535c796c01ca970ee5b6396",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 4,
        "height" : 11185,
        "time" : 1620406320,
        "txcount" : 2
    },
    {
        "hash" : "0079265b34d729fde68fa6711da1c65e863e234a427d2491dee099c3975eff32",
        "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "confirmations" : 3,
        "height" : 11186,
        "time" : 1620406364,
        "txcount" : 1
    },
    {
        "hash" : "008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 2,
        "height" : 11187,
        "time" : 1621075754,
        "txcount" : 1
    },
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 1,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    }
]

---

Do you mean now on 1aXH the height is 11188? Did you run reconsiderblock there?

This is what's going on: after lot of attempts we have many blocks in different chains conflicting with each other. The block on 11191 was bad. The chain we created couple of hours ago on 1Yjc is good:

 {"method":"listblocks","params":["-3"],"id":"95528427-1621790069","chain_name":"clicktochain"}

[
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 3,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    },
    {
        "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
        "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "confirmations" : 2,
        "height" : 11189,
        "time" : 1621195165,
        "txcount" : 3
    },
    {
        "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 1,
        "height" : 11190,
        "time" : 1621195229,
        "txcount" : 5
    }
]


We want to see it on ALL nodes (I don't care at this stage what are the blocks 11191+), the important thing, the block with height 11190 should have hash  00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

I'm pretty sure the block with height 11188 is 00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1 on all nodes.

Can you please check on all 3 nodes what is the state of the chain? If you don't see this on some of the nodes, please run

getblock 00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

there

---

Yes 1aXH height of 11188, after setlastblock.
I did run reconsider on this node before that.

getblock 00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

is currently

{
    "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
    "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
    "confirmations" : -1,
    "size" : 1462,
    "height" : 11190,
    "version" : 3,
    "merkleroot" : "586d549a79d648352ebf1da81a4fe6130d6733f36aa2e0f4021dc1f447cd7680",
    "tx" : [
        "f50d17e3d335485aec6ca28c7bc2a27b7ae4a4327ddcd5d1090fff57b1047970",
        "0c35019f6a986854d64f687dbae6dfb8929b917c129f6b57ae89cfccc9d3f8fe",
        "e327497312f71cd9cb7b1583b2c132b9cdc76e20112c1d1207c805ee66af5103",
        "f24efe59483b37ce19bc9cc283999f71ec59ac5fc27107348f255fe268e11a09",
        "16f187c0cfe3656a2e74e87226f860b8b8e74c22ad66639de2bce6be35fb7ec9"
    ],
    "time" : 1621195229,
    "nonce" : 449,
    "bits" : "2000ffff",
    "difficulty" : 5.96046447753906e-8,
    "chainwork" : "00000000000000000000000000000000000000000000000000000000002bb700",
    "previousblockhash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b"
}

---

All nodes have 11188 with 00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1

The other two nodes have the following
[
    {
        "hash" : "008f8d8925ae4f2e47cf5b953c0870210c44b857d8f608f02fd50b51e38e2621",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 4,
        "height" : 11187,
        "time" : 1621075754,
        "txcount" : 1
    },
    {
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
        "confirmations" : 3,
        "height" : 11188,
        "time" : 1621091604,
        "txcount" : 1
    },
    {
        "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
        "miner" : "1Yjcr4p4qex1vpV9QDpuomNocGxu45EM9RQUoD",
        "confirmations" : 2,
        "height" : 11189,
        "time" : 1621195165,
        "txcount" : 3
    },
    {
        "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
        "miner" : "1En1g5QmTvbNDxRgz66PEPXkyPvFaGekvfFLUj",
        "confirmations" : 1,
        "height" : 11190,
        "time" : 1621195229,
        "txcount" : 5
    }
]

---

"The other two nodes" looks great, so the problem is only with 1aXH

I understand you ran the following:
pause mining, incoming
setlastblock 11188
setruntimeparam bantx <first tx in the output of getblock above>

But did you run

resume mining,incoming?

Anyway, it is safe to run

reconsiderblock 00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b
reconsiderblock
00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402
resume mining,incoming

---

I ran it as you said. The height is still 11188. I definitely resumed mining and incoming

getinfo
 "incomingpaused" : false,
 "miningpaused" : false,

I tried it again and then again - same answer.

---

Something prevents the node from being switched to the correct chain, can you check whether there is something in debug.log?

---

This is interesting.

2021-05-23 20:11:40 Banned transaction set:    0 f91434b4daa6af64a1ef56eaeeb2d0420c96e7feecd247461b94317c67833cad
2021-05-23 20:11:58 VerifyBlockMiner: Invalid block 00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963 (height 11189)
2021-05-23 20:11:58 VerifyBlockMiner: Block 00538028d2a61347c3bf41a520f80309434fcc6dd28cc17d2087ba854c570963 (height 11189) miner verification failed
2021-05-23 20:12:20 Node paused state is set to 00000000

---

This is OK, we made this block invalid, the question is why it doesn't take
correct blocks instead?

what about

getblock 00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b

---

{"method":"getblock","params":["00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b"],"id":"60112918-162180
2081","chain_name":"clicktochain"}

{
    "hash" : "00fc47d5bee8b3fcec665ac04f85515f662ebcac165e45d980c8b185cba0675b",
    "miner" : "1aXHYVYvJGLqvxi1SLmgfSQ6ozFPYPcay8G6NY",
    "confirmations" : -1,
    "size" : 874,
    "height" : 11189,
    "version" : 3,
    "merkleroot" : "8a3f99a04fcefab645fdde3bc38a32f0eb3a0ed28d27f137532176e47ead79c0",
    "tx" : [
        "fabae173ac522a46de2cb7dcc0c28c9587037873bfeb6921fc88ae3a1f869cf7",
        "5ae4263e791d32a918bf36cd89d05f42425dd40c69ff406bad7198a469d76eb1",
        "43946ef41e9a4e7b53c91d85734a7244f63a5279ad9f357f9d9eb0df0a552c4f"
    ],
    "time" : 1621195165,
    "nonce" : 425,
    "bits" : "2000ffff",
    "difficulty" : 5.96046447753906e-8,
    "chainwork" : "00000000000000000000000000000000000000000000000000000000002bb600",
    "previousblockhash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1"
}

---

It's strange. I will try to simulate this behavior tomorrow.

One more thing for today. Can you please run

getchaintips

---

[
    {
        "height" : 11191,
        "hash" : "008db70d580736377573e35fb1f0170b56c998c4aa65af00d1be0ca6bb5bb321",
        "branchlen" : 3,
        "status" : "invalid"
    },
    {
        "height" : 11190,
        "hash" : "00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402",
        "branchlen" : 2,
        "status" : "valid-headers"
    },
    {
        "height" : 11190,
        "hash" : "00991edcac11e65860c1255441c9bcd1cc210b037028a2db4ad26a78d27ce1ac",
        "branchlen" : 2,
        "status" : "valid-headers"
    },
    {
        "height" : 11189,
        "hash" : "00141787b45d1bc2a512428691a965b2e12bb9ac57020e5ff5bb398fdc34072d",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 11189,
        "hash" : "002ae8cf5d45c4a9424b0e25102c8d820de5fe159056eb504578eb6785c7f483",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 11188,
        "hash" : "00a3daff2a7ec6c1da63e566ae2397080d13acd3c089c45be7c950b734fdd1c1",
        "branchlen" : 0,
        "status" : "active"
    },
    {
        "height" : 11187,
        "hash" : "009bc6a7edaf92e49e53bc9c61ceaa603c88dc1f165376be285f05815f4ba606",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 11161,
        "hash" : "0073b581e04207dc0fd8b2e433c7076949ff929cb0ccd67550511afb9bc6753a",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 10834,
        "hash" : "00690cd115fbc521659fdd9f30bc84d597346d08d053c241125338f1c0f51580",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 9022,
        "hash" : "00e5f616a8f84b628213382b42c6411579c87af023a701958a30cf089164aa15",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 6400,
        "hash" : "0081783293a579258c3bd7005309ad96580050ec5f47faf7047f889638ec89d0",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 6250,
        "hash" : "0036f85652ec92dad70b3339371ed284543518ccca0d720e7a84aaa5dff1a13e",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 4111,
        "hash" : "00d1d3423293c47168f06cfd9f3e68ff5e4866895466ce764063ed8f2385f73f",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 3262,
        "hash" : "00b57e2846342d04a581c57271edff615b8f419ec267f80cd6e870aee6f5d7fe",
        "branchlen" : 1,
        "status" : "invalid"
    },
    {
        "height" : 2632,
        "hash" : "00e1771f80db97b1e3b6fec19845eaa2b375b6bf7406692759b1765014601d9d",
        "branchlen" : 1,
        "status" : "invalid"
    }
]

---

Thanks, what a mess...

I see our branch marked as "valid-header". It is a good sign, I will check tomorrow how we can revive it.

---

It looks like solution is very simple, on 1aXH:

pause mining, incoming
setlastblock 00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

if you see in response
00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

you should see identical output of lisblocks -4 on all nodes and block 11190 should be 00707cb8a1dca8531b83d980bc5ce5110952f0a42c53bd6dbf8453ce06ade402

Then, just run

resume mining,incoming on 1aXH

and everything should be fixed

---

I ran this, seems promising went to height of 11193 straight away, I then send 0 to an address and now I have 11195 on all nodes.

---

I take it, the fork could not be resolved and we went ahead of it? Or what would the explanation be for that block?

---

Great, it looks like the problem is finally resolved!

The last bit was related to the fact that 1aXH had the chain longer than our "correct" branch. Since it was longer, 1aXH had no reason to switch to the shorter branch. Other nodes, on the other hand didn't switch to that bad long branch because we probably banned it in some way couple of weeks ago. Then we made block on the longer chain invalid - very intrusive and non-standard action, in most cases node switches to another (even shorter) branch automatically, but not in this case (I will not bother you with details, it is also related to the wrong sequence of operations we made). We just had to make the switch explicitly (with the last  setlastblock).

But the source of the problem is mining-diversity=1 (it is also related to the question you asked in another thread). If it would be, say, 0.6, simple revoke on one of the admin nodes would solve the problem - you are absolutely right, this is how it should work (but not for mining-diversity=1). Even more, you would not be aware of the problem at all - 3 nodes would continue mining without waiting for non-existent 1C8A.

But when you granted mining permission to 1C8A, with mining-diversity=1 you said "this address should mine every 4th block". But it is the whole point of blockchain - it is immutable ledger - you cannot change any fact stored on blockchain without agreement from the majority of the miners, and with mining-diversity=1 "majority" means ALL miners, including broken 1C8A (which never agree).

That's why the recovery procedure is so complicated - it is against the nature of blockchain, it cannot be simple. And, of course, it requires agreement from majority of the miners (ALL in our case) and these miners should be perfectly synchronized  to do this properly.  

There is no problem with granting/revoking mine permission, but with mining-diversity=1 you should make sure you have private key of the miner you grant permission.  BTW, it is also true for existing miners (1Y, 1A and 1E) - if you lose one of these keys  - you will be stuck again. The simplest way to do this is to back up the wallet (https://www.multichain.com/developers/backing-up-restoring-nodes/)

---

Thanks for your help. Much appreciated. I did not understand mining-diversity=1 was as strict as it is when I did this back then. thanks again.

---

Unfortunately now its not returning any stream items.

https://www.multichain.com/qa/32542/streams-have-no-stream-items