Port number problem after a node's IP address was changed

Question

I have 3 servers on AWS, let's call them S1, S2 and S3, which are all on the same chain. S1 used IP address 10.0.1.250 and port 2775, S2 uses 10.0.1.233:2775 and S3 uses 10.,0.1.248:2775. The S1 instance crashed and had to be rebuilt, but this caused its IP address to change to 10.0.3.328. I have tried connecting S2 and S3 to S1, and S1 to S2 and S3 by using the addnode command, but updates I perform on S1 do not seem to be propagating to S2 and S3. S2 and S3 are still communicating with each other, but S1 appears to be cut off completely.

When I run getpeerinfo on each server I get peculiar results as I expect all the port numbers to be 2775, but sometimes they are totally different. Here is the output from S1:

C:\multichain>multichain-cli chain1 getpeerinfo
{"method":"getpeerinfo","params":[],"id":"87620777-1576427198","chain_name":"chain1"}

[
{
"id" : 4736,
"addr" : "10.0.1.233:2775",
"addrlocal" : "10.0.3.238:60924",
"services" : "0000000000000001",
"lastsend" : 1576427194,
"lastrecv" : 1576427194,
"bytessent" : 2648,
"bytesrecv" : 6428,
"conntime" : 1576427161,
"pingtime" : 0.09599,
"version" : 70002,
"subver" : "/MultiChain:0.2.0.11/",
"handshakelocal" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
"handshake" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
"inbound" : false,
"encrypted" : false,
"startingheight" : 1152,
"banscore" : 0,
"synced_headers" : 1152,
"synced_blocks" : 963,
"inflight" : [
964,
965,
966,
967,
968,
969,
970,
971,
972,
973,
974,
975,
976,
977,
978,
979
],
"whitelisted" : false
}
]

C:\multichain>

Here is the output from S2:

C:\multichain>multichain-cli chain1 getpeerinfo
{"method":"getpeerinfo","params":[],"id":"42786950-1576427533","chain_name":"chain1"}

[
{
"id" : 2,
"addr" : "10.0.1.248:2775",
"addrlocal" : "10.0.1.233:51214",
"services" : "0000000000000001",
"lastsend" : 1576427532,
"lastrecv" : 1576427532,
"bytessent" : 1146430,
"bytesrecv" : 1146518,
"conntime" : 1576248189,
"pingtime" : 0.046886,
"version" : 70002,
"subver" : "/MultiChain:0.2.0.11/",
"handshakelocal" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
"handshake" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
"inbound" : false,
"encrypted" : false,
"startingheight" : 1152,
"banscore" : 0,
"synced_headers" : 1152,
"synced_blocks" : 1152,
"inflight" : [
],
"whitelisted" : false
},
{
"id" : 2374,
"addr" : "10.0.3.238:61014",
"addrlocal" : "10.0.1.233:2775",
"services" : "0000000000000001",
"lastsend" : 1576427524,
"lastrecv" : 1576427524,
"bytessent" : 6976,
"bytesrecv" : 2840,
"conntime" : 1576427463,
"pingtime" : 0.062524,
"version" : 70002,
"subver" : "/MultiChain:0.2.0.11/",
"handshakelocal" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
"handshake" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
"inbound" : true,
"encrypted" : false,
"startingheight" : 1075,
"banscore" : 0,
"synced_headers" : -1,
"synced_blocks" : -1,
"inflight" : [
],
"whitelisted" : false
}
]

C:\multichain>

Here is the output from S3:

C:\multichain>multichain-cli chain1 getpeerinfo
{"method":"getpeerinfo","params":[],"id":"49302041-1576427671","chain_name":"chain1"}

[
{
"id" : 2,
"addr" : "10.0.1.233:51214",
"addrlocal" : "10.0.1.248:2775",
"services" : "0000000000000001",
"lastsend" : 1576427670,
"lastrecv" : 1576427671,
"bytessent" : 1147382,
"bytesrecv" : 1147294,
"conntime" : 1576248189,
"pingtime" : 0.03125,
"version" : 70002,
"subver" : "/MultiChain:0.2.0.11/",
"handshakelocal" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
"handshake" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
"inbound" : true,
"encrypted" : false,
"startingheight" : 1152,
"banscore" : 0,
"synced_headers" : 1152,
"synced_blocks" : 1152,
"inflight" : [
],
"whitelisted" : false
},
{
"id" : 2378,
"addr" : "10.0.3.238:61069",
"addrlocal" : "10.0.1.248:2775",
"services" : "0000000000000001",
"lastsend" : 1576427668,
"lastrecv" : 1576427668,
"bytessent" : 1219,
"bytesrecv" : 1218,
"conntime" : 1576427648,
"pingtime" : 0.062477,
"version" : 70002,
"subver" : "/MultiChain:0.2.0.11/",
"handshakelocal" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
"handshake" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
"inbound" : true,
"encrypted" : false,
"startingheight" : 1075,
"banscore" : 0,
"synced_headers" : -1,
"synced_blocks" : -1,
"inflight" : [
],
"whitelisted" : false
}
]

C:\multichain>

How can I get all 3 nodes to communicate with each other?

Answer 1

First, the surprising port numbers you're seeing are due to the fact that each connection between peers is initiated in one particular direction. The port number of the initiator will not be 2775, because it is an outgoing rather than a listening port, and cannot block incoming connections. You'll see that for any connection where inbound is false, the addr port is 2775. And for any connection where inbound is true, the addrlocal port is 2775.

Second, it looks like the nodes are successfully connected to each other. My best guess as to the problem is that the clocks on the nodes are too far out of sync with each other, so they are not accepting each other's data. The output of these API calls shows a skew of 5+ minutes.

Comments

That time difference was due to the delay between logging on to each server, running the getperrinfo command, copying the output. I have just logged onto each of the 3 servers and the clocks are all in perfect sync, so that cannot be the problem.

Is there any significance in the number of 'inflight' entries in the data for S1?

When I try to update the node labels for each of the 3 nodes the S2 and S3 nodes show each other's updates but not the change for S1. Server S1 still does not show any changes for S2 and S3.

I have tried using S1 to update the permissions of S2 and S3, but these all show as 'pending'.

The 3 nodes may be able to see each other, but there is no traffic between S1 and the other 2 nodes.

---

I've forwarded this to the dev team to take a closer look.

---

If you see "inflight" in getpeerinfo, it means S1is still not synchronized with S2 and S3. You can check also "synced_blocks" on all 3 nodes. When is this state, it is likely transactions sent by S1 will be ignored by S2/S3 and vice versa, transactions made by S2/S3 can be ignored by S1. For example, you will not be able to change permissions for S2/S3 from S1.

But this out-of-sync state is temporary. If, for some reason, it takes too much time, you can try to start multichaind with -debug=net and see in debug.log why the blocks are not processed normally.

Can you please also clarify what to you mean by "update the node labels"?

---

By "update the node label" I mean using the "publishfrom" command to change the label for a node's address.

I have run -debug=net on S1 and produced a log file, but it is too large to include in this textbox. Do you have an email address so that I can send it as an attachment?

---

Yes, please send it to multichain dot debug at gmail dot com.

---

Thanks - the file has been received and we are discussing internally. It looks like the issue was caused by the fact that the protocol was upgraded while the network was disconnected, and you have mining-diversity set low enough to allow a long-distance fork. Will confirm more details shortly.

Answer 2

Thanks for your patience. We have concluded that this is a bug in MultiChain that was triggered by the unusual case of the protocol being upgraded on a node while it was not in sync with the network. It will be fixed in the next maintenance release. In the meantime your workaround, assuming this is not a production network, is to destroy S1 and recreate it, attaching it to the network. Then perform the upgrade only once it is in sync. Please let me know if this resolves the situation.

Comments

I have destroyed S1, reinstalled the previous version of the software (1.0.7), recreated S1 (which has given it a new address)), attached it to the network and upgraded to the latest version (2.0.3). When I try to change the label I get the following error:

Function: setLabel, code: -716, message: No unspent outputs are available. Please send a transaction, with zero amount, to this node or address first and wait for its confirmation.

What does "send a transaction with zero amount" mean. The only screen I see with amounts is "Send Asset", but I don't have any assets.

---

An address cannot sent out a transaction until it has been sent something (even if no actual value) from another address. Usually this is not an issue because the act of granting a permission to an address does the job. But it seems like you are running an open network for testing without these permission grants. So instead just do something like this from the command-line of another node:

send <put-new-address-here> 0

---

I have created 3 closed streams called items, pubkeys and access to which all three nodes have been given write access. I have issued the send command to node S1 from both S2 and S3, but even an hour later I am still getting that -716 error.

---

If you run "listunspent 0" on S1 you'll see if it has seen this transaction. If not, you're back to the old problem of the nodes not being able to connect or sync.

---

When I try "listunspent 0" on node S1 it shows all the transactions which I have sent from S2 and S3. If I publish something on the "items" stream in S2 and S3 then they are visible to S1, but I cannot publish anything from S1. All I see is the same error:

Error: -716 No unspent outputs are available.Please send a transaction, with zero amount, to this node or address first and wait for its confirmation.

Can I do anything on S1 to confirm these transactions?

I have also noticed that while the two transactions I published from S2 and S2 are visible on all 3 nodes, they are still marked as "unconfirmed".

---

I'm not sure what is going wrong here, this is a simple scenario so probably something simple is forgotten. Please post the output of running the following commands on S1:

getblockchainparams
listaddresses
listunspent 0
listpermissions

---

C:\multichain>multichain-cli chain1 getblockchainparams
{"method":"getblockchainparams","params":[],"id":"85821710-1577601926","chain_name":"chain1"}

{
    "chain-protocol" : "multichain",
    "chain-description" : "MultiChain chain1",
    "root-stream-name" : "root",
    "root-stream-open" : true,
    "chain-is-testnet" : false,
    "target-block-time" : 15,
    "maximum-block-size" : 8388608,
    "maximum-chunk-size" : 1048576,
    "maximum-chunk-count" : 1024,
    "default-network-port" : 2775,
    "default-rpc-port" : 2774,
    "anyone-can-connect" : false,
    "anyone-can-send" : false,
    "anyone-can-receive" : false,
    "anyone-can-receive-empty" : true,
    "anyone-can-create" : false,
    "anyone-can-issue" : false,
    "anyone-can-mine" : false,
    "anyone-can-activate" : false,
    "anyone-can-admin" : false,
    "support-miner-precheck" : true,
    "allow-arbitrary-outputs" : false,
    "allow-p2sh-outputs" : true,
    "allow-multisig-outputs" : true,
    "setup-first-blocks" : 60,
    "mining-diversity" : 0.3,
    "admin-consensus-upgrade" : 0.5,
    "admin-consensus-txfilter" : 0.5,
    "admin-consensus-admin" : 0.5,
    "admin-consensus-activate" : 0.5,
    "admin-consensus-mine" : 0.5,
    "admin-consensus-create" : 0,
    "admin-consensus-issue" : 0,
    "lock-admin-mine-rounds" : 10,
    "mining-requires-peers" : true,
    "mine-empty-rounds" : 10,
    "mining-turnover" : 0.5,
    "first-block-reward" : -1,
    "initial-block-reward" : 0,
    "reward-halving-interval" : 52560000,
    "reward-spendable-delay" : 1,
    "minimum-per-output" : 0,
    "maximum-per-output" : 100000000000000,
    "minimum-offchain-fee" : 0,
    "minimum-relay-fee" : 0,
    "native-currency-multiple" : 100000000,
    "skip-pow-check" : false,
    "pow-minimum-bits" : 8,
    "target-adjust-freq" : -1,
    "allow-min-difficulty-blocks" : false,
    "only-accept-std-txs" : true,
    "max-std-tx-size" : 4194304,
    "max-std-op-returns-count" : 10,
    "max-std-op-return-size" : 2097152,
    "max-std-op-drops-count" : 5,
    "max-std-element-size" : 8192,
    "chain-name" : "chain1",
    "protocol-version" : 20011,
    "network-message-start" : "f6c6e3f8",
    "address-pubkeyhash-version" : "00947a4c",
    "address-scripthash-version" : "057056d6",
    "private-key-version" : "80be6233",
    "address-checksum-value" : "925da43c",
    "genesis-pubkey" : "033f63f59c22a67d5df6eff72c73d19714a45ad393b9b7f628de7f7bfcd1f15f6b",
    "genesis-version" : 1,
    "genesis-timestamp" : 1508375844,
    "genesis-nbits" : 536936447,
    "genesis-nonce" : 189,
    "genesis-pubkey-hash" : "60f299fb28b49dc82f81bb1f9d01f4efbaee2321",
    "genesis-hash" : "0098346b2a9428e44283bb8462cc9c306118adf305fda21663db3fbeff9808f0",
    "chain-params-hash" : "800f7c30cd15242fac1b7a732217759530967df0c53fdc743513cf327f80edce"
}

C:\multichain>multichain-cli chain1 listaddresses
{"method":"listaddresses","params":[],"id":"16171758-1577602067","chain_name":"chain1"}

[
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "ismine" : true
    }
]

---

C:\multichain>multichain-cli chain1 listunspent 0
{"method":"listunspent","params":[0],"id":"55284279-1577602108","chain_name":"chain1"}

[
    {
        "txid" : "355b820d6df5eda3643c19cd984f9c297dc004bfff309938490477cbf2eca436",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "558ced337a61dcd69a5cde24b6ff2895657c8ea895e04c693b77539147ba57a9",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "827f866906562bfc89a423381e97738819856fd1c043cd8657edfba42b3c67ba",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b703731000000000000ffffffffd0dd015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : true,
                "issue" : true,
                "mine" : true,
                "admin" : true,
                "activate" : true,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577180624
            }
        ]
    },
    {
        "txid" : "88acd749ec9c4689cea8e956008ada9efdb7f11d187f36af6f7bfb752a337d14",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "903b3871280af540c1148a487d13e35c6c3c04920070182d3f78313f6497ae58",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b700700000000000000ffffffff12cd015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : false,
                "issue" : false,
                "mine" : false,
                "admin" : false,
                "activate" : false,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577176338
            }
        ]
    },
    {
        "txid" : "b5ce0b2219424c3fce28951a4833bf49f6f236f6fa6859d1da5faca9122ac92c",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b703731000000000000ffffffffcdde015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : true,
                "issue" : true,
                "mine" : true,
                "admin" : true,
                "activate" : true,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577180877
            }
        ]
    },
    {
        "txid" : "f2426d8285dc1b4f0193a706288a977d2acc7ae4ebebfbbf24086e2d6f3ce5b4",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    }
]

---

C:\multichain>multichain-cli chain1 listpermissions
{"method":"listpermissions","params":[],"id":"43503830-1577602190","chain_name":"chain1"}

[
    {
        "address" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
        "for" : null,
        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
        "for" : null,
        "type" : "send",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6",
        "for" : null,
        "type" : "receive",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "send",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "receive",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "create",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "send",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "receive",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "create",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "activate",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
        "for" : null,
        "type" : "issue",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "issue",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "activate",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "send",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "receive",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "create",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "issue",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "activate",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 4294967295
    },
    {
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295
    }
]

---

Thanks. Everything looks in order there. Can you please post the exact command you are using to try to "change the label" of S1? If you are using the Web Demo you can uncomment line 41 in the functions.php file and this will be displayed in the web browser.

---

These are the three lines which are displayed:

{"id":1577687394,"method":"publishfrom","params":["1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL","root","","31302e302e332e323338202842433129"]}

Error: -716 No unspent outputs are available. Please send a transaction, with zero amount, to this node or address first and wait for its confirmation.

{"id":1577687394,"method":"liststreampublishers","params":["root","*",true,10000]}

---

OK, and if you run this via multichain-cli do you see the same error:

publishfrom 1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL root '' 31302e302e332e323338202842433129

---

Yes. This is the output:

C:\multichain>multichain-cli chain1 publishfrom 1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL root '' 31302e302e332e323338202842433129
{"method":"publishfrom","params":["1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL","root","''","31302e302e332e323338202842433129"],"id":"56179692-1577861676","chain_name":"chain1"}

error code: -716
error message:
No unspent outputs are available. Please send a transaction, with zero amount, to this node or address first and wait for its confirmation.

C:\multichain>

---

Thanks for your patience. I'm referring this to the dev team to take a look and will be in touch shortly.

---

Hi, can you please again stop the node, then zip up the blockchain directory for that node, and send it to us at multichain dot debug at gmail dot com. Thanks!

---

We used the files you sent, and see empty output for "listunspent 0". Is it possible you have two nodes running, and are not talking to the one you expect?

---

I have 3 nodes which I shall call S1, S2 and S3.
S1 - IP address 10.0.3.238, node address 1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL
S2 - IP address 10.0.1.233, node address 1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz
S3 - IP address 10.0.1.248, node address 1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE

S1 was originally 10.0.1.250 and 1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6, but got corrupted and had to be rebuilt, which resulted in a new IP address. When I added it to the exiting network it was given a new node address.

I sent zero amounts from nodes S2 and S3 to S1, but they have never been confirmed.

---

OK, but you posted the output of some API calls on this thread. When we ran the same API calls on a node configured from your blockchain directory we got a different answer. This suggests that, as part of the blockchain sorting itself out after the fork (caused by the upgrade bug), those transactions that you had sent, and you had seen, were lost. You can confirm this by re-running "listunspent 0" on S1.

If all your nodes are now synchronized and connected to each other (see getpeerinfo and getblockchaininfo) you could try sending a zero amount to S1 again, and then it should work fine.

---

This is the result of "listunspent 0" on node S1:

C:\multichain>multichain-cli chain1 listunspent 0
{"method":"listunspent","params":[0],"id":"10181279-1578121495","chain_name":"chain1"}

[
    {
        "txid" : "355b820d6df5eda3643c19cd984f9c297dc004bfff309938490477cbf2eca436",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "558ced337a61dcd69a5cde24b6ff2895657c8ea895e04c693b77539147ba57a9",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "827f866906562bfc89a423381e97738819856fd1c043cd8657edfba42b3c67ba",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b703731000000000000ffffffffd0dd015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : true,
                "issue" : true,
                "mine" : true,
                "admin" : true,
                "activate" : true,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577180624
            }
        ]
    },
    {
        "txid" : "88acd749ec9c4689cea8e956008ada9efdb7f11d187f36af6f7bfb752a337d14",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    },
    {
        "txid" : "903b3871280af540c1148a487d13e35c6c3c04920070182d3f78313f6497ae58",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b700700000000000000ffffffff12cd015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : false,
                "issue" : false,
                "mine" : false,
                "admin" : false,
                "activate" : false,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577176338
            }
        ]
    },
    {
        "txid" : "b5ce0b2219424c3fce28951a4833bf49f6f236f6fa6859d1da5faca9122ac92c",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac1473706b703731000000000000ffffffffcdde015e75",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
            {
                "for" : null,
                "connect" : true,
                "send" : true,
                "receive" : true,
                "create" : true,
                "issue" : true,
                "mine" : true,
                "admin" : true,
                "activate" : true,
                "custom" : [
                ],
                "startblock" : 0,
                "endblock" : 4294967295,
                "timestamp" : 1577180877
            }
        ]
    },
    {
        "txid" : "f2426d8285dc1b4f0193a706288a977d2acc7ae4ebebfbbf24086e2d6f3ce5b4",
        "vout" : 0,
        "address" : "1AQ9wsU9dewdzcSgXcfT57aUwSpb7QX3Pg8qdL",
        "account" : "",
        "scriptPubKey" : "76a914458b47d7b0f1d29ceead9fd19705591b38a3ca3788ac",
        "amount" : 0,
        "confirmations" : 0,
        "cansend" : true,
        "spendable" : true,
        "assets" : [
        ],
        "permissions" : [
        ]
    }
]

C:\multichain>

---

Note that if I stop the multichain daemon on nodes S2 and S3 I get exactly the same result. All those transactions I created since rebuilding node S1 remain unconfirmed.

---

So we installed the directory you sent us from S1, and ran the same API commands, and see a different output. This is what is confusing, since the API output is determined entirely by the node's state, and that state is determined entirely by the blockchain directory's contents.

Is S1's blockchain synchronized with S2 and S3? You can verify this by running getblockchaininfo on each node and comparing the "blocks" and "bestblockhash" fields.

---

Here is the output from node S1:

C:\multichain>multichain-cli chain1 getblockchaininfo
{"method":"getblockchaininfo","params":[],"id":"50766014-1578402040","chain_name":"chain1"}

{
    "chain" : "main",
    "chainname" : "chain1",
    "description" : "MultiChain chain1",
    "protocol" : "multichain",
    "setupblocks" : 60,
    "reindex" : false,
    "blocks" : 1152,
    "headers" : 1152,
    "bestblockhash" : "00d230e03d6af4a76e1aa274ede17fdab2a9eb4b997dbc38ba46111229e64ebe",
    "difficulty" : 5.96046447753906e-8,
    "verificationprogress" : 1,
    "chainwork" : "0000000000000000000000000000000000000000000000000000000000048100",
    "chainrewards" : 0
}

C:\multichain>

---

Here is the output from node S2:

C:\multichain>multichain-cli chain1 getblockchaininfo
{"method":"getblockchaininfo","params":[],"id":"18286690-1578402098","chain_name":"chain1"}

{
    "chain" : "main",
    "chainname" : "chain1",
    "description" : "MultiChain chain1",
    "protocol" : "multichain",
    "setupblocks" : 60,
    "reindex" : false,
    "blocks" : 1152,
    "headers" : 1152,
    "bestblockhash" : "00d230e03d6af4a76e1aa274ede17fdab2a9eb4b997dbc38ba46111229e64ebe",
    "difficulty" : 5.96046447753906e-8,
    "verificationprogress" : 1,
    "chainwork" : "0000000000000000000000000000000000000000000000000000000000048100",
    "chainrewards" : 0
}

C:\multichain>

---

Here is the output from node S3:

C:\multichain>multichain-cli chain1 getblockchaininfo
{"method":"getblockchaininfo","params":[],"id":"15886104-1578402130","chain_name":"chain1"}

{
    "chain" : "main",
    "chainname" : "chain1",
    "description" : "MultiChain chain1",
    "protocol" : "multichain",
    "setupblocks" : 60,
    "reindex" : false,
    "blocks" : 1152,
    "headers" : 1152,
    "bestblockhash" : "00d230e03d6af4a76e1aa274ede17fdab2a9eb4b997dbc38ba46111229e64ebe",
    "difficulty" : 5.96046447753906e-8,
    "verificationprogress" : 1,
    "chainwork" : "0000000000000000000000000000000000000000000000000000000000048100",
    "chainrewards" : 0
}

C:\multichain>

---

I have published entries on the "items" stream from both node S2 and S3, and these show up on all three nodes as unconfirmed. The fact that they are visible on all three nodes means that all three nodes are connected to the same chain, but the problem appears to be that they are not being confirmed.

---

OK, it seems that in general your chain/nodes got in a muddle because of this upgrade problem. From our perspective it would be ideal if you could give us access to the three nodes to understand in detail what happened, but I don't know if this is an option.

Apart from that, assuming you're still in development, I would recommend just starting a new chain from scratch, and that will clear up all this mess.

---

I cannot give you access to these three nodes as they are behind a password-protected VPN. All I could do is provide a copy of the multichain directory from each one.

---

OK, I'll check if we can work with this and revert.

---

Sure, send them over please and we'll take a look.

---

We checked the directories you sent to us. The problem is - your chain doesn't have miner node. I suppose it was S1 (the first node you created), but after the crash, address with mining permission was permanently lost. MultiChain doesn't allow  spending of unconfirmed outputs sent from other nodes (S2/S3) and since these outputs are never confirmed, you cannot publish from S1.

To fix the problem, you should grant mining permission to one of the addresses, say, S2. Please note that you have 3 admins in your chain - S2, S3 and "lost S1". You need at least two of them to grant mining permissions, so you should do this from both S2 and S3:

grant  1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz mine

When first block is mined (after several seconds), you should be able to see outputs in 'listunspent 1' and publish from S1

If you want to continue to use this chain, I would also suggest you to revoke permissions from the "lost S1" address:

On S2 and S3:

revoke 1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6 admin,mine

---

Thanks for that suggestion, but I have deleted all the multichain directories on all 3 nodes and restarted from scratch. Everything is now working as expected.

---

I have tried restoring the previous data and followed your suggestions, but any change to permissions on any node is marked as "pending", as shown in the following:

C:\multichain>multichain-cli chain1 listpermissions * 1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE true
{"method":"listpermissions","params":["*","1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",true],"id":"11576586-1579174595","chain_name":"chain1"}

[
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "connect",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
        ],
        "pending" : [
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "send",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
        ],
        "pending" : [
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "receive",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
        ],
        "pending" : [
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "issue",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
        ],
        "pending" : [
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "create",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
        ],
        "pending" : [
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "mine",
        "startblock" : 0,
        "endblock" : 0,
        "admins" : [
            "1Qon9APHsiY89PV4J5k2CRqmRGzTv1RqUgpTbz",
            "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6"
        ],
        "pending" : [
            {
                "startblock" : 0,
                "endblock" : 4294967295,
                "admins" : [
                    "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
                ],
                "required" : 1
            }
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "admin",
        "startblock" : 0,
        "endblock" : 4294967295,
        "admins" : [
            "1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6"
        ],
        "pending" : [
            {
                "startblock" : 0,
                "endblock" : 4294967295,
                "admins" : [
                    "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
                ],
                "required" : 1
            }
        ]
    },
    {
        "address" : "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE",
        "for" : null,
        "type" : "activate",
        "startblock" : 0,
        "endblock" : 0,
        "admins" : [
        ],
        "pending" : [
            {
                "startblock" : 0,
                "endblock" : 4294967295,
                "admins" : [
                    "1VYiJ7xrc9X1cZxyPCwXf93x6bEpZ9xT7xjsLE"
                ],
                "required" : 1
            }
        ]
    }
]

C:\multichain>

---

Sorry, my mistake. It is impossible to restore the chain if the private key of the only miner (1E6yHM3qBbh7WBL2X8whecLJch6cB5eh9WcvX6) is lost. This is because any new miner grant should be confirmed in block and there is no miner in the chain to confirm this.

I'm afraid, unless you backed up wallet of S1 before crash, the chain is useless. You can see the data published on that chain, but you cannot mine new blocks.