Timeout downloading block from peer, disconnecting

Question

Hi,

I have a master node with 900+ blocks. Now I try to connect a new node, but when it starts syncing up with the master, it disconnects after some blocks complaining about a timeout:

receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=959, us=174.16.213.31:53738, peer=2
Added time data, samples 2, offset +0 (+0 minutes)
mchn: Parameter set from peer=2 verified
Pre-allocating up to position 0x100000 in rev00000.dat
UpdateTip:            new best=00f4c23c213136008518e74414f17605c024ad3cf34d81cb53f8130266ee815b  height=1  log2_work=9  tx=2  date=2019-01-28 08:48:07 progress=1.000000  cache=0
UpdateTip:            new best=0000b25e3031363046cc69e138f209868484bbd15f6fee3f0dc42b5f51be3565  height=2  log2_work=9.5849625  tx=3  date=2019-01-28 08:48:08 progress=1.000000  cache=0
...

...
UpdateTip:            new best=00113d9451141fc321bc88e5bc2f2388580f683fa3c9af4705a754485c3fbdac  height=126  log2_work=14.988685  tx=1284  date=2019-01-28 14:10:10 progress=1.000000  cache=0
UpdateTip:            new best=005f0110f6d32bf8e68a9598dd7533852a30cff92ec65341e25e55149f976eff  height=127  log2_work=15  tx=1540  date=2019-01-28 14:10:14 progress=1.000000  cache=0
UpdateTip:            new best=00f59a8a840365aa1c3bac5fe717753c07f4cf27f6e95926423531ab33488af8  height=128  log2_work=15.011227  tx=1760  date=2019-01-28 14:10:16 progress=1.000000  cache=0
ResendWalletTransactions()
UpdateTip:            new best=008a866cb66d2f98894fd98ebe2b7eb78f5d9ff5460d6e723ef0db38ba6757f2  height=129  log2_work=15.022368  tx=1908  date=2019-01-28 14:10:18 progress=1.000000  cache=0
UpdateTip:            new best=00e7442c16a2008681791394d95966db86e5155cee8668ff25566c332953d6f0  height=130  log2_work=15.033423  tx=2015  date=2019-01-28 14:10:20 progress=1.000000  cache=0
UpdateTip:            new best=0012883f0df88e470804ec4f5aefe78fbbfd0ced34b02737a321bb496b0f63d2  height=131  log2_work=15.044394  tx=2112  date=2019-01-28 14:10:23 progress=1.000000  cache=0
UpdateTip:            new best=00d26a69b99e592f5687a8704a217b87be3516f4f72dfdb80d734d939aa52663  height=132  log2_work=15.055282  tx=2273  date=2019-01-28 14:10:25 progress=1.000000  cache=0
UpdateTip:            new best=00a163113a22b08250698cd44280b44e4a7f952ddf9dbb9aa33f88a8c57741d9  height=133  log2_work=15.066089  tx=2410  date=2019-01-28 14:10:27 progress=1.000000  cache=0
UpdateTip:            new best=006862bab537d132727e6abc9f1adef8e2520c9d26d5d1f5f727d7ab24f4e8b4  height=134  log2_work=15.076816  tx=2521  date=2019-01-28 14:10:28 progress=1.000000  cache=0
UpdateTip:            new best=00c77418d1248a4118363b5df2c620023d2e3fd28ed6b83f3366eeed99a04118  height=135  log2_work=15.087463  tx=2613  date=2019-01-28 14:10:30 progress=1.000000  cache=0
ResendWalletTransactions()
UpdateTip:            new best=0081d6e61d0b716b1b318ddd48260d1c620be5b100fdad271e2732311fe7b049  height=136  log2_work=15.098032  tx=2789  date=2019-01-28 14:10:33 progress=1.000000  cache=0
Timeout downloading block 00e9dfcfc91067a07d32a80e241d944089d33a5aeea12335422970dd270dcbe1 from peer=2, disconnecting

What is this timeout about? Why does it happen? Is there any parameter that allows me to increase such timeout?

Thanks!

Comments

Also `getblockchaininfo` states that there are 959 blocks, but the node could only sync up with 136:

```
{
    "chain": "main",
    "chainname": "ipregister",
    "description": "IP Register",
    "protocol": "multichain",
    "setupblocks": 10,
    "reindex": false,
    "blocks": 136,
    "headers": 959,
    "bestblockhash": "0081d6e61d0b716b1b318ddd48260d1c620be5b100fdad271e2732311fe7b049",
    "difficulty": 5.96046447753906e-8,
    "verificationprogress": 1,
    "chainwork": "0000000000000000000000000000000000000000000000000000000000008e00",
    "chainrewards": 0
}
```

---

Can you please tell me the size of blocks and the lowest speed of your connection between nodes?

---

Blocks are around 1000 txs with low 1ms latency in local VLAN

---

My use case is that I'm importing data into a stream as JSON documents. Logs show:

MultiChainMiner: Block Found - 00d94e0967d1de29f629aee60a2258327e338ba80b058932d643b1aad8fb33dc, prev: 00d69eb55a6bdbabe0b5e9ef8e108d928133dd079c7e6416d6385670116f9eae, height: 40, txs: 745
MultiChainMiner: Block Found - 001ce0a25f81e7210b095835596f318c822da215a57bf25056234d866cf90999, prev: 00d94e0967d1de29f629aee60a2258327e338ba80b058932d643b1aad8fb33dc, height: 41, txs: 162
MultiChainMiner: Block Found - 0079672d33a259779c2e83a0c51b2471720e5cb79372f9aaaf239ea428dc5443, prev: 001ce0a25f81e7210b095835596f318c822da215a57bf25056234d866cf90999, height: 42, txs: 151
MultiChainMiner: Block Found - 009fc9562d3d8fd6754705351e9566c87947497924fd54e45556839ca518989f, prev: 0079672d33a259779c2e83a0c51b2471720e5cb79372f9aaaf239ea428dc5443, height: 43, txs: 109
...
MultiChainMiner: Block Found - 005152736b33af48cfed2a9b64b73df04971f34d8dd5456e0800cb97d3a9dbb2, prev: 006faa229fc13ef54f28ea4571d2f938f02d448f49ce0f3e4bd2b7ef8722daf5, height: 48, txs: 333
MultiChainMiner: Block Found - 008028254e1869c0e5eed8aa37c5e38403703c12ede670fd5a9f4e363ff38b5b, prev: 005152736b33af48cfed2a9b64b73df04971f34d8dd5456e0800cb97d3a9dbb2, height: 49, txs: 124
MultiChainMiner: Block Found - 0055b8801cec866fd50f115928799e901c8e76cbefa30775e74481e3a78bbec6, prev: 008028254e1869c0e5eed8aa37c5e38403703c12ede670fd5a9f4e363ff38b5b, height: 50, txs: 5
MultiChainMiner: Block Found - 008e573e4d15f3bc7d4540a86086e85c4557c51fb337b4aa647c8c476d4d4848, prev: 0055b8801cec866fd50f115928799e901c8e76cbefa30775e74481e3a78bbec6, height: 51, txs: 18
MultiChainMiner: Block Found - 0088163c6c2e8398723131d13c0e84e1c2305ad5ffdd731863ecf97361d6c9c0, prev: 008e573e4d15f3bc7d4540a86086e85c4557c51fb337b4aa647c8c476d4d4848, height: 52, txs: 204
MultiChainMiner: Block Found - 00e67edf2d21d3c9590c1cd124b46367f55e87d34beab7fd7eb9396dd43a031f, prev: 0088163c6c2e8398723131d13c0e84e1c2305ad5ffdd731863ecf97361d6c9c0, height: 53, txs: 138
MultiChainMiner: Block Found - 00fc913a374074db7b2cd6b209646e0e117a06a12b60983b14d4d2e33410eeec, prev: 00e67edf2d21d3c9590c1cd124b46367f55e87d34beab7fd7eb9396dd43a031f, height: 54, txs: 249
MultiChainMiner: Block Found - 00d90356b5d85ea55f7e0c4050201370b94b4666ca4f3efc0bfaa4a79825fcfe, prev: 00fc913a374074db7b2cd6b209646e0e117a06a12b60983b14d4d2e33410eeec, height: 55, txs: 73
MultiChainMiner: Block Found - 0061ac2e2aad8fa1dc1928cb1bd9238d39b3c853ef1b2ace0614ab4e7912f302, prev: 00d90356b5d85ea55f7e0c4050201370b94b4666ca4f3efc0bfaa4a79825fcfe, height: 56, txs: 233
MultiChainMiner: Block Found - 005b8f07754b68afd4e660b6d64bcd2e2daafeaed73040197d866b9f10214325, prev: 0061ac2e2aad8fa1dc1928cb1bd9238d39b3c853ef1b2ace0614ab4e7912f302, height: 57, txs: 26
MultiChainMiner: Block Found - 00228af9f41c1d8886fbe900e699465710226f046c3d7fe8ce32bca04e31b23a, prev: 005b8f07754b68afd4e660b6d64bcd2e2daafeaed73040197d866b9f10214325, height: 58, txs: 145
MultiChainMiner: Block Found - 00f163e41a6bb6461fbb9427e5c36ea5f954f8eaa77f51c2733530f05243a844, prev: 00228af9f41c1d8886fbe900e699465710226f046c3d7fe8ce32bca04e31b23a, height: 59, txs: 320
MultiChainMiner: Block Found - 00e14b44e1c360df8b0c1c2cef10728efbe75f4e97a90742be00b32cabcd302d, prev: 00f163e41a6bb6461fbb9427e5c36ea5f954f8eaa77f51c2733530f05243a844, height: 60, txs: 15
MultiChainMiner: Block Found - 008763023a7d93b440f5a62ac3d12bec9639d52f3936ec73364ebd85461920a8, prev: 00e14b44e1c360df8b0c1c2cef10728efbe75f4e97a90742be00b32cabcd302d, height: 61, txs: 190
MultiChainMiner: Block Found - 00ac4d7ac3ade8698f8da2d1567b109f567ea3e5b3de8eb3a74f49d1e900883a, prev: 008763023a7d93b440f5a62ac3d12bec9639d52f3936ec73364ebd85461920a8, height: 62, txs: 5
MultiChainMiner: Block Found - 004f9a195eb292342782bba6aa29fc131cd44f8611a89ef135c704f83333ebd0, prev: 00ac4d7ac3ade8698f8da2d1567b109f567ea3e5b3de8eb3a74f49d1e900883a, height: 63, txs: 242
MultiChainMiner: Block Found - 00bc61ef974452af75ee4f694724441facd8a694b213d3893f1e5187328a4a6d, prev: 004f9a195eb292342782bba6aa29fc131cd44f8611a89ef135c704f83333ebd0, height: 64, txs: 8
MultiChainMiner: Block Found - 0011c82654aafd41463c3148762b87f7860c9bfa8261e57ffa2ba181d4032b30, prev: 00bc61ef974452af75ee4f694724441facd8a694b213d3893f1e5187328a4a6d, height: 65, txs: 238
MultiChainMiner: Block Found - 00f5e930387af9109762479310b953968d3c10bf9567cd2c48d8a599d1534913, prev: 0011c82654aafd41463c3148762b87f7860c9bfa8261e57ffa2ba181d4032b30, height: 66, txs: 108
MultiChainMiner: Block Found - 008f58c6fbe4c14297fadc030d76d9a019f4ab34539f62cee554e123beeee4dc, prev: 00f5e930387af9109762479310b953968d3c10bf9567cd2c48d8a599d1534913, height: 67, txs: 114
MultiChainMiner: Block Found - 006fe17f133c64e3d1f4b07a16f114dddd836693b3b6a1870725df408ba8ab08, prev: 008f58c6fbe4c14297fadc030d76d9a019f4ab34539f62cee554e123beeee4dc, height: 68, txs: 172
MultiChainMiner: Block Found - 00c30a2b2deb0c0f7c8b9e4f6bdfca638a81e7bf89abdbf2d30815346be81ec4, prev: 006fe17f133c64e3d1f4b07a16f114dddd836693b3b6a1870725df408ba8ab08, height: 69, txs: 176
MultiChainMiner: Block Found - 00e04cc212d652aece138e5303e3667b5b0f501d0857605ab4426d54a7b8ff35, prev: 00c30a2b2deb0c0f7c8b9e4f6bdfca638a81e7bf89abdbf2d30815346be81ec4, height: 70, txs: 435
MultiChainMiner: Block Found - 00097f020ece9ba426ff22f949387c0d4e577edb9d35e578c4d137e9f9b7ddfc, prev: 00e04cc212d652aece138e5303e3667b5b0f501d0857605ab4426d54a7b8ff35, height: 71, txs: 155
MultiChainMiner: Block Found - 005884e75348bea7dd35ffb5450755902d46271a24525e964f8cf089a303d7d4, prev: 00097f020ece9ba426ff22f949387c0d4e577edb9d35e578c4d137e9f9b7ddfc, height: 72, txs: 172
MultiChainMiner: Block Found - 009e86f2a9b8e63b06d14a781afabf2ffd570ad5790ec2b0090b2be880856f3d, prev: 005884e75348bea7dd35ffb5450755902d46271a24525e964f8cf089a303d7d4, height: 73, txs: 185
MultiChainMiner: Block Found - 0010ad9404b75e79ada7a87a95e5aee208aadcf3ceb1e4a269b3914aaaf0bcad, prev: 009e86f2a9b8e63b06d14a781afabf2ffd570ad5790ec2b0090b2be880856f3d, height: 74, txs: 153
MultiChainMiner: Block Found - 00e7ee53cfdfae924095aff8599e83510ef571f35639063df010c00466fc41a4, prev: 0010ad9404b75e79ada7a87a95e5aee208aadcf3ceb1e4a269b3914aaaf0bcad, height: 75, txs: 496
MultiChainMiner: Block Found - 0097c168e6e9ac4dcf2a1aa0e022fbe3118bf760bb05d847523432adf622e160, prev: 00e7ee53cfdfae924095aff8599e83510ef571f35639063df010c00466fc41a4, height: 76, txs: 174
MultiChainMiner: Block Found - 0009253c7c20366f4596a1aa7e112383c92166563c20921cc2790e4e4e39c234, prev: 0097c168e6e9ac4dcf2a1aa0e022fbe3118bf760bb05d847523432adf622e160, height: 77, txs: 210
MultiChainMiner: Block Found - 0063f025b442b95573d92f3575a27000379da6d803b4afc5ac676052f085c404, prev: 0009253c7c20366f4596a1aa7e112383c92166563c20921cc2790e4e4e39c234, height: 78, txs: 91
MultiChainMiner: Block Found - 0087cb8b9882695c7b6f184f3d189b42e73a470543593f0db81e47492edd7bf3, prev: 0063f025b442b95573d92f3575a27000379da6d803b4afc5ac676052f085c404, height: 79, txs: 475
MultiChainMiner: Block Found - 00e94d0d02d852d3aee18d97cffefae0cefe20ca369d0e3afd7ac51fe50899c0, prev: 0087cb8b9882695c7b6f184f3d189b42e73a470543593f0db81e47492edd7bf3, height: 80, txs: 416
MultiChainMiner: Block Found - 00182cd9b71aa1c476f93e951033b39c5b9bedf62b5bd5b59a7efab46f294946, prev: 00e94d0d02d852d3aee18d97cffefae0cefe20ca369d0e3afd7ac51fe50899c0, height: 81, txs: 573
MultiChainMiner: Block Found - 0037e3da20a2cb12301d314b71ffa14981f54602d6a62124187e8de84abafdc7, prev: 00182cd9b71aa1c476f93e951033b39c5b9bedf62b5bd5b59a7efab46f294946, height: 82, txs: 329
MultiChainMiner: Block Found - 00b3f0094fc99ff3205ba25dc576853bbd486c5af1807f3912ed18bd00c4accf, prev: 0037e3da20a2cb12301d314b71ffa14981f54602d6a62124187e8de84abafdc7, height: 83, txs: 438
MultiChainMiner: Block Found - 009d2e4fb48a150512b1beb05fdbe2ffbdfb3f9434018c4a73ce93cf731eda76, prev: 00b3f0094fc99ff3205ba25dc576853bbd486c5af1807f3912ed18bd00c4accf, height: 84, txs: 920

Let me know if I can provide any more information.

---

Also sometimes I see this in the logs and then the client node becomes totally unresponsive:

socket sending timeout: 61s
Still rescanning. At block 25. Progress=1.000000

And then the master node shows:

socket send error Connection reset by peer (104)

---

Also if I stop the client node, and then start it up without -reindex, it tries to catch up from the first pending block, but always fails with the timeout, even after some few seconds:

MultiChainMiner started
opencon thread start
msghand thread start
addcon thread start
dumpaddr thread start
Node started
Seed node is set, trying to connect to ochlt-dck301:31187...
mchn: Sending minimal parameter set to 10.136.106.51:31187
receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=297, us=10.136.106.51:64222, peer=1
Added time data, samples 2, offset +0 (+0 minutes)
mchn: Parameter set from peer=1 verified
UpdateTip:            new best=002a7d646ea44984c9ae47d52f8a1c7d6a3781f9201b4afc31cb506488f3f8ed  height=29  log2_work=12.906891  tx=3957  date=2019-01-31 09:07:10 progress=1.000000  cache=0
UpdateTip:            new best=00cf3053c8f7215d219fad3a717da864d3a612fac9213e4cbede9565186e731c  height=30  log2_work=12.954196  tx=4460  date=2019-01-31 09:07:14 progress=1.000000  cache=0
UpdateTip:            new best=0062f2a269a361cf0d6913d7110166d1adfa0d96a7d0b6c4c5ada536e28dd944  height=31  log2_work=13  tx=4851  date=2019-01-31 09:07:16 progress=1.000000  cache=0
ResendWalletTransactions()
UpdateTip:            new best=00d9cb1657363e2c622d486ebd3d7057d8a05ce777dbe574c0c49d5e44aeb6df  height=32  log2_work=13.044394  tx=5179  date=2019-01-31 09:07:18 progress=1.000000  cache=0
UpdateTip:            new best=0005db3114b590a114719eb319bada245021e6f4dd3c78929b8358d79e329c89  height=33  log2_work=13.087463  tx=6094  date=2019-01-31 09:07:23 progress=1.000000  cache=0
UpdateTip:            new best=0067a5547c14c69f3285983adb6c8be899ce9721d94c1744d4d6956b62973c7b  height=34  log2_work=13.129283  tx=6872  date=2019-01-31 09:07:28 progress=1.000000  cache=0
UpdateTip:            new best=005c5b8c6dc04fb27b4307d76442e0b09d974a8d192a186fc8fbf12ce3509b00  height=35  log2_work=13.169925  tx=7149  date=2019-01-31 09:07:30 progress=1.000000  cache=0
ResendWalletTransactions()
UpdateTip:            new best=006d3b4c41970b3049d712a561b5d65e427d2bb278bca53cffe1459cd70226d7  height=36  log2_work=13.209453  tx=7349  date=2019-01-31 09:07:31 progress=1.000000  cache=0
UpdateTip:            new best=00391dd6e037cebad1a4343a6631494144e2c129c99820ab717571230318870c  height=37  log2_work=13.247928  tx=7703  date=2019-01-31 09:07:33 progress=1.000000  cache=0
UpdateTip:            new best=007d50b59310e30bc3b184ae533aea46a2b3b874f53bfabd620e1df431202e3d  height=38  log2_work=13.285402  tx=8193  date=2019-01-31 09:07:36 progress=1.000000  cache=0
UpdateTip:            new best=00609b4df4c9be0c37b2b161085dd178cc62871716c7ad256f349e5b5dbe2d43  height=39  log2_work=13.321928  tx=8251  date=2019-01-31 09:07:37 progress=1.000000  cache=0
UpdateTip:            new best=00702ebe825cab092fd06fcd8da3f5ea85956f167c7d6ae96442778310f7933c  height=40  log2_work=13.357552  tx=8260  date=2019-01-31 09:07:37 progress=1.000000  cache=0
UpdateTip:            new best=003bd0851986e566ba3350c09166a19021158376b2a424ac2dea752e5a05052a  height=41  log2_work=13.392317  tx=8817  date=2019-01-31 09:07:40 progress=1.000000  cache=0
UpdateTip:            new best=002a72fc80ede1a7658e8c2dff89b265c2e1526f093ace2403425ab84ead847c  height=42  log2_work=13.426265  tx=9186  date=2019-01-31 09:07:42 progress=1.000000  cache=0
UpdateTip:            new best=00b250f060c463327d523854dc9697c1a0ab0232eee1e3f8fb365a7283af9aa8  height=43  log2_work=13.459432  tx=9503  date=2019-01-31 09:07:45 progress=1.000000  cache=0
Timeout downloading block 00c1c792616342727f7a2afb49d1b466970bbecb7b7e9bf3eeeb31b3a5433d57 from peer=1, disconnecting

---

More feedback...

If I create the master node without importing any data, then join the client node, then start importing data through the master node in batches of 1000 (not 10.000 as before), then everything seems to be working properly, and the client node does not throw any timeout and can keep up with the master.

However, if I start a master node, then import data also in batches of 1000, then start up a client node, the client node cannot catch up with the master whatsoever due to constant timeout errors like the ones explained in the comments above this one.

IMHO, this is definitively a performance or capacity issue in Multichain.

What is your take on this from the Multichain's team?

---

We're taking a look and will get back to you shortly.

Answer 1

Based on your review, it looks like this is simple case of overloading the node that is doing all of the mining, so it doesn't manage to deliver the blocks to clients while everything else is going on. Like any software MultiChain does have capacity limits. You may want to consider increasing your block time, and mining across multiple nodes. Also please take a look at these general performance tips here:

https://www.multichain.com/developers/performance-optimization/

Comments

Actually, it's the other way round: when I start up both nodes, master and client, at the same time, and then start to import massive amounts of data using the master, the client can keep up with all the blocks without issues, as it basically updates its local repository as they are mined by the master.

However, if I repeat the same process, but only with the master, and then once everything has been imported, I start up a client, in this scenario the client cannot catch up with the master due to the timeout issue, even when everything has already been mined by the master.

---

And there are no new transactions being added to the master at the same time that the client is unable to catch up?

---

No, it's just the client starting up.

Can I turn any logging which I can send to you with more details? For me it's easily reproducible.

---

Yes, please don't use the -printtoconsole option and then the log files should have timestamps in them. You can then post some extracts of the log files (from both nodes here) and we'll see if we can make progress from that.

---

Here are some log extracts with two timeouts, but if you could provide an email where I could send the complete logs, it would be great:

2019-02-06 08:53:54 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-02-06 08:53:54 mchn: Wallet optimization after block
2019-02-06 08:53:54 mchn: Wallet optimization
2019-02-06 08:53:54 mchn-block: Chain activated:   009eb62bf12e807c99d9760f7ce725a6434b0a41fbf01d55c67370bd23f3048c (height 12), can-mine: 0
2019-02-06 08:53:54 mchn-P: Block   12 (  11); Txs:   1001; T:    0.373; TX:    0.000 (   0.000%); BL:    0.361 (  96.832%);
2019-02-06 08:53:54 mchn-block: Received block:   0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038,  peer=2
2019-02-06 08:53:54 VerifyBlockMiner: Block: 13, Fork: 12, Chain: 12
2019-02-06 08:53:54 VerifyBlockMiner: Rolled back to block 12
2019-02-06 08:53:54 VerifyBlockMiner: Verifying block 13
2019-02-06 08:53:54 VerifyBlockMiner: New block 0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038 (height 13)
2019-02-06 08:53:54 VerifyBlockMiner: Restoring chain, block 13
2019-02-06 08:53:54 VerifyBlockMiner: Restored on block 12
2019-02-06 08:53:55 mchn-block: New block index:   0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038, prev: 009eb62bf12e807c99d9760f7ce725a6434b0a41fbf01d55c67370bd23f3048c, height: 13, mined-by-me: 0, can-mine: 0
2019-02-06 08:53:55 mchn: Connecting block 0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038 (height 13) ...
2019-02-06 08:53:55 mchn: Checking block signature and miner permissions...
2019-02-06 08:53:55 mchn: Checking Block with 1390 transactions
2019-02-06 08:53:55 mchn-miner: miner not found, height: 12, gap:    0.000s, miners: (tot: 1, max: 1, pool: 1)
2019-02-06 08:53:56 mchn: Committing permission changes for block 13...
2019-02-06 08:53:56 UpdateTip:            new best=0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038  height=13  log2_work=11.807355  tx=3403  date=2019-02-04 17:27:29 progress=1.000000  cache=0
2019-02-06 08:53:56 mchn-block: height: 13, miner: 1bPJ5rh5ENGCLofp37F9xg4GLPvjKxeKtGJY5R
2019-02-06 08:53:56 mchn: Replaying memory pool (0 new transactions, total 0)
2019-02-06 08:53:56 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-02-06 08:53:56 mchn: Wallet optimization after block
2019-02-06 08:53:56 mchn: Wallet optimization
2019-02-06 08:53:56 mchn-block: Chain activated:   0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038 (height 13), can-mine: 0
2019-02-06 08:53:56 mchn-P: Block   13 (  12); Txs:   1390; T:    0.507; TX:    0.000 (   0.000%); BL:    0.495 (  97.503%);
2019-02-06 08:53:56 ResendWalletTransactions()
2019-02-06 08:53:56 Timeout downloading block 00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11 from peer=2, disconnecting
2019-02-06 08:53:56 mchn-miner: miner not found, height: 13, gap:    0.000s, miners: (tot: 1, max: 1, pool: 1)
2019-02-06 08:53:58 mchn: Sending minimal parameter set to 10.136.106.13:8099
2019-02-06 08:53:58 receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=34, us=10.136.106.51:57271, peer=3
2019-02-06 08:53:58 mchn: Parameter set from peer=3 verified
2019-02-06 08:53:58 ResendWalletTransactions()
2019-02-06 08:53:58 mchn: Sent 3 known addresses
2019-02-06 08:53:58 mchn: received addr: 3
2019-02-06 08:53:58 mchn-block: Received headers: 34-34,  peer=3
2019-02-06 08:53:58 mchn-block: Received block:   00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11,  peer=3
2019-02-06 08:53:58 VerifyBlockMiner: Block: 14, Fork: 13, Chain: 13
2019-02-06 08:53:58 VerifyBlockMiner: Rolled back to block 13
2019-02-06 08:53:58 VerifyBlockMiner: Verifying block 14
2019-02-06 08:53:58 VerifyBlockMiner: New block 00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11 (height 14)
2019-02-06 08:53:58 VerifyBlockMiner: Restoring chain, block 14
2019-02-06 08:53:58 VerifyBlockMiner: Restored on block 13
2019-02-06 08:53:58 mchn-block: New block index:   00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11, prev: 0001aa087d382592c33f3c78c375d249fbff462bac196f10648829236c4c3038, height: 14, mined-by-me: 0, can-mine: 0
2019-02-06 08:53:58 mchn: Connecting block 00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11 (height 14) ...
2019-02-06 08:53:58 mchn: Checking block signature and miner permissions...
2019-02-06 08:53:58 mchn: Checking Block with 765 transactions
2019-02-06 08:53:59 mchn: Committing permission changes for block 14...
2019-02-06 08:53:59 UpdateTip:            new best=00d8e9a8bca986758e801ecb8190ac132f554c32917d1dde2c5f3440991c7a11  height=14  log2_work=11.906891  tx=4168  date=2019-02-04 17:27:32 progress=1.000000  cache=0
2019-02-06 08:53:59 mchn-block: height: 14, miner: 1bPJ5rh5ENGCLofp37F9xg4GLPvjKxeKtGJY5R
2019-02-06 08:53:59 mchn: Replaying memory pool (0 new transactions, total 0)
...
2019-02-06 08:54:04 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-02-06 08:54:04 mchn: Wallet optimization after block
2019-02-06 08:54:04 mchn: Wallet optimization
2019-02-06 08:54:04 mchn-block: Chain activated:   006e679dfba5859650a80ef23f40598f520898f8ab09bc666e74d03f090b91ec (height 22), can-mine: 0
2019-02-06 08:54:04 mchn-P: Block   22 (  21); Txs:    422; T:    0.962; TX:    0.000 (   0.000%); BL:    0.726 (  75.416%);
2019-02-06 08:54:04 Timeout downloading block 0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1 from peer=3, disconnecting
2019-02-06 08:54:05 mchn-miner: miner not found, height: 22, gap:    0.000s, miners: (tot: 1, max: 1, pool: 1)
2019-02-06 08:54:05 mchn: Sending minimal parameter set to 10.136.106.13:8099
2019-02-06 08:54:05 receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=34, us=174.16.212.0:42057, peer=4
2019-02-06 08:54:05 mchn: Parameter set from peer=4 verified
2019-02-06 08:54:05 mchn: Sent 3 known addresses
2019-02-06 08:54:05 mchn: received addr: 3
2019-02-06 08:54:05 mchn-block: Received headers: 34-34,  peer=4
2019-02-06 08:54:06 mchn-block: Received block:   0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1,  peer=4
2019-02-06 08:54:06 VerifyBlockMiner: Block: 23, Fork: 22, Chain: 22
2019-02-06 08:54:06 VerifyBlockMiner: Rolled back to block 22
2019-02-06 08:54:06 VerifyBlockMiner: Verifying block 23
2019-02-06 08:54:06 VerifyBlockMiner: New block 0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1 (height 23)
2019-02-06 08:54:06 VerifyBlockMiner: Restoring chain, block 23
2019-02-06 08:54:06 VerifyBlockMiner: Restored on block 22
2019-02-06 08:54:06 mchn-block: New block index:   0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1, prev: 006e679dfba5859650a80ef23f40598f520898f8ab09bc666e74d03f090b91ec, height: 23, mined-by-me: 0, can-mine: 0
2019-02-06 08:54:06 mchn: Connecting block 0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1 (height 23) ...
2019-02-06 08:54:06 mchn: Checking block signature and miner permissions...
2019-02-06 08:54:06 mchn: Checking Block with 1009 transactions
2019-02-06 08:54:07 mchn: Committing permission changes for block 23...
2019-02-06 08:54:07 UpdateTip:            new best=0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1  height=23  log2_work=12.584963  tx=8358  date=2019-02-04 17:27:49 progress=1.000000  cache=0
2019-02-06 08:54:07 mchn-block: height: 23, miner: 1bPJ5rh5ENGCLofp37F9xg4GLPvjKxeKtGJY5R
2019-02-06 08:54:07 mchn: Replaying memory pool (0 new transactions, total 0)
2019-02-06 08:54:07 ReacceptWalletTransactions: 0 txs in unconfirmed pool
2019-02-06 08:54:07 mchn: Wallet optimization after block
2019-02-06 08:54:07 mchn: Wallet optimization
2019-02-06 08:54:07 mchn-block: Chain activated:   0025ebc1010cbc7b8ead77fedb94377aeb6815bc15dd0517174963eb6fb3a5b1 (height 23), can-mine: 0

---

Thanks, please send full logs to multichain dot debug at gmail dot com.

---

More feedback...

If I just import a reduced set of data, then the clients can catch up with the master node, even if they are started up after the import process. However, as I start the client without the "-txindex", when I try to send a command to the blockchain, like subscribing to a stream, then I face the issue again, this time showing the following in the console:

multichain-cli.exe ipregister -rpcpassword=m4lt1ch41nRpC! -rpcuser=multichainrpc subscribe ipregister

...
socket sending timeout: 61s
Still rescanning. At block 13. Progress=1.000000
Still rescanning. At block 14. Progress=1.000000
Still rescanning. At block 15. Progress=1.000000
Still rescanning. At block 16. Progress=1.000000
Still rescanning. At block 19. Progress=1.000000
....

If I just wait until it reaches the last block, then my pending subscription command executes properly and everything works from now onwards:

...
Still rescanning. At block 24. Progress=1.000000
socket recv error An established connection was aborted by the software in your host machine.  (10053)
mchn: Sending minimal parameter set to 10.136.106.13:8099
receive version message: /MultiChain:0.2.0.6/: version 70002, blocks=36, us=174.16.87.0:41153, peer=3
mchn: Parameter set from peer=3 verified
mchn: Synced with node 3 on block 36 - requesting mempool

---

It's normal that during a blockchain rescan, new blocks temporarily won't come through.

---

Ok, thanks, I'll review my configuration settings and watch for performance issues.