Chain stuck on loading addresses for a long time

Question

I have been trying to restart a node and it has been stuck on loading addresses for quite a long time now about 20+ minutes, one of the nodes wasn't responding to cli or rpc calls and I tried to restart it, the node has been in loading address state for a quite long time now. The debug log file also had different entries than the usual ones.

2021-06-27 13:12:56 CommitTransaction: aa72c9cd6873052eaf91d32ff2da47e9221595355e1e3f76cefda35c7ed98edf, vin: 1, vout: 2
2021-06-27 16:56:07 CommitTransaction: a4428818ab3c62f7120e7e114cc263e1242aa82fc80922a1e57b45ae227b3145, vin: 1, vout: 2
2021-06-27 18:17:27 CommitTransaction: d8e8606cd1ea65cdfe91f5847e8c0f3164327aaedb7f4ba4bcb6ee6fa15b2a68, vin: 1, vout: 2
2021-06-28 04:18:45 CommitTransaction: 5f4620b7517473dd51cbfb1918a42034091cf16fd7ad404d553a97a16311ade9, vin: 1, vout: 2
2021-06-28 04:19:09 CommitTransaction: 82af0f23752a4ea7c27d6e78b56cc5f41d9cee7656d06eab230826e3090dd79e, vin: 1, vout: 2
2021-06-28 04:35:16 CommitTransaction: 7f51ac9df331b45ece6d62470bac977088be6c6a16fe2d902ca4195e18e45180, vin: 1, vout: 2
2021-06-28 04:35:41 CommitTransaction: 67142d62563b7213f0c273e70c0248feebc50a494dec7efb58e5dda92bcd92f4, vin: 1, vout: 2
2021-06-28 04:36:05 CommitTransaction: 9d9dd83ff10adee42b2f13a72e2e76f34a560f629e35ade5d8ca45ef55444457, vin: 1, vout: 2
2021-06-28 04:36:31 CommitTransaction: 1830a692cc544905235eaff6e991f623c116f777d2b88faf4762d0cabe2b72c2, vin: 1, vout: 2
2021-06-28 04:38:56 CommitTransaction: 233c4820d954f51ff9be6a56023bf69d98202696f8ceb679308d80cdb09fc8b9, vin: 1, vout: 2
2021-06-28 04:39:19 CommitTransaction: b951e9171f38a0816f26ef3caa523ed264e92bfce25ee836416ac934e4a21448, vin: 1, vout: 2
2021-06-28 04:50:33 CommitTransaction: e346ed325d62e49f3ac01eba13baccdc378b38446eb97f851437481783be1522, vin: 1, vout: 2
2021-06-28 04:51:05 CommitTransaction: abd8fb86d7fbf9e3b5bba1fc86f6c9d8f4b53e5c54eb44c054ab9f91b0731fd6, vin: 1, vout: 2

the debug.log file is filled with messages of the following type which is different from the usual debug messages.

what are the next steps I should do to restart the node properly?

 

We are running

MultiChain version build 2.0.2 protocol 20010

There are currently two nodes in the chain with both having mining permisssion.

Comments

Can you please clarify what you mean by "the node has been in loading address state", i.e. where you see this information and what exactly is shown? The CommitTransaction lines are normal and noting the creation of a new transaction using a high-level API command.

---

Hello, thanks for the reply, whenever I try to restart the chain using multichaind chain-name -daemon, the process starts but is not put into the background the prompt doesn't return to the terminal as it should be when the -daemon flag is used. If I create another session to the same machine and try with multichain-cli chain-name then after putting any command it shows the loading addresses message and doesn't respond to any of the cli or rpc calls.

This is the debug log of one of my other chains on the same machine running on the same multichain binary

2021-06-29 12:37:24 UpdateTip:            new best=008cc0897dc3a741e72a84d6e818d57d91115d6c00c0b70eaf5f57861a70ab66  height=187700  log2_work=25.518077  tx=233241  date=2021-06-29 12:37:18 progress=1.000000  cache=0
2021-06-29 12:37:36 MultiChainMiner: Block Found - 0094da29fe67ce25c88fe8cca097275c7dbabe88742f6b24bad7e3551cb3cd15, prev: 008cc0897dc3a741e72a84d6e818d57d91115d6c00c0b70eaf5f57861a70ab66, height: 187701, txs: 1
2021-06-29 12:37:36 UpdateTip:            new best=0094da29fe67ce25c88fe8cca097275c7dbabe88742f6b24bad7e3551cb3cd15  height=187701  log2_work=25.518084  tx=233242  date=2021-06-29 12:37:35 progress=1.000000  cache=0
2021-06-29 12:37:47 UpdateTip:            new best=003100a96b2c4b7340bd9310a76c0ed42256ac4009c0986a29af358c08180b6a  height=187702  log2_work=25.518092  tx=233243  date=2021-06-29 12:37:40 progress=1.000000  cache=0
2021-06-29 12:37:49 ResendWalletTransactions()
2021-06-29 12:38:04 MultiChainMiner: Block Found - 00a8c2fddd080e1ba49fef480885e19d73f742292b8713263c2ca5bb77f295c5, prev: 003100a96b2c4b7340bd9310a76c0ed42256ac4009c0986a29af358c08180b6a, height: 187703, txs: 1
2021-06-29 12:38:04 UpdateTip:            new best=00a8c2fddd080e1ba49fef480885e19d73f742292b8713263c2ca5bb77f295c5  height=187703  log2_work=25.5181  tx=233244  date=2021-06-29 12:38:01 progress=1.000000  cache=0
2021-06-29 12:38:05 ResendWalletTransactions()
2021-06-29 12:38:15 UpdateTip:            new best=009a05e71aa47b33c8a7465e048750ea98519c1eb854f9ade7ed3c930ef88cf4  height=187704  log2_work=25.518108  tx=233245  date=2021-06-29 12:38:08 progress=1.000000  cache=0
2021-06-29 12:38:22 MultiChainMiner: Block Found - 00b6b1c906a7a73a652674e997694c67a68989653d450f40c5beff504de91c9e, prev: 009a05e71aa47b33c8a7465e048750ea98519c1eb854f9ade7ed3c930ef88cf4, height: 187705, txs: 1
2021-06-29 12:38:22 UpdateTip:            new best=00b6b1c906a7a73a652674e997694c67a68989653d450f40c5beff504de91c9e  height=187705  log2_work=25.518115  tx=233246  date=2021-06-29 12:38:22 progress=1.000000  cache=0


this is different from the debug log of the chain which gets stuck, the debug log used to have similar messages.

Answer 1

It is OK that different nodes have different  rows in debug.log, "UpdateTip" rows should appear on all nodes, but "CommitTransaction" rows - only on the node that sent the transaction.

When node starts, it generates several rows in debug.log, starting from

init message: Loading addresses...

and ending with

init message: Done loading

I suppose you don't see the ending message. Can you, please send rows starting from the last "Loading addresses..." to the end of the file (there should be about 15-20 at most)?

Comments

HI Michael, here is the debug log after when I try to restart the chain

2021-06-28 17:23:54 MultiChain version build 2.0.2 protocol 20010 (May 19 2019, 09:53:03)
2021-06-28 17:23:54 Using OpenSSL version OpenSSL 1.0.1h 5 Jun 2014
2021-06-28 17:23:54 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2021-06-28 17:23:54 Default data directory /home/ubuntu/.multichain/prod-1
2021-06-28 17:23:54 Using data directory /home/ubuntu/.multichain/prod-1
2021-06-28 17:23:54 Using config file /home/ubuntu/.multichain/prod-1/prod-1.conf
2021-06-28 17:23:54 Using at most 125 connections (1024 file descriptors available)
2021-06-28 17:23:54 Using 2 threads for script verification
2021-06-28 17:23:54 Using wallet wallet.dat
2021-06-28 17:23:54 init message: Verifying wallet...
2021-06-28 17:23:54 Wallet file exists. WalletDBVersion: 3.
2021-06-28 17:23:54 init message: Initializing multichain...
2021-06-28 17:23:54 mchn: Parameter set is valid - initializing blockchain parameters...
2021-06-28 17:24:00 Wallet mode: 00000103
2021-06-28 17:24:00 nFileVersion = 100000
2021-06-28 17:24:00 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-28 17:24:00 Node paused state is set to 00000000
2021-06-28 17:24:00 Binding RPC on address :: port 7010 (IPv4+IPv6 bind any: 1)
2021-06-28 17:24:00 Bound to [::]:6010
2021-06-28 17:24:00 Bound to 0.0.0.0:6010
2021-06-28 17:24:00 Filter initialization completed
2021-06-28 17:24:00 Setting banned transaction list:    0 transactions
2021-06-28 17:24:00 init message: Loading block index...
2021-06-28 17:24:00 Opening LevelDB in /home/ubuntu/.multichain/prod-1/blocks/index
2021-06-28 17:24:00 Opened LevelDB successfully
2021-06-28 17:24:00 Opening LevelDB in /home/ubuntu/.multichain/prod-1/chainstate
2021-06-28 17:24:00 Opened LevelDB successfully
2021-06-28 17:24:00 nFileVersion = 100000
2021-06-28 17:24:00 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-28 17:24:01 LoadBlockIndexDB: last block file = 0
2021-06-28 17:24:01 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=46604, size=123223062, heights=0...44595, time=2021-02-24...2021-06-28)
2021-06-28 17:24:01 Checking all blk files are present...
2021-06-28 17:24:01 LoadBlockIndexDB(): transaction index enabled
2021-06-28 17:24:01 LoadBlockIndexDB(): hashBestChain=0046eb52a1bf9bc42db6aa70ae0b4c0c06d748d17dc6fa12e8baffc9e1b51593 height=44595 date=2021-06-28 05:45:02 progress=1.000000
2021-06-28 17:24:01 init message: Verifying blocks...
2021-06-28 17:24:01 Verifying last 288 blocks at level 3
2021-06-28 17:24:01 No coin database inconsistencies in last 289 blocks (377 transactions)
2021-06-28 17:24:01  block index            1141ms
2021-06-28 17:24:01 MultiChain protocol version: 20010
2021-06-28 17:24:01 init message: Loading wallet...
2021-06-28 17:24:01 nFileVersion = 100000
2021-06-28 17:24:01 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-28 17:24:01  wallet                    1ms
2021-06-28 17:24:01 mapBlockIndex.size() = 46604
2021-06-28 17:24:01 nBestHeight = 44595
2021-06-28 17:24:01 setKeyPool.size() = 2
2021-06-28 17:24:01 oldWallet.size() = 0
2021-06-28 17:24:01 mapWallet.size() = 496578
2021-06-28 17:24:01 mapAddressBook.size() = 1
2021-06-28 17:24:01 init message: Loading addresses...
2021-06-28 17:24:01 Loaded 2 addresses from peers.dat  1ms
2021-06-28 17:24:01 AddLocal(11.0.2.19:6010,1)
2021-06-28 17:24:01 Discover: IPv4 ens5: 11.0.2.19
2021-06-28 17:24:01 AddLocal([fe80::fd:8cff:fe1d:a122]:6010,1)
2021-06-28 17:24:01 Discover: IPv6 ens5: fe80::fd:8cff:fe1d:a122
2021-06-28 17:24:01 dnsseed thread start
2021-06-28 17:24:01 net thread start
2021-06-28 17:24:01 addcon thread start
2021-06-28 17:24:01 opencon thread start
2021-06-28 17:24:01 msghand thread start
2021-06-28 17:24:01 dumpaddr thread start
2021-06-28 17:24:01 MultiChainMiner started
2021-06-28 17:24:12 Loading addresses from DNS seeds (could take a while)
2021-06-28 17:24:12 0 addresses found from DNS seeds
2021-06-28 17:24:12 dnsseed thread exit

after that there are no lines in the debug.log and I have to forcefully kill the process by kill command since the restart takes too much time.

---

All looks good, nothing is missing, the next record is supposed to be "Done loading"

Let's try to add two more debug flags:

-debug=mchn -debug=wallet

---

Hi Michael,

I did try to start the node with the given flags, it is still taking a lot of time to start

here is the debug log

021-06-30 18:22:58 MultiChain version build 2.0.2 protocol 20010 (May 19 2019, 09:53:03)
2021-06-30 18:22:58 Using OpenSSL version OpenSSL 1.0.1h 5 Jun 2014
2021-06-30 18:22:58 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2021-06-30 18:22:58 Default data directory /home/ubuntu/.multichain/prod-1
2021-06-30 18:22:58 Using data directory /home/ubuntu/.multichain/prod-1
2021-06-30 18:22:58 Using config file /home/ubuntu/.multichain/prod-1/prod-1.conf
2021-06-30 18:22:58 Using at most 125 connections (1024 file descriptors available)
2021-06-30 18:22:58 Using 2 threads for script verification
2021-06-30 18:22:58 Using wallet wallet.dat
2021-06-30 18:22:58 init message: Verifying wallet...
2021-06-30 18:22:58 Wallet file exists. WalletDBVersion: 3.
2021-06-30 18:22:58 init message: Initializing multichain...
2021-06-30 18:22:58 mchn: Parameter set is valid - initializing blockchain parameters...
2021-06-30 18:22:58 mchn: Burn address: 1XXXXXXX9vXXXXXXo2XXXXXXPuXXXXXXWitjhP
2021-06-30 18:22:58 wtxs: Loaded 1 unspent outputs for import 0
2021-06-30 18:23:05 Wallet mode: 00000103
2021-06-30 18:23:05 nFileVersion = 100000
2021-06-30 18:23:05 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-30 18:23:05 mchn: Default connect  address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default send     address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default receive  address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default create   address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default issue    address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default mine     address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default admin    address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 mchn: Default activate address: 1VYeT7jHTFPg3xwmgiQ43W85Lb7Cbj4oHW7W7R
2021-06-30 18:23:05 Node paused state is set to 00000000
2021-06-30 18:23:05 Binding RPC on address :: port 7010 (IPv4+IPv6 bind any: 1)
2021-06-30 18:23:05 Bound to [::]:6010
2021-06-30 18:23:05 Bound to 0.0.0.0:6010
2021-06-30 18:23:05 Filter initialization completed
2021-06-30 18:23:05 Setting banned transaction list:    0 transactions
2021-06-30 18:23:05 init message: Loading block index...
2021-06-30 18:23:05 Opening LevelDB in /home/ubuntu/.multichain/prod-1/blocks/index
2021-06-30 18:23:05 Opened LevelDB successfully
2021-06-30 18:23:05 Opening LevelDB in /home/ubuntu/.multichain/prod-1/chainstate
2021-06-30 18:23:05 Opened LevelDB successfully
2021-06-30 18:23:05 nFileVersion = 100000
2021-06-30 18:23:05 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-30 18:23:06 LoadBlockIndexDB: last block file = 0
2021-06-30 18:23:06 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=46604, size=123223062, heights=0...44595, time=2021-02-24...2021-06-28)
2021-06-30 18:23:06 Checking all blk files are present...
2021-06-30 18:23:06 LoadBlockIndexDB(): transaction index enabled
2021-06-30 18:23:06 mchn: Rolling back permission DB to height 44595
2021-06-30 18:23:06 mchn: Rolling back asset DB to height 44595
2021-06-30 18:23:06 mchn: Rolling back wallet txs DB to height 44595
2021-06-30 18:23:06 LoadBlockIndexDB(): hashBestChain=0046eb52a1bf9bc42db6aa70ae0b4c0c06d748d17dc6fa12e8baffc9e1b51593 height=44595 date=2021-06-28 05:45:02 progress=1.000000
2021-06-30 18:23:06 init message: Verifying blocks...
2021-06-30 18:23:06 Verifying last 288 blocks at level 3
2021-06-30 18:23:07 No coin database inconsistencies in last 289 blocks (377 transactions)
2021-06-30 18:23:07  block index            1392ms
2021-06-30 18:23:07 MultiChain protocol version: 20010
2021-06-30 18:23:07 init message: Loading wallet...
2021-06-30 18:23:07 nFileVersion = 100000
2021-06-30 18:23:07 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2021-06-30 18:23:07  wallet                    1ms
2021-06-30 18:23:07 mchn: Found 0 assets in 0 groups
2021-06-30 18:23:07 mchn: Asset Grouping. Assets: 0. Group Size: 1. Group Count: 0. Single-Asset Groups: 0.
2021-06-30 18:23:07 mchn: Unspent list initialized: Total: 0, Unspent: 0
2021-06-30 18:23:07 mapBlockIndex.size() = 46604
2021-06-30 18:23:07 nBestHeight = 44595
2021-06-30 18:23:07 setKeyPool.size() = 2
2021-06-30 18:23:07 oldWallet.size() = 0
2021-06-30 18:23:07 mapWallet.size() = 496578
2021-06-30 18:23:07 mapAddressBook.size() = 1
2021-06-30 18:23:07 init message: Loading addresses...
2021-06-30 18:23:07 Loaded 2 addresses from peers.dat  4ms
2021-06-30 18:23:07 AddLocal(11.0.2.19:6010,1)
2021-06-30 18:23:07 Discover: IPv4 ens5: 11.0.2.19
2021-06-30 18:23:07 AddLocal([fe80::fd:8cff:fe1d:a122]:6010,1)
2021-06-30 18:23:07 Discover: IPv6 ens5: fe80::fd:8cff:fe1d:a122
2021-06-30 18:23:07 ReacceptWalletTransactions: 5604 txs in unconfirmed pool
2021-06-30 18:23:07 Unconfirmed wtx: e07895db0ddcd7569e65e0bd8a41c4a81755c905397b4c1342846d83597e3964, depth: -1
2021-06-30 18:23:07 Reaccepting wtx e07895db0ddcd7569e65e0bd8a41c4a81755c905397b4c1342846d83597e3964
2021-06-30 18:23:07 msghand thread start
2021-06-30 18:23:07 dumpaddr thread start
2021-06-30 18:23:07 opencon thread start
2021-06-30 18:23:07 addcon thread start
2021-06-30 18:23:07 MultiChainMiner started
2021-06-30 18:23:07 dnsseed thread start
2021-06-30 18:23:07 net thread start
2021-06-30 18:23:07 wtxs: Found 8 entities in tx e07895db0ddcd7569e65e0bd8a41c4a81755c905397b4c1342846d83597e3964, flags: 00030004, import 0
2021-06-30 18:23:07 Unconfirmed wtx: 13bff04454c74a94287e64c2e35cd40ba37543afbec0199271c86424f40f0026, depth: -1
2021-06-30 18:23:07 Reaccepting wtx 13bff04454c74a94287e64c2e35cd40ba37543afbec0199271c86424f40f0026
2021-06-30 18:23:07 wtxs: Found 22 entities in tx 13bff04454c74a94287e64c2e35cd40ba37543afbec0199271c86424f40f0026, flags: 00030004, import 0
2021-06-30 18:23:07 Unconfirmed wtx: 8a29e69b947cd75059591342e8b967d48d18b96573c71fd24ec54813e91143e5, depth: -1
2021-06-30 18:23:07 Reaccepting wtx 8a29e69b947cd75059591342e8b967d48d18b96573c71fd24ec54813e91143e5
2021-06-30 18:23:07 wtxs: Found 22 entities in tx 8a29e69b947cd75059591342e8b967d48d18b96573c71fd24ec54813e91143e5, flags: 00030004, import 0
.
.
2021-06-30 18:35:01 wtxs: Found 20 entities in tx 0d7647944cd0692e6e0143bde4bd9b5fae374171e093f2aab01e676de23c31ff, flags: 00030004, import 0
2021-06-30 18:35:01 Unconfirmed wtx: 0584785a633580956abe3782e11b4a5888ed0c612f6dd2730817f66a296b1566, depth: -1
2021-06-30 18:35:01 Reaccepting wtx 0584785a633580956abe3782e11b4a5888ed0c612f6dd2730817f66a296b1566
2021-06-30 18:35:01 wtxs: Found 8 entities in tx 0584785a633580956abe3782e11b4a5888ed0c612f6dd2730817f66a296b1566, flags: 00030004, import 0
2021-06-30 18:35:01 Unconfirmed wtx: 1a59f2232b5e61fa36f2a8fc87d08e2cee63c6e5dee4117d29c32128de3423ad, depth: -1
2021-06-30 18:35:01 Reaccepting wtx 1a59f2232b5e61fa36f2a8fc87d08e2cee63c6e5dee4117d29c32128de3423ad
2021-06-30 18:35:01 wtxs: Found 10 entities in tx 1a59f2232b5e61fa36f2a8fc87d08e2cee63c6e5dee4117d29c32128de3423ad, flags: 00030004, import 0
2021-06-30 18:35:01 Unconfirmed wtx: 768cb31011fbaf0a8472621038a131564c47d2b59ecc4dafb76109909c41bb28, depth: -1
2021-06-30 18:35:01 Reaccepting wtx 768cb31011fbaf0a8472621038a131564c47d2b59ecc4dafb76109909c41bb28
2021-06-30 18:35:03 wtxs: Found 262 entities in tx 768cb31011fbaf0a8472621038a131564c47d2b59ecc4dafb76109909c41bb28, flags: 00030004, import 0
2021-06-30 18:35:03 Unconfirmed wtx: a499595bc4697ab032333cc02a9bc24c185d716db26c01afaea68151ea9b6088, depth: -1
2021-06-30 18:35:03 Reaccepting wtx a499595bc4697ab032333cc02a9bc24c185d716db26c01afaea68151ea9b6088

have trimmed the lines in mid.

---

OK, this was expected. You have lot of (5604) unconfirmed transactions, which were created by this node. When node starts, it loads these transactions. Some transactions are very big (look at 768cb31011fbaf0a8472621038a131564c47d2b59ecc4dafb76109909c41bb28 - it took 2s to process) and have lot of (262) "entities" - (addresses, stream items, assets, ...). So, it takes time. You can count "Reaccepting wtx" rows to estimate how much - it should be 5604 in total.

We will improve "Loading addresses..." message and exchange it by more suitable "Loading unconfirmed transactions...". Anyway, the node cannot respond to API calls before this process is completed.

But this raises important question: why do you have so many unconfirmed transactions? These transactions are very heavy (like 768cb31011fbaf0a8472621038a131564c47d2b59ecc4dafb76109909c41bb28), it takes 2s to process even if it is sent for the first time. It looks like your node simple doesn't cope with the load.

You can check this by looking at the output of getmempoolinfo. It can go up and down, but if the overall long-term trend is up - it makes sense to check what's going on with this node. Please note - it may be a problem with this specific node. I don't know what your "entities" are (streams, assets), but the node subscribed to these entities generally works much harder.

---

Hi Michael, thanks for explaining this I think the entities are transactions we are sending some raw transaction with about 200 publish sub transactions, this may be a reason and the most important problem I think that the node is not mining properly, we have set the mining diversity of this chain to 0 as we wanted any available node to mine, also the same problem is happening with one of our other chains it is also accumulating mempool and not confirming them, the rest 12 chains are working fine atm. Needed to ask even if the diversity is set to 0 is it possible that this is happening due to the other node being designated to mine the transaction but that node going offline and the txns remaining in mempool or just because load is too high?

---

I don't think mining is the issue. Most likely explanation - load is too high. The time for processing transaction like 768c... (2s) is very high. If your network produces transactions like this more frequently than once per 2s, there is no chance node can cope with the load.
And you should take offline time into account, of course. If node, which is barely able to deal with the network traffic, goes offline, it may take very long time until it catches the rest of the nodes.
You can try to optimize your network behavior by unsubscribing from some streams on some nodes - transaction processing will be significantly faster. But you can do this, of course, only when node is started, i.e. after 5604 txs are reaccepted.

---

Thanks Michael, will check and try to fix it and let you know if anything else comes up, thanks again.