Corrupted block database detected.

Question

In the scenario where we have 3 nodes, lets say we made node 1 first then node 2 then 3. Nodes 2 and 3 are replicas of 1. If we insert assets from node 2 into the wallet of node 1 the assets are generated at a
high rate. At one point during "insert from 2 to 1" speed at which assets were being inserted dropped significantly (it seems that node 1 was choking) then node 1 crashed and speed of inserting got back to high.

when i tried to restart node 1 i got an error saying :

: Corrupted block database detected.

Please restart multichaind with reindex=1.

i assume this was probably a hardware glitch or something but i wanted to ask you what can cause this error and also what causes the "choke" that caused node 1 to crash in the first place? Should we add a small sleeep(5s) every 100 sec or so to avoid this or something?

 

debug log:

 

2016-03-22 08:55:48 MultiChain version build 1.0 alpha 18 protocol 10005 (2016-03-13 08:58:54 +0200)
2016-03-22 08:55:48 Using OpenSSL version OpenSSL 1.0.1h 5 Jun 2014
2016-03-22 08:55:48 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2016-03-22 08:55:48 Default data directory /home/sloba/.multichain/theone
2016-03-22 08:55:48 Using data directory /home/sloba/.multichain/theone
2016-03-22 08:55:48 Using config file /home/sloba/.multichain/theone/theone.conf
2016-03-22 08:55:48 Using at most 125 connections (1024 file descriptors available)
2016-03-22 08:55:48 Using 4 threads for script verification
2016-03-22 08:55:48 Using wallet wallet.dat
2016-03-22 08:55:48 init message: Verifying wallet...
2016-03-22 08:55:48 CDBEnv::Open : LogDir=/home/sloba/.multichain/theone/database ErrorFile=/home/sloba/.multichain/theone/db.log
2016-03-22 08:55:49 init message: Initializing multichain...
2016-03-22 08:55:50 nFileVersion = 100000
2016-03-22 08:55:50 Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total
2016-03-22 08:55:50 mchn: Parameter set is valid - initializing blockchain parameters...
2016-03-22 08:55:50 Binding RPC on address ::1 port 7724 (IPv4+IPv6 bind any: 0)
2016-03-22 08:55:50 Binding RPC on address 127.0.0.1 port 7724 (IPv4+IPv6 bind any: 0)
2016-03-22 08:55:50 Bound to [::]:7725
2016-03-22 08:55:50 Bound to 0.0.0.0:7725
2016-03-22 08:55:50 init message: Loading block index...
2016-03-22 08:55:50 Opening LevelDB in /home/sloba/.multichain/theone/blocks/index
2016-03-22 08:55:50 Opened LevelDB successfully
2016-03-22 08:55:50 Opening LevelDB in /home/sloba/.multichain/theone/chainstate
2016-03-22 08:55:50 Opened LevelDB successfully
2016-03-22 08:55:50 LoadBlockIndexDB: last block file = 0
2016-03-22 08:55:50 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=1404, size=16787062, heights=0...1396, time=2016-03-21...2016-03-22)
2016-03-22 08:55:50 Checking all blk files are present...
2016-03-22 08:55:50 LoadBlockIndexDB(): transaction index enabled
2016-03-22 08:55:51 LoadBlockIndexDB(): hashBestChain=00004bfc0db5821879c2b167ba2c6cefe8c18320b5eabb9043d2032dcf178dcb height=1396 date=2016-03-22 08:20:46 progress=1.000000
2016-03-22 08:55:51 init message: Verifying blocks...
2016-03-22 08:55:51 Verifying last 288 blocks at level 3
2016-03-22 08:55:51 ERROR: VerifyDB() : *** found bad permission data at 1396, hash=00004bfc0db5821879c2b167ba2c6cefe8c18320b5eabb9043d2032dcf178dcb

2016-03-22 08:55:51 : Corrupted block database detected.

Please restart multichaind with reindex=1.
2016-03-22 08:55:51 Aborted block database rebuild. Exiting.
2016-03-22 08:55:51 Shutdown: In progress...
2016-03-22 08:55:51 StopNode()
2016-03-22 08:55:51 Node stopped
2016-03-22 08:55:51 Shutdown: done
2016-03-22 08:55:51 CDBEnv::EnvShutdown : Error 22 shutting down database environment: Invalid argument

Answer 1

Thanks for this. First would you be able to zip up the blockchain directory and send it to us for debugging, before running with the reindex command?

Apart from that, what do you mean by "insert asset from node 2 into node 1"? Are the nodes not connected together, in which case they synchronize their asset databases automatically via the blockchain?

Comments

yeah sorry we already did the reindex but then we got a new error so you will probbably still get some info out of the zipped file we will send. New situation was i tried following commands
multichaind theone -daemon reindex=1  >>> nothing happened

multichaind theone  reindex=1 -daemon >>> nothing happened

multichaind theone -daemon -reindex=1  >>> then we got a new error

  vMerkleTree:  1df6a2dee9869f6f0ca856eb15785c787109fe4186d7fe15ec72cfe265204b2f 1ec417e12ab00a7f442c9764b27edac2491d5a5e17096a2e0eb5df84b92ce12e 829e17d56e8187b0fdd36d6e99602d6238aceacd3de733c284f53ed86d3cc6b5

2016-03-22 10:23:44 generated 0.00
2016-03-22 10:23:44 ERROR: ReadBlockFromDisk(CBlock&, CBlockIndex*) : GetHash() doesn't match index
2016-03-22 10:23:44 *** Failed to read block
2016-03-22 10:23:44 Error: Error: A fatal internal error occured, see debug.log for details
2016-03-22 10:23:44 ERROR: ProcessNewBlock : ActivateBestChain failed
2016-03-22 10:23:44 ERROR: MultiChainMiner : ProcessNewBlock, block not accepted
2016-03-22 10:23:44 Running MultiChainMiner with 2 transactions in block (19034 bytes)
2016-03-22 10:23:44 MultiChainMiner terminated
2016-03-22 10:23:44 StopNode()
2016-03-22 10:23:44 Node stopped
2016-03-22 10:23:55 Shutdown: done

---

could it be that blockchain named "theone" attracted the attention of agent Smith and now he is hacking us from the inside wrecking havoc on the system?

---

Follow the white rabbit...

---

"Apart from that, what do you mean by "insert asset from node 2 into node 1"? Are the nodes not connected together, in which case they synchronize their asset databases automatically via the blockchain?"

what i mean by this is were using wallet adress of node 1 as a parameter for the issue asset command which we execute on node 2.
example of terminal comand on server 3 (server 3=node 3 server2=node2 etc.)

[slobodan@blockserver3 theone]$ multichain-cli theone issue XYZ_adressof_node1_onserver1_XYZ    name_of_the_asset 1000 0.01

---

OK, understood. In terms of what caused the choke on node 1, it's hard for us to know. If you still have the blockchain directory with the corrupted database, it would be very helpful to zip it up and send it to us. Otherwise, please do this next time it happens. It seems you are pushing the issuance of a large number of assets harder than other users of Multichain :)

---

http://bit.ly/22txM3u thanks for reminding me I forgot to share the link :)

Yeah i guess we are pushing it kinda hard. Had a conversation with the bosses today and they mentioned "75 million isn't enough" and suddenly my enthusiasm about breaching 500 000 after choking at 20 000 for weeks just withered and died. Glad you noticed we are trying hard :) ,and you made our day at the office with the follow up on Matrix jokes.

---

Thanks, we will look into it and see what we find.

---

I've just had the same issue. I created an asset in node1 a few days ago, and use issuemore to an address in node2. Worked a few times over the course of days. This morning, everything came down with this error.

node1 goes nuts by running at high speed:

2016-04-18 07:42:06 generated 0.00
2016-04-18 07:42:06 ERROR: ConnectTip() : ConnectBlock 0000a29243ebfd464db852c9b7645cdf78906e724a15ff1202c7cd31fae372ca failed
2016-04-18 07:42:06 ERROR: ProcessNewBlock : ActivateBestChain failed
2016-04-18 07:42:06 ERROR: MultiChainMiner : ProcessNewBlock, block not accepted
2016-04-18 07:42:06 Running MultiChainMiner with 2 transactions in block (532 bytes)
2016-04-18 07:42:06 MultiChainMiner:
2016-04-18 07:42:06 proof-of-work found  
  hash: 0000343d2ae031e9f9dcfbf9e4ed572d0dc71aefaa652300767dc737d6b7f34a  
target: 0000ffff00000000000000000000000000000000000000000000000000000000
2016-04-18 07:42:06 CBlock(hash=0000343d2ae031e9f9dcfbf9e4ed572d0dc71aefaa652300767dc737d6b7f34a, ver=3, hashPrevBlock=000000099e7eeebe97452a0107b04936601efee306b7f8b43658013ef11b6127, hashMerkleRoot=85db8436ce13fe854c6a85ef242911447a0189c05bad54dd86f91f235134ebeb, nTime=1460965326, nBits=1f00ffff, nNonce=70516, vtx=2)
  CTransaction(hash=d3a1248bdf, ver=1, vin.size=1, vout.size=1, nLockTime=0)
    CTxIn(COutPoint(0000000000, 4294967295), coinbase 02230d027121062f503253482f)
    CTxOut(nValue=0.00000000, scriptPubKey=OP_RETURN 53504b62473045022100)

  CTransaction(hash=c34cc0ba76, ver=1, vin.size=1, vout.size=3, nLockTime=0)
    CTxIn(COutPoint(0ed9714009, 2), scriptSig=304402200e2dd9be80c9924d)
    CTxOut(nValue=0.00000000, scriptPubKey=OP_DUP OP_HASH160 f973409bdd53)
    CTxOut(nValue=0.00000000, scriptPubKey=OP_RETURN 1665880147)
    CTxOut(nValue=0.00000000, scriptPubKey=OP_DUP OP_HASH160 70c0626b5afa)

  vMerkleTree:  d3a1248bdfa9ff92c1450590ad3b1b986cff429ac18c0803e9b5107c75feeb7e c34cc0ba76a8e7d12fc4d718574383305b07365566cac134ca5a5c09c2ee838f 85db8436ce13fe854c6a85ef242911447a0189c05bad54dd86f91f235134ebeb

---

It looks like the same issue:

http://www.multichain.com/qa/1429/chain-folder-keeps-growing?show=1437#c1437

Can one of you zip entire blockchain directory and post it for debugging, please? If it is too big, debug.log can be removed

---

I was overly exited and deleted the chain :( next time!