Backup and Disaster Recovery

Question

Hello, I need sure directions for backup and Disaster Recovery because I'm in production now.
I understand the way to do the backup is stop the multichain, copy the folder, restart the multichain but:
1. When stop the multichain the service is not guaranteed
2. I would lose all transactions between last backup and failure
So, I made some slave node that have only connect permission. Think at it as a start IT infrastructure: one master node where all transactions are done and some slaves, for backup purposes, web explorer support.
Q1 - For backup purposes, do I can stop regularly one of the slave node and backup datas on directory in slave node ? At this point can I copy this data on master node directory, restart the multichain and all magically works ?
Q2 - In case of failure of master node do I can simply restart the multichain on the master (of course after deleting the data multichain directory) giving as address one of the slave node ? The fact that slave node have only connect permission (no send no receive etc etc) have some influence on the restart of the master ?
Q3 - In case of failure of master node, can I simply stop a slave node, copy the directory on master node and restart the multichain on master node ?
Q4 - Do you suggest could be useful to have another slave node with all permission availables ?
Q5 - Have done a little research on Q&A section and I have seen that there's a command dumpprivkey that download the private key of addresses, but I do not understand the utility. And I must download key address by address ! 500 address for now on my multichain and growing.

Thanks for any help. 

Comments

Thats a good question.This isnt an answer but a little sidebar.. We have made a message queue of items to be processed to the chain, in effect creating a safe pre 'unconfirmed' scenario. We can then control the stop/start a little more safely. Especially on other random points of failures, we still have the queue.
We can also stop everything and our websites would still be pushing to the queue, which would then be relayed to the chain once up and running.

Answer 1

To answer your questions in one go, you can't automatically directly copy the full blockchain directory on a slave (connect only) node to the master node because each node has a different wallet and tracks different addresses. 

So you have two sensible approaches to backup and disaster recovery.

The first is to share the private keys (and any watch-only addresses) from the master to slave using dumpprivkey, importprivkey and importaddress and then the nodes are effectively full replicas of each other. At any time you can stop one of the nodes, back up its directory, and then later that directory can be used to start up another node, between master and slave and vice versa.

The second is to only keep a backup of a node's private keys, and let it recover everything else from other nodes on the network (which could be within other organizations). In this case you recover a node by starting it with -initprivkey passing the private key of the address required for connecting, and then use importprivkey for the other private keys it should hold. Or even simpler, back up the wallet.dat file, and use that for restoring the node's wallet after which you run MultiChain with -rescan=1.

One important note: if two nodes are sharing a private key, you should not simultaneously create transactions for an address corresponding to that private key in both nodes. Use one or the other, and only switch if there's an actual failure. Otherwise you can get double-spend failures.

Comments

Hi, thank you for your reply, if you don't mind, I would like to recap if I really understand, also because I'm new in multichain town and my english is not good as I would like to be, so forgive me if maybe I have too much questions:
First approach seems good but require to download keys, so I have two objections:
1. I ought not be able to download user keys, as in theory I can use that keys to move money from a wallet to the other. Correct ? Why I can do that (I tried with dumpwallet() and it works) ? there's a way to avoid this ? As I can see, on BlockChain generally I can read my privatekey only one time and if I lost it, there's no way to recover it. How I can replicate this behaviour ?
2. What happens if a new user join the master or the slave node ? I must replicate the same download, upload process for all keys ? just for that user ?
Second approach seems more interesting, just a couple of questions:
1. When you say "-initprivkey passing the private key of the address required for connecting" what's the address ? the one in multichain.conf on master node directory ? Or the one in the slave directory ? or the one I had to authorize for connection, send, receive ? (maybe last two are the same, I'm confused, seems address have at least two different meaning)
The second way, backup the wallet.dat file require to be done with multichain stopped ? So basically, I have a slave node and a wallet.dat, I stop the slave node, copy the wallet.dat over the old one, run the multichain on the slave with the option -rescan=1 and I have a full working copy as the original ? No transaction lost (of course if no new user join in the middle) ?
Sorry for to be so tedious, but you know when backup need, backup must work, and in this case there are user money at the stake.
I think it would be nice if there would be a tutorial.
Thanks you.

---

To answer 1 and 2, there is no multiuser security within a single node's wallet in MultiChain right now. Any user who has access to a node's API can access all of the private keys held by that node, and sign transactions using them. If you want multitenant security on a single MultiChain node, you need to hold the private keys outside the node's wallet. See this tutorial: https://www.multichain.com/developers/external-key-management/

You need to pass a private key with initprivkey, as obtained using dumpprivkey passing the corresponding address.

And yes, you should stop MultiChain before backing up wallet.dat, just in case some API call is in the process of modifying it. But better is actually to use dumpwallet and importwallet which can be used while MultiChain is running.

Finally we'll be producing more documentation regarding maintenance, like backup and recovery, over the coming months.