MultiChain feed file format

Build your own adapter for reading MultiChain Enterprise feed files

MultiChain feeds are real-time on-disk binary logs of the events relating to one or more blockchain streams, or the blocks in a chain. They are designed for reading by external processes, and can be used to keep an external database synchronized with the state of the chain. MultiChain feeds are available in MultiChain 2.0 Enterprise, a demo of which is available for free download.

The open source MultiChain Feed Adapter, written in Python, can read a feed and automatically replicate its contents to several popular databases. For most purposes, we recommended extending or modifying this adapter, which includes everything you need to parse feed files correctly.

However some developers may wish to create their own solution for reading feed files in a different programming language. The information below explains the format of the files, which are designed for efficient reading as binary data. The files use a simple unified structure based mostly on 1-byte identifiers and small-endian 32-bit integers.

Overall File Structure

Files in a feed directory are named consecutively in the form feedXXXXXX.dat where XXXXXX is a six-digit integer with leading zeros. The first file is named feed000000.dat and each file ends with a record indicating the number of the next file to move to. The feed moves on to the next file when the current file reaches 128 MB in size or after 24 hours – whichever is sooner. Files are append-only, and it is up to the parser to keep track of the current file number and position.

Each file contains one or more records, and each record contains one or more fields. Most records and fields relating to blockchain events are optional, and their presence will depend on the feed’s options as set through the JSON-RPC API commands addtofeed and updatefeed.

Each record in a file takes the following form:

Field Size Description
Record identifier 1 byte Identifier for type of record to follow.
Record size 4 bytes Byte length of remainder of record, as a 32-bit small-endian integer.
Repeat the below for each field in the record
Field identifier 1 byte Identifier for type of field to follow.
Field size 4 bytes Byte length of field data, as a 32-bit small-endian integer. This is included even if the field length is fixed.
Field data Variable Appropriate data for the field identifier.

Parsing a Feed File

Each feed file begins with a Previous File record (starting 0x04). After that, records are written in batches, where each batch is bracketed by a Batch Start and Batch End record.

As it proceeds through the file, the parser must wait until it sees a 0x01 byte indicating a Batch Start record before proceeding to read until the end of that batch. If the next byte does not indicate a Batch Start record, the parser should wait, because the next batch has not yet been written.

Below is a description of each of the possible record types that can appear in a feed file, along with the fields that can appear within the record. The title of each section shows the 1-byte record identifier. Any records or fields with unknown identifiers should be safely ignored. If the size of a field indicated in the file does not match the size in the tables below, the field can also be ignored.

Previous File – 0x04

This record appears at the start of each feed file, with the following fields:

Field identifier Size Description Required?
0x01 4 bytes Byte length of previous feed file, as a 32-bit small-endian integer. Yes
0x02 4 bytes Number of previous feed file, as a 32-bit small-endian integer. Yes
0x05 4 bytes File creation time, as Unix timestamp in small-endian order. Yes

Batch Start and Incomplete Batch – 0x01 and 0x03

One of these records appears at the start of a batch of records that were written to a file in a single write operation, with the following field:

Field identifier Size Description Required?
0x01 4 bytes Byte length of batch, including this record and the Batch End record, as 32-bit small-endian integer. This is optimized for forward scanning – the next batch (if it exists) starts this many bytes after the current one. Yes

When writing a batch to a feed file, MultiChain writes the batch prefixed by an Incomplete Batch record with identifier 0x03, then flushes the write operation to disk and updates its internal pointer. The Incomplete Batch record is then converted to a Start Batch record by changing its identifier to 0x01, meaning that the batch is ready to parse.

Batch End – 0x02

This record appears at the end of a batch of records that were written to a file in a single operation, with the following fields:

Field identifier Size Description Required?
0x01 4 bytes Byte length of batch, including the Batch Start record but not this Batch End record, as a 32-bit small-endian integer. This is optimized for backward scanning – the batch started this many bytes back. Yes
0x02 4 bytes If this is the last batch in this file, the number of the next feed file, as a 32-bit small-endian integer. No

Block Add Start and Block Add End – 0x26 and 0x27

These records are written before and after attaching a new valid block to the end of the chain, and can contain the following fields:

Field identifier Size Description Required?
0x20 4 bytes Height of block, as a 32-bit small-endian integer. No
0x21 32 bytes Hash of block, with bytes ordered to match JSON-RPC API (not network protocol). Yes
0x22 4 bytes Number of transactions in block, as a 32-bit small-endian integer. No
0x23 4 bytes Confirmation time of block, as Unix timestamp in small-endian order. No
0x24 Variable Address of the block miner, as a regular base58-encoded string. No
0x25 4 bytes Byte size of block, as a 32-bit small-endian integer. No

Block Remove Start and Block Remove End – 0x23 and 0x24

These records are written before and after detaching a block from the end of the chain, due to a (rare) blockchain reorganization. They can contain the following fields:

Field identifier Size Description Required?
0x20 4 bytes Height of block, as a 32-bit small-endian integer. No
0x21 32 bytes Hash of block, with bytes ordered to match JSON-RPC API (not network protocol). Yes

Stream Item Received – 0x30

This is the first feed record written for each stream item, with the following fields:

Field identifier Size Description Required?
0x30 20 bytes ID of item in feed, for reference in other feed records. Yes
0x32 Variable Name of stream, as a UTF-8 encoded string. No
0x2A 32 bytes Transaction ID containing item, with bytes ordered to match JSON-RPC API (not network protocol). No
0x2B 4 bytes Number of transaction output (vout) containing item, as a 32-bit small-endian integer. No
0x33 Variable Address of item publisher, as a regular base58-encoded string. This field will appear multiple times if the item has multiple publishers. No
0x34 Variable Item key, as a UTF-8 encoded string. This field will appear multiple times if the item has multiple keys. No
0x35 4 bytes Item format, where 0 means binary, 1 means text, and 2 means JSON. No
0x36 8 bytes Item data size in bytes, as a 64-bit small-endian integer. No
0x37 Variable Data for binary items – see maxshowndata note below. No
0x38 Variable Data for text items in UTF-8 encoding – see maxshowndata note below. No
0x39 Variable Data for JSON items in regular JSON format – see maxshowndata note below. No
0x29 4 bytes Time item was received, as a Unix timestamp in small-endian order. No
0x3B 1 byte Item flags. Current flags are 1 if the item data is available, 2 if the item data is off-chain, and 4 is the item is off-chain and contains multiple chunks. No
0x2C 40 bytes Item dataref, as a binary string – see maxshowndata note below. No

In order to save disk space, if an item’s data as stored in the transaction is longer than the feed’s maxshowndata setting, it will not be included within the feed file. Instead, that data can be retrieved by passing the item’s dataref to the getdatarefdata or datareftobinarycache JSON-RPC APIs. Note also that if an item’s data is off-chain, that data may not yet be available – see the Offchain Data Available record below.

Stream Item Confirmed – 0x31

This record is written when a stream item is confirmed in a block, with the following fields:

Field identifier Size Description Required?
0x30 20 bytes ID of item in feed, for reference in other feed records. Yes
0x32 Variable Name of stream, as a UTF-8 encoded string. No
0x2A 32 bytes Transaction ID containing item, with bytes ordered to match JSON-RPC API (not network protocol). No
0x2B 4 bytes Number of transaction output (vout) containing item, as a 32-bit small-endian integer. No
0x20 4 bytes Height of block, as a 32-bit small-endian integer. No
0x21 32 bytes Hash of block, with bytes ordered to match JSON-RPC API (not network protocol). No
0x23 4 bytes Confirmation time of block, as Unix timestamp in small-endian order. No
0x2D 4 bytes Byte offset of transaction output containing item in block, as a 32-bit small-endian integer. No
0x2C 40 bytes Item dataref, as a binary string. This may be different from a dataref provided for the item in a previous record. If so, both datarefs will work, but the latest one will offer better performance. No

Stream Item Unconfirmed – 0x32

This record is written when the block confirming a stream item is removed, due to a (rare) blockchain reorganization, with the following fields:

Field identifier Size Description Required?
0x30 20 bytes ID of item in feed, for reference in other feed records. Yes
0x32 Variable Name of stream, as a UTF-8 encoded string. No
0x2A 32 bytes Transaction ID containing item, with bytes ordered to match JSON-RPC API (not network protocol). No
0x2B 4 bytes Number of transaction output (vout) containing item, as a 32-bit small-endian integer. No

Stream Item Invalid – 0x33

This record is written when an unconfirmed transaction containing a stream item becomes invalid, so the item should be removed from the stream. This is an unusual and unlikely event – possible causes include a double spend of one of the transaction’s inputs or a revocation of stream write permissions being confirmed before the stream item. The record can contain the same fields as a Stream Item Unconfirmed record, as well as two more below:

Field identifier Size Description Required?
0x10 1 byte Internal code, useful for debugging purposes. No
0x32 Variable Short message explaining cause of invalidity. No

Offchain Data Available – 0x34

This record is written when the data for an off-chain stream item becomes available, with the following fields:

Field identifier Size Description Required?
0x30 20 bytes ID of item in feed, for reference in other feed records. Yes
0x32 Variable Name of stream, as a UTF-8 encoded string. No
0x35 4 bytes Item format, where 0 means binary, 1 means text, and 2 means JSON. No
0x36 8 bytes Item data size in bytes, as a 64-bit small-endian integer. No
0x37 Variable Data for binary items – see maxshowndata note below. No
0x38 Variable Data for text items in UTF-8 encoding – see maxshowndata note below. No
0x39 Variable Data for JSON items in regular JSON format – see maxshowndata note below. No
0x29 4 bytes Time off-chain data was received, as a Unix timestamp in small-endian order. No
0x3B 1 byte Updated item flags. Current flags are 1 if the item data is available, 2 if the item data is off-chain, and 4 is the item is off-chain and contains multiple chunks. No
0x2C 40 bytes Updated item dataref, as a binary string – see maxshowndata note below. This may be different from a dataref provided for the item in a previous record. If so, both datarefs will work, but the latest one will offer better performance. No

As with the Item Received record, if an item’s data as stored in the transaction is longer than the feed’s maxshowndata setting, it will not be included within the feed file. Instead, that data can be retrieved by passing the dataref to the getdatarefdata or datareftobinarycache JSON-RPC APIs.

Offchain Data Purged – 0x35

This record is written when the off-chain data for a stream item is purged from local storage, with the following fields:

Field identifier Size Description Required?
0x30 20 bytes ID of item in feed, for reference in other feed records. Yes
0x32 Variable Name of stream, as a UTF-8 encoded string. No