Store (mavkit-shell-libs.Mavryk_store

Description

This component handles the on-disk storage of static objects such as blocks, operations, block's metadata, protocols and chain data. The store also handle the chain's current state: current head, invalid blocks, active testchains, ...

Concurrency

This module is designed to handle concurrent accesses to the store. Data-races and deadlocks might still happen through the use of potential dangerous calls that are documented as such. Both a mutex and a lockfile are present to handle concurrent accesses at process levels.

Context handling

The store manages the context and thus handles the context initialization through Lib_context.Context.init. This is done, among others reasons, for a consistency checking when storing a block. The store never commits a context on-disk.

History mode handling

This store handles the three different Mavryk_base.History_mode.t:

Archive: maintains every block that is part of the chain including their metadata.

Full <offset>: maintains every block that is part of the chain but prune the metadata for blocks that are below the following threshold level: last_preserved_block_level of the current head
offset cycles.

Rolling <offset>: maintains rolling windows which contain recent blocks that are part of the chain, along with their metadata. It prunes everything that is below the following threshold level: last_preserved_block_level of the current head - offset cycles.

Protocol store

The store has a global protocol store which may be shared by different chain stores.

Chain stores and merging

The store has a toplevel chain store which is called the main_chain_store and which can be accessed through the store's interface. It is either created if the associated directory is not present or loaded if it was previously created. This main_chain_store is able to recursively spawn testchains which are also chain stores. Those testchains are also recursively able to spawn their own testchains.

Each chain store possesses its own block store and therefore its own historic. It also maintains a state throughout its run which is composed of:

current_head: the current head of the chain.

target: the optional block for which the chain must pass. The store will not allow to store blocks past this target's block if they are not its successors.

checkpoint: the block that represents the no fork point level: blocks that are below this block are discarded.

savepoint: the block that represents the lowest block with metadata in the chain.

caboose: the block that represents the lowest block known in the chain (with or without metadata).

More details and invariants on these points are provided in the Chain module description below.

When a block is promoted as head of the chain (through Chain.set_head) the following happens:

A check is made if this head is consistent (i.e. if it's not below the checkpoint);

If the last_preserved_block_level of the head is different from the previous head's one, then we can establish that a cycle has been completed and we can start cementing this cycle by "triggering a merge".

A merge phase consists of establishing the interval of blocks to cement, which is trivially last_preserved_block_level(new_head) to last_preserved_block_level(prev_head), but also, for Full and Rolling history modes, keep some extra blocks so that we make sure to keep blocks above max_operation_ttl(last_preserved_block_level(checkpoint)). This is done to make sure that we can export snapshots at the checkpoint level later on. This merging operation is asynchronous, the changes will be committed on disk only when the merge succeeds. Before that, we only retain the changes in RAM so we may keep storing new blocks without blocking. If the process crashes while a merge is happening, the state is reloaded before the merging point. More details are given in Chain.set_head.

Files hierarchy

The store directory is organized as follows:

/<protocol_dir>/ the directory containing stored protocols

/<protocol_dir>/<protocol_hash_b58>* files containing the encoded protocol.

/<chain_id_b58>/ the chain_store_dir directory containing the main chain store.

chain_store_dir is a symbol for all chain stores directory hierarchy.

chain_store_dir/<lock> the lockfile.

chain_store_dir/<config.json> the chain store's configuration as a JSON file.

chain_store_dir/<block_store> contains every file mentioned in Block_store's format.

chain_store_dir/<stored_data>* files containing encoded simple data structures such as: genesis block, checkpoint, savepoint, caboose, protocol levels, forked chains, invalid blocks, etc.

chain_store_dir/testchains/<testchain_id_b58>/ contains the chains_store_dir's test chain, based on a similar hierarchy.

type t

The abstract type to manipulate the global store.

type store = t

The type alias for the global store.

type chain_store

The abstract type for a chain store. Equivalent to Chain.t.

Initialization

val init : 
  ?patch_context:
    (Mavryk_protocol_environment.Context.t ->
      Mavryk_protocol_environment.Context.t Mavryk_base.TzPervasives.tzresult
        Lwt.t) ->
  ?commit_genesis:
    (chain_id:Mavryk_base.TzPervasives.Chain_id.t ->
      Mavryk_base.TzPervasives.Context_hash.t Mavryk_base.TzPervasives.tzresult
        Lwt.t) ->
  ?history_mode:Mavryk_shell_services.History_mode.t ->
  ?readonly:bool ->
  ?block_cache_limit:int ->
  store_dir:string ->
  context_dir:string ->
  allow_testchains:bool ->
  Mavryk_base.TzPervasives.Genesis.t ->
  store Mavryk_base.TzPervasives.tzresult Lwt.t

init ?patch_context ?commit_genesis ?history_mode ?block_cache_limit ~store_dir ~context_dir ~allow_testchains genesis initializes the store and a main chain store. If store_dir (resp. context_dir) does not exist, a fresh store (resp. context) is created. Otherwise, it loads the store (resp. context) from reading the adequate directory. If allow_testchains is passed, the store will be able to fork chains and instantiate testchain's sub chain stores, for all chains contained in the store. The chain store created is based on the genesis provided. Its chain identifier will be computed using the Chain_id.of_block_hash function.

parameter patch_context
the handle called when initializing the context. It usually is passed when creating a sandboxed chain.

parameter commit_genesis
overrides the default initial genesis commit when the node is created. This is used for when the context must be in readonly (e.g. started by an external validator) and we want to prevent writing in the context. Warning passing this argument will initialize the context in readonly. Default: Context.commit_genesis is called with genesis

parameter history_mode
the history mode used throughout the store. If a directory already exists and the given history_mode is different, the initialization will fail. Default: History_mode.default (which should correspond to full with 5 extra preserved cycles.)

parameter block_cache_limit
allows to override the size of the block cache to use. The minimal value is 1.

parameter readonly
a flag that, if set to true, prevent writing throughout the store and context. Default: false

val main_chain_store : store -> chain_store

main_chain_store global_store returns the main chain store.

val close_store : store -> unit Lwt.t

close_store global_store closes the underlying block store and context along with every opened file descriptors for every chain store and testchain present in global_store. If the store is already closed, this function is idempotent.

val may_switch_history_mode : 
  store_dir:string ->
  context_dir:string ->
  Mavryk_base.TzPervasives.Genesis.t ->
  new_history_mode:Mavryk_shell_services.History_mode.t ->
  unit Mavryk_base.TzPervasives.tzresult Lwt.t

may_switch_history_mode ?patch_context ~store_dir ~context_dir genesis ~new_history_mode tries switching the store located at store_dir (if present) to new_history_mode when possible.

Accessors

val directory : store -> [ `Store_dir ] Mavryk_store_shared.Naming.directory

directory global_store returns the path where global_store is stored. This corresponds to the store_dir argument passed to the init function.

val context_index : store -> Mavryk_context_ops.Context_ops.index

context_index global_store returns the context's index initialized in global_store.

val allow_testchains : store -> bool

allow_testchains global_store returns true if the store is allowed to fork testchains.

val all_chain_stores : store -> chain_store list Lwt.t

all_chain_stores global_store returns every initialized chain store in global_store. The resulting list also comprises the initialized testchains. If allow_testchains is false, the list will only contain a single element.

val get_chain_store : 
  store ->
  Mavryk_base.TzPervasives.Chain_id.t ->
  chain_store Mavryk_base.TzPervasives.tzresult Lwt.t

get_chain_store global_store chain_id returns the initialized chain store in global_store associated to chain_id.

val get_chain_store_opt : 
  store ->
  Mavryk_base.TzPervasives.Chain_id.t ->
  chain_store option Lwt.t

get_chain_store_opt global_store chain_id optional version of get_chain_store.

val make_pp_store : store -> (Stdlib.Format.formatter -> unit -> unit) Lwt.t

val make_pp_chain_store : 
  chain_store ->
  (Stdlib.Format.formatter -> unit -> unit) Lwt.t

module Block : sig ... end

The module for handling block-related operations such as storing and reading data associated to a single block.

module Chain : sig ... end

The module for handling chain-related operations such as setting the head of the chain, updating the chain state, forking testchains, ...

val global_block_watcher : 
  store ->
  (Chain.chain_store * Block.block) Lwt_stream.t * Lwt_watcher.stopper

global_block_watcher global_store instantiates a new block watcher for every chain store active in global_store.

module Protocol : sig ... end

The module for handling protocol-related operations.

module Chain_traversal : sig ... end

The utility module used to traverse the chain.

val v_3_0_upgrade : 
  store_dir:string ->
  Mavryk_base.TzPervasives.Genesis.t ->
  unit Mavryk_base.TzPervasives.tzresult Lwt.t

Upgrade a v_2 to v_3 store by rewriting the block store and the protocol level's table.

Warning Not backward-compatible.