Module Mavryk_store_unix.Cemented_block_store

Persistent block store with linear history

The cemented block store is a store where blocks are stored linearly (by level) in chunks. Blocks in this store should not be reorganized anymore and are thus *cemented*. As these blocks should not be accessed regularly and especially their optionally stored metadata, the later are compressed using a zip format to save disk space. For each chunk of blocks, a dedicated file is used. Moreover, to enable easy access and to prevent too much on-disk reading, two indexed maps are used to retrieve blocks hash from their level and their level from the block hash.

The cemented block store contains a set of files updated each time a new chunk is added to the store. These files indicate which interval of blocks (w.r.t. their levels) are stored in it.

Invariants

This store is expected to respect the following invariants:

      ∀f_x=(i,j) ∈ F ∧ x < |F|, ∃f_y =(i', j'), x = y - 1 ∧ j + 1 = j'

meaning the concatenation of every chunk must be continuous.

Files format

The cemented block store is composed of the following files:

| <n> × <offset> | <n> × <block> |

where n is (j - i + 1), <offset> is a 8 bytes integer representing the absolute offset of a block where the k-th (with 0 <= k < n) offset stands for the absolute offset of the k-th block in the file and with <block>, a Block_repr.t value encoded using Block_repr.encoding (thus prefixed by the its size).

module Cemented_block_level_index : Index.S with type key = Mavryk_store_shared.Block_key.t and type value = Mavryk_store_shared.Block_level.t

On-disk index of block's hash to level.

module Cemented_block_hash_index : Index.S with type key = Mavryk_store_shared.Block_level.t and type value = Mavryk_store_shared.Block_key.t

On-disk index of block's level to hash.

type t

The type of the cemented block store

type cemented_metadata_file = {
  1. start_level : int32;
  2. end_level : int32;
  3. metadata_file : [ `Cemented_blocks_metadata ] Mavryk_store_shared.Naming.file;
}
type cemented_blocks_file = {
  1. start_level : int32;
  2. end_level : int32;
  3. file : [ `Cemented_blocks_file ] Mavryk_store_shared.Naming.file;
}

The type for cemented block chunks file description

val init : ?log_size:int -> [< `Chain_dir ] Mavryk_store_shared.Naming.directory -> readonly:bool -> t Mavryk_base.TzPervasives.tzresult Lwt.t

init ?log_size ~cemented_blocks_dir ~readonly creates or loads an existing cemented block store at path cemented_blocks_dir. cemented_blocks_dir will be created if it does not exists. If readonly is true, cementing blocks will result in an error. log_size determines the index cache size.

val close : t -> unit

close cemented_store closes the cemented_store opened files: its indexes.

val cemented_blocks_files : t -> cemented_blocks_file array option

cemented_blocks_files cemented_store returns the current array of cemented blocks chunks files. The returned array is sorted in ascending order such that the first element of the array is the lowest known cycle of the store.

val cemented_metadata_files : t -> cemented_metadata_file array option Mavryk_base.TzPervasives.tzresult Lwt.t

cemented_metadata_files cemented_store returns the current array of cemented metadata files. The returned array is sorted in ascending order such that the first element of the array is the lowest known cycle of the store.

val cemented_block_level_index : t -> Cemented_block_level_index.t

cemented_block_level_index block_store returns the hash to level index.

val cemented_block_hash_index : t -> Cemented_block_hash_index.t

cemented_block_hash_index block_store returns the level to hash index.

val load_table : [ `Cemented_blocks_dir ] Mavryk_store_shared.Naming.directory -> cemented_blocks_file array option Mavryk_base.TzPervasives.tzresult Lwt.t

load_table ~cemented_blocks_dir reads the cemented_blocks_dir directory and instantiate the cemented blocks chunks files.

val load_metadata_table : [ `Cemented_blocks_dir ] Mavryk_store_shared.Naming.directory -> cemented_metadata_file array option Mavryk_base.TzPervasives.tzresult Lwt.t

load_metadata_table ~cemented_blocks_dir similar to load_table, but for the cemented metadata files.

val find_block_file : t -> int32 -> cemented_blocks_file option

find_block_file cemented_store block_level lookups the cemented_store to find the cemented block chunk file that contains the block at level block_level. Returns None if the block cannot be found.

val is_cemented : t -> Mavryk_base.TzPervasives.Block_hash.t -> bool

is_cemented cemented_store block_hash checks if the block_hash is stored in the cemented_store.

val get_cemented_block_level : t -> Mavryk_base.TzPervasives.Block_hash.t -> int32 option

get_cemented_block_level cemented_store block_hash returns the level of the block_hash if present in cemented_store. Returns None otherwise.

val get_cemented_block_hash : t -> int32 -> Mavryk_base.TzPervasives.Block_hash.t option

get_cemented_block_hash cemented_store block_level returns the hash of the block at block_level if present in cemented_store. Returns None otherwise.

val read_block_metadata : t -> int32 -> Mavryk_store_shared.Block_repr.metadata option Mavryk_base.TzPervasives.tzresult Lwt.t

read_block_metadata cemented_store block_level returns the metadata of the block at block_level if present in cemented_store. Returns None otherwise.

val cement_blocks_metadata : t -> Mavryk_store_shared.Block_repr.t list -> unit Mavryk_base.TzPervasives.tzresult Lwt.t

cement_blocks_metadata cemented_store chunk compresses and stores the metadata of blocks present in chunk. If no block of the given chunk contains metadata, nothing is done. Otherwise, for every block containing metadata, an entry is written in the dedicated .zip metadata file.

We assume that the blocks containing metadata are contiguous and if at least one block has metadata, then the blocks from that block with metadata to the last block of chunk must have metadata. However, we do not check the validity of this assumption.

val get_lowest_cemented_level : t -> int32 option

get_lowest_cemented_level cemented_store returns the lowest cemented block in cemented_store, if it exists.

val get_highest_cemented_level : t -> int32 option

get_highest_cemented_level cemented_store returns the highest cemented block in cemented_store if it exists.

val get_cemented_block_by_level : t -> read_metadata:bool -> int32 -> Mavryk_store_shared.Block_repr.block option Mavryk_base.TzPervasives.tzresult Lwt.t

get_cemented_block_by_level cemented_store ~read_metadata level reads the cemented block at level in cemented_store, if it exists. It also tries to retrieves the metadata depending on read_metadata but do not fail if no metadata is available.

val get_cemented_block_by_hash : read_metadata:bool -> t -> Mavryk_base.TzPervasives.Block_hash.t -> Mavryk_store_shared.Block_repr.block option Mavryk_base.TzPervasives.tzresult Lwt.t

get_cemented_block_by_hash cemented_store hash reads the cemented block of hash in cemented_store, if it exists. It also retrieves the metadata depending on read_metadata.

type chunk_iterator = {
  1. chunk_length : int;
  2. reading_sequence : (Mavryk_base.TzPervasives.Block_hash.t * int * bytes) Mavryk_base.TzPervasives.tzresult Lwt.t Mavryk_base.TzPervasives.Seq.t;
}

The type used to describe reading sequences used to perform buffered block cementing.

val make_chunk_iterator : Mavryk_store_shared.Block_repr.block list -> chunk_iterator

make_chunk_iterator bl is an utility function that transforms a Block_repr.block list into a chunk_iterator

val cement_blocks : ?check_consistency:bool -> t -> write_metadata:bool -> chunk_iterator -> unit Mavryk_base.TzPervasives.tzresult Lwt.t

cement_blocks ?check_consistency cemented_store ~write_metadata chunk_iterator iterates from the lazy-bufferized chunk_iterator to cement blocks and to write their metadata if the flag write_metadata is set. check_consistency (default is true) ensures that the cycles are contiguous.

val trigger_gc : t -> Mavryk_shell_services.History_mode.t -> unit Lwt.t

trigger_gc cemented_store history_mode garbage collects metadata chunks and/or chunks from the cemented_store depending on the History_mode.t:

  • in Archive mode, nothing is done;
  • in Full offset mode, only offset chunks of metadata are kept;
  • in Rolling offset mode, only offset chunks of metadata and chunks are kept.

Important: when purging chunks of blocks, it is necessary to rewrite the index to remove garbage collected blocks. Therefore, the higher the offset is, the longest the GC phase will last.

val iter_cemented_file : (Mavryk_store_shared.Block_repr.block -> unit Lwt.t) -> cemented_blocks_file -> unit Mavryk_base.TzPervasives.tzresult Lwt.t

iter_cemented_file ~cemented_block_dir f block_file reads from the cemented block_file located in cemented_block_dir and applies f on every block.

Warning: in this version, exceptions are caught. Use raw_iter_cemented_file for manual exception management.

val raw_iter_cemented_file : (Mavryk_store_shared.Block_repr.block -> unit Lwt.t) -> cemented_blocks_file -> unit Lwt.t

Unsafe version of iter_cemented_file where internal exceptions/errors are not caught.

val check_indexes_consistency : ?post_step:(unit -> unit Lwt.t) -> ?genesis_hash:Mavryk_base.TzPervasives.Block_hash.t -> t -> unit Mavryk_base.TzPervasives.tzresult Lwt.t

check_indexes_consistency ?post_step ?genesis_hash cemented_store history_mode iterates over a partially initialized cemented_store that contains both chunks of blocks and indexes then check the consistency of each block: (hashes, predecessors and levels). The hash is not checked for genesis_hash and post_step is called after each treated chunk. This is used for snapshot imports.

val get_and_upgrade_offsets : Lwt_unix.file_descr -> int -> bytes Lwt.t

get_and_upgrade_offsets fd nb_blocks obtains the list of nb_blocks offsets from the beginning of the file given by file descriptor fd and upgrades their sizes from 32-bits to 64-bits, adjusting their values for the following blocks in the cemented blocks file.

val v_3_2_upgrade : [ `Chain_dir ] Mavryk_store_shared.Naming.directory -> unit Mavryk_base.TzPervasives.tzresult Lwt.t