Supported Open Metrics

The Mavkit node is able to produce metrics information and serve them in the Open Metrics format, an emerging standard for exposing metrics data, especially used in cloud-based systems.

The Mavkit node supports the following metrics, characterized by: the name of the metric, the type of the metric as in the open metrics specification, a user friendly description on the metric and a list of labels (that can be used to aggregate or query metrics).

For more information check the openmetrics specification: https://openmetrics.io/

Name	Type	Description	Labels
mavkit_distributed_db_message_block_header_messages	Counter	Number of block_header messages	action
mavkit_distributed_db_message_checkpoint_messages	Counter	Number of checkpoint messages	action
mavkit_distributed_db_message_current_branch_messages	Counter	Number of current_branch messages	action
mavkit_distributed_db_message_current_head_messages	Counter	Number of current_head messages	action
mavkit_distributed_db_message_deactivate_messages	Counter	Number of deactivate messages	action
mavkit_distributed_db_message_get_block_headers_messages	Counter	Number of get_block_headers messages	action
mavkit_distributed_db_message_get_checkpoint_messages	Counter	Number of get_checkpoint messages	action
mavkit_distributed_db_message_get_current_branch_messages	Counter	Number of get_current_branch messages	action
mavkit_distributed_db_message_get_current_head_messages	Counter	Number of get_current_head messages	action
mavkit_distributed_db_message_get_operations_for_blocks_messages	Counter	Number of get_operations_for_blocks messages	action
mavkit_distributed_db_message_get_operations_messages	Counter	Number of get_operations messages	action
mavkit_distributed_db_message_get_predecessor_header_messages	Counter	Number of get_predecessor_header messages	action
mavkit_distributed_db_message_get_protocol_branch_messages	Counter	Number of get_protocol_branch messages	action
mavkit_distributed_db_message_get_protocols_messages	Counter	Number of get_protocols messages	action
mavkit_distributed_db_message_operation_messages	Counter	Number of operation messages	action
mavkit_distributed_db_message_operations_for_block_messages	Counter	Number of operations_for_block messages	action
mavkit_distributed_db_message_predecessor_header_messages	Counter	Number of predecessor_header messages	action
mavkit_distributed_db_message_protocol_branch_messages	Counter	Number of protocol_branch messages	action
mavkit_distributed_db_message_protocol_messages	Counter	Number of protocol messages	action
mavkit_distributed_db_requester_table_length	Gauge	Number of entries (to grab) from the network present	requester_kind;entry_type
mavkit_mempool_pending_branch_delayed	Gauge	Mempool pending branch delayed operations count
mavkit_mempool_pending_branch_refused	Gauge	Mempool pending branch refused operations count
mavkit_mempool_pending_outdated	Gauge	Mempool pending outdated operations count
mavkit_mempool_pending_refused	Gauge	Mempool pending refused operations count
mavkit_mempool_pending_unprocessed	Gauge	Mempool pending unprocessed operations count
mavkit_mempool_pending_validated	Gauge	Mempool pending validated operations count
mavkit_mempool_worker_completion_count	Counter	Number of requests completed the block validator worker
mavkit_mempool_worker_error_count	Counter	Number of errors encountered by the block validator worker
mavkit_mempool_worker_request_count	Counter	Number of requests received by the block validator worker
mavkit_p2p_connections_active	Gauge	Number of active connections
mavkit_p2p_connections_incoming	Gauge	Number of incoming connections
mavkit_p2p_connections_outgoing	Gauge	Number of outgoing connections
mavkit_p2p_connections_private	Gauge	Number of private connections
mavkit_p2p_io_scheduler_current_inflow	Gauge	Current ingoing data rate
mavkit_p2p_io_scheduler_current_outflow	Gauge	Current outgoing data rate
mavkit_p2p_io_scheduler_total_recv	Gauge	Total amount of received data
mavkit_p2p_io_scheduler_total_sent	Gauge	Total amount of sent data (in bytes)
mavkit_p2p_messages_advertise_received	Counter	Number of advertise received
mavkit_p2p_messages_advertise_sent	Counter	Number of advertise sent
mavkit_p2p_messages_bootstrap_received	Counter	Number of bootstrap received
mavkit_p2p_messages_bootstrap_sent	Counter	Number of bootstrap sent
mavkit_p2p_messages_broadcast_message_sent	Counter	Number of user message sent by broadcasting
mavkit_p2p_messages_swap_ack_received	Counter	Number of swap acks received
mavkit_p2p_messages_swap_ack_sent	Counter	Number of swap acks sent
mavkit_p2p_messages_swap_requests_received	Counter	Number of swap received
mavkit_p2p_messages_swap_requests_sent	Counter	Number of swap sent
mavkit_p2p_messages_user_message_received	Counter	Number of user message received
mavkit_p2p_messages_user_message_received_error	Counter	Number of user message received that resulted in error
mavkit_p2p_messages_user_message_sent	Counter	Number of user message sent
mavkit_p2p_peers_accepted	Gauge	Number of accepted connections
mavkit_p2p_peers_disconnected	Gauge	Number of disconnected peers
mavkit_p2p_peers_running	Gauge	Number of running peers
mavkit_p2p_points_accepted	Gauge	Number of accepted points
mavkit_p2p_points_disconnected	Gauge	Number of disconnected points
mavkit_p2p_points_greylisted	Gauge	Number of greylisted points
mavkit_p2p_points_running	Gauge	Number of running points
mavkit_p2p_points_trusted	Gauge	Number of trusted points
mavkit_p2p_swap_fail	Counter	Number of failed swap
mavkit_p2p_swap_ignored	Counter	Number of ignored swap
mavkit_p2p_swap_success	Counter	Number of successful swap
mavkit_rpc_calls	Summary	RPC endpoint call counts and sum of execution times.	endpoint;method
mavkit_store_alternate_heads_count	Gauge	Current number of alternated heads known
mavkit_store_caboose_level	Gauge	Current caboose level
mavkit_store_checkpoint_level	Gauge	Current checkpoint level
mavkit_store_invalid_blocks	Gauge	Number of blocks known to be invalid stored on disk
mavkit_store_last_merge_time	Gauge	Time, in seconds, for the completion of the last store merge
mavkit_store_last_written_block_size	Gauge	Size, in bytes, of the last block written in store
mavkit_store_savepoint_level	Gauge	Current savepoint level
mavkit_validator_block_already_commited_blocks_count	Counter	Number of requests to validate a block already handled
mavkit_validator_block_last_finished_request_completion_timestamp	Gauge	Timestamp at which the latest request handled by the worker was completed
mavkit_validator_block_last_finished_request_push_timestamp	Gauge	Reception timestamp of the latest request handled by the worker
mavkit_validator_block_last_finished_request_treatment_timestamp	Gauge	Timestamp at which the worker started processing of the latest request it handled
mavkit_validator_block_operations_per_pass	Gauge	Number of operations per pass for the last validated block	pass_id
mavkit_validator_block_outdated_blocks_count	Counter	Number of requests to validate a block older than the node’s checkpoint
mavkit_validator_block_preapplication_errors_count	Counter	Number of refused application simulations of blocks
mavkit_validator_block_preapplied_blocks_count	Counter	Number of successful application simulations of blocks
mavkit_validator_block_precheck_failed_count	Counter	Number of block validation requests where the prechecking of a block failed
mavkit_validator_block_validated_blocks_count	Counter	Number of requests to validate a valid block
mavkit_validator_block_validation_errors_after_precheck_count	Counter	Number of requests to validate an invalid but precheckable block
mavkit_validator_block_validation_errors_count	Counter	Number of requests to validate an invalid block
mavkit_validator_block_worker_completion_count	Counter	Number of requests completed the block validator worker
mavkit_validator_block_worker_error_count	Counter	Number of errors encountered by the block validator worker
mavkit_validator_block_worker_request_count	Counter	Number of requests received by the block validator worker
mavkit_validator_chain_branch_switch_count	Counter	Number of times the chain_validator switched branch	chain_id
mavkit_validator_chain_head_consumed_gas	Gauge	Gas consumed in the current node’s head	chain_id
mavkit_validator_chain_head_cycle	Gauge	Cycle of the current node’s head	chain_id
mavkit_validator_chain_head_increment_count	Counter	Number of times the chain_validator incremented its head for a direct successor	chain_id
mavkit_validator_chain_head_level	Gauge	Level of the current node’s head	chain_id
mavkit_validator_chain_head_round	Gauge	Round of the current node’s head	chain_id
mavkit_validator_chain_ignored_head_count	Counter	Number of requests where the chain validator ignored a new valid block with a lower fitness than its current head	chain_id
mavkit_validator_chain_is_bootstrapped	Gauge	Returns 1 if the node has bootstrapped, 0 otherwise.	chain_id
mavkit_validator_chain_last_finished_request_completion_timestamp	Gauge	Timestamp at which the latest request handled by the worker was completed	chain_id
mavkit_validator_chain_last_finished_request_push_timestamp	Gauge	Reception timestamp of the latest request handled by the worker	chain_id
mavkit_validator_chain_last_finished_request_treatment_timestamp	Gauge	Timestamp at which the worker started processing of the latest request it handled	chain_id
mavkit_validator_chain_synchronisation_status	Gauge	Returns 0 if the node is unsynchronised, 1 if the node is synchronised, 2 if the node is stuck.	chain_id
mavkit_validator_chain_worker_completion_count	Counter	Number of requests completed the block validator worker	chain_id
mavkit_validator_chain_worker_error_count	Counter	Number of errors encountered by the block validator worker	chain_id
mavkit_validator_chain_worker_request_count	Counter	Number of requests received by the block validator worker	chain_id
mavkit_validator_peer_connections	Counter	Number of time we connected to a peer.
mavkit_validator_peer_invalid_block	Counter	Number of time we received an invalid block from a peer.
mavkit_validator_peer_invalid_locator	Counter	Number of time we received an invalid locator from a peer.
mavkit_validator_peer_new_branch_completed	Counter	Number of time we successfuly completed a new branch request from a peer.
mavkit_validator_peer_new_head_completed	Counter	Number of time we successfuly completed a new head request from a peer.
mavkit_validator_peer_on_no_request_count	Counter	Number of time we did no hear new messages from a peer since the last timeout.
mavkit_validator_peer_operations_fetching_canceled_new_branch	Counter	Number of time we canceled the fetching of operations on a new branch request for a peer.
mavkit_validator_peer_operations_fetching_canceled_new_known_valid_head	Counter	Number of time we canceled the fetching of operations on a new head request for a peer.
mavkit_validator_peer_operations_fetching_canceled_new_unknown_head	Counter	Number of time we canceled the fetching of operations on a new head request or an unknown head for a peer.
mavkit_validator_peer_system_error	Counter	Number of time a request trigerred a system error from a peer.
mavkit_validator_peer_too_short_locator	Counter	Number of time we received a too short locator from a peer.
mavkit_validator_peer_unavailable_protocol	Counter	Number of time we received an unknown protocol from a peer.
mavkit_validator_peer_unknown_ancestor	Counter	Number of time we received a locator with an unknown ancestor from a peer.
mavkit_validator_peer_unknown_error	Counter	Number of time an unknown error happened for a peer.
mavkit_version	Gauge	Node version	version;chain_name;distributed_db_version;p2p_version;commit_hash;commit_date
ocaml_gc_allocated_bytes	Counter	Total number of bytes allocated since the program was started.
ocaml_gc_compactions	Counter	Number of heap compactions since the program was started.
ocaml_gc_heap_words	Gauge	Total size of the major heap, in words.
ocaml_gc_major_collections	Counter	Number of major collection cycles completed since the program was started.
ocaml_gc_major_words	Counter	Number of words allocated in the major heap since the program was started.
ocaml_gc_minor_collections	Counter	Number of minor collection cycles completed since the program was started.
ocaml_gc_top_heap_words	Counter	Maximum size reached by the major heap, in words.
process_cpu_seconds_total	Counter	Total user and system CPU time spent in seconds.
process_start_time_seconds	Counter	Start time of the process since unix epoch in seconds.
prometheus_logs_messages_total	Counter	Total number of messages logged	level;src

Example

In the following, we indicate a typical monitoring setup for Mavkit developers. For more details on setting up the node for monitoring see Monitoring an Mavkit Node.

To instruct the Mavkit node to produce metrics, the user needs to pass the option --metrics-addr=<ADDR>:<PORT>. The port specified on the command line is the port where the integrated open metrics server will be available (9932 by default). The address defaults to localhost. When the option is not supplied at all, no metrics are produced. Ex.:

mavkit-node run --metrics-addr=:9091

To query the open metrics server the user can simply query the node.

Ex.:

curl http://<node_addr>:9091/metrics

Collecting metrics

Different third-party tools can be used to query the Mavkit node and collect metrics from it. Let us illustrates this with the example of a Prometheus server.

Update the Prometheus configuration file (typically, prometheus.yml) to add a “scrape job” - that is how Prometheus is made aware of a new data source - using adequate values:

• job_name: Use a unique name among other scrape jobs. All metrics collected through this job will have automatically a ‘job’ label with this value added to it

• targets: The URL of Mavkit node.

- job_name: 'mavkit-metrics'
    scheme: http
    static_configs:
      - targets: ['localhost:9091']