# State Sync
Learn about Tendermint Core state sync and support offered by the Cosmos SDK.
Note: Only curious about how to sync a node with the network? Skip to this section.
# Tendermint Core State Sync
State sync allows a new node to join a network by fetching a snapshot of the network state at a recent height, instead of fetching and replaying all historical blocks. Since application state is smaller than the combination of all blocks, and restoring state is faster than replaying blocks, this reduces the time to sync with the network from days to minutes.
This section of the document provides a brief overview of the Tendermint state sync protocol, and how to sync a node. For more details, refer to the ABCI Application Guide (opens new window) and the ABCI Reference Documentation (opens new window).
# State Sync Snapshots
A guiding principle when designing Tendermint state sync was to give applications as much flexibility as possible. Therefore, Tendermint does not care what snapshots contain, how they are taken, or how they are restored. It is only concerned with discovering existing snapshots in the network, fetching them, and passing them to applications via ABCI. Tendermint uses light client verification to check the final app hash of a restored application against the chain app hash, but any further verification must be done by the application itself during restoration.
Snapshots consist of binary chunks in an arbitrary format. Chunks cannot be larger than 16 MB, but otherwise there are no restrictions. Snapshot metadata (opens new window), exchanged via ABCI and P2P, contains the following fields:
uint64): height at which the snapshot was taken
uint32): arbitrary application-specific format identifier (eg. version)
uint32): number of binary chunks in the snapshot
bytes): arbitrary snapshot hash for comparing snapshots across nodes
bytes): arbitrary binary snapshot metadata for use by applications
format field allows applications to change their snapshot format in a backwards-compatible manner, by providing snapshots in multiple formats, and choosing which formats to accept during restoration. This is useful when, for example, changing serialization or compression formats: as nodes may be able to provide snapshots to peers running older verions, or make use of old snapshots when starting up with a newer version.
hash field contains an arbitrary snapshot hash. Snapshots that have identical
metadata fields (including
hash) across nodes are considered identical, and
chunks will be fetched from any of these nodes. The
hash cannot be trusted, and is not verified by Tendermint itself, which guards against inadvertent nondeterminism in snapshot generation. The
hash may be verified by the application instead.
metadata field can contain any arbitrary metadata needed by the application. For example, the application may want to include chunk checksums to discard damaged
chunks, or Merkle proofs (opens new window) to verify each chunk individually against the chain app hash. In Protobuf (opens new window)-encoded form, snapshot
metadata messages cannot exceed 4 MB.
# Taking, Serving Snapshots
To enable state sync, some nodes in the network must take and serve snapshots. When a peer is attempting to state sync, an existing Tendermint node will call the following ABCI methods on the application to provide snapshot data to this peer:
ListSnapshots(opens new window): returns a list of available snapshots, with metadata
LoadSnapshotChunk(opens new window): returns binary chunk data
Snapshots should typically be generated at regular intervals rather than on-demand: this improves state sync performance, since snapshot generation can be slow, and avoids a denial-of-service vector where an adversary floods a node with such requests. Older snapshots can usually be removed, but it may be useful to keep at least the two most recent to avoid deleting the previous snapshot while a node is restoring it.
It is entirely up to the application to decide how to take snapshots, but it should strive to satisfy the following guarantees:
- Asynchronous: snapshotting should not halt block processing, and it should therefore happen asynchronously, eg. in a separate thread
- Consistent: snapshots should be taken at isolated heights, and should not be affected by concurrent writes, eg. due to block processing in the main thread
- Deterministic: snapshot
metadatashould be identical (at the byte level) across all nodes for a given
format, to ensure good availability of
As an example, this can be implemented as follows:
- Use a data store that supports transactions with snapshot isolation, such as RocksDB or BadgerDB.
- Start a read-only database transaction in the main thread after committing a block.
- Pass the database transaction handle into a newly spawned thread.
- Iterate over all data items in a deterministic order (eg. sorted by key)
- Serialize data items (eg. using Protobuf (opens new window)), and write them to a byte stream.
- Hash the byte stream, and split it into fixed-size chunks (eg. of 10 MB)
- Store the chunks in the file system as separate files.
- Write the snapshot metadata to a database or file, including the byte stream hash.
- Close the database transaction and exit the thread.
Applications may want to take additional steps as well, such as compressing the data, checksumming chunks, generating proofs for incremental verification, and removing old snapshots.
# Restoring Snapshots
When Tendermint starts, it will check whether the local node has any state (ie. whether
LastBlockHeight == 0), and if it doesn't, it will begin discovering snapshots via the P2P network. These snapshots will be provided to the local application via the following ABCI calls:
OfferSnapshot(snapshot, apphash)(opens new window): offers a discovered snapshot to the application
ApplySnapshotChunk(index, chunk, sender)(opens new window): applies a snapshot chunk
Discovered snapshots are offered to the application and it can respond by accepting the snapshot, rejecting it, rejecting the format, rejecting the senders, aborting state sync, and so on.
Once a snapshot is accepted, Tendermint will fetch chunks from across available peers, and apply them sequentially to the application, which can choose to accept the chunk, refetch it, reject the snapshot, reject the sender, abort state sync, and so on.
Once all chunks have been applied, Tendermint will call the
Info ABCI method (opens new window) on the application, and check that the app hash and height correspond to the trusted values from the chain. It will then switch to fast sync to fetch any remaining blocks (if enabled), before finally joining normal consensus operation.
How snapshots are actually restored is entirely up to the application, but will generally be the inverse of how they are generated. Note, however, that Tendermint only verifies snapshots after all chunks have been restored, and does not reject any P2P peers on its own. As long as the trusted hash and application code are correct, it is not possible for an adversary to cause a state synced node to have incorrect state when joining consensus, but it is up to the application to counteract state sync denial-of-service (eg. by implementing incremental verification, rejecting invalid peers).
Note that state synced nodes will have a truncated block history starting at the height of the restored snapshot, and there is currently no backfill of all block data (opens new window). Networks should consider broader implications of this, and may want to ensure at least a few archive nodes retain a complete block history, for both auditability and backup.
# Cosmos SDK State Sync
Cosmos SDK (opens new window) v0.40+ includes automatic support for state sync, so application developers only need to enable it to take advantage. They will not need to implement the state sync protocol described in the above section on Tendermint themselves.
# State Sync Snapshots
Tendermint Core handles most of the grunt work of discovering, exchanging, and verifying state data for state sync, but the application must take snapshots of its state at regular intervals, and make these available to Tendermint via ABCI calls, and be able to restore these when syncing a new node.
The Cosmos SDK stores application state in a data store called IAVL (opens new window), and each module can set up its own IAVL stores. At regular height intervals (which are configurable), the Cosmos SDK will export the contents of each store at that height, Protobuf (opens new window)-encode and compress it, and save it to a snapshot store in the local filesystem. Since IAVL keeps historical versions of data, these snapshots can be generated simultaneously with new blocks being executed. These snapshots will then be fetched by Tendermint via ABCI when a new node is state syncing.
Note that only IAVL stores that are managed by the Cosmos SDK can be snapshotted. If the application stores additional data in external data stores, there is currently no mechanism to include these in state sync snapshots, so the application therefore cannot make use of automatic state sync via the SDK. However, it is free to implement the state sync protocol itself as described in the ABCI Documentation (opens new window).
When a new node is state synced, Tendermint will fetch a snapshot from peers in the network and provide it to the local (empty) application, which will import it into its IAVL stores. Tendermint then verifies the application's app hash against the main blockchain using light client verification, and proceeds to execute blocks as usual. Note that a state synced node will only restore the application state for the height the snapshot was taken at, and will not contain historical data nor historical blocks.
# Enabling State Sync Snapshots
To enable state sync snapshots, an application using the CosmosSDK
BaseApp needs to set up a snapshot store (with a database and filesystem directory) and configure the snapshotting interval and the number of historical snapshots to keep. A minimal exmaple of this follows:
When starting the application with the appropriate flags, (eg.
--state-sync.snapshot-interval 1000 --state-sync.snapshot-keep-recent 2) it should generate snapshots and output log messages:
Note that the snapshot interval must currently be a multiple of the
pruning-keep-every (defaults to 100), to prevent heights from being pruned while taking snapshots. It's also usually a good idea to keep at least 2 recent snapshots, such that the previous snapshot isn't removed while a node is attempting to state sync using it.
# State Syncing a Node
Once a few nodes in a network have taken state sync snapshots, new nodes can join the network using state sync. To do this, the node should first be configured as usual, and the following pieces of information must be obtained for light client verification:
- Two available RPC servers (at least)
- Trusted height
- Block ID hash of trusted height
The trusted hash must be obtained from a trusted source (eg. a block explorer), but the RPC servers do not need to be trusted. Tendermint will use the hash to obtain trusted app hashes from the blockchain in order to verify restored application snapshots. The app hash and corresponding height are the only pieces of information that can be trusted when restoring snapshots. Everything else can be forged by adversaries.
In this guide we use Ubuntu 20.04
# Prepare system
Set the node name
# Use commands below for Testnet setup
Reset the node
Change config files (set the node name, add persistent peers, set indexer = "null")
Set the variables for start from snapshot
Output example (numbers will be different):
If output is OK do next
# Run uptickd
The node is now state synced, having joined the network in seconds