주요 콘텐츠로 건너뛰기

Introduction to Ardor v3

This document aims to summarize the changes done in Ardor version 3 in order to help developers who built on Ardor 2 migrate their applications to Ardor 3.

 

Introduction

A major innovation in Ardor is the pruning of child chain transactions. Such pruning is possible because transactions that don't modify the ARDR balances (or in other ways affect the proof-of-stake calculations) are not required for finding the chain with best cumulative difficulty. Once a node has found the best chain, it can trust the validity of any data in a given block in that best chain with the same confidence it can trust the validity of a payment included in that block (as far as the protocol enforces forgers to validate this data).

So we exploit this observation by storing the result of the execution of all transactions, i.e. the state of the ledger, into an authenticated data structure, and also including a short digest of this ledger state in each block. A node that processed the blockchain can trust (with certain level of confidence) the validity of the digest if it got enough number of confirmations. Having this digest, the node can download and recreate the state at the respective height from other not trusted nodes.

 

Merkelized State Tree

Similarly to other protocols, the authenticated data structure is a tree, which is "Merkelized" - meaning that a pointer to a node in the tree is the hash of that tree node. Unlike other protocols, Ardor has a complex state, which contains not only balances, but various other types of objects with certain relations between them. This complexity is caused by the big variety of features supported by Ardor's child chains. We won't get into details about the structure of the Merkelized State Tree, what is important for this document is that it contains only data necessary for the consensus rules and also that each block now contains the root hash of the Merkelized State Tree to which the block is applied. After applying the block (and unless the block is empty), the Merkelized State Tree changes, and so does its root. The new root is stored in the next block. For conciseness, we will use the term "snapshot" to denote this data structure.

 

Block byte format change

The root hash of the Merkelized State Tree is a 256-bit value, which is now stored between the previous block hash and the block signature in the block bytes. Here is the relevant piece of code which builds the block bytes:

ByteBuffer buffer = ByteBuffer.allocate(size);
buffer.order(ByteOrder.LITTLE_ENDIAN);
buffer.putInt(version);
buffer.putInt(timestamp);
buffer.putLong(previousBlockId);
buffer.putInt(getFxtTransactions().size());
buffer.putLong(totalFeeFQT);
buffer.put(payloadHash);
buffer.put(getGeneratorPublicKey());
buffer.put(generationSignature);
buffer.put(previousBlockHash);
if (version >= SNAPSHOT_BLOCK_VERSION) {
    buffer.put(stateHashBeforeBlock);
}
if (blockSignature != null) {
    buffer.put(blockSignature);
}

 

Public key announcement in ChildBlock transaction.

Pruning is only possible for transactions that don't affect the proof-of-stake calculations. A problem which we didn't identify before the Ardor launch is that the Proof-of-Stake algorithm depends on the height at which the forger's public key is announced or added to the database. Since all transactions (including child chain transactions) can announce the public key, a node processing the parent chain will need such child chain transactions in order to create a public key record at the correct height and not refuse blocks generated by accounts that announced their public key with pruned transaction. To mitigate this problem we are adding all announced public keys to the ChildBlockAttachment data - so that they are available to blockchain processors even after the transactions have been pruned. The ChildBlock transaction is considered invalid if a child transaction is announcing a public key which is not added to the ChildBlockAttachment by the bundler (so bundlers have the incentive to announce the public keys). And while changing the public key announcement code, we also removed the sender's public key from all transactions and created a SenderPublicKeyAnnouncement appendix. This will save 24 redundant bytes from all transactions created by some account except from the first one.

 

Recipient public key announcement on parent chain

Another change in Ardor v3 is that the recipient's public key can be announced with transaction on the parent (ARDR) chain.

 

Transaction byte format change

In Ardor v2 we have a PublicKeyAnnouncement transaction appendix which announces the public key of the recipient account. The sender's public key is part of the transaction bytes - it is not an appendix. In Ardor v3 we extracted the sender's public key as a new SenderPublicKeyAnnouncement transaction appendix and in the transaction bytes we store only the sender ID. The SenderPublicKeyAnnouncement is mandatory only for the first transaction issued by an account. This saves 24 bytes from each transaction and also simplifies the announcement of public keys in ChildBlock transactions.

Here is the relevant piece of code which builds the transaction bytes:

ByteBuffer buffer = ByteBuffer.allocate(includePrunable ? getFullSize() : getSize());
buffer.order(ByteOrder.LITTLE_ENDIAN);
buffer.putInt(getChain().getId());
buffer.put(getType().getType());
buffer.put(getType().getSubtype());
buffer.put(getVersion());
buffer.putInt(getTimestamp());
buffer.putShort(getDeadline());
if (getVersion() < SENDER_PUBLIC_KEY_ANNOUNCEMENT_VERSION) {
    buffer.put(getSenderPublicKey());
} else {
    buffer.putLong(getSenderId());
}
buffer.putLong(getRecipientId());
buffer.putLong(getAmount());
buffer.putLong(getFee());
buffer.put(getSignature() != null ? getSignature() : new byte[64]);
buffer.putInt(getECBlockHeight());
buffer.putLong(getECBlockId());
putAppendages(buffer, includePrunable);

And there is a new SenderPublicKeyAnnouncement appendix with type 128 (2^7)

 

Referencing pruned transactions

There is a feature in Ardor (actually inherited from Nxt) that a transaction can reference another transaction. The consensus rules require that the referenced transaction is included in the blockchain before the referencing one can be included. Moreover, since during the Nxt development we were not sure how pruning will work, we added another consensus rule that for any reference chain of length up to 10, the time duration between the first and the last transaction cannot exceed 60 days. Now that we assume a child chain transaction is valid once it has been in the blockchain for more than 720 blocks, we can modify these restrictions to not require storing child transactions for long periods of time.

Apart from that, there is a by-transaction phasing voting model which allows the linked transaction to be in the blockchain before the phased with by-transaction voting transaction. Similarly to the referenced transaction, back in the day we added a requirement that the linked transaction, if it is already in the blockchain, should not be older than 60 days before the phased transaction. Today, this rule cannot be consistently checked between synced from snapshot nodes and nodes that have all child transactions. This is because the nodes which processed child chains will have some older than 60 days transaction and will not accept a phased transaction that links to it, while a node that synced from snapshot will not have the old transaction and will allow the phased transaction assuming that the linked transaction will be provided in the future. So we had to revise this rule.

To summarize, the changes we make for Ardor v3 are as follows:

  1. Instead of keeping all child transactions, we only keep the hashes of all child transactions for at least 60*1440 blocks before the current height. When a node syncs from snapshot, it also restores the child transaction hashes from the last 60*1440 blocks before the snapshot height.
  2. When checking the referenced transaction, we check that only the directly referenced transaction (not the whole reference chain) is in the blockchain in the last 60*1440 blocks before current height
  3. We removed the validation of the linked by phasing transactions - since it cannot be consistently performed. And we changed the logic of counting votes in the by-transaction model: we count linked child transactions only if they are accepted not more than (MAX_REFERENCED_TRANSACTION_INTERVAL - MAX_PHASING_DURATION) blocks before the phased transaction.

 

Coin Exchange

Transactions that exchange ARDR for some child chain token must set their EC block to a block not older than 1440 blocks before the current height. This is to prevent replaying these transactions on long range attacker's chain.

 

Blockchain processing

As already mentioned, we introduced a new type of Ardor full node - one which synced from snapshot and didn't process the whole child transactions history. The old way of processing the blockchain is still functional. Switching between these types of nodes is done via the nxt.pruneChildChains property. The property cannot be changed from true to false if the blockchain database is already created. When nxt.pruneChildChains is false there shouldn't be any difference in the blockchain processing and the available data in the DB, compared to Ardor v2.

Downloading the blockchain when nxt.pruneChildChains is true

  1. Download only the parent chain until we get the most recent possible block.
  2. Choose a snapshot block and make sure it is not too old. If it is, we are likely on a fork, so continue trying to download the parent chain. The snapshot block must also have certain number of confirmations - preferably more than 720. Or else the node risks to download rogue snapshot. After the snapshot is downloaded and imported, popping-off before the snapshot height is not allowed. See the nxt.stateSnapshotConfirmations configuration.
  3. Download the snapshot data and incrementally verify its authenticity by using the root hash from the snapshot block.
  4. Restore all child transaction hashes from the last 60*1440 blocks
  5. Restore phased transactions that are not finished at snapshot height. Their IDs are stored in the snapshot, together with any votes accepted before the snapshot block
  6. Import the snapshot into the relational DB used by the consensus rules.

Processing of all chains

Nodes with nxt.pruneChildChains=false and nodes that successfully synced from snapshot are processing the transactions from all chains in similar fashion as in Ardor v2 (of course there are changes in the consensus rules as specified above). A big difference from Ardor v2 is that now all nodes are updating not only the relational DB but also the snapshot data structure. They need to do this in order to check the validity of the root hash of each block. Nodes with nxt.pruneChildChains=true are additionally pruning old child transactions.

 

Missing data in snapshot-synced nodes

The snapshot contains only data necessary for the consensus rules. During the blockchain processing, Ardor additionally generates in its relational DB historic data which is not needed by consensus like executed trades and others. This data won't be available in nodes synced from snapshot, for the blocks before the snapshot height. Here is a list of DB tables which are not restored from snapshot:

Child chain schema

transaction

Child transactions created before the snapshot height will be missing. Transactions are also pruned.

prunable_message

All messages before the snapshot height are pruned. Currently restoring messages of pruned transactions fails

tagged_data (Feature: Data Cloud)

All data before the snapshot height is pruned. Currently restoring tagged data of pruned transactions fails

exchange (Feature: Monetary System)

Affected APIs:

  • GetAllExchanges
  • GetExchanges
  • GetLastExchanges
  • GetExchangesByExchangeRequest
  • GetExchangesByOffer
exchange_request (Feature: Monetary System)

Affected APIs: GetAccountExchangeRequests

buy_offer and sell_offer (Feature: Monetary System)

The creation_height of the currency exchange offers is set to the snapshot height.

trade (Feature: Asset Exchange)

Affected APIs:

  • GetAllTrades
  • GetOrderTrades
  • GetTrades
  • GetLastTrades
asset_dividend (Feature: Asset Exchange)

Affected APIs:

  • GetAssetDividends
vote (Feature: Voting System)

To save snapshot space, poll votes are deleted after the poll has finished. So a finished poll won't restore its votes from the snapshot. A currently active poll will restore them.

Affected APIs:

  • GetPollVotes
  • GetPollVote
poll_result (Feature: Voting System)

Affected APIs:

  • GetPollResult
purchase_feedback (Feature: Digital Goods Store)

The feedbackNotes field of any purchase object returned by the API will miss feedback created before the snapshot height

purchase_public_feedback (Feature: Digital Goods Store)

The publicFeedbacks field of any purchase object returned by the API will miss public feedback created before the snapshot height

 

PUBLIC schema

asset_history (Feature: Asset Exchange)

Affected APIs:

  • GetAssetHistory
asset_transfer (Feature: Asset Exchange)

Affected APIs:

  • GetAssetTransfers
  • The numberOfTransfers field of any asset objects returned by API
currency_transfer (Feature: Monetary System)

Affected APIs:

  • GetCurrencyTransfers
  • The numberOfTransfers field of any currency objects returned by API
coin_trade_fxt (Feature: Coin Exchange)

Affected APIs:

  • GetCoinExchangeTrade
  • GetCoinExchangeTrades
account_ledger (Feature: Account Ledger)

For heights before snapshot block account ledger will have only events for ARDR chain

 

GetState API

In the response of the GetState API, if includeCounts parameter is true, the following counts will not include records created before the snapshot height:

  • numberOfExchanges
  • numberOfExchangeRequests
  • numberOfTrades
  • numberOfVotes
  • numberOfTransfers
  • numberOfCurrencyTransfers

 

API additions

getBlockchainStatus

Added blockchainProcessingStage to the response. Possible values:

  • PARENT_CHAIN_ONLY
  • DOWNLOAD_SNAPSHOT_DATA
  • RESTORE_TRANSACTION_HASHES
  • DOWNLOAD_PHASED_TRANSACTIONS
  • IMPORT_SNAPSHOT_DATA
  • ALL_CHAINS
APIs that return blocks

Added stateHash field to the block JSON object. This is a 256-bit value encoded as hex string

trimDerivedTables

Trims also the old records of the state snapshot and prunes eligible transactions if nxt.pruneChildChains is true

labelConvert utility

This is a new API call which allows converting labels used in the snapshot data structure to more human-readable formats like 64-bit IDs or account IDs.

 

Peer network changes

GetStateData and StateDataMessage

These are two messages for requesting and returning blocks of the snapshot data structure

GetNextParentBlocks

Same as the currently available GetNextBlocks message, but the result BlocksMessage contains only parent chain transactions

 

Future work

  • Restoring pruned transactions
  • Expiration of the ask and bid asset orders. Without this, an order is a permanent data until canceled, so if not canceled, it will be stored in the state tree forever.
  • Expiration of coin orders
  • Upper limit on the expiration height of MS exchange offers, so that it cannot be set too far in the future.
  • Expiration of marketplace goods or higher fee for permanent goods