Skip to content

Latest commit

 

History

History
113 lines (94 loc) · 5.17 KB

tip-382.md

File metadata and controls

113 lines (94 loc) · 5.17 KB
Raw
tip: 382
title: Optimize the data structure of account assets
author: wb_bupt@163.com
discussions to: https://github.com/tronprotocol/TIPs/issues/382
status: Final
type: Standards Track
category: Core
created: 2022-02-28

Simple Summary

This TIP is to optimize the data structure of account assets.

Abstract

In the process of executing the transaction, most of the time is spent in the deserialization operation of the account. Later analysis, it is caused by the account asset list being too large. Some accounts have thousands of assets. During the deserialization process, it takes a lot of time.

The asset data structure is a map. When performing asset operations, I found that the map has no size limit, and there are certain loopholes in the design.

message Account {
  ...
  map<string, int64> asset = 6;
  map<string, int64> assetV2 = 56;
  ...
}

This article details the optimization of the data structure of account assets.

Motivation

According to the latest online statistics, there are currently 79,185,490 accounts and 6,595 assets. The statistics of the number of account assets are as follows..

range of assets number of accounts
> 1000 2
500 ~ 1000 51
200 ~ 500 1910
100 ~ 200 13219
50 ~ 100 161214

Through the performance analysis of the flame graph, it can be seen that the deserialization of the account takes a lot of time. Randomly select a batch of accounts, and perform 10,000 deserialization of complete data and clear asset data respectively. The analysis results are as follows:

complete data clear asset data
total cost 340,123ms 29,575ms
average cost 69ms 6ms

It can be seen that the performance of the deserialization operation of the account data is improved by more than ten times after the assets are cleared.

In order to improve the performance of the blockchain, increase the TPS, it is necessary to reduce the transaction execution time. In order to reduce the deserialization time of the account, it is necessary to redesign the asset data structure.

Rationale

In order to prevent the infinite expansion of asset data, I have come up with the following two solutions, and you are welcome to propose better solutions.

  1. Limit the asset map size. Set the number of assets per account to no more than max-asset-count(1000), when account A transfers asset(1) to account B, The following two situations need to be considered. a. If the number of assets in account B is less than max-asset-count, the transfer can be done normally. b. If the number of assets in account B is greater than or equal to max-asset-count, and if account B contains asset(1), the transfer can be performed normally, otherwise the transfer is refused.

  2. Use k-v data to store assets, key is a string that connects the account ID and the asset number, and the value is the asset balance. The method of getting and setting asset(1) of account A (ID:12...) is as follows. a. get asset interface: assetStore.get("12...1") b. set asset interface: assetStore.put("12...1", 100)

The advantage of scheme 1 is that the implementation logic is relatively simple, but when the assets are full, new asset transfers cannot be accepted, and the user experience is not very good. Scheme 2 does not affect the asset transfer, but the implementation logic is more complicated.

Implementation

Choose to use the k-v storage scheme to achieve, mainly involves the import and splitting of assets.

Import assets only when asset operations are involved.

public long getAssetV2(String key) {
    importAsset(key.getBytes());
    Long balance = this.account.getAssetV2Map().get(key);
    return balance == null ? 0 : balance;
  }

  public Map<String, Long> getAssetMapV2() {
    importAllAsset();
    Map<String, Long> assetMap = this.account.getAssetV2Map();
    if (assetMap.isEmpty()) {
      assetMap = Maps.newHashMap();
    }
    return assetMap;
  }

When the account is written to the DB, asset separation is performed.

private void processAccount(Map<WrappedByteArray, WrappedByteArray> batch) {
    AccountAssetStore assetStore = ChainBaseManager.getInstance().getAccountAssetStore();
    Map<WrappedByteArray, WrappedByteArray> accounts = new HashMap<>();
    Map<WrappedByteArray, WrappedByteArray> assets = new HashMap<>();
    batch.forEach((k, v) -> {
      if (ByteArray.isEmpty(v.getBytes())) {
        accounts.put(k, v);
        assets.putAll(assetStore.getDeletedAssets(k.getBytes()));
      } else {
        AccountCapsule item = new AccountCapsule(v.getBytes());
        if (!item.getAssetOptimized()) {
          assets.putAll(assetStore.getDeletedAssets(k.getBytes()));
          item.setAssetOptimized(true);
        }
        assets.putAll(assetStore.getAssets(item.getInstance()));
        item.clearAsset();
        accounts.put(k, WrappedByteArray.of(item.getData()));
      }
    });
    ((Flusher) db).flush(accounts);
    if (assets.size() > 0) {
      assetStore.updateByBatch(AccountAssetStore.convert(assets));
    }
  }