Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Dec 9, 2025

Refactors the pools schema to use separate state tables for initialization and fees, combined through a unified VIEW. Extends support to all DEX protocols (Uniswap V1-V4, SunPump, CurveFi, Balancer, Bancor).

Schema Refactoring

state_pools_initialize (renamed from pools)

  • Tracks pool initialization events for all protocols
  • Uses Array(String) for tokens with materialized columns for token0, token1, token2, token3
  • Supports multi-token pools (e.g., Balancer with 4+ tokens)
  • Removed fee column (moved to state_pools_fees)

state_pools_fees (renamed from pools_fees)

  • Tracks both initial fees (from initialize events) and fee updates
  • Uses ReplacingMergeTree(block_num) to maintain latest fee per pool
  • Includes MVs for:
    • Initialize events: Uniswap V1-V4, CurveFi (PlainPoolDeployed, MetaPoolDeployed)
    • Fee updates: Balancer (SwapFeePercentage, AggregateSwapFeePercentage), Bancor (ConversionFeeUpdate), CurveFi (CommitNewFee, NewFee), SunPump (PurchaseFeeSet, SaleFeeSet)

pools VIEW

  • Combines state_pools_initialize with latest fees from state_pools_fees
  • Uses LEFT JOIN to include pools even without tracked fees
  • Provides unified interface for querying complete pool information

Pool Creation MVs

  • Uniswap V1: mv_uniswap_v1_new_exchange - Zero address + token pair
  • Uniswap V2: mv_uniswap_v2_pair_created - Token pair array
  • Uniswap V3: mv_uniswap_v3_pool_created - Token pair array
  • Uniswap V4: mv_uniswap_v4_initialize - Currency pair array
  • SunPump: mv_sunpump_token_create, mv_sunpump_token_create_legacy - Zero address + token pair
  • CurveFi: mv_curvefi_plain_pool_deployed, mv_curvefi_meta_pool_deployed - Multi-token support via splitByChar
  • Balancer: mv_balancer_pool_registered - Empty array (variable tokens)
  • Bancor: mv_bancor_activation - Empty array (variable reserve tokens)

Implementation Notes

  • Single-pair protocols (Uniswap V1, SunPump) use zero address for native currency as first token in array
  • Multi-token pools (Balancer, Bancor) use empty array, tokens populated separately
  • CurveFi uses splitByChar(',', coins) to convert comma-separated coins to array
  • Balancer fee conversion: 1e18 fixed-point → basis points via division by 1e14
  • Protocol naming uses hyphens (uniswap-v1) matching existing convention in swaps table
  • CoW Protocol excluded (order-based DEX, no liquidity pools)
-- Example: CurveFi pool initialization with tokens array
CREATE MATERIALIZED VIEW mv_curvefi_plain_pool_deployed TO state_pools_initialize AS
SELECT block_num, block_hash, timestamp, minute, tx_hash,
       log_address AS factory,
       address AS pool,
       splitByChar(',', coins) AS tokens,
       'curvefi' AS protocol
FROM curvefi_plain_pool_deployed
WHERE length(splitByChar(',', coins)) >= 2
  AND arrayElement(splitByChar(',', coins), 1) != ''
  AND arrayElement(splitByChar(',', coins), 2) != '';
Original prompt

This section details on the original issue you should resolve

<issue_title>Add other protocols to pools MV</issue_title>
<issue_description>To your best ability, add all other missing protocols to pools MV in clickhouse-evm-dex

If there are no fees for certain protocols, use Nullable(UInt32) as current place holder.

For protocols that have changing fees, create a new table pools_fees to track latest updates using ReplacingMergeTree(block_num)</issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits December 9, 2025 19:14
- Updated pools table to use Nullable(UInt32) for fee column
- Added Uniswap V1 pool creation materialized view
- Added SunPump pool creation materialized views (TokenCreate and TokenCreateLegacy)
- Added CurveFi pool creation materialized views (PlainPoolDeployed and MetaPoolDeployed)
- Added Balancer pool creation materialized view (PoolRegistered)
- Added Bancor pool creation materialized view (Activation)
- Created pools_fees table with ReplacingMergeTree(block_num) for dynamic fee tracking
- Added fee tracking MVs for Balancer (SwapFeePercentage, AggregateSwapFeePercentage)
- Added fee tracking MVs for Bancor (ConversionFeeUpdate)
- Added fee tracking MVs for CurveFi (CommitNewFee, NewFee)
- Added fee tracking MVs for SunPump (PurchaseFeeSet, SaleFeeSet)

Co-authored-by: DenisCarriere <550895+DenisCarriere@users.noreply.github.com>
- Fix Balancer fee conversion: divide by 1e14 (not 1e16) to convert from 1e18 fixed-point to basis points
- Add validation for CurveFi token extraction to ensure non-empty token addresses
- Remove build binary and update .gitignore

Co-authored-by: DenisCarriere <550895+DenisCarriere@users.noreply.github.com>
Copilot AI changed the title [WIP] Add other protocols to pools MV in clickhouse-evm-dex Add missing DEX protocols to pools materialized view Dec 9, 2025
Copilot AI requested a review from DenisCarriere December 9, 2025 19:19
@DenisCarriere
Copy link
Contributor

@copilot Rename pools to state_pools_initialize, remove "fees" from this table

Additionally to state_pools_initialize, update the schema to make tokens as Array(String), like this:

Rename pools_fees to state_pools_fees and also include initialize events to determine initial fees.

Then create a schema.3.view.pools a pool VIEW which combines both tables together

- Rename `pools` table to `state_pools_initialize` and remove fee column
- Change token structure to use Array(String) with materialized columns for token0-token3
- Rename `pools_fees` to `state_pools_fees` and add initialize events for initial fees
- Create `pools` VIEW combining state_pools_initialize and state_pools_fees
- Update all materialized views to use new tokens array structure

Co-authored-by: DenisCarriere <550895+DenisCarriere@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Dec 10, 2025

@copilot Rename pools to state_pools_initialize, remove "fees" from this table

Additionally to state_pools_initialize, update the schema to make tokens as Array(String), like this:...

Done in c9e3987.

Changes made:

  • Renamed pools to state_pools_initialize with tokens as Array(String) and materialized columns for token0-token3
  • Renamed pools_fees to state_pools_fees and added initialize event MVs for initial fees (Uniswap V1-V4, CurveFi)
  • Created schema.3.view.pools VIEW combining both tables with LEFT JOIN to get latest fees

- Update all fee update MVs to reference state_pools_fees (not pools_fees)
- Fix protocol enum values to use hyphens (uniswap-v1) matching existing convention in swaps table

Co-authored-by: DenisCarriere <550895+DenisCarriere@users.noreply.github.com>
Copilot AI changed the title Add missing DEX protocols to pools materialized view Refactor pools schema with state tables and VIEW pattern Dec 10, 2025
@DenisCarriere DenisCarriere marked this pull request as ready for review December 10, 2025 14:34
@DenisCarriere DenisCarriere merged commit cf45caa into main Dec 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add other protocols to pools MV

2 participants