refactor(rooch-da): introduce new data struct for DA and approach to …

…improve robust of DA submission (#2731)

* wip: refactor da/proposer

* feat(rooch-types): add DA batch and chunk serialization

Introduce DA batch structures and implement chunk serialization logic.
Moved DA-related modules to rooch-types and updated dependencies.

* feat(rooch-store): add DA store for block submission tracking

Introduce DAMetaDBStore to handle DA block submission and cursors. Refactor tests and configuration to integrate new DA store functionalities.

* fix(rooch-sequencer): correct typo and add TODO for transaction

Corrected a typo in the sequencer initialization comment and added a TODO note to save sequencer_info and transaction in a single transaction. This change ensures clarity in the code and highlights a necessary improvement.

* docs(rooch-da): update DA server responsibilities in README

Revised README to clarify DA server responsibilities and structure. Removed outdated internal and Celestia server README files.

* refactor(rooch-da): rename DABatch constructors

Renamed DABatch constructors for clarity. 'new' is now 'new_no_sign' and 'new_with_sign' is now 'new' to better reflect their functionality.

* refactor(rooch-config): rename InternalDAServerConfig to DABackendConfig

Renamed InternalDAServerConfig and associated types to DABackendConfig for clarity and consistency. Updated related references in scripts and configuration files accordingly.

* feat(rooch-da): simplify DA backend and message handling

Remove actor and message handling layers for DA backend, and streamline Celestia and OpenDA backend logic. This refactor reduces complexity and improves maintainability by consolidating DA backend responsibilities.

* chore

* feat(rooch-store): enhance block range calculations

Improve block range calculations for better handling of tx order continuity and maximum transactions per block constraint. Adjust test cases to validate changes and ensure accurate block range partitioning.

* feat(rooch-da): add new constructor for DABatchMeta

Added a constructor for DABatchMeta to initialize its fields. Also, updated SignedDABatchMeta and DABatch to include meta_signature field.

* fix(rooch-store): reduce MAX_TXS_PER_BLOCK_IN_FIX

Update MAX_TXS_PER_BLOCK_IN_FIX to 8192 to prevent OOM errors during block submission fixes. This change helps manage memory usage more effectively.

* feat(rooch-da): implement batch metadata signing and submission

Update batch message structure, return type, and implement batch metadata signing. Introduce background submission logic and remove unused dependencies.

* chore(rooch-executor): remove hex dependency

Removed the hex crate from rooch-executor's Cargo.toml as it is no longer needed.
This cleanup helps to minimize the dependencies and can potentially reduce the final binary size.

* feat(rooch-rpc-server): integrate DAServer with sequencer

Updated the DA module integration by switching from DAActor to DAServerActor and corresponding proxy changes. Added sequencer order tracking to improve transaction handling with the new DAServerActor setup.

* fix(rooch-da): improve error handling in append_submitting_block

Enhanced error message formatting in append_submitting_block for better debugging. This provides more context about the block numbers involved when an error occurs.

* feat(rooch-proposer): replace DAProxy with DAServerProxy

Update the `ProposerActor` and `StateCommitmentChain` to use `DAServerProxy` instead of `DAProxy`. This change enhances consistency with server handling. Additionally, update dependencies in Cargo.lock to latest versions.

* refactor(rooch-da): improve error handling and code clarity

Refactor error handling logic and code readability. Replaced use of deprecated macros and improved conditional checks for better maintainability.

* refactor(rooch-da): simplify DA backend and server logic

Removed unused imports and redundant code. Refactored backend submission logic for efficiency and readability.

* feat(rooch-da): refactor DABatch to use typed tx_list

Refactored DABatch to accept Vec<LedgerTransaction> instead of Vec<u8>. This improves type safety and clarity, and involves modifying methods, tests, and related message structures.

* feat(rooch-da): use genesis digest as open-da namespace

1. simply da config
2. use genesis digest as open-da namespace when start server

* fix lint issue

.github/workflows/docker_build.yml

@@ -92,5 +92,5 @@ jobs:
           chmod 600 private_key.pem
           sudo apt update
           sudo apt install -y --no-install-recommends openssh-server
-          ssh -o StrictHostKeyChecking=no -i private_key.pem $USER@$HOST bash -c "'sleep 30' && docker image prune -a -f && docker ps | grep main_debug | awk '{print \$1}' | xargs -r docker stop && docker ps -a | grep main_debug | awk '{print \$1}' | xargs -r docker rm -f && docker pull 'ghcr.io/rooch-network/rooch:main_debug' && docker run --rm -v /root:/root ghcr.io/rooch-network/rooch:main_debug server clean -n dev -f && docker run -d -v /root:/root -p 6767:6767 -p 9184:9184 'ghcr.io/rooch-network/rooch:main_debug' server start -n dev --btc-rpc-url '${{secrets.BTC_REGTEST_RPC_URL}}' --btc-rpc-username rooch-regtest --btc-rpc-password '${{secrets.BTC_REGTEST_RPC_PWD}}' --da '{\"internal-da-server\": {\"servers\": [{\"open-da\": {\"scheme\": \"fs\"}}]}}' --traffic-burst-size 100000 --traffic-per-second 1"
+          ssh -o StrictHostKeyChecking=no -i private_key.pem $USER@$HOST bash -c "'sleep 30' && docker image prune -a -f && docker ps | grep main_debug | awk '{print \$1}' | xargs -r docker stop && docker ps -a | grep main_debug | awk '{print \$1}' | xargs -r docker rm -f && docker pull 'ghcr.io/rooch-network/rooch:main_debug' && docker run --rm -v /root:/root ghcr.io/rooch-network/rooch:main_debug server clean -n dev -f && docker run -d -v /root:/root -p 6767:6767 -p 9184:9184 'ghcr.io/rooch-network/rooch:main_debug' server start -n dev --btc-rpc-url '${{secrets.BTC_REGTEST_RPC_URL}}' --btc-rpc-username rooch-regtest --btc-rpc-password '${{secrets.BTC_REGTEST_RPC_PWD}}' --da '{\"da-backend\": {\"backends\": [{\"open-da\": {\"scheme\": \"fs\"}}]}}' --traffic-burst-size 100000 --traffic-per-second 1"
           ssh -o StrictHostKeyChecking=no -i private_key.pem $USER@$HOST "cd /root/rooch && git pull origin main && bash scripts/check_dev_deploy_status.sh main_debug '${{ secrets.TESTNET_MNEMONIC_PHRASE }}'"

crates/rooch-config/Cargo.toml

@@ -22,6 +22,9 @@ dirs = { workspace = true }
 once_cell = { workspace = true }
 clap = { workspace = true }
 dirs-next = { workspace = true }
+celestia-types = { workspace = true }
+hex = { workspace = true }
 
 rooch-types = { workspace = true }
 moveos-config = { workspace = true }
+moveos-types = { workspace = true }

crates/rooch-config/src/config.rs

@@ -4,17 +4,9 @@
 use anyhow::Context;
 use serde::de::DeserializeOwned;
 use serde::Serialize;
-use std::collections::HashMap;
-use std::error::Error;
 use std::fs;
 use std::path::{Path, PathBuf};
 
-pub enum MapConfigValueSource {
-    ConfigKey,   // Value came from the presence of a key in the configuration
-    Environment, // Value came from the environment
-    Default,     // Value came from a defined default value
-}
-
 pub trait Config
 where
     Self: DeserializeOwned + Serialize,
@@ -82,113 +74,3 @@ impl<C> std::ops::DerefMut for PersistedConfig<C> {
         &mut self.inner
     }
 }
-
-pub fn parse_hashmap(
-    s: &str,
-) -> Result<HashMap<String, String>, Box<dyn Error + Send + Sync + 'static>> {
-    s.split(',')
-        .filter(|kv| !kv.is_empty())
-        .map(|kv| {
-            let mut parts = kv.splitn(2, '=');
-            match (parts.next(), parts.next()) {
-                (Some(key), Some(value)) if !key.trim().is_empty() => {
-                    Ok((key.to_string(), value.to_string()))
-                }
-                (Some(""), Some(_)) => Err("key is missing before '='".into()),
-                _ => {
-                    Err("each key=value pair must be separated by a comma and contain a key".into())
-                }
-            }
-        })
-        .collect()
-}
-
-// value order:
-// 1. key value
-// 2. env value
-// 3. default value
-pub fn retrieve_map_config_value(
-    config: &mut HashMap<String, String>,
-    key: &str,
-    env_var: Option<&str>,
-    default_var: &str,
-) -> MapConfigValueSource {
-    if config.contains_key(key) {
-        return MapConfigValueSource::ConfigKey;
-    }
-
-    if let Some(env_var) = env_var {
-        if let Ok(env_var_value) = std::env::var(env_var) {
-            // env_var exists
-            config.insert(key.to_string(), env_var_value.clone());
-            return MapConfigValueSource::Environment;
-        }
-    }
-
-    // Use the default
-    config.insert(key.to_string(), default_var.to_string());
-    MapConfigValueSource::Default
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_parse_hashmap_ok() {
-        let input = "key1=VALUE1,key2=value2";
-        let output = parse_hashmap(input).unwrap();
-
-        let mut expected = HashMap::new();
-        expected.insert("key1".to_string(), "VALUE1".to_string());
-        expected.insert("key2".to_string(), "value2".to_string());
-
-        assert_eq!(output, expected);
-    }
-
-    #[test]
-    fn test_parse_hashmap_empty_value() {
-        let input = "key1=,key2=value2";
-        let output = parse_hashmap(input).unwrap();
-
-        let mut expected = HashMap::new();
-        expected.insert("key1".to_string(), "".to_string());
-        expected.insert("key2".to_string(), "value2".to_string());
-
-        assert_eq!(output, expected);
-    }
-
-    #[test]
-    fn test_parse_hashmap_empty_string() {
-        let input = "";
-        let output = parse_hashmap(input).unwrap();
-
-        let expected = HashMap::new();
-
-        assert_eq!(output, expected);
-    }
-
-    #[test]
-    fn test_parse_hashmap_missing_value() {
-        let input = "key1,key2=value2";
-        let output = parse_hashmap(input);
-
-        assert!(output.is_err());
-    }
-
-    #[test]
-    fn test_parse_hashmap_missing_key() {
-        let input = "=value1,key2=value2";
-        let output = parse_hashmap(input);
-
-        assert!(output.is_err());
-    }
-
-    #[test]
-    fn test_parse_hashmap_no_equals_sign() {
-        let input = "key1value1,key2=value2";
-        let output = parse_hashmap(input);
-
-        assert!(output.is_err());
-    }
-}