Configuration using viper and cobra (#41)

### TL;DR

Implemented a configuration system and restructured the project to use Cobra for command-line interface.

### What changed?

- Added a new configuration system using Viper
- Restructured the project to use Cobra for CLI
- Introduced new configuration files: `config.yml` and `secrets.example.yml`
- Updated existing components to use the new configuration system
- Removed environment variable loading
- Added support for command-line arguments, environment variables, and configuration files

### How to test?

1. Copy `configs/secrets.example.yml` to `configs/secrets.yml` and add your secrets
2. Run the application with different configuration methods:
   - Using config file: `go run main.go --config ./configs/config.yml`
   - Using environment variables: `RPC_URL=https://my-rpc.com go run main.go`
   - Using command-line arguments: `go run main.go --rpc-url https://my-rpc.com`
3. Test different commands:
   - API: `go run main.go api`
   - Orchestrator: `go run main.go orchestrator`

### Why make this change?

This change improves the flexibility and maintainability of the application by:

1. Providing a centralized configuration system
2. Allowing easy configuration through multiple methods (files, env vars, CLI args)
3. Improving the command-line interface for better user experience
4. Enhancing modularity and separation of concerns in the codebase

.gitignore

@@ -25,3 +25,6 @@ go.work.sum
 .env
 
 .vscode
+
+configs/secrets*
+!configs/secrets.example.yml

README.md

@@ -1 +1,193 @@
-# indexer
+# indexer
+
+## Configuration
+
+You can configure the application in 3 ways.
+The order of priority is
+1. Command line arguments
+2. Environment variables
+3. Configuration files
+
+### Configuration using command line arguments
+You can configure the application using command line arguments.
+For example to configure the `rpc.url` configuration, you can use the `--rpc-url` command line argument.
+Only select arguments are implemented. More on this below.
+
+### Configuration using environment variables
+You can also configure the application using environment variables. You can configure any configuration in the `config.yml` file using environment variables by replacing the `.` in the configuration with `_` and making the variable name uppercase.  
+For example to configure the `rpc.url` configuration to `https://my-rpc.com`, you can set the `RPC_URL` environment variable to `https://my-rpc.com`.  
+
+### Configuration using configuration files
+The default configuration file is `configs/config.yml`. You can use the `--config` flag to specify a different configuration file.  
+If you want to add secrets to the configuration file, you can copy `configs/secrets.example.yml` to `configs/secrets.yml` and add the secrets. They won't be commited to the repository or the built image.
+
+### Supported configurations:
+
+#### RPC URL
+URL to use as the RPC client.
+
+cmd: `--rpc-url`
+env: `RPC_URL`
+yaml:
+```yaml
+rpc:
+  url: https://rpc.com
+```
+
+#### Log Level
+Log level for the logger. Default is `warn`.
+
+cmd: `--log-level`
+env: `LOG_LEVEL`
+yaml:
+```yaml
+log:
+  level: debug
+```
+
+#### Pretty log
+Whether to print logs in a pretty format. Affects performance. Default is `false`.
+
+env: `PRETTY_LOG`
+yaml:
+```yaml
+log:
+  pretty: true
+```
+
+#### Poller
+Whether to enable the poller. Default is `true`.
+
+cmd: `--poller`
+env: `POLLER_ENABLED`
+yaml:
+```yaml
+poller:
+  enabled: true
+```
+
+#### Poller Interval
+Poller trigger interval in millisecons. Default is `1000`.
+
+env: `POLLER_INTERVAL`
+yaml:
+```yaml
+poller:
+  interval: 3000
+```
+
+#### Poller Batch Size
+How many blocks to poll each interval. Default is `10`.
+
+env: `POLLER_BATCH_SIZE`
+yaml:
+```yaml
+poller:
+  batchSize: 3
+```
+
+#### Poller From Block
+From which block to start polling. Default is `0`.
+
+env: `POLLER_FROM_BLOCK`
+yaml:
+```yaml
+poller:
+  fromBlock: 20000000
+```
+
+#### Poller Until Block
+Until which block to poll. If not set, it will poll until the latest block.
+
+env: `POLLER_UNTIL_BLOCK`
+yaml:
+```yaml
+poller:
+  untilBlock: 20000010
+```
+
+#### Commiter
+Whether to enable the commiter. Default is `true`.
+
+cmd: `--commiter`
+env: `COMMITER_ENABLED`
+yaml:
+```yaml
+commiter:
+  enabled: true
+```
+
+#### Commiter Interval
+Commiter trigger interval in millisecons. Default is `250`.
+
+env: `COMMITER_INTERVAL`
+yaml:
+```yaml
+commiter:
+  interval: 3000
+```
+
+#### Commiter Batch Size
+How many blocks to commit each interval. Default is `10`.
+
+env: `COMMITER_BATCH_SIZE`
+yaml:
+```yaml
+commiter:
+  batchSize: 3
+```
+
+#### Failure Recoverer
+Whether to enable the failure recoverer. Default is `true`.
+
+cmd: `--failure-recoverer`
+env: `FAILURE_RECOVERER_ENABLED`
+yaml:
+```yaml
+failureRecoverer:
+  enabled: true
+```
+
+#### Failure Recoverer Interval
+Failure recoverer trigger interval in millisecons. Default is `1000`.
+
+env: `FAILURE_RECOVERER_INTERVAL`
+yaml:
+```yaml
+failureRecoverer:
+  interval: 3000
+```
+
+#### Failure Recoverer Batch Size
+How many blocks to recover each interval. Default is `10`.
+
+env: `FAILURE_RECOVERER_BATCH_SIZE`
+yaml:
+```yaml
+failureRecoverer:
+  batchSize: 3
+```
+
+#### Storage
+This application has 3 kinds of strorage. Main, Staging and Orchestrator.
+Each of them takes similar configuration depending on the driver you want to use.
+
+There are no defaults, so this needs to be configured.
+
+```yaml
+storage:
+  main:
+    driver: "clickhouse"
+    clickhouse:
+      host: "localhost"
+      port: 3000
+      user: "admin"
+      password: "admin"
+      database: "base"
+  staging:
+    driver: "memory"
+    memory:
+      maxItems: 1000000
+  orchestrator:
+    driver: "memory"
+```

cmd/api.go

@@ -0,0 +1,29 @@
+package cmd
+
+import (
+	"net/http"
+
+	"github.com/rs/zerolog/log"
+	"github.com/spf13/cobra"
+
+	"github.com/go-chi/chi/v5"
+	"github.com/thirdweb-dev/indexer/internal/handlers"
+)
+
+var (
+	apiCmd = &cobra.Command{
+		Use:   "api",
+		Short: "TBD",
+		Long:  "TBD",
+		Run: func(cmd *cobra.Command, args []string) {
+			var r *chi.Mux = chi.NewRouter()
+			handlers.Handler(r)
+
+			log.Info().Msg("Starting Server on port 3000")
+			err := http.ListenAndServe("localhost:3000", r)
+			if err != nil {
+				log.Error().Err(err).Msg("Error starting server")
+			}
+		},
+	}
+)

cmd/orchestrator.go

@@ -0,0 +1,30 @@
+package cmd
+
+import (
+	"github.com/rs/zerolog/log"
+	"github.com/spf13/cobra"
+	"github.com/thirdweb-dev/indexer/internal/common"
+	"github.com/thirdweb-dev/indexer/internal/orchestrator"
+)
+
+var (
+	orchestratorCmd = &cobra.Command{
+		Use:   "orchestrator",
+		Short: "TBD",
+		Long:  "TBD",
+		Run: func(cmd *cobra.Command, args []string) {
+			log.Info().Msg("Starting indexer")
+			rpc, err := common.InitializeRPC()
+			if err != nil {
+				log.Fatal().Err(err).Msg("Failed to initialize RPC")
+			}
+
+			orchestrator, err := orchestrator.NewOrchestrator(*rpc)
+			if err != nil {
+				log.Fatal().Err(err).Msg("Failed to create orchestrator")
+			}
+
+			orchestrator.Start()
+		},
+	}
+)

cmd/root.go

@@ -0,0 +1,55 @@
+package cmd
+
+import (
+	"os"
+
+	"github.com/rs/zerolog/log"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+	configs "github.com/thirdweb-dev/indexer/configs"
+	customLogger "github.com/thirdweb-dev/indexer/internal/log"
+)
+
+var (
+	// Used for flags.
+	cfgFile string
+
+	rootCmd = &cobra.Command{
+		Use:   "indexer",
+		Short: "TBD",
+		Long:  "TBD",
+		Run: func(cmd *cobra.Command, args []string) {
+			log.Info().Msg("TODO: Starting indexer & api both")
+		},
+	}
+)
+
+func Execute() {
+	if err := rootCmd.Execute(); err != nil {
+		os.Exit(1)
+	}
+}
+
+func init() {
+	cobra.OnInitialize(initConfig)
+
+	rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is ./configs/config.yml)")
+	rootCmd.PersistentFlags().String("rpc-url", "", "RPC Url to use for the indexer")
+	rootCmd.PersistentFlags().String("log-level", "", "Log level to use for the application")
+	rootCmd.PersistentFlags().Bool("poller", true, "Toggle poller")
+	rootCmd.PersistentFlags().Bool("commiter", true, "Toggle commiter")
+	rootCmd.PersistentFlags().Bool("failure-recoverer", true, "Toggle failure recoverer")
+	viper.BindPFlag("rpc.url", rootCmd.PersistentFlags().Lookup("rpc-url"))
+	viper.BindPFlag("log.level", rootCmd.PersistentFlags().Lookup("log-level"))
+	viper.BindPFlag("poller.enabled", rootCmd.PersistentFlags().Lookup("poller"))
+	viper.BindPFlag("commiter.enabled", rootCmd.PersistentFlags().Lookup("commiter"))
+	viper.BindPFlag("failure-recoverer.enabled", rootCmd.PersistentFlags().Lookup("failure-recoverer"))
+
+	rootCmd.AddCommand(orchestratorCmd)
+	rootCmd.AddCommand(apiCmd)
+}
+
+func initConfig() {
+	configs.LoadConfig(cfgFile)
+	customLogger.InitLogger()
+}