Duneapi package integration (#6)

Unfortunately, this integration comes with a bigger refactor.

The package offers the ability to wait until a query calculation is finished. This is really nice and allows us to cut the amount of python script by a factor of 2: Previously, we had one script for executing the queries and a second one for fetching the results. Now, with the waiting feature, we can do it all in one go.

The duneapi package also allowed me to store the fetched dune data in a cleaner structure, as the parsing of the fetched result is better. Due to that I also needed to touch the rust code.

There is still a lot to be refactored to make the repo clean, but this should be okay as a first go.

testplan:

cd dune_api_scripts
python3 -m venv env
source ./env/bin/activate
pip install -r requirements.txt
source .env
export DUNE_DATA_FOLDER=./data/dune_data/
export APP_DATA_REFERRAL_RELATION_FILE=./data/referral_data/app_data_referral_relationship.json
python -m dune_api_scripts.store_query_result_all_distinct_app_data
Now, the referral data needs to be downloaded from ipfs. This takes quite long, if this should be done for all appdata. Hence, I would recommend deleting all entries, but 1, from the newly created distinct_app_data.json file in the data folder.
Once it is done, one can run

cargo run
And wait until app_data_referral_relationship.json is written.

Now, one can run:

python -m dune_api_scripts.store_query_result_for_todays_trading_data
cargo run
and check that trading volume was downloaded and is served by checking:

http://127.0.0.1:8080/api/v1/profile/0x019d6ad53704fc851f1cc9effc12e89de4a1fc2c

.env.example

@@ -1,4 +1,4 @@
-export DUNE_PASSWORD=<PWD>
-export DUNE_USER=<User name> 
-export DUNE_DATA_FOLDER=./data/dune_data/
-export APP_DATA_REFERRAL_RELATION_FILE=./data/referral_data/app_data_referral_relationship.json 
+DUNE_PASSWORD=<PWD>
+DUNE_USER=<User name> 
+DUNE_DATA_FOLDER=./data/dune_data/
+APP_DATA_REFERRAL_RELATION_FILE=./data/referral_data/app_data_referral_relationship.json 

.github/workflows/pull-request.yaml

@@ -10,19 +10,16 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
-      - name: Setup Python 3.9
+      - name: Setup Python 3.10
         uses: actions/setup-python@v2
         with:
-          python-version: '3.9'
+          python-version: '3.10'
       - name: Install Project Requirements
-        run:
-          pip install -r dune_api_scripts/requirements.txt
+        run: pip install -r dune_api_scripts/requirements.txt
       - name: Run Linter
-        run:
-          pylint dune_api_scripts/
+        run: pylint dune_api_scripts/
       - name: Run Tests
-        run:
-          pytest dune_api_scripts/tests
+        run: pytest dune_api_scripts/tests
   rust-misc:
     runs-on: ubuntu-latest
     steps:
@@ -45,4 +42,4 @@ jobs:
     steps:
       - uses: actions/checkout@v2
       - uses: Swatinem/rust-cache@v1
-      - run: cargo test --locked --all-features
+      - run: cargo test --locked --all-features

.gitignore

@@ -1,4 +1,7 @@
 /target
 /dune_api_scripts/**/__pycache__
 .env
-data/
+data/
+/dune_api_scripts/env/*
+.vscode
+env/

.pylintrc

@@ -52,7 +52,7 @@ persistent=yes
 
 # Min Python version to use for version dependend checks. Will default to the
 # version used to run pylint.
-py-version=3.9
+py-version=3.10
 
 # When enabled, pylint would attempt to guess common misconfiguration and emit
 # user-friendly hints instead of false-positive error messages.

Readme.md

@@ -39,24 +39,17 @@ Setting some envs:
 cp .env.example .env
 ```
 and adjust the values.
-
 ### Download data:
 
 Pulling new query results:
 
 ```
 python -m dune_api_scripts.store_query_result_all_distinct_app_data
-python -m dune_api_scripts.store_query_result_for_entire_history_trading_data
 python -m dune_api_scripts.store_query_result_for_todays_trading_data
+python -m dune_api_scripts.store_query_result_for_entire_history_trading_data
 ```
 
-
-Update query:
-```
-python -m dune_api_scripts.modify_and_execute_dune_query_for_entire_history_trading_data
-python -m dune_api_scripts.modify_and_execute_dune_query_for_todays_trading_data
-python -m dune_api_scripts.execute_dune_query_for_all_app_data
-```
+The last command might take a while, as downloading the whole history takes quite some time.
 
 Alternatively, the scripts can also be run via docker:
 ```