From 2b8f62de033bdc2adb2050104276ae3d2c10a1bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 17:44:50 +0200 Subject: [PATCH 01/17] restructure Readme content so we have first things first - how does usage look like, then setup, then how to extend/develop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- README.md | 112 +++++++++++++++++++++++++++--------------------------- 1 file changed, 55 insertions(+), 57 deletions(-) diff --git a/README.md b/README.md index 6b8ad10..84841ee 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,62 @@ -# FLEXMEASURES-WEATHER - a plugin for FlexMeasures to integrate multiple Weather API services +# FLEXMEASURES-WEATHER - a plugin for FlexMeasures to integrate weather forecasts -### Configuration This plugin currently supports two Weather API services: [OpenWeatherMap One Call API](https://openweathermap.org/api/one-call-3) and [Weather API](https://www.weatherapi.com/). The configuration is controlled via your FlexMeasures config file. + +## Usage + +To register a new weather sensor: + +`flexmeasures weather register-weather-sensor --name "wind speed" --latitude 30 --longitude 40` + +Currently supported: wind speed, temperature & irradiance. + +To collect weather forecasts: + +`flexmeasures weather get-weather-forecasts --location 30,40` + +This saves forecasts for your registered sensors in the database. + +Use the `--help`` option for more options, e.g. for specifying two locations and requesting that a number of weather stations cover the bounding box between them (where the locations represent top left and bottom right). + +Notes about weather sensor setup: + +- Weather sensors are public assets in FlexMeasures. They are accessible by all accounts on a FlexMeasures server. +- The resolution is one hour. Weather also supports minutely data within the upcoming hour(s), but that is not supported here. + +An alternative usage is to save raw results in JSON files (for later processing), like this: + +`flexmeasures weather get-weather-forecasts --location 30,40 --store-as-json-files --region somewhere` + +This saves the complete response from the Weather Provider in a local folder (i.e. no sensor registration needed, this is a direct way to use Weather APIs, without FlexMeasures integration). `region` will become a subfolder. + +Finally, note that these APIs allow free calls, but not without limits. +For instance, currently 1000 free calls per day can be made to the OpenWeatherMap API, +so you can make a call every 15 minutes for up to 10 locations or every hour for up to 40 locations (or get a paid account). + + +## Setup + +### Installation + +To add as plugin to an existing FlexMeasures system, add "/path/to/flexmeasures-weather-repo/flexmeasures_weather" to your FlexMeasures config file, +using the FLEXMEASURES_PLUGINS setting (a list). + +Alternatively, if you installed this plugin as a package (e.g. via `python setup.py install`, `pip install -e` or `pip install flexmeasures_weather` after this project is on Pypi), then "flexmeasures_weather" suffices. + +To enable weather forecast functionality, two PostgreSQL extensions must be installed. Run the following SQL commands in your database: + +``` +CREATE EXTENSION IF NOT EXISTS cube; +CREATE EXTENSION IF NOT EXISTS earthdistance; +``` + +These extensions provide support for geographical calculations such as `ll_to_earth` and `earth_distance`, which we use to find the nearest weather station asset. + + +### Configuration + Add the following entries to your config: ```ini @@ -57,61 +110,6 @@ To expand the plugin's coverage to additional weather API services: > This modular structure allows for seamless integration of additional services while maintaining consistency and clarity in data handling. -## Usage - -To register a new weather sensor: - -`flexmeasures weather register-weather-sensor --name "wind speed" --latitude 30 --longitude 40` - -Currently supported: wind speed, temperature & irradiance. - -Notes about weather sensor setup: - -- Weather sensors are public. They are accessible by all accounts on a FlexMeasures server. TODO: maybe limit this to a list of account roles. -- The resolution is one hour. Weather also supports minutely data within the upcoming hour(s), but that is not supported here. - -To collect weather forecasts: - -Enable required Postgres Extensions - -To enable weather forecast functionality, two PostgreSQL extensions must be installed. Run the following SQL commands in your database: - -``` -CREATE EXTENSION IF NOT EXISTS cube; -CREATE EXTENSION IF NOT EXISTS earthdistance; -``` - -These extensions provide support for geographical calculations such as ll_to_earth and earth_distance, which are used to determine proximity between coordinates. - -`flexmeasures weather get-weather-forecasts --location 30,40` - -This saves forecasts for your registered sensors in the database. - -Use the `--help`` option for more options, e.g. for specifying two locations and requesting that a number of weather stations cover the bounding box between them (where the locations represent top left and bottom right). - -An alternative usage is to save raw results in JSON files (for later processing), like this: - -`flexmeasures weather get-weather-forecasts --location 30,40 --store-as-json-files --region somewhere` - -This saves the complete response from the Weather Provider in a local folder (i.e. no sensor registration needed, this is a direct way to use Weather, without FlexMeasures integration). `region` will become a subfolder. - -Finally, note that currently 1000 free calls per day can be made to the OpenWeatherMap API, -so you can make a call every 15 minutes for up to 10 locations or every hour for up to 40 locations (or get a paid account). - - -## Installation - -To install locally, run - - make install - -To add as plugin to an existing FlexMeasures system, add "/path/to/FLEXMEASURES-WEATHER/flexmeasures_weather" to your FlexMeasures (>v0.7.0dev8) config file, -using the FLEXMEASURES_PLUGINS setting (a list). - -Alternatively, if you installed this plugin as a package (e.g. via `python setup.py install`, `pip install -e` or `pip install flexmeasures_weather` after this project is on Pypi), then "flexmeasures_weather" suffices. - - - ## Development We use pre-commit to keep code quality up. From 68c3b7066bde5746f1e9641c5da1eca7b0ffdf84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:25:19 +0200 Subject: [PATCH 02/17] remove unneeded DS_Store file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .DS_Store | Bin 8196 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 .DS_Store diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 614c19722a9eb759a83dce957a18851310999c7b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8196 zcmeHMzl#$=6n>M7$%a!%aUO>vELaJrCKyz>CKn^NY7P}tV)n8otDBve-54PRQfhaV zAU1-QHX>Lb7B*t*A8=@4ua@GQABj6z_sDrc4xNFSZ!+`V%(vg{%X>s)8Q)+lwYQN3neaqM0-yKs(rLCx{(4tH?sHk#bH z>Uee9E77i3_qlIHbQl>UQ_U{+`?+QFOn$Iz_Va}T*3ae&gF(hPc=Y7Tg-aXau9~Ek+hZOC;H&BszHet6zV_!Y-)F*(fk4hFN0q|M~du|FfA)kD?XO3hYS* zNU~fhmyp`%b#am0F!0fuEE#k757- From 22d9b1bda71b7ca2f719ef73c93f75c6e7a8c4d2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:31:02 +0200 Subject: [PATCH 03/17] version updates in GitHub Actions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/lint-and-test.yml | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index 61e0fcd..a074832 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -6,13 +6,14 @@ on: push jobs: check: runs-on: ubuntu-latest - name: Check (on Python3.9) + name: Check (on Python 3.11) steps: - - uses: actions/setup-python@v2 + - uses: actions/setup-python@v4 with: - python-version: 3.9 - - uses: actions/checkout@v2 - - uses: pre-commit/action@v2.0.0 + python-version: 3.11 + - uses: actions/checkout@v3 + - uses: pre-commit/action@v3.0.0 + test: needs: check @@ -20,7 +21,7 @@ jobs: strategy: fail-fast: false matrix: - py_version: [ '3.9' ] + py_version: [ '3.11' ] name: "Test (on Python ${{ matrix.py_version }})" steps: - uses: actions/setup-python@v2 @@ -47,7 +48,7 @@ jobs: # Label used to access the service container postgres: # Docker Hub image - image: postgres:12.5 + image: postgres:14.17 env: POSTGRES_USER: flexmeasures_test POSTGRES_PASSWORD: flexmeasures_test From 6de8017dedebb6feb4c94c4db610ba79f5b15b66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:34:45 +0200 Subject: [PATCH 04/17] upgrade pre-commit tooling/library versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .pre-commit-config.yaml | 4 ++-- README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 06e3982..57ebf02 100755 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,11 +1,11 @@ repos: - repo: https://github.com/pycqa/flake8 - rev: 6.0.0 # New version tags can be found here: https://github.com/pycqa/flake8/tags + rev: 7.1.1 # New version tags can be found here: https://github.com/pycqa/flake8/tags hooks: - id: flake8 name: flake8 (code linting) - repo: https://github.com/psf/black - rev: 22.10.0 # New version tags can be found here: https://github.com/psf/black/tags + rev: 24.8.0 # New version tags can be found here: https://github.com/psf/black/tags hooks: - id: black name: black (code formatting) diff --git a/README.md b/README.md index 84841ee..92fc269 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ We use pre-commit to keep code quality up. Install necessary tools with: - pip install pre-commit black flake8 mypy + pip install pre-commit pre-commit install or: From 4fdff62af72cfbcdcc2c59fe58c921f189601e58 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:38:59 +0200 Subject: [PATCH 05/17] run black MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- flexmeasures_weather/utils/weather.py | 72 +++++++++++++-------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/flexmeasures_weather/utils/weather.py b/flexmeasures_weather/utils/weather.py index 13b69f1..f27f91a 100644 --- a/flexmeasures_weather/utils/weather.py +++ b/flexmeasures_weather/utils/weather.py @@ -48,8 +48,7 @@ def get_supported_sensors_str() -> str: def process_weatherapi_data( - data: List[Dict[str, Any]], - hour_no: int + data: List[Dict[str, Any]], hour_no: int ) -> List[Dict[str, Any]]: """ Processes raw WeatherAPI forecast data into a format similar to OpenWeatherMap's format. @@ -63,14 +62,14 @@ def process_weatherapi_data( List[Dict[str, Any]]: A list of 48 hourly forecast entries, each mapped to the expected structure with fields like temperature, humidity, wind, and condition. """ - first_day = data[0]['hour'] - second_day = data[1]['hour'] - third_day = data[2]['hour'] + first_day = data[0]["hour"] + second_day = data[1]["hour"] + third_day = data[2]["hour"] combined = first_day + second_day + third_day - - relevant = combined[hour_no: hour_no + 48] + + relevant = combined[hour_no : hour_no + 48] # relevant = combined - + def map_weather_api_to_owm(weather_api_data: Dict[str, Any]) -> Dict[str, Any]: """ Converts a single hour of WeatherAPI data to an OpenWeatherMap-style dictionary. @@ -105,7 +104,7 @@ def map_weather_api_to_owm(weather_api_data: Dict[str, Any]) -> Dict[str, Any]: "pop": weather_api_data["chance_of_rain"] / 100, } return game - + converted = [map_weather_api_to_owm(hour) for hour in relevant] return converted @@ -133,10 +132,8 @@ def call_openweatherapi( return time_of_api_call, data["hourly"] -def call_weatherapi( - api_key: str, - location: Tuple[float, float], - days: int = 3 +def call_weatherapi( + api_key: str, location: Tuple[float, float], days: int = 3 ) -> Tuple[datetime, List[Dict]]: """ Makes a request to the WeatherAPI to retrieve hourly weather forecast data. @@ -147,44 +144,43 @@ def call_weatherapi( days (int, optional): Number of days to request the forecast for (default is 3, including current day). Returns: - Tuple[datetime, List[Dict]]: + Tuple[datetime, List[Dict]]: - The timestamp of the API call. - A list of hourly forecast data as dictionaries. Note that the first forecast is about the current hour. Raises: AssertionError: If the response from the Weather API is not successful (HTTP status 200). """ - + latitude, longitude = location[0], location[1] - + query_str = f"http://api.weatherapi.com/v1/forecast.json?key={api_key}&q={latitude},{longitude}&days={days}&aqi=yes&alerts=yes" res = requests.get(query_str) - + assert ( res.status_code == 200 ), f"Weather API returned status code {res.status_code}: {res.text}" - + data = res.json() - + # get the time of the api call - time_of_call = int(data['location']['localtime_epoch']) - local_timezone = ZoneInfo(data['location']['tz_id']) + time_of_call = int(data["location"]["localtime_epoch"]) + local_timezone = ZoneInfo(data["location"]["tz_id"]) local_time = datetime.fromtimestamp(time_of_call, local_timezone) time_of_api_call = as_server_time(local_time) time_of_api_call = time_of_api_call.replace(second=0, microsecond=0) - + print(f"Time of API call in WAPI is {time_of_api_call}") - - relevant = data['forecast']['forecastday'] + + relevant = data["forecast"]["forecastday"] hour_no = local_time.hour - + hourly = process_weatherapi_data(relevant, hour_no) return time_of_api_call, hourly def call_api( - api_key: str, - location: Tuple[float, float] + api_key: str, location: Tuple[float, float] ) -> Tuple[datetime, List[Dict]]: """ Dispatches the weather API call based on the configured provider. @@ -194,7 +190,7 @@ def call_api( location (Tuple[float, float]): Latitude and longitude tuple. Returns: - Tuple[datetime, List[Dict]]: + Tuple[datetime, List[Dict]]: - Timestamp of the API call. - List of hourly forecast data. @@ -203,10 +199,12 @@ def call_api( """ provider = str(current_app.config.get("WEATHER_PROVIDER", "")) - if provider not in ['OWM', 'WAPI']: - raise Exception("Invalid provider name. Please set WEATHER_PROVIDER setting in config file to either OWM or WAPI, the two permissible options.") - - if provider == 'OWM': + if provider not in ["OWM", "WAPI"]: + raise Exception( + "Invalid provider name. Please set WEATHER_PROVIDER setting in config file to either OWM or WAPI, the two permissible options." + ) + + if provider == "OWM": click.secho("Calling Open Weather Map") return call_openweatherapi(api_key, location) else: @@ -230,9 +228,9 @@ def save_forecasts_in_db( ) for location in locations: click.echo("[FLEXMEASURES] %s, %s" % location) - weather_sensors: Dict[ - str, Sensor - ] = {} # keep track of the sensors to save lookups + weather_sensors: Dict[str, Sensor] = ( + {} + ) # keep track of the sensors to save lookups db_forecasts: Dict[Sensor, List[TimedBelief]] = {} # collect beliefs per sensor now = server_now() @@ -251,7 +249,9 @@ def save_forecasts_in_db( fc_datetime = as_server_time( datetime.fromtimestamp(fc["dt"], get_timezone()) ) - click.echo(f"[FLEXMEASURES-WEATHER] Processing forecast for {fc_datetime} ...") + click.echo( + f"[FLEXMEASURES-WEATHER] Processing forecast for {fc_datetime} ..." + ) data_source = get_or_create_owm_data_source() for sensor_specs in mapping: sensor_name = str(sensor_specs["fm_sensor_name"]) From cc5cbab0bb003f3f8e11144aca1de61257e8031a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:43:27 +0200 Subject: [PATCH 06/17] run more black on omore files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- flexmeasures_weather/cli/commands.py | 4 +--- flexmeasures_weather/cli/tests/test_get_forecasts.py | 4 +--- flexmeasures_weather/utils/locating.py | 3 ++- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/flexmeasures_weather/cli/commands.py b/flexmeasures_weather/cli/commands.py index 26469b6..a95c435 100644 --- a/flexmeasures_weather/cli/commands.py +++ b/flexmeasures_weather/cli/commands.py @@ -164,9 +164,7 @@ def collect_weather_data(location, asset_id, store_in_db, num_cells, method, reg api_key = str(current_app.config.get("WEATHERAPI_KEY", "")) if api_key == "": - raise Exception( - "[FLEXMEASURES-WEATHER] Setting WEATHERAPI_KEY not available." - ) + raise Exception("[FLEXMEASURES-WEATHER] Setting WEATHERAPI_KEY not available.") if asset_id is not None: locations = [get_location_by_asset_id(asset_id)] elif location is not None: diff --git a/flexmeasures_weather/cli/tests/test_get_forecasts.py b/flexmeasures_weather/cli/tests/test_get_forecasts.py index 3454ae1..68aae3a 100644 --- a/flexmeasures_weather/cli/tests/test_get_forecasts.py +++ b/flexmeasures_weather/cli/tests/test_get_forecasts.py @@ -63,7 +63,5 @@ def test_get_weather_forecasts_no_close_sensors( ["--location", f"{weather_station.latitude-5},{weather_station.longitude}"], ) print(result.output) - assert ( - "Reported task get-weather-forecasts status as True" in result.output - ) + assert "Reported task get-weather-forecasts status as True" in result.output assert "no sufficiently close weather sensor found" in caplog.text diff --git a/flexmeasures_weather/utils/locating.py b/flexmeasures_weather/utils/locating.py index c07bf62..555872c 100644 --- a/flexmeasures_weather/utils/locating.py +++ b/flexmeasures_weather/utils/locating.py @@ -123,6 +123,7 @@ def get_location_by_asset_id(asset_id: int) -> Tuple[float, float]: ) if asset is None: raise Exception( - "[FLEXMEASURES-WEATHER] No asset found for the given asset id %s." % asset_id + "[FLEXMEASURES-WEATHER] No asset found for the given asset id %s." + % asset_id ) return (asset.latitude, asset.longitude) From a004d18778103a2d64eb5700372fdab87dd796f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 19:49:27 +0200 Subject: [PATCH 07/17] upgrade another version in GitHub Actions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/lint-and-test.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index a074832..83e75ae 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -28,11 +28,12 @@ jobs: with: python-version: ${{ matrix.py_version }} - name: Check out src from Git - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Get history and tags for SCM versioning to work run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* + - name: Install SQL extensions sudo apt-get update sudo apt-get -y install postgresql-client psql -h $PGHOST -p $PGPORT --file scripts/load-psql-extensions.sql -U $PGUSER $PGDB; From 33cbe7407a2896304147ab169d1171b93d0143fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Tue, 27 May 2025 20:06:41 +0200 Subject: [PATCH 08/17] fix GH Action syntax MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/lint-and-test.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index 83e75ae..c2f4f21 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -34,6 +34,7 @@ jobs: git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Install SQL extensions + run: | sudo apt-get update sudo apt-get -y install postgresql-client psql -h $PGHOST -p $PGPORT --file scripts/load-psql-extensions.sql -U $PGUSER $PGDB; From b59c8dbdbf6ae210d7b37a5f4d24aec91c18b9a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Thu, 29 May 2025 18:46:38 +0200 Subject: [PATCH 09/17] no use for the version right now, maybe get that back when we are packaging one day MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/lint-and-test.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index c2f4f21..ca1c401 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -29,10 +29,6 @@ jobs: python-version: ${{ matrix.py_version }} - name: Check out src from Git uses: actions/checkout@v3 - - name: Get history and tags for SCM versioning to work - run: | - git fetch --prune --unshallow - git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Install SQL extensions run: | sudo apt-get update From 54bcc2ed3e132d8ab2cef8a8d11772ae5b4f3046 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 00:47:23 +0200 Subject: [PATCH 10/17] install modern way, less depenencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- Makefile | 2 +- requirements/app.in | 1 - requirements/app.txt | 443 +----------------------------------------- requirements/dev.txt | 14 +- requirements/test.in | 10 +- requirements/test.txt | 74 ------- 6 files changed, 12 insertions(+), 532 deletions(-) diff --git a/Makefile b/Makefile index df1acf2..78ef750 100644 --- a/Makefile +++ b/Makefile @@ -25,7 +25,7 @@ install-deps: pip-sync requirements/app.txt install-flexmeasures-weather: - python setup.py develop + pip install -e . install-pip-tools: pip3 install -q "pip-tools>=6.2" diff --git a/requirements/app.in b/requirements/app.in index f7e8120..aa34d7a 100644 --- a/requirements/app.in +++ b/requirements/app.in @@ -1,4 +1,3 @@ -flexmeasures>=0.9.3 pvlib # the following three are optional in pvlib, but we use them netCDF4 diff --git a/requirements/app.txt b/requirements/app.txt index f8a2295..5872da0 100644 --- a/requirements/app.txt +++ b/requirements/app.txt @@ -4,493 +4,64 @@ # # pip-compile --output-file=requirements/app.txt requirements/app.in # -alembic==1.10.2 - # via - # flask-migrate - # flexmeasures -altair==4.2.2 - # via flexmeasures -arrow==1.2.3 - # via - # flexmeasures - # rq-dashboard -async-timeout==4.0.2 - # via - # flexmeasures - # redis -attrs==22.2.0 - # via - # flexmeasures - # jsonschema -babel==2.12.1 - # via - # flexmeasures - # py-moneyed -bcrypt==4.0.1 - # via flexmeasures beautifulsoup4==4.11.1 # via siphon -blinker==1.5 - # via - # flask-mail - # flask-principal - # flask-security-too - # flexmeasures - # sentry-sdk certifi==2022.12.7 - # via - # flexmeasures - # requests - # sentry-sdk + # via requests cftime==1.6.1 # via netcdf4 charset-normalizer==3.1.0 - # via - # flexmeasures - # requests -click==8.1.3 - # via - # click-default-group - # flask - # flexmeasures - # rq -click-default-group==1.2.2 - # via flexmeasures -colour==0.1.5 - # via flexmeasures -contourpy==1.0.7 - # via - # flexmeasures - # matplotlib -convertdate==2.4.0 - # via - # flexmeasures - # workalendar -cycler==0.11.0 - # via - # flexmeasures - # matplotlib -deprecated==1.2.13 - # via - # flexmeasures - # sktime -dill==0.3.6 - # via - # flexmeasures - # openturns -dnspython==2.3.0 - # via - # email-validator - # flexmeasures -email-validator==1.3.1 - # via - # flask-security-too - # flexmeasures -entrypoints==0.4 - # via - # altair - # flexmeasures -filelock==3.10.7 - # via - # flexmeasures - # tldextract -flask==2.1.2 - # via - # flask-classful - # flask-cors - # flask-json - # flask-login - # flask-mail - # flask-marshmallow - # flask-migrate - # flask-principal - # flask-security-too - # flask-sqlalchemy - # flask-sslify - # flask-wtf - # flexmeasures - # rq-dashboard - # sentry-sdk -flask-classful==0.14.2 - # via flexmeasures -flask-cors==3.0.10 - # via flexmeasures -flask-json==0.3.5 - # via flexmeasures -flask-login==0.6.1 - # via - # flask-security-too - # flexmeasures -flask-mail==0.9.1 - # via flexmeasures -flask-marshmallow==0.14.0 - # via flexmeasures -flask-migrate==4.0.4 - # via flexmeasures -flask-principal==0.4.0 - # via - # flask-security-too - # flexmeasures -flask-security-too==5.1.2 - # via flexmeasures -flask-sqlalchemy==2.5.1 - # via - # flask-migrate - # flexmeasures -flask-sslify==0.1.5 - # via flexmeasures -flask-wtf==1.1.1 - # via - # flask-security-too - # flexmeasures -flexmeasures==0.13.3 - # via -r requirements/app.in -fonttools==4.39.3 - # via - # flexmeasures - # matplotlib -greenlet==2.0.2 - # via - # flexmeasures - # sqlalchemy + # via requests h5py==3.7.0 # via pvlib -humanize==4.6.0 - # via flexmeasures idna==3.4 - # via - # email-validator - # flexmeasures - # requests - # tldextract -importlib-metadata==6.1.0 - # via - # flexmeasures - # timely-beliefs -importlib-resources==5.12.0 - # via flexmeasures -inflect==6.0.2 - # via flexmeasures -inflection==0.5.1 - # via flexmeasures -iso8601==1.1.0 - # via flexmeasures -isodate==0.6.1 - # via - # flexmeasures - # timely-beliefs -itsdangerous==2.1.2 - # via - # flask - # flask-security-too - # flask-wtf - # flexmeasures -jinja2==3.1.2 - # via - # altair - # flask - # flexmeasures -joblib==1.2.0 - # via - # flexmeasures - # scikit-learn -jsonschema==4.17.3 - # via - # altair - # flexmeasures -kiwisolver==1.4.4 - # via - # flexmeasures - # matplotlib -llvmlite==0.39.1 - # via - # flexmeasures - # numba -lunardate==0.2.0 - # via - # flexmeasures - # workalendar -mako==1.2.4 - # via - # alembic - # flexmeasures -markupsafe==2.1.2 - # via - # flexmeasures - # jinja2 - # mako - # wtforms -marshmallow==3.19.0 - # via - # flask-marshmallow - # flexmeasures - # marshmallow-polyfield - # marshmallow-sqlalchemy - # webargs -marshmallow-polyfield==5.11 - # via flexmeasures -marshmallow-sqlalchemy==0.29.0 - # via flexmeasures -matplotlib==3.7.1 - # via - # flexmeasures - # timetomodel + # via requests netcdf4==1.6.0 # via -r requirements/app.in -numba==0.56.4 - # via - # flexmeasures - # sktime numexpr==2.8.3 # via tables numpy==1.23.5 # via - # altair # cftime - # contourpy - # flexmeasures # h5py - # matplotlib # netcdf4 - # numba # numexpr # pandas - # patsy - # properscoring # pvlib - # scikit-learn # scipy # siphon - # sktime - # statsmodels # tables - # timely-beliefs - # timetomodel - # uniplot -openturns==1.20.post3 - # via - # flexmeasures - # timely-beliefs packaging==23.0 # via - # flexmeasures - # marshmallow - # marshmallow-sqlalchemy - # matplotlib # numexpr - # statsmodels # tables - # webargs pandas==1.5.3 # via - # altair - # flexmeasures # pvlib # siphon - # sktime - # statsmodels - # timely-beliefs - # timetomodel -passlib==1.7.4 - # via - # flask-security-too - # flexmeasures -patsy==0.5.3 - # via - # flexmeasures - # statsmodels -pillow==9.4.0 - # via - # flexmeasures - # matplotlib -pint==0.20.1 - # via flexmeasures -ply==3.11 - # via - # flexmeasures - # pyomo -properscoring==0.1 - # via - # flexmeasures - # timely-beliefs protobuf==4.21.5 # via siphon -pscript==0.7.7 - # via flexmeasures -psutil==5.9.4 - # via - # flexmeasures - # openturns -psycopg2-binary==2.9.5 - # via - # flexmeasures - # timely-beliefs pvlib==0.9.2 # via -r requirements/app.in -py-moneyed==3.0 - # via flexmeasures -pydantic==1.10.7 - # via - # flexmeasures - # inflect -pyluach==2.2.0 - # via - # flexmeasures - # workalendar -pymeeus==0.5.12 - # via - # convertdate - # flexmeasures -pyomo==6.5.0 - # via flexmeasures -pyparsing==3.0.9 - # via - # flexmeasures - # matplotlib -pyrsistent==0.19.3 - # via - # flexmeasures - # jsonschema python-dateutil==2.8.2 - # via - # arrow - # flexmeasures - # matplotlib - # pandas - # timetomodel - # workalendar -python-dotenv==1.0.0 - # via flexmeasures + # via pandas pytz==2023.3 # via - # flexmeasures # pandas # pvlib - # timely-beliefs - # timetomodel -redis==4.5.4 - # via - # flexmeasures - # rq - # rq-dashboard requests==2.28.2 # via - # flexmeasures # pvlib - # requests-file # siphon - # tldextract -requests-file==1.5.1 - # via - # flexmeasures - # tldextract -rq==1.13.0 - # via - # flexmeasures - # rq-dashboard -rq-dashboard==0.6.1 - # via flexmeasures -scikit-learn==1.2.2 - # via - # flexmeasures - # sktime - # timetomodel scipy==1.10.1 - # via - # flexmeasures - # properscoring - # pvlib - # scikit-learn - # sktime - # statsmodels - # timely-beliefs - # timetomodel -sentry-sdk[flask]==1.18.0 - # via flexmeasures + # via pvlib siphon==0.9 # via -r requirements/app.in six==1.16.0 - # via - # flask-cors - # flask-marshmallow - # flexmeasures - # isodate - # patsy - # python-dateutil - # requests-file -sktime==0.16.1 - # via - # flexmeasures - # timely-beliefs + # via python-dateutil soupsieve==2.3.2.post1 # via beautifulsoup4 -sqlalchemy==1.4.47 - # via - # alembic - # flask-sqlalchemy - # flexmeasures - # marshmallow-sqlalchemy - # timely-beliefs - # timetomodel -statsmodels==0.13.5 - # via - # flexmeasures - # timetomodel tables==3.7.0 # via -r requirements/app.in -tabulate==0.9.0 - # via flexmeasures -threadpoolctl==3.1.0 - # via - # flexmeasures - # scikit-learn -timely-beliefs[forecast]==1.19.0 - # via flexmeasures -timetomodel==0.7.2 - # via flexmeasures -tldextract==3.4.0 - # via flexmeasures -toolz==0.12.0 - # via - # altair - # flexmeasures -typing-extensions==4.5.0 - # via - # alembic - # flexmeasures - # py-moneyed - # pydantic -uniplot==0.10.0 - # via flexmeasures urllib3==1.26.15 - # via - # flexmeasures - # requests - # sentry-sdk -webargs==8.2.0 - # via flexmeasures -werkzeug==2.0.3 - # via - # flask - # flask-login - # flexmeasures -workalendar==17.0.0 - # via flexmeasures -wrapt==1.15.0 - # via - # deprecated - # flexmeasures -wtforms==3.0.1 - # via - # flask-security-too - # flask-wtf - # flexmeasures -xlrd==2.0.1 - # via flexmeasures -zipp==3.15.0 - # via - # flexmeasures - # importlib-metadata - -# The following packages are considered to be unsafe in a requirements file: -# setuptools + # via requests diff --git a/requirements/dev.txt b/requirements/dev.txt index 31be6ee..8cde839 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -9,16 +9,11 @@ black==22.8.0 cfgv==3.3.1 # via pre-commit click==8.1.3 - # via - # -c requirements/app.txt - # -c requirements/test.txt - # black + # via black distlib==0.3.6 # via virtualenv filelock==3.10.7 - # via - # -c requirements/app.txt - # virtualenv + # via virtualenv flake8==5.0.4 # via -r requirements/dev.in flake8-blind-except==0.2.1 @@ -37,8 +32,7 @@ nodeenv==1.7.0 # via pre-commit packaging==23.0 # via - # -c requirements/app.txt - # -c requirements/test.txt + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # setuptools-scm pathspec==0.10.1 # via black @@ -62,13 +56,11 @@ toml==0.10.2 # via pre-commit tomli==2.0.1 # via - # -c requirements/test.txt # black # mypy # setuptools-scm typing-extensions==4.5.0 # via - # -c requirements/app.txt # mypy # setuptools-scm virtualenv==20.16.5 diff --git a/requirements/test.in b/requirements/test.in index 3f3e08b..75a3140 100644 --- a/requirements/test.in +++ b/requirements/test.in @@ -1,11 +1,3 @@ -c app.txt -pytest -pytest-flask -pytest-sugar -pytest-cov -# lets tests run successfully in containers -# Upper limit because latest versions do not support latest Redis sometimes -fakeredis >2.14, <2.17.0 -# required with fakeredis, maybe because we use rq -lupa +# there are no extra requirements beyond what FlexMeasures installs to test (test.txt in your FlexMeasures requirements) diff --git a/requirements/test.txt b/requirements/test.txt index dfa9c59..e8c8063 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -4,77 +4,3 @@ # # pip-compile --output-file=requirements/test.txt requirements/test.in # -async-timeout==4.0.2 - # via - # -c requirements/app.txt - # redis -attrs==22.2.0 - # via - # -c requirements/app.txt - # pytest -click==8.1.3 - # via - # -c requirements/app.txt - # flask -coverage[toml]==6.4.4 - # via pytest-cov -fakeredis==2.16.0 - # via -r requirements/test.in -flask==2.1.2 - # via - # -c requirements/app.txt - # pytest-flask -iniconfig==1.1.1 - # via pytest -itsdangerous==2.1.2 - # via - # -c requirements/app.txt - # flask -jinja2==3.1.2 - # via - # -c requirements/app.txt - # flask -lupa==1.13 - # via -r requirements/test.in -markupsafe==2.1.2 - # via - # -c requirements/app.txt - # jinja2 -packaging==23.0 - # via - # -c requirements/app.txt - # pytest - # pytest-sugar -pluggy==1.0.0 - # via pytest -py==1.11.0 - # via pytest -pytest==7.1.3 - # via - # -r requirements/test.in - # pytest-cov - # pytest-flask - # pytest-sugar -pytest-cov==3.0.0 - # via -r requirements/test.in -pytest-flask==1.2.0 - # via -r requirements/test.in -pytest-sugar==0.9.5 - # via -r requirements/test.in -redis==4.5.4 - # via - # -c requirements/app.txt - # fakeredis -sortedcontainers==2.4.0 - # via fakeredis -termcolor==2.0.1 - # via pytest-sugar -tomli==2.0.1 - # via - # coverage - # pytest -werkzeug==2.0.3 - # via - # -c requirements/app.txt - # flask - # pytest-flask From 6058182b5244f25c6668d318211e92b303aecef9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 00:56:01 +0200 Subject: [PATCH 11/17] install some libraries to get NetCDF to work in containers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/lint-and-test.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index ca1c401..e352357 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -34,6 +34,9 @@ jobs: sudo apt-get update sudo apt-get -y install postgresql-client psql -h $PGHOST -p $PGPORT --file scripts/load-psql-extensions.sql -U $PGUSER $PGDB; + - name: Install necessary items for netcdf + run: | + sudo apt-get install libhdf5-serial-dev netcdf-bin libnetcdf-dev - run: make test env: PGHOST: 127.0.0.1 From 963991159ab302d4326b93699f4e56dd4822635f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 01:05:26 +0200 Subject: [PATCH 12/17] upgrade dependencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- requirements/app.txt | 76 ++++++++++++++++++++++++++++---------------- requirements/dev.txt | 57 +++++++++++++++++---------------- 2 files changed, 79 insertions(+), 54 deletions(-) diff --git a/requirements/app.txt b/requirements/app.txt index 5872da0..bce252b 100644 --- a/requirements/app.txt +++ b/requirements/app.txt @@ -4,24 +4,35 @@ # # pip-compile --output-file=requirements/app.txt requirements/app.in # -beautifulsoup4==4.11.1 +beautifulsoup4==4.13.4 # via siphon -certifi==2022.12.7 - # via requests -cftime==1.6.1 +blosc2==3.3.4 + # via tables +certifi==2025.4.26 + # via + # netcdf4 + # requests +cftime==1.6.4.post1 # via netcdf4 -charset-normalizer==3.1.0 +charset-normalizer==3.4.2 # via requests -h5py==3.7.0 +h5py==3.13.0 # via pvlib -idna==3.4 +idna==3.10 # via requests -netcdf4==1.6.0 +msgpack==1.1.0 + # via blosc2 +ndindex==1.10.0 + # via blosc2 +netcdf4==1.7.2 # via -r requirements/app.in -numexpr==2.8.3 - # via tables -numpy==1.23.5 +numexpr==2.10.2 + # via + # blosc2 + # tables +numpy==2.2.6 # via + # blosc2 # cftime # h5py # netcdf4 @@ -31,37 +42,48 @@ numpy==1.23.5 # scipy # siphon # tables -packaging==23.0 - # via - # numexpr - # tables -pandas==1.5.3 +packaging==25.0 + # via tables +pandas==2.2.3 # via # pvlib # siphon -protobuf==4.21.5 +platformdirs==4.3.8 + # via blosc2 +protobuf==6.31.1 # via siphon -pvlib==0.9.2 +pvlib==0.12.0 # via -r requirements/app.in -python-dateutil==2.8.2 +py-cpuinfo==9.0.0 + # via + # blosc2 + # tables +python-dateutil==2.9.0.post0 # via pandas -pytz==2023.3 +pytz==2025.2 # via # pandas # pvlib -requests==2.28.2 +requests==2.32.3 # via + # blosc2 # pvlib # siphon -scipy==1.10.1 +scipy==1.15.3 # via pvlib -siphon==0.9 +siphon==0.10.0 # via -r requirements/app.in -six==1.16.0 +six==1.17.0 # via python-dateutil -soupsieve==2.3.2.post1 +soupsieve==2.7 # via beautifulsoup4 -tables==3.7.0 +tables==3.10.1 # via -r requirements/app.in -urllib3==1.26.15 +typing-extensions==4.13.2 + # via + # beautifulsoup4 + # tables +tzdata==2025.2 + # via pandas +urllib3==2.4.0 # via requests diff --git a/requirements/dev.txt b/requirements/dev.txt index 8cde839..230a6cb 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -4,68 +4,71 @@ # # pip-compile --output-file=requirements/dev.txt requirements/dev.in # -black==22.8.0 +black==25.1.0 # via -r requirements/dev.in -cfgv==3.3.1 +cfgv==3.4.0 # via pre-commit -click==8.1.3 +click==8.2.1 # via black -distlib==0.3.6 +distlib==0.3.9 # via virtualenv -filelock==3.10.7 +filelock==3.18.0 # via virtualenv -flake8==5.0.4 +flake8==7.2.0 # via -r requirements/dev.in flake8-blind-except==0.2.1 # via -r requirements/dev.in -identify==2.5.5 +identify==2.6.12 # via pre-commit mccabe==0.7.0 # via flake8 -mypy==0.971 +mypy==1.16.0 # via -r requirements/dev.in -mypy-extensions==0.4.3 +mypy-extensions==1.1.0 # via # black # mypy -nodeenv==1.7.0 +nodeenv==1.9.1 # via pre-commit -packaging==23.0 +packaging==25.0 # via # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # black # setuptools-scm -pathspec==0.10.1 - # via black -platformdirs==2.5.2 +pathspec==0.12.1 + # via + # black + # mypy +platformdirs==4.3.8 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # black # virtualenv -pre-commit==2.20.0 +pre-commit==4.2.0 # via -r requirements/dev.in -pycodestyle==2.9.1 +pycodestyle==2.13.0 # via flake8 -pyflakes==2.5.0 +pyflakes==3.3.2 # via flake8 -pytest-runner==6.0.0 +pytest-runner==6.0.1 # via -r requirements/dev.in -pyyaml==6.0 +pyyaml==6.0.2 # via pre-commit -setuptools-scm==7.0.5 +setuptools-scm==8.3.1 # via -r requirements/dev.in -toml==0.10.2 - # via pre-commit -tomli==2.0.1 +tomli==2.2.1 # via # black # mypy # setuptools-scm -typing-extensions==4.5.0 +typing-extensions==4.13.2 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # black # mypy - # setuptools-scm -virtualenv==20.16.5 +virtualenv==20.31.2 # via pre-commit -watchdog==2.1.9 +watchdog==6.0.0 # via -r requirements/dev.in # The following packages are considered to be unsafe in a requirements file: From 5749b1c64b364866c89565829c3ba4f4011b802b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 01:13:57 +0200 Subject: [PATCH 13/17] add back test requirements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- requirements/dev.txt | 7 ++++- requirements/test.in | 10 ++++++- requirements/test.txt | 64 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 2 deletions(-) diff --git a/requirements/dev.txt b/requirements/dev.txt index 230a6cb..5652582 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -9,7 +9,9 @@ black==25.1.0 cfgv==3.4.0 # via pre-commit click==8.2.1 - # via black + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt + # black distlib==0.3.9 # via virtualenv filelock==3.18.0 @@ -33,6 +35,7 @@ nodeenv==1.9.1 packaging==25.0 # via # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt # black # setuptools-scm pathspec==0.12.1 @@ -58,12 +61,14 @@ setuptools-scm==8.3.1 # via -r requirements/dev.in tomli==2.2.1 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt # black # mypy # setuptools-scm typing-extensions==4.13.2 # via # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt # black # mypy virtualenv==20.31.2 diff --git a/requirements/test.in b/requirements/test.in index 75a3140..3f3e08b 100644 --- a/requirements/test.in +++ b/requirements/test.in @@ -1,3 +1,11 @@ -c app.txt -# there are no extra requirements beyond what FlexMeasures installs to test (test.txt in your FlexMeasures requirements) +pytest +pytest-flask +pytest-sugar +pytest-cov +# lets tests run successfully in containers +# Upper limit because latest versions do not support latest Redis sometimes +fakeredis >2.14, <2.17.0 +# required with fakeredis, maybe because we use rq +lupa diff --git a/requirements/test.txt b/requirements/test.txt index e8c8063..e171f93 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -4,3 +4,67 @@ # # pip-compile --output-file=requirements/test.txt requirements/test.in # +async-timeout==5.0.1 + # via redis +blinker==1.9.0 + # via flask +click==8.2.1 + # via flask +coverage[toml]==7.8.2 + # via pytest-cov +exceptiongroup==1.3.0 + # via pytest +fakeredis==2.16.0 + # via -r requirements/test.in +flask==3.1.1 + # via pytest-flask +iniconfig==2.1.0 + # via pytest +itsdangerous==2.2.0 + # via flask +jinja2==3.1.6 + # via flask +lupa==2.4 + # via -r requirements/test.in +markupsafe==3.0.2 + # via + # flask + # jinja2 + # werkzeug +packaging==25.0 + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # pytest + # pytest-sugar +pluggy==1.6.0 + # via pytest +pytest==8.3.5 + # via + # -r requirements/test.in + # pytest-cov + # pytest-flask + # pytest-sugar +pytest-cov==6.1.1 + # via -r requirements/test.in +pytest-flask==1.3.0 + # via -r requirements/test.in +pytest-sugar==1.0.0 + # via -r requirements/test.in +redis==6.2.0 + # via fakeredis +sortedcontainers==2.4.0 + # via fakeredis +termcolor==3.1.0 + # via pytest-sugar +tomli==2.2.1 + # via + # coverage + # pytest +typing-extensions==4.13.2 + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # exceptiongroup +werkzeug==3.1.3 + # via + # flask + # pytest-flask From 55358e845f28781c2871593693ab5d4937467f6c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 01:19:38 +0200 Subject: [PATCH 14/17] modern use of importlib metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- flexmeasures_weather/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures_weather/__init__.py b/flexmeasures_weather/__init__.py index 09f367c..b76cca0 100644 --- a/flexmeasures_weather/__init__.py +++ b/flexmeasures_weather/__init__.py @@ -8,7 +8,7 @@ """ -from importlib_metadata import version, PackageNotFoundError +from importlib.metadata import version, PackageNotFoundError from flask import Blueprint From 32cabb09369a0dfb96f779031e81e70a9deee704 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 09:42:24 +0200 Subject: [PATCH 15/17] we do depend on FlexMeasures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- requirements/app.in | 1 + requirements/app.txt | 372 +++++++++++++++++++++++++++++++++++++++++- requirements/dev.txt | 10 +- requirements/test.txt | 31 +++- 4 files changed, 397 insertions(+), 17 deletions(-) diff --git a/requirements/app.in b/requirements/app.in index aa34d7a..b26161a 100644 --- a/requirements/app.in +++ b/requirements/app.in @@ -1,3 +1,4 @@ +flexmeasures pvlib # the following three are optional in pvlib, but we use them netCDF4 diff --git a/requirements/app.txt b/requirements/app.txt index bce252b..7191af4 100644 --- a/requirements/app.txt +++ b/requirements/app.txt @@ -4,24 +4,203 @@ # # pip-compile --output-file=requirements/app.txt requirements/app.in # +alembic==1.16.1 + # via flask-migrate +altair==5.5.0 + # via flexmeasures +argon2-cffi==23.1.0 + # via flexmeasures +argon2-cffi-bindings==21.2.0 + # via argon2-cffi +arrow==1.3.0 + # via rq-dashboard +async-timeout==5.0.1 + # via redis +attrs==25.3.0 + # via + # jsonschema + # referencing +babel==2.17.0 + # via py-moneyed +backports-datetime-fromisoformat==2.0.3 + # via marshmallow +bcrypt==4.0.1 + # via flexmeasures beautifulsoup4==4.13.4 # via siphon +blinker==1.9.0 + # via + # flask + # flask-mail + # flask-principal + # sentry-sdk blosc2==3.3.4 # via tables certifi==2025.4.26 # via # netcdf4 # requests + # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings cftime==1.6.4.post1 # via netcdf4 charset-normalizer==3.4.2 # via requests +click==8.2.1 + # via + # click-default-group + # flask + # flexmeasures + # rq +click-default-group==1.2.4 + # via flexmeasures +contourpy==1.3.2 + # via matplotlib +convertdate==2.4.0 + # via workalendar +cycler==0.12.1 + # via matplotlib +dill==0.4.0 + # via openturns +dnspython==2.7.0 + # via email-validator +email-validator==2.2.0 + # via + # flask-security-too + # flexmeasures +filelock==3.18.0 + # via tldextract +flask==3.1.1 + # via + # flask-classful + # flask-cors + # flask-json + # flask-login + # flask-mail + # flask-marshmallow + # flask-migrate + # flask-principal + # flask-security-too + # flask-sqlalchemy + # flask-sslify + # flask-wtf + # flexmeasures + # rq-dashboard + # sentry-sdk +flask-classful==0.16.0 + # via flexmeasures +flask-cors==6.0.0 + # via flexmeasures +flask-json==0.4.0 + # via flexmeasures +flask-login==0.6.3 + # via + # flask-security-too + # flexmeasures +flask-mail==0.10.0 + # via flexmeasures +flask-marshmallow==1.3.0 + # via flexmeasures +flask-migrate==4.1.0 + # via flexmeasures +flask-principal==0.4.0 + # via flask-security-too +flask-security-too==5.6.2 + # via flexmeasures +flask-sqlalchemy==3.1.1 + # via + # flask-migrate + # flexmeasures +flask-sslify==0.1.5 + # via flexmeasures +flask-wtf==1.2.2 + # via + # flask-security-too + # flexmeasures +flexcache==0.3 + # via pint +flexmeasures==0.25.0 + # via -r requirements/app.in +flexparser==0.4 + # via pint +fonttools==4.58.1 + # via matplotlib +greenlet==3.2.2 + # via sqlalchemy h5py==3.13.0 # via pvlib +humanize==4.12.3 + # via flexmeasures idna==3.10 - # via requests + # via + # email-validator + # requests + # tldextract +importlib-metadata==8.7.0 + # via + # flexmeasures + # timely-beliefs +importlib-resources==6.5.2 + # via flask-security-too +inflect==6.0.2 + # via flexmeasures +inflection==0.5.1 + # via flexmeasures +iso8601==2.1.0 + # via flexmeasures +isodate==0.7.2 + # via + # flexmeasures + # timely-beliefs +itsdangerous==2.2.0 + # via + # flask + # flask-wtf +jinja2==3.1.6 + # via + # altair + # flask +joblib==1.4.2 + # via + # scikit-learn + # sktime +jsonschema==4.24.0 + # via altair +jsonschema-specifications==2025.4.1 + # via jsonschema +kiwisolver==1.4.8 + # via matplotlib +lunardate==0.2.2 + # via workalendar +mako==1.3.10 + # via alembic +markupsafe==3.0.2 + # via + # flask + # flask-security-too + # jinja2 + # mako + # sentry-sdk + # werkzeug + # wtforms +marshmallow==4.0.0 + # via + # flask-marshmallow + # flexmeasures + # marshmallow-polyfield + # marshmallow-sqlalchemy + # webargs +marshmallow-polyfield==5.11 + # via flexmeasures +marshmallow-sqlalchemy==1.4.2 + # via flexmeasures +matplotlib==3.10.3 + # via timetomodel msgpack==1.1.0 # via blosc2 +narwhals==1.41.0 + # via altair ndindex==1.10.0 # via blosc2 netcdf4==1.7.2 @@ -30,60 +209,237 @@ numexpr==2.10.2 # via # blosc2 # tables -numpy==2.2.6 +numpy==1.26.4 # via # blosc2 # cftime + # contourpy # h5py + # matplotlib # netcdf4 # numexpr # pandas + # patsy + # properscoring # pvlib + # scikit-learn # scipy # siphon + # sktime + # statsmodels # tables + # timely-beliefs + # timetomodel + # uniplot +openturns==1.24 + # via timely-beliefs packaging==25.0 - # via tables -pandas==2.2.3 # via + # altair + # matplotlib + # sktime + # statsmodels + # tables + # webargs +pandas==2.2.1 + # via + # flexmeasures # pvlib # siphon + # sktime + # statsmodels + # timely-beliefs + # timetomodel +passlib==1.7.4 + # via flask-security-too +patsy==1.0.1 + # via statsmodels +pillow==11.2.1 + # via + # flexmeasures + # matplotlib +pint==0.24.4 + # via flexmeasures platformdirs==4.3.8 - # via blosc2 + # via + # blosc2 + # pint +ply==3.11 + # via pyomo +properscoring==0.1 + # via timely-beliefs protobuf==6.31.1 # via siphon +psutil==7.0.0 + # via openturns +psycopg2-binary==2.9.10 + # via + # flexmeasures + # timely-beliefs pvlib==0.12.0 # via -r requirements/app.in py-cpuinfo==9.0.0 # via # blosc2 # tables +py-moneyed==3.0 + # via flexmeasures +pycparser==2.22 + # via cffi +pydantic==1.10.22 + # via + # flexmeasures + # inflect +pyluach==2.2.0 + # via workalendar +pymeeus==0.5.12 + # via convertdate +pyomo==6.9.2 + # via flexmeasures +pyparsing==3.2.3 + # via matplotlib python-dateutil==2.9.0.post0 - # via pandas + # via + # arrow + # matplotlib + # pandas + # timetomodel + # workalendar +python-dotenv==1.1.0 + # via flexmeasures pytz==2025.2 # via + # flexmeasures # pandas # pvlib + # timely-beliefs + # timetomodel +pyyaml==6.0.2 + # via flexmeasures +redis==6.2.0 + # via + # flexmeasures + # redis-sentinel-url + # rq + # rq-dashboard +redis-sentinel-url==1.0.1 + # via rq-dashboard +referencing==0.36.2 + # via + # jsonschema + # jsonschema-specifications requests==2.32.3 # via # blosc2 # pvlib + # requests-file # siphon + # tldextract +requests-file==2.1.0 + # via tldextract +rpds-py==0.25.1 + # via + # jsonschema + # referencing +rq==2.3.3 + # via + # flexmeasures + # rq-dashboard +rq-dashboard==0.8.2.2 + # via flexmeasures +scikit-base==0.12.3 + # via sktime +scikit-learn==1.6.1 + # via + # sktime + # timetomodel scipy==1.15.3 - # via pvlib + # via + # properscoring + # pvlib + # scikit-learn + # sktime + # statsmodels + # timely-beliefs + # timetomodel +sentry-sdk[flask]==2.29.1 + # via flexmeasures siphon==0.10.0 # via -r requirements/app.in six==1.17.0 # via python-dateutil +sktime==0.37.0 + # via timely-beliefs soupsieve==2.7 # via beautifulsoup4 +sqlalchemy==2.0.41 + # via + # alembic + # flask-sqlalchemy + # flexmeasures + # marshmallow-sqlalchemy + # timely-beliefs + # timetomodel +statsmodels==0.14.4 + # via timetomodel tables==3.10.1 # via -r requirements/app.in +tabulate==0.9.0 + # via flexmeasures +threadpoolctl==3.6.0 + # via scikit-learn +timely-beliefs[forecast]==3.2.0 + # via flexmeasures +timetomodel==0.7.3 + # via flexmeasures +tldextract==5.3.0 + # via flexmeasures +tomli==2.2.1 + # via alembic +types-python-dateutil==2.9.0.20250516 + # via arrow typing-extensions==4.13.2 # via + # alembic + # altair # beautifulsoup4 + # flexcache + # flexparser + # marshmallow + # pint + # py-moneyed + # pydantic + # referencing + # sqlalchemy # tables tzdata==2025.2 # via pandas +uniplot==0.21.1 + # via flexmeasures urllib3==2.4.0 - # via requests + # via + # requests + # sentry-sdk +vl-convert-python==1.8.0 + # via flexmeasures +webargs==8.7.0 + # via flexmeasures +werkzeug==3.1.3 + # via + # flask + # flask-cors + # flask-login + # flexmeasures +workalendar==17.0.0 + # via flexmeasures +wtforms==3.2.1 + # via + # flask-security-too + # flask-wtf +xlrd==2.0.1 + # via flexmeasures +zipp==3.22.0 + # via importlib-metadata + +# The following packages are considered to be unsafe in a requirements file: +# setuptools diff --git a/requirements/dev.txt b/requirements/dev.txt index 5652582..197b782 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -10,12 +10,15 @@ cfgv==3.4.0 # via pre-commit click==8.2.1 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt # black distlib==0.3.9 # via virtualenv filelock==3.18.0 - # via virtualenv + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # virtualenv flake8==7.2.0 # via -r requirements/dev.in flake8-blind-except==0.2.1 @@ -56,11 +59,14 @@ pyflakes==3.3.2 pytest-runner==6.0.1 # via -r requirements/dev.in pyyaml==6.0.2 - # via pre-commit + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # pre-commit setuptools-scm==8.3.1 # via -r requirements/dev.in tomli==2.2.1 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/test.txt # black # mypy diff --git a/requirements/test.txt b/requirements/test.txt index e171f93..0edfa68 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -5,11 +5,17 @@ # pip-compile --output-file=requirements/test.txt requirements/test.in # async-timeout==5.0.1 - # via redis + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # redis blinker==1.9.0 - # via flask + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # flask click==8.2.1 - # via flask + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # flask coverage[toml]==7.8.2 # via pytest-cov exceptiongroup==1.3.0 @@ -17,17 +23,24 @@ exceptiongroup==1.3.0 fakeredis==2.16.0 # via -r requirements/test.in flask==3.1.1 - # via pytest-flask + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # pytest-flask iniconfig==2.1.0 # via pytest itsdangerous==2.2.0 - # via flask + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # flask jinja2==3.1.6 - # via flask + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # flask lupa==2.4 # via -r requirements/test.in markupsafe==3.0.2 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # flask # jinja2 # werkzeug @@ -51,13 +64,16 @@ pytest-flask==1.3.0 pytest-sugar==1.0.0 # via -r requirements/test.in redis==6.2.0 - # via fakeredis + # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt + # fakeredis sortedcontainers==2.4.0 # via fakeredis termcolor==3.1.0 # via pytest-sugar tomli==2.2.1 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # coverage # pytest typing-extensions==4.13.2 @@ -66,5 +82,6 @@ typing-extensions==4.13.2 # exceptiongroup werkzeug==3.1.3 # via + # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # flask # pytest-flask From 277009887a9e13abb59f2e00a0b84d30d7ef830b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 09:55:59 +0200 Subject: [PATCH 16/17] do not limit fakeredis version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- requirements/test.in | 3 +-- requirements/test.txt | 3 ++- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/requirements/test.in b/requirements/test.in index 3f3e08b..8c1f66e 100644 --- a/requirements/test.in +++ b/requirements/test.in @@ -5,7 +5,6 @@ pytest-flask pytest-sugar pytest-cov # lets tests run successfully in containers -# Upper limit because latest versions do not support latest Redis sometimes -fakeredis >2.14, <2.17.0 +fakeredis # required with fakeredis, maybe because we use rq lupa diff --git a/requirements/test.txt b/requirements/test.txt index 0edfa68..d20c75f 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -20,7 +20,7 @@ coverage[toml]==7.8.2 # via pytest-cov exceptiongroup==1.3.0 # via pytest -fakeredis==2.16.0 +fakeredis==2.29.0 # via -r requirements/test.in flask==3.1.1 # via @@ -80,6 +80,7 @@ typing-extensions==4.13.2 # via # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt # exceptiongroup + # fakeredis werkzeug==3.1.3 # via # -c /home/nicolas/workspace/seita/flexmeasures-weather/requirements/app.txt From 5e86aa39d5ea4a5a4ab483ae452f2ef074726b11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Fri, 30 May 2025 13:44:37 +0200 Subject: [PATCH 17/17] limit marshmallow as FexMeasures does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- requirements/app.in | 2 ++ requirements/app.txt | 7 +++---- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/requirements/app.in b/requirements/app.in index b26161a..86168a4 100644 --- a/requirements/app.in +++ b/requirements/app.in @@ -1,4 +1,6 @@ flexmeasures +# remove when FlexMeasures also removes this +marshmallow>=3,<4 pvlib # the following three are optional in pvlib, but we use them netCDF4 diff --git a/requirements/app.txt b/requirements/app.txt index 7191af4..adac73d 100644 --- a/requirements/app.txt +++ b/requirements/app.txt @@ -22,8 +22,6 @@ attrs==25.3.0 # referencing babel==2.17.0 # via py-moneyed -backports-datetime-fromisoformat==2.0.3 - # via marshmallow bcrypt==4.0.1 # via flexmeasures beautifulsoup4==4.13.4 @@ -184,8 +182,9 @@ markupsafe==3.0.2 # sentry-sdk # werkzeug # wtforms -marshmallow==4.0.0 +marshmallow==3.26.1 # via + # -r requirements/app.in # flask-marshmallow # flexmeasures # marshmallow-polyfield @@ -236,6 +235,7 @@ openturns==1.24 packaging==25.0 # via # altair + # marshmallow # matplotlib # sktime # statsmodels @@ -405,7 +405,6 @@ typing-extensions==4.13.2 # beautifulsoup4 # flexcache # flexparser - # marshmallow # pint # py-moneyed # pydantic