diff --git a/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.mo b/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.mo new file mode 100644 index 000000000..43f3693ac Binary files /dev/null and b/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.mo differ diff --git a/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.po b/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.po new file mode 100644 index 000000000..ba8fd2de6 --- /dev/null +++ b/docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.po @@ -0,0 +1,45 @@ +msgid "" +msgstr "" +"Project-Id-Version: DSMR Reader\n" +"Report-Msgid-Bugs-To: Dennis Siemensma \n" +"Last-Translator: Dennis Siemensma \n" +"Language: nl\n" +"Language-Team: Dennis Siemensma \n" +"Plural-Forms: nplurals=2; plural=(n != 1);\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=utf-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Generated-By: Babel 2.9.0\n" +"X-Generator: Poedit 2.3\n" + +#: ../../how-to/troubleshooting/lagging-statistics.rst:2 +msgid "Common error resolution: How do I fix ``Day statistics are lagging behind``?" +msgstr "Veel voorkomende fouten: Hoe los ik ``Day statistics are lagging behind`` op?" + +#: ../../how-to/troubleshooting/lagging-statistics.rst:4 +msgid "The error::" +msgstr "De foutmelding::" + +#: ../../how-to/troubleshooting/lagging-statistics.rst:8 +msgid "Means that the statistics module did not run (successfully) or lacks new data. This module runs once a day, by default at midnight." +msgstr "Betekent dat de statistiekenmodule niet (of onsuccesvol) heeft gedraaid, of gegevens mist. De module draait dagelijks, doorgaans na middernacht." + +#: ../../how-to/troubleshooting/lagging-statistics.rst:10 +msgid "Due to unforeseen circumstances the module may have not ran. E.g. caused by power outage or restart of the docker image." +msgstr "De module kan niet (goed) hebben gedraaid door onvoorziene omstandigheden. Bijvoorbeeld door stroomuitval of het herstarten van een Docker image." + +#: ../../how-to/troubleshooting/lagging-statistics.rst:12 +msgid "Wait until the next scheduled run and see if the problem gets resolved automatically. Alternatively (if you can't wait), go to ``/admin/dsmr_backend/scheduledprocess/`` and change the scheduled time of the ``Generate day and hour statistics`` task into the past, causing it to be executed momentarily." +msgstr "Wacht tot de volgende run om te zien of dit het probleem oplost. Eventueel kun je ook zelf naar ``/admin/dsmr_backend/scheduledprocess/`` gaan en daar de geplande datumtijd van ``Generate day and hour statistics`` in het verleden zetten. Hierdoor wordt de taak direct opgepakt." + +#: ../../how-to/troubleshooting/lagging-statistics.rst:15 +msgid "If that doesn't resolve the issue, dive into the DEBUG logs and try to find the problem." +msgstr "Als dit het probleem niet oplost, schakel DEBUG logging in om verder te zoeken." + +#: ../../how-to/troubleshooting/lagging-statistics.rst:19 +msgid "Check the logs for ``Stats:`` statements, which may identify a root cause." +msgstr "Bekijk de logfiles voor regels met ``Stats:``, die een mogelijke oorzaak aangeven." + +#: ../../how-to/troubleshooting/lagging-statistics.rst:23 +msgid ":doc:`See here for how to enable DEBUG logging `." +msgstr ":doc:`Bekijk hier hoe je DEBUG-logging inschakelt `." diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index dca3194f2..0a637f9d1 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -22,7 +22,8 @@ How to's troubleshooting/smart-meter-replacement troubleshooting/smart-meter-invalid-date-or-times troubleshooting/recalculate-prices - + troubleshooting/lagging-statistics + database/data-limits database/postgresql-restore-backup database/postgresql-migrate-day-hour-statistics diff --git a/docs/how-to/installation/quick.rst b/docs/how-to/installation/quick.rst index 4c5d37e57..7539c3649 100644 --- a/docs/how-to/installation/quick.rst +++ b/docs/how-to/installation/quick.rst @@ -66,7 +66,7 @@ Continue:: sudo chown -R dsmr:dsmr /var/www/dsmrreader/ # Code checkout - sudo git clone https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader + sudo git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader sudo chown -R dsmr:dsmr /home/dsmr/ # Virtual env diff --git a/docs/how-to/troubleshooting/lagging-statistics.rst b/docs/how-to/troubleshooting/lagging-statistics.rst new file mode 100644 index 000000000..d3e0a0900 --- /dev/null +++ b/docs/how-to/troubleshooting/lagging-statistics.rst @@ -0,0 +1,23 @@ +Common error resolution: How do I fix ``Day statistics are lagging behind``? +============================================================================ + +The error:: + + Day statistics are lagging behind (1 day, 16 hours ago) + +Means that the statistics module did not run (successfully) or lacks new data. This module runs once a day, by default at midnight. + +Due to unforeseen circumstances the module may have not ran. E.g. caused by power outage or restart of the docker image. + +Wait until the next scheduled run and see if the problem gets resolved automatically. +Alternatively (if you can't wait), go to ``/admin/dsmr_backend/scheduledprocess/`` and change the scheduled time of the ``Generate day and hour statistics`` task into the past, causing it to be executed momentarily. + +If that doesn't resolve the issue, dive into the DEBUG logs and try to find the problem. + +.. hint:: + + Check the logs for ``Stats:`` statements, which may identify a root cause. + +.. seealso:: + + :doc:`See here for how to enable DEBUG logging `. diff --git a/docs/reference/changelog.rst b/docs/reference/changelog.rst index 1d3c1c19c..3a480bd1a 100644 --- a/docs/reference/changelog.rst +++ b/docs/reference/changelog.rst @@ -17,6 +17,19 @@ Current version :doc:`How to update` *(minor updates only)* +v4.20.0 - 2022-02-07 +-------------------- + +.. note:: + + This is the last release of DSMR-reader ``v4.x``. You can upgrade to ``v5.x`` for future support/features/rework. + +- ``Added`` Doc update, FAQ regarding "lagging statistics" - by @balk77 [`#1530 `_] + +- ``Fixed`` Periode energiecontracten mist laatste dag [`#1534 `_] +- ``Fixed`` Geen grafiek "Verhouding tarieven" als 100% nachtverbruik [`#1523 `_] + + v4.19.1 - 2022-02-01 -------------------- @@ -32,8 +45,6 @@ v4.19.0 - 2021-10-23 This is the last **feature** release of DSMR-reader ``v4.x``. Upcoming new features will probably only be added to ``v5.x``. - *The v4.20 release following this one, will be used to mark the end of this major version and contain an how-to for upgrading to v5 later.* - .. warning:: There has been a bug in the hour statistics since ``v2.10``, offsetting the values by one hour. diff --git a/docs/tutorial/installation/step-by-step.rst b/docs/tutorial/installation/step-by-step.rst index 9d13d4ada..b93e0f4cf 100644 --- a/docs/tutorial/installation/step-by-step.rst +++ b/docs/tutorial/installation/step-by-step.rst @@ -134,7 +134,7 @@ Now is the time to clone the code from the repository into the homedir we create - Clone the repository:: - git clone https://github.com/dsmrreader/dsmr-reader.git + git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git This may take a few seconds. When finished, you should see a new folder called ``dsmr-reader``, containing a clone of the GitHub repository. diff --git a/dsmr_backend/apps.py b/dsmr_backend/apps.py index 29d2cee3d..09f6a28da 100644 --- a/dsmr_backend/apps.py +++ b/dsmr_backend/apps.py @@ -20,7 +20,7 @@ class BackendAppConfig(AppConfig): name = 'dsmr_backend' verbose_name = _('Backend (dsmr_backend)') - def ready(self) -> NoReturn: + def ready(self) -> NoReturn: # pragma: no cover @register(Tags.compatibility, deploy=True) def system_checks(app_configs, **kwargs) -> List: """ @see https://docs.djangoproject.com/en/3.1/topics/checks/""" diff --git a/dsmr_consumption/services.py b/dsmr_consumption/services.py index a85fd0b23..7aff50b2e 100644 --- a/dsmr_consumption/services.py +++ b/dsmr_consumption/services.py @@ -509,9 +509,11 @@ def summarize_energy_contracts() -> List[Dict]: } for current in EnergySupplierPrice.objects.all().order_by('-start'): + end_date = current.end or timezone.now().date() summary = dsmr_stats.services.range_statistics( start=current.start, - end=current.end or timezone.now().date() + # Note: +1 day is due to range_statistics()'s query (#1534) + end=end_date + timezone.timedelta(days=1) ) # Override this one, since it's only good when ALL price fields are set. diff --git a/dsmr_datalogger/admin.py b/dsmr_datalogger/admin.py index 982d33ed1..3a419b523 100644 --- a/dsmr_datalogger/admin.py +++ b/dsmr_datalogger/admin.py @@ -72,39 +72,39 @@ class DsmrReadingAdmin(ReadOnlyAdminModel): ('timestamp', DateTimeRangeFilter), ) - def formatted_electricity_delivered_1(self, obj: DsmrReading) -> str: + def formatted_electricity_delivered_1(self, obj: DsmrReading) -> str: # pragma: no cover if not obj.electricity_delivered_1: return '-' - return obj.electricity_delivered_1 + return str(obj.electricity_delivered_1) formatted_electricity_delivered_1.short_description = 'electricity 1' - def formatted_electricity_delivered_2(self, obj: DsmrReading) -> str: + def formatted_electricity_delivered_2(self, obj: DsmrReading) -> str: # pragma: no cover if not obj.electricity_delivered_2: return '-' - return obj.electricity_delivered_2 + return str(obj.electricity_delivered_2) formatted_electricity_delivered_2.short_description = 'electricity 2' - def formatted_electricity_returned_1(self, obj: DsmrReading) -> str: + def formatted_electricity_returned_1(self, obj: DsmrReading) -> str: # pragma: no cover if not obj.electricity_returned_1: return '-' - return obj.electricity_returned_1 + return str(obj.electricity_returned_1) formatted_electricity_returned_1.short_description = 'electricity returned 1' - def formatted_electricity_returned_2(self, obj: DsmrReading) -> str: + def formatted_electricity_returned_2(self, obj: DsmrReading) -> str: # pragma: no cover if not obj.electricity_returned_2: return '-' - return obj.electricity_returned_2 + return str(obj.electricity_returned_2) formatted_electricity_returned_2.short_description = 'electricity returned 2' - def formatted_extra_device_delivered(self, obj: DsmrReading) -> str: + def formatted_extra_device_delivered(self, obj: DsmrReading) -> str: # pragma: no cover if not obj.extra_device_delivered: return '-' - return obj.extra_device_delivered + return str(obj.extra_device_delivered) formatted_extra_device_delivered.short_description = 'gas' diff --git a/dsmr_frontend/migrations/0044_v4_20_0_release.py b/dsmr_frontend/migrations/0044_v4_20_0_release.py new file mode 100644 index 000000000..b57bbc296 --- /dev/null +++ b/dsmr_frontend/migrations/0044_v4_20_0_release.py @@ -0,0 +1,37 @@ +# Generated by Django 3.1 on 2020-09-07 19:35 + +from django.db import migrations +from django.utils.translation import gettext_lazy + + +def migrate_forward(apps, schema_editor): + import dsmr_frontend.services + import dsmr_backend.services.backend + + if dsmr_backend.services.backend.is_recent_installation(): + # Skip for new installations. + return + + Notification = apps.get_model('dsmr_frontend', 'Notification') + Notification.objects.create( + message=dsmr_frontend.services.get_translated_string(text=gettext_lazy( + 'This is the final release on v4. Switching to v5 requires you to manually upgrade. ' + 'Click the link button for more information.' + )), + redirect_to='frontend:v5-upgrade-redirect' + ) + + +def migrate_backward(apps, schema_editor): + pass + + +class Migration(migrations.Migration): + + operations = [ + migrations.RunPython(migrate_forward, migrate_backward), + ] + + dependencies = [ + ('dsmr_frontend', '0043_default_color_update_tariff_2'), + ] diff --git a/dsmr_frontend/tests/webinterface/test_generic.py b/dsmr_frontend/tests/webinterface/test_generic.py index 10bc431de..a3771d95f 100644 --- a/dsmr_frontend/tests/webinterface/test_generic.py +++ b/dsmr_frontend/tests/webinterface/test_generic.py @@ -200,7 +200,7 @@ class TestAlwaysRequireLoginDisabled(TestCase): 'docs-redirect', 'feedback-redirect', 'donations-redirect', - 'v4-upgrade-redirect', + 'v5-upgrade-redirect', 'configuration', 'status', 'export', diff --git a/dsmr_frontend/urls.py b/dsmr_frontend/urls.py index 0f37b8c10..1f809bd6f 100644 --- a/dsmr_frontend/urls.py +++ b/dsmr_frontend/urls.py @@ -11,7 +11,7 @@ from dsmr_frontend.views.compare import Compare, CompareXhrSummary from dsmr_frontend.views.export import Export, ExportAsCsv from dsmr_frontend.views.generic import ChangelogRedirect, DocsRedirect, FeedbackRedirect, DonationsRedirect, \ - XhrHeader, V4UpgradeRedirect, StatusRedirectView + XhrHeader, V5UpgradeRedirect, StatusRedirectView from dsmr_frontend.views.energy_contracts import EnergyContracts from dsmr_frontend.views.live_graphs import LiveGraphs, LiveXhrElectricityConsumption, LiveXhrGasConsumption, \ LiveXhrTemperature @@ -58,7 +58,7 @@ path('docs-redirect', DocsRedirect.as_view(), name='docs-redirect'), path('feedback-redirect', FeedbackRedirect.as_view(), name='feedback-redirect'), path('donations-redirect', DonationsRedirect.as_view(), name='donations-redirect'), - path('v4-upgrade-redirect', V4UpgradeRedirect.as_view(), name='v4-upgrade-redirect'), + path('v5-upgrade-redirect', V5UpgradeRedirect.as_view(), name='v5-upgrade-redirect'), # Views always requiring authentication. path('configuration', Configuration.as_view(), name='configuration'), diff --git a/dsmr_frontend/views/generic.py b/dsmr_frontend/views/generic.py index ab3efb539..9a17d173a 100644 --- a/dsmr_frontend/views/generic.py +++ b/dsmr_frontend/views/generic.py @@ -54,6 +54,6 @@ class DonationsRedirect(ReadTheDocsRedirectView): subpage = 'donations.html' -class V4UpgradeRedirect(ReadTheDocsRedirectView): - subpage = 'faq/v4_upgrade.html' - branch = 'v4' +class V5UpgradeRedirect(ReadTheDocsRedirectView): + subpage = 'tutorial/upgrading/to-v5.html' + branch = 'v5' diff --git a/dsmr_frontend/views/trends.py b/dsmr_frontend/views/trends.py index 7d3bcdbb2..11fe08c21 100644 --- a/dsmr_frontend/views/trends.py +++ b/dsmr_frontend/views/trends.py @@ -98,18 +98,19 @@ def get(self, request): 'electricity1': frontend_settings.tariff_1_delivered_name.capitalize(), 'electricity2': frontend_settings.tariff_2_delivered_name.capitalize(), } + electricity_tariff_percentage = dsmr_stats.services.electricity_tariff_percentage( + start=form.cleaned_data['start_date'], + end=form.cleaned_data['end_date'], + ) result = {} - if not capabilities[Capability.ANY] or not DayStatistics.objects.exists(): + if not capabilities[Capability.ANY] or not DayStatistics.objects.exists() or not electricity_tariff_percentage: return JsonResponse(result) result['data'] = [ {'name': translation_mapping[k], 'value': v} for k, v in - dsmr_stats.services.electricity_tariff_percentage( - start=form.cleaned_data['start_date'], - end=form.cleaned_data['end_date'], - ).items() + electricity_tariff_percentage.items() ] return JsonResponse(result) diff --git a/dsmr_stats/admin.py b/dsmr_stats/admin.py index 297ff1416..52a933258 100644 --- a/dsmr_stats/admin.py +++ b/dsmr_stats/admin.py @@ -75,25 +75,25 @@ class HourStatisticsAdmin(admin.ModelAdmin): ('hour_start', DateTimeRangeFilter), ) - def formatted_electricity_merged(self, obj: HourStatistics) -> str: + def formatted_electricity_merged(self, obj: HourStatistics) -> str: # pragma: no cover if not obj.electricity_merged: return '-' return obj.electricity_merged formatted_electricity_merged.short_description = 'electricity delivered' - def formatted_electricity_returned_merged(self, obj: HourStatistics) -> str: + def formatted_electricity_returned_merged(self, obj: HourStatistics) -> str: # pragma: no cover if not obj.electricity_returned_merged: return '-' return obj.electricity_returned_merged formatted_electricity_returned_merged.short_description = 'electricity returned' - def formatted_gas(self, obj: HourStatistics) -> str: + def formatted_gas(self, obj: HourStatistics) -> str: # pragma: no cover if not obj.gas: return '-' - return obj.gas + return str(obj.gas) formatted_gas.short_description = 'gas' diff --git a/dsmr_stats/tests/test_services.py b/dsmr_stats/tests/test_services.py index 42f523859..d45038379 100644 --- a/dsmr_stats/tests/test_services.py +++ b/dsmr_stats/tests/test_services.py @@ -641,10 +641,11 @@ def test_electricity_tariff_percentage(self): # Now try again without data. DayStatistics.objects.all().delete() - dsmr_stats.services.electricity_tariff_percentage( + percentages = dsmr_stats.services.electricity_tariff_percentage( start=target_date.date(), end=timezone.now().date(), ) + self.assertIsNone(percentages) @mock.patch('dsmr_stats.services.update_electricity_statistics') def test_dsmr_update_electricity_statistics_signal(self, service_mock): diff --git a/dsmrreader/__init__.py b/dsmrreader/__init__.py index 1502393ba..d7d410736 100644 --- a/dsmrreader/__init__.py +++ b/dsmrreader/__init__.py @@ -1,5 +1,5 @@ from django.utils.version import get_version -VERSION = (4, 19, 1, 'final', 0) +VERSION = (4, 20, 0, 'final', 0) __version__ = get_version(VERSION) diff --git a/dsmrreader/locales/nl/LC_MESSAGES/django.mo b/dsmrreader/locales/nl/LC_MESSAGES/django.mo index 6b5311b0f..43ad2acfd 100644 Binary files a/dsmrreader/locales/nl/LC_MESSAGES/django.mo and b/dsmrreader/locales/nl/LC_MESSAGES/django.mo differ diff --git a/dsmrreader/locales/nl/LC_MESSAGES/django.po b/dsmrreader/locales/nl/LC_MESSAGES/django.po index a4332a1f0..fd3afbd88 100644 --- a/dsmrreader/locales/nl/LC_MESSAGES/django.po +++ b/dsmrreader/locales/nl/LC_MESSAGES/django.po @@ -900,6 +900,9 @@ msgstr "DSMR-reader v4.1: Ondersteuning toegevoegd voor het uitlezen van netwerk msgid "DSMR-reader will eventually be moved to its own organisation on GitHub in the near future. It should be redirected automatically, but if not, the new location will be: https://github.com/dsmrreader/dsmr-reader" msgstr "DSMR-reader wordt uiteindelijk onder een eigen 'organisatie' ondergebracht op GitHub. Het doorverwijzen zou t.z.t automatisch moeten gaan, maar zo niet, dan wordt de nieuwe locatie: https://github.com/dsmrreader/dsmr-reader" +msgid "This is the final release on v4. Switching to v5 requires you to manually upgrade. Click the link button for more information." +msgstr "Dit is de laatste release op v4. Wisselen naar v5 vereist je om handmatig te upgraden. Klik op de link/knop voor meer informatie." + msgid "Application notification" msgstr "Applicatienotificatie" diff --git a/dsmrreader/provisioning/downgrade/v4.20.0.sh b/dsmrreader/provisioning/downgrade/v4.20.0.sh new file mode 100644 index 000000000..adaae1c9b --- /dev/null +++ b/dsmrreader/provisioning/downgrade/v4.20.0.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash + +# Dump for DSMR-reader v4.20.0 +./manage.py migrate dsmr_api 0003_create_api_user +./manage.py migrate dsmr_backend 0015_backend_restart_required +./manage.py migrate dsmr_backup 0013_dropbox_setting_refactoring +./manage.py migrate dsmr_consumption 0019_energy_supplier_price_decimals +./manage.py migrate dsmr_datalogger 0030_override_telegram_timestamp +./manage.py migrate dsmr_dropbox 0001_schedule_dropbox +./manage.py migrate dsmr_frontend 0044_v4_20_0_release +./manage.py migrate dsmr_influxdb 0004_client_settings_update +./manage.py migrate dsmr_mindergas 0005_schedule_mindergas_export +./manage.py migrate dsmr_mqtt 0020_drop_mqtt_qos_setting +./manage.py migrate dsmr_notification 0008_dummy_notification_provider +./manage.py migrate dsmr_pvoutput 0004_pvoutput_setting_refactoring +./manage.py migrate dsmr_stats 0017_day_statistics_reading_history_retroactive +./manage.py migrate dsmr_weather 0006_schedule_weather_update diff --git a/tools/install.sh b/tools/install.sh index bdf3ee0ac..64848c6c0 100755 --- a/tools/install.sh +++ b/tools/install.sh @@ -19,7 +19,7 @@ sudo mkdir -p /var/www/dsmrreader/static sudo chown -R dsmr:dsmr /var/www/dsmrreader/ echo "Cloning DSMR-reader from GitHub" -sudo git clone https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader +sudo git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader sudo chown -R dsmr:dsmr /home/dsmr/ echo "Setting up virtualenv" diff --git a/tools/quick-test.sh b/tools/quick-test.sh index 2c7bc9eef..5e4ff01b5 100755 --- a/tools/quick-test.sh +++ b/tools/quick-test.sh @@ -14,7 +14,7 @@ export DJANGO_SETTINGS_MODULE=dsmrreader.config.test echo "" echo "--- Testing with SQLite..." export DJANGO_DATABASE_ENGINE=django.db.backends.sqlite3 -pytest --cov --cov-report=html $ARGS +pytest --cov --cov-report=html --cov-report=term $ARGS if [ $? -ne 0 ]; then echo "[!] Tests failed [!]"