Merge branch 'wip/v4.20' into v4

docs/_locale/nl/LC_MESSAGES/how-to/troubleshooting/lagging-statistics.po

@@ -0,0 +1,45 @@
+msgid ""
+msgstr ""
+"Project-Id-Version: DSMR Reader\n"
+"Report-Msgid-Bugs-To: Dennis Siemensma <github@dennissiemensma.nl>\n"
+"Last-Translator: Dennis Siemensma <github@dennissiemensma.nl>\n"
+"Language: nl\n"
+"Language-Team: Dennis Siemensma <github@dennissiemensma.nl>\n"
+"Plural-Forms: nplurals=2; plural=(n != 1);\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.0\n"
+"X-Generator: Poedit 2.3\n"
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:2
+msgid "Common error resolution: How do I fix ``Day statistics are lagging behind``?"
+msgstr "Veel voorkomende fouten: Hoe los ik ``Day statistics are lagging behind`` op?"
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:4
+msgid "The error::"
+msgstr "De foutmelding::"
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:8
+msgid "Means that the statistics module did not run (successfully) or lacks new data. This module runs once a day, by default at midnight."
+msgstr "Betekent dat de statistiekenmodule niet (of onsuccesvol) heeft gedraaid, of gegevens mist. De module draait dagelijks, doorgaans na middernacht."
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:10
+msgid "Due to unforeseen circumstances the module may have not ran. E.g. caused by power outage or restart of the docker image."
+msgstr "De module kan niet (goed) hebben gedraaid door onvoorziene omstandigheden. Bijvoorbeeld door stroomuitval of het herstarten van een Docker image."
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:12
+msgid "Wait until the next scheduled run and see if the problem gets resolved automatically. Alternatively (if you can't wait), go to ``/admin/dsmr_backend/scheduledprocess/`` and change the scheduled time of the ``Generate day and hour statistics`` task into the past, causing it to be executed momentarily."
+msgstr "Wacht tot de volgende run om te zien of dit het probleem oplost. Eventueel kun je ook zelf naar ``/admin/dsmr_backend/scheduledprocess/`` gaan en daar de geplande datumtijd van ``Generate day and hour statistics`` in het verleden zetten. Hierdoor wordt de taak direct opgepakt."
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:15
+msgid "If that doesn't resolve the issue, dive into the DEBUG logs and try to find the problem."
+msgstr "Als dit het probleem niet oplost, schakel DEBUG logging in om verder te zoeken."
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:19
+msgid "Check the logs for ``Stats:`` statements, which may identify a root cause."
+msgstr "Bekijk de logfiles voor regels met ``Stats:``, die een mogelijke oorzaak aangeven."
+
+#: ../../how-to/troubleshooting/lagging-statistics.rst:23
+msgid ":doc:`See here for how to enable DEBUG logging </how-to/troubleshooting/enabling-debug-logging>`."
+msgstr ":doc:`Bekijk hier hoe je DEBUG-logging inschakelt </how-to/troubleshooting/enabling-debug-logging>`."

docs/how-to/index.rst

@@ -22,7 +22,8 @@ How to's
    troubleshooting/smart-meter-replacement
    troubleshooting/smart-meter-invalid-date-or-times
    troubleshooting/recalculate-prices
-
+   troubleshooting/lagging-statistics
+
    database/data-limits
    database/postgresql-restore-backup
    database/postgresql-migrate-day-hour-statistics

docs/how-to/installation/quick.rst

@@ -66,7 +66,7 @@ Continue::
     sudo chown -R dsmr:dsmr /var/www/dsmrreader/
 
     # Code checkout
-    sudo git clone https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader
+    sudo git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader
     sudo chown -R dsmr:dsmr /home/dsmr/
 
     # Virtual env

docs/how-to/troubleshooting/lagging-statistics.rst

@@ -0,0 +1,23 @@
+Common error resolution: How do I fix ``Day statistics are lagging behind``?
+============================================================================
+
+The error::
+
+    Day statistics are lagging behind (1 day, 16 hours ago)
+
+Means that the statistics module did not run (successfully) or lacks new data. This module runs once a day, by default at midnight.
+
+Due to unforeseen circumstances the module may have not ran. E.g. caused by power outage or restart of the docker image.
+
+Wait until the next scheduled run and see if the problem gets resolved automatically.
+Alternatively (if you can't wait), go to ``/admin/dsmr_backend/scheduledprocess/`` and change the scheduled time of the ``Generate day and hour statistics`` task into the past, causing it to be executed momentarily.
+
+If that doesn't resolve the issue, dive into the DEBUG logs and try to find the problem.
+
+.. hint::
+
+    Check the logs for ``Stats:`` statements, which may identify a root cause.
+
+.. seealso::
+
+    :doc:`See here for how to enable DEBUG logging </how-to/troubleshooting/enabling-debug-logging>`.

docs/reference/changelog.rst

@@ -17,6 +17,19 @@ Current version
     :doc:`How to update</how-to/upgrading/upgrade>` *(minor updates only)*
 
 
+v4.20.0 - 2022-02-07
+--------------------
+
+.. note::
+
+    This is the last release of DSMR-reader ``v4.x``. You can upgrade to ``v5.x`` for future support/features/rework.
+
+- ``Added`` Doc update, FAQ regarding "lagging statistics" - by @balk77 [`#1530 <https://github.com/dsmrreader/dsmr-reader/issues/1530>`_]
+
+- ``Fixed`` Periode energiecontracten mist laatste dag [`#1534 <https://github.com/dsmrreader/dsmr-reader/issues/1534>`_]
+- ``Fixed`` Geen grafiek "Verhouding tarieven" als 100% nachtverbruik [`#1523 <https://github.com/dsmrreader/dsmr-reader/issues/1523>`_]
+
+
 v4.19.1 - 2022-02-01
 --------------------
 
@@ -32,8 +45,6 @@ v4.19.0 - 2021-10-23
 
     This is the last **feature** release of DSMR-reader ``v4.x``. Upcoming new features will probably only be added to ``v5.x``.
 
-    *The v4.20 release following this one, will be used to mark the end of this major version and contain an how-to for upgrading to v5 later.*
-
 .. warning::
 
     There has been a bug in the hour statistics since ``v2.10``, offsetting the values by one hour.

docs/tutorial/installation/step-by-step.rst

@@ -134,7 +134,7 @@ Now is the time to clone the code from the repository into the homedir we create
 
 - Clone the repository::
 
-    git clone https://github.com/dsmrreader/dsmr-reader.git
+    git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git
 
 This may take a few seconds. When finished, you should see a new folder called ``dsmr-reader``, containing a clone of the GitHub repository.
 

dsmr_backend/apps.py

@@ -20,7 +20,7 @@ class BackendAppConfig(AppConfig):
     name = 'dsmr_backend'
     verbose_name = _('Backend (dsmr_backend)')
 
-    def ready(self) -> NoReturn:
+    def ready(self) -> NoReturn:  # pragma: no cover
         @register(Tags.compatibility, deploy=True)
         def system_checks(app_configs, **kwargs) -> List:
             """ @see https://docs.djangoproject.com/en/3.1/topics/checks/"""

dsmr_consumption/services.py

@@ -509,9 +509,11 @@ def summarize_energy_contracts() -> List[Dict]:
     }
 
     for current in EnergySupplierPrice.objects.all().order_by('-start'):
+        end_date = current.end or timezone.now().date()
         summary = dsmr_stats.services.range_statistics(
             start=current.start,
-            end=current.end or timezone.now().date()
+            # Note: +1 day is due to range_statistics()'s query (#1534)
+            end=end_date + timezone.timedelta(days=1)
         )
 
         # Override this one, since it's only good when ALL price fields are set.

dsmr_datalogger/admin.py

@@ -72,39 +72,39 @@ class DsmrReadingAdmin(ReadOnlyAdminModel):
         ('timestamp', DateTimeRangeFilter),
     )
 
-    def formatted_electricity_delivered_1(self, obj: DsmrReading) -> str:
+    def formatted_electricity_delivered_1(self, obj: DsmrReading) -> str:  # pragma: no cover
         if not obj.electricity_delivered_1:
             return '-'
 
-        return obj.electricity_delivered_1
+        return str(obj.electricity_delivered_1)
     formatted_electricity_delivered_1.short_description = 'electricity 1'
 
-    def formatted_electricity_delivered_2(self, obj: DsmrReading) -> str:
+    def formatted_electricity_delivered_2(self, obj: DsmrReading) -> str:  # pragma: no cover
         if not obj.electricity_delivered_2:
             return '-'
 
-        return obj.electricity_delivered_2
+        return str(obj.electricity_delivered_2)
     formatted_electricity_delivered_2.short_description = 'electricity 2'
 
-    def formatted_electricity_returned_1(self, obj: DsmrReading) -> str:
+    def formatted_electricity_returned_1(self, obj: DsmrReading) -> str:  # pragma: no cover
         if not obj.electricity_returned_1:
             return '-'
 
-        return obj.electricity_returned_1
+        return str(obj.electricity_returned_1)
     formatted_electricity_returned_1.short_description = 'electricity returned 1'
 
-    def formatted_electricity_returned_2(self, obj: DsmrReading) -> str:
+    def formatted_electricity_returned_2(self, obj: DsmrReading) -> str:  # pragma: no cover
         if not obj.electricity_returned_2:
             return '-'
 
-        return obj.electricity_returned_2
+        return str(obj.electricity_returned_2)
     formatted_electricity_returned_2.short_description = 'electricity returned 2'
 
-    def formatted_extra_device_delivered(self, obj: DsmrReading) -> str:
+    def formatted_extra_device_delivered(self, obj: DsmrReading) -> str:  # pragma: no cover
         if not obj.extra_device_delivered:
             return '-'
 
-        return obj.extra_device_delivered
+        return str(obj.extra_device_delivered)
     formatted_extra_device_delivered.short_description = 'gas'
 
 

dsmr_frontend/migrations/0044_v4_20_0_release.py

@@ -0,0 +1,37 @@
+# Generated by Django 3.1 on 2020-09-07 19:35
+
+from django.db import migrations
+from django.utils.translation import gettext_lazy
+
+
+def migrate_forward(apps, schema_editor):
+    import dsmr_frontend.services
+    import dsmr_backend.services.backend
+
+    if dsmr_backend.services.backend.is_recent_installation():
+        # Skip for new installations.
+        return
+
+    Notification = apps.get_model('dsmr_frontend', 'Notification')
+    Notification.objects.create(
+        message=dsmr_frontend.services.get_translated_string(text=gettext_lazy(
+            'This is the final release on v4. Switching to v5 requires you to manually upgrade. '
+            'Click the link button for more information.'
+        )),
+        redirect_to='frontend:v5-upgrade-redirect'
+    )
+
+
+def migrate_backward(apps, schema_editor):
+    pass
+
+
+class Migration(migrations.Migration):
+
+    operations = [
+        migrations.RunPython(migrate_forward, migrate_backward),
+    ]
+
+    dependencies = [
+        ('dsmr_frontend', '0043_default_color_update_tariff_2'),
+    ]

dsmr_frontend/tests/webinterface/test_generic.py

@@ -200,7 +200,7 @@ class TestAlwaysRequireLoginDisabled(TestCase):
         'docs-redirect',
         'feedback-redirect',
         'donations-redirect',
-        'v4-upgrade-redirect',
+        'v5-upgrade-redirect',
         'configuration',
         'status',
         'export',

dsmr_frontend/urls.py

@@ -11,7 +11,7 @@
 from dsmr_frontend.views.compare import Compare, CompareXhrSummary
 from dsmr_frontend.views.export import Export, ExportAsCsv
 from dsmr_frontend.views.generic import ChangelogRedirect, DocsRedirect, FeedbackRedirect, DonationsRedirect, \
-    XhrHeader, V4UpgradeRedirect, StatusRedirectView
+    XhrHeader, V5UpgradeRedirect, StatusRedirectView
 from dsmr_frontend.views.energy_contracts import EnergyContracts
 from dsmr_frontend.views.live_graphs import LiveGraphs, LiveXhrElectricityConsumption, LiveXhrGasConsumption, \
     LiveXhrTemperature
@@ -58,7 +58,7 @@
     path('docs-redirect', DocsRedirect.as_view(), name='docs-redirect'),
     path('feedback-redirect', FeedbackRedirect.as_view(), name='feedback-redirect'),
     path('donations-redirect', DonationsRedirect.as_view(), name='donations-redirect'),
-    path('v4-upgrade-redirect', V4UpgradeRedirect.as_view(), name='v4-upgrade-redirect'),
+    path('v5-upgrade-redirect', V5UpgradeRedirect.as_view(), name='v5-upgrade-redirect'),
 
     # Views always requiring authentication.
     path('configuration', Configuration.as_view(), name='configuration'),

dsmr_frontend/views/generic.py

@@ -54,6 +54,6 @@ class DonationsRedirect(ReadTheDocsRedirectView):
     subpage = 'donations.html'
 
 
-class V4UpgradeRedirect(ReadTheDocsRedirectView):
-    subpage = 'faq/v4_upgrade.html'
-    branch = 'v4'
+class V5UpgradeRedirect(ReadTheDocsRedirectView):
+    subpage = 'tutorial/upgrading/to-v5.html'
+    branch = 'v5'

dsmr_frontend/views/trends.py

@@ -98,18 +98,19 @@ def get(self, request):
             'electricity1': frontend_settings.tariff_1_delivered_name.capitalize(),
             'electricity2': frontend_settings.tariff_2_delivered_name.capitalize(),
         }
+        electricity_tariff_percentage = dsmr_stats.services.electricity_tariff_percentage(
+            start=form.cleaned_data['start_date'],
+            end=form.cleaned_data['end_date'],
+        )
         result = {}
 
-        if not capabilities[Capability.ANY] or not DayStatistics.objects.exists():
+        if not capabilities[Capability.ANY] or not DayStatistics.objects.exists() or not electricity_tariff_percentage:
             return JsonResponse(result)
 
         result['data'] = [
             {'name': translation_mapping[k], 'value': v}
             for k, v in
-            dsmr_stats.services.electricity_tariff_percentage(
-                start=form.cleaned_data['start_date'],
-                end=form.cleaned_data['end_date'],
-            ).items()
+            electricity_tariff_percentage.items()
         ]
 
         return JsonResponse(result)

dsmr_stats/admin.py

@@ -75,25 +75,25 @@ class HourStatisticsAdmin(admin.ModelAdmin):
         ('hour_start', DateTimeRangeFilter),
     )
 
-    def formatted_electricity_merged(self, obj: HourStatistics) -> str:
+    def formatted_electricity_merged(self, obj: HourStatistics) -> str:  # pragma: no cover
         if not obj.electricity_merged:
             return '-'
 
         return obj.electricity_merged
     formatted_electricity_merged.short_description = 'electricity delivered'
 
-    def formatted_electricity_returned_merged(self, obj: HourStatistics) -> str:
+    def formatted_electricity_returned_merged(self, obj: HourStatistics) -> str:  # pragma: no cover
         if not obj.electricity_returned_merged:
             return '-'
 
         return obj.electricity_returned_merged
     formatted_electricity_returned_merged.short_description = 'electricity returned'
 
-    def formatted_gas(self, obj: HourStatistics) -> str:
+    def formatted_gas(self, obj: HourStatistics) -> str:  # pragma: no cover
         if not obj.gas:
             return '-'
 
-        return obj.gas
+        return str(obj.gas)
     formatted_gas.short_description = 'gas'
 
 

dsmr_stats/tests/test_services.py

@@ -641,10 +641,11 @@ def test_electricity_tariff_percentage(self):
 
         # Now try again without data.
         DayStatistics.objects.all().delete()
-        dsmr_stats.services.electricity_tariff_percentage(
+        percentages = dsmr_stats.services.electricity_tariff_percentage(
             start=target_date.date(),
             end=timezone.now().date(),
         )
+        self.assertIsNone(percentages)
 
     @mock.patch('dsmr_stats.services.update_electricity_statistics')
     def test_dsmr_update_electricity_statistics_signal(self, service_mock):

dsmrreader/__init__.py

@@ -1,5 +1,5 @@
 from django.utils.version import get_version
 
-VERSION = (4, 19, 1, 'final', 0)
+VERSION = (4, 20, 0, 'final', 0)
 
 __version__ = get_version(VERSION)

dsmrreader/locales/nl/LC_MESSAGES/django.po

@@ -900,6 +900,9 @@ msgstr "DSMR-reader v4.1: Ondersteuning toegevoegd voor het uitlezen van netwerk
 msgid "DSMR-reader will eventually be moved to its own organisation on GitHub in the near future. It should be redirected automatically, but if not, the new location will be: https://github.com/dsmrreader/dsmr-reader"
 msgstr "DSMR-reader wordt uiteindelijk onder een eigen 'organisatie' ondergebracht op GitHub. Het doorverwijzen zou t.z.t automatisch moeten gaan, maar zo niet, dan wordt de nieuwe locatie: https://github.com/dsmrreader/dsmr-reader"
 
+msgid "This is the final release on v4. Switching to v5 requires you to manually upgrade. Click the link button for more information."
+msgstr "Dit is de laatste release op v4. Wisselen naar v5 vereist je om handmatig te upgraden. Klik op de link/knop voor meer informatie."
+
 msgid "Application notification"
 msgstr "Applicatienotificatie"
 

dsmrreader/provisioning/downgrade/v4.20.0.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+# Dump for DSMR-reader v4.20.0
+./manage.py migrate dsmr_api 0003_create_api_user
+./manage.py migrate dsmr_backend 0015_backend_restart_required
+./manage.py migrate dsmr_backup 0013_dropbox_setting_refactoring
+./manage.py migrate dsmr_consumption 0019_energy_supplier_price_decimals
+./manage.py migrate dsmr_datalogger 0030_override_telegram_timestamp
+./manage.py migrate dsmr_dropbox 0001_schedule_dropbox
+./manage.py migrate dsmr_frontend 0044_v4_20_0_release
+./manage.py migrate dsmr_influxdb 0004_client_settings_update
+./manage.py migrate dsmr_mindergas 0005_schedule_mindergas_export
+./manage.py migrate dsmr_mqtt 0020_drop_mqtt_qos_setting
+./manage.py migrate dsmr_notification 0008_dummy_notification_provider
+./manage.py migrate dsmr_pvoutput 0004_pvoutput_setting_refactoring
+./manage.py migrate dsmr_stats 0017_day_statistics_reading_history_retroactive
+./manage.py migrate dsmr_weather 0006_schedule_weather_update

tools/install.sh

@@ -19,7 +19,7 @@ sudo mkdir -p /var/www/dsmrreader/static
 sudo chown -R dsmr:dsmr /var/www/dsmrreader/
 
 echo "Cloning DSMR-reader from GitHub"
-sudo git clone https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader
+sudo git clone --branch v4 https://github.com/dsmrreader/dsmr-reader.git /home/dsmr/dsmr-reader
 sudo chown -R dsmr:dsmr /home/dsmr/
 
 echo "Setting up virtualenv"

tools/quick-test.sh

@@ -14,7 +14,7 @@ export DJANGO_SETTINGS_MODULE=dsmrreader.config.test
 echo ""
 echo "--- Testing with SQLite..."
 export DJANGO_DATABASE_ENGINE=django.db.backends.sqlite3
-pytest --cov --cov-report=html $ARGS
+pytest --cov --cov-report=html --cov-report=term $ARGS
 
 if [ $? -ne 0 ]; then
     echo "[!] Tests failed [!]"