Merged development branch to master for v1.5.2

.gitignore

@@ -5,4 +5,5 @@
 /dsmrreader/settings.py
 /coverage_report/
 /backups/
+/logs/
 /.coverage

docs/api.rst

@@ -1,6 +1,6 @@
 API
 ===
-The application has a simple, one-command API for remote dataloggers.
+The application has a simple, one-command API for remote dataloggers. It allows you to run the datalogger and application separately.
 
 
 .. contents::
@@ -12,8 +12,10 @@ Configuration
 Enable API
 ^^^^^^^^^^
 
-By default the API is disabled in the application. You may enable it in your configuration or admin settings.
+The API is disabled by default in the application. You may enable it in your configuration or admin settings.
 
+Example
+~~~~~~~
 .. image:: _static/screenshots/admin_api_settings.png
     :target: _static/screenshots/admin_api_settings.png
     :alt: API admin settings
@@ -22,16 +24,16 @@ Authentication
 ^^^^^^^^^^^^^^
 Besides allowing the API to listen for requests, you will also need use the generated API Auth Key. 
 It can be found on the same page as in the screenshot above. The configuration page will also display it, but only partly.
-Feel free to alter the API Auth Key when required. The application initially randomly generates one for you. 
+Feel free to alter the API Auth Key when required. The application initially generates one randomly for you. 
 
 You should pass it in the header of every API call. The header should be defined as ``X-AUTHKEY``. See below for an example. 
 
 Examples
-^^^^^^^^
+~~~~~~~~
 
 Using ``cURL``::
 
-   curl http://YOUR-DSMR-URL/api/v1/endpointX \
+   curl http://YOUR-DSMR-URL/api/v1/datalogger/dsmrreading \
         -d 'telegram=xxxxx' \
         -H 'X-AUTHKEY: YOUR-DSMR-API-AUTHKEY'
         
@@ -49,8 +51,13 @@ API calls
 
 POST ``/api/v1`` ``/datalogger/dsmrreading``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Description
+~~~~~~~~~~~
 This allows you to insert a raw telegram, read from your meter remotely, into the application as if it was read locally using the serial cable.
 
+Parameters
+~~~~~~~~~~
 - Method: ``POST``
 - Data: ``telegram`` (as raw string containing all linefeeds ``\n``, and carriage returns ``\r``, as well!)
 - Status code returned: ``HTTP 200`` on success, any other on failure.
@@ -100,4 +107,145 @@ Example
     if response.status_code != 200:
         # Or you will find the error (hint) in the reponse body on failure.
         print('Error: {}'.format(response.text))
-    
+
+
+Script
+------
+Below is a more detailed script you can use to run via Supervisor. It will send telegrams to one or multiple instances of DSMR-reader.
+
+
+.. note::
+
+    You will still require the ``dsmr`` user and VirtualEnv, :doc:`as discussed in the install guide<installation>` in **chapters 3 and 6**!
+
+**VirtualEnv**::
+
+    sudo su - dsmr
+    pip install pyserial==3.2.1
+    pip install requests==2.12.4
+
+
+.. note::
+
+    The serial connection in this example is based on ``DSMR v4``.
+
+.. warning::
+
+    Don't forget to insert your own configuration below in ``API_SERVERS``.
+
+Client file in ``/home/dsmr/dsmr_datalogger_api_client.py``::
+
+    from time import sleep
+
+    from serial.serialutil import SerialException
+    import requests
+    import serial
+    
+    
+    API_SERVERS = (
+        ('http://HOST-OR-IP-ONE/api/v1/datalogger/dsmrreading', 'APIKEY-BLABLABLA-ABCDEFGHI'),
+    ###    ('http://HOST-OR-IP-TWO/api/v1/datalogger/dsmrreading', 'APIKEY-BLABLABLA-JKLMNOPQR'),
+    )
+    
+    
+    def main():
+        print ('Starting...')
+    
+        while True:
+            telegram = read_telegram()
+            print('Read telegram', telegram)
+    
+            for current_server in API_SERVERS:
+                api_url, api_key = current_server
+                send_telegram(telegram, api_url, api_key)
+                print('Sent telegram to:', api_url)
+    
+            sleep(1)
+    
+    
+    def read_telegram():
+        """ Reads the serial port until we can create a reading point. """
+        serial_handle = serial.Serial()
+        serial_handle.port = '/dev/ttyUSB0'
+        serial_handle.baudrate = 115200
+        serial_handle.bytesize = serial.EIGHTBITS
+        serial_handle.parity = serial.PARITY_NONE
+        serial_handle.stopbits = serial.STOPBITS_ONE
+        serial_handle.xonxoff = 1
+        serial_handle.rtscts = 0
+        serial_handle.timeout = 20
+    
+        # This might fail, but nothing we can do so just let it crash.
+        serial_handle.open()
+    
+        telegram_start_seen = False
+        telegram = ''
+    
+        # Just keep fetching data until we got what we were looking for.
+        while True:
+            try:
+                data = serial_handle.readline()
+            except SerialException as error:
+                # Something else and unexpected failed.
+                print('Serial connection failed:', error)
+                return
+    
+            try:
+                # Make sure weird characters are converted properly.
+                data = str(data, 'utf-8')
+            except TypeError:
+                pass
+    
+            # This guarantees we will only parse complete telegrams. (issue #74)
+            if data.startswith('/'):
+                telegram_start_seen = True
+    
+            # Delay any logging until we've seen the start of a telegram.
+            if telegram_start_seen:
+                telegram += data
+    
+            # Telegrams ends with '!' AND we saw the start. We should have a complete telegram now.
+            if data.startswith('!') and telegram_start_seen:
+                serial_handle.close()
+                return telegram
+    
+    
+    def send_telegram(telegram, api_url, api_key):
+        # Register telegram by simply sending it to the application with a POST request.
+        response = requests.post(
+            api_url,
+            headers={'X-AUTHKEY': api_key},
+            data={'telegram': telegram},
+        )
+    
+        # You will receive a status 200 when successful.
+        if response.status_code != 200:
+            # Or you will find the error (hint) in the response body on failure.
+            print('[!] Error: {}'.format(response.text))
+    
+    
+    if __name__ == '__main__':
+        main()
+
+
+Supervisor config in ``/etc/supervisor/conf.d/dsmr-client.conf``::
+
+    [program:dsmr_client_datalogger]
+    command=/usr/bin/nice -n 5 /home/dsmr/.virtualenvs/dsmrclient/bin/python3 -u /home/dsmr/dsmr_datalogger_api_client.py
+    pidfile=/var/tmp/dsmrreader--%(program_name)s.pid
+    user=dsmr
+    group=dsmr
+    autostart=true
+    autorestart=true
+    startsecs=1
+    startretries=100
+    stopwaitsecs=20
+    stdout_logfile=/var/log/supervisor/%(program_name)s.log
+    stdout_logfile_maxbytes=10MB
+    stdout_logfile_backups=3
+
+
+**Supervisor**::
+
+    sudo supervisorctl reread
+    sudo supervisorctl update 

docs/application.rst

@@ -47,7 +47,10 @@ Data preservation & backups
 
 - The SD card is by far **the weakest link** of this setup and **will** fail you some day.
 
-- The application will, by default, create a backup every night. However, as the data is still stored **locally** on your 'vulnerable' SD card, you must export it off your RaspberryPi as well. 
+- The application will, by default, create a database backup every night. 
+
+ - You can find them in the ``backups`` folder of the application. They either have a ``.sql`` or ``.gz`` extension, depending on whether compression is enabled in the backup configuration.
+ - However, as the data is still stored **locally** on your 'vulnerable' SD card, you must export it off your RaspberryPi as well. 
 
 - There is an builtin option to have backups synced to your **Dropbox**, *without exposing your Dropbox account and your private files in it*. 
 

docs/changelog.rst

@@ -17,6 +17,20 @@ Please make sure you have a fresh **database backup** before upgrading! Upgradin
     - `About upgrading <http://dsmr-reader.readthedocs.io/en/latest/application.html#application-updates-bug-fixes-new-features>`_.
 
 
+
+v1.5.2 - 2017-01-09
+^^^^^^^^^^^^^^^^^^^
+
+**Tickets resolved in this release:**
+
+- Automatic refresh of dashboard charts (`#210 <https://github.com/dennissiemensma/dsmr-reader/issues/210>`_).
+- Mindergas.nl API: Tijdstip van verzending willekeurig maken (`#204 <https://github.com/dennissiemensma/dsmr-reader/issues/204>`_).
+- Extend API docs with additional example (`#185 <https://github.com/dennissiemensma/dsmr-reader/issues/185>`_).
+- Docs: How to restore backup (`#190 <https://github.com/dennissiemensma/dsmr-reader/issues/190>`_).
+- Log errors occured to file (`#181 <https://github.com/dennissiemensma/dsmr-reader/issues/181>`_).
+
+
+
 v1.5.1 - 2017-01-04
 ^^^^^^^^^^^^^^^^^^^
 
@@ -30,6 +44,7 @@ v1.5.1 - 2017-01-04
 - Fix for issues `#200 <https://github.com/dennissiemensma/dsmr-reader/issues/200>`_ & `#217 <https://github.com/dennissiemensma/dsmr-reader/issues/217>`_, which is caused by omitting the switch to the VirtualEnv. This was not documented well enough in early versions of this project, causing failed upgrades. 
 
 
+
 v1.5.0 - 2017-01-01
 ^^^^^^^^^^^^^^^^^^^
 

docs/faq.rst

@@ -135,8 +135,8 @@ This application displays separate tariffs by default, but supports merging them
 Just make sure that you apply the **same price to both electricity 1 and 2** and enable the option ``Merge electricity tariffs`` in the frontend configuration.
 
 
-I want to see my electricity phases as well
--------------------------------------------
+I want to see the load of each electricity phase as well
+---------------------------------------------------------
 Since ``DSMR-reader v1.5`` it's possible to track your ``P+`` (consumption) phases as well. You will need to enable this in the ``Datalogger configuration``.
 There is a setting called ``Track electricity phases``. When active, this will log the current usage of those phases and plot these on the Dashboard page.
 
@@ -171,6 +171,58 @@ The gas meter positions are only be updated once per hour (for DSMR v4).
 The Status page will give you insight in this as well.
 
 
+How do I restore a database backup?
+-----------------------------------
+
+.. warning::
+
+    Restoring a backup will replace any existing data stored in the database and is irreversible! 
+
+.. note::
+
+    Do you need a complete reinstall of DSMR-reader as well? 
+    Then please :doc:`follow the install guide<installation>` and restore the database backup **using the notes at the end of chapter 1**. 
+
+Only want to restore the database?
+
+- This asumes you are still running the same application version as the backup was created in.
+
+- Stop the application first with ``sudo supervisorctl stop all``. This will disconnect it from the database as well.
+
+For **PostgreSQL** restores::
+
+    sudo sudo -u postgres dropdb dsmrreader
+    sudo sudo -u postgres createdb -O dsmrreader dsmrreader
+    
+    # Either restore an uncompressed (.sql) backup:
+    sudo sudo -u postgres psql dsmrreader -f <PATH-TO-POSTGRESQL-BACKUP.sql>
+    
+    # OR
+    
+    # Restore a compressed (.gz) backup with:
+    zcat <PATH-TO-POSTGRESQL-BACKUP.sql.gz> | sudo sudo -u postgres psql dsmrreader
+
+
+For **MySQL** restores::
+
+    sudo mysqladmin create dsmrreader
+    sudo mysqladmin drop dsmrreader
+    
+    # Either restore an uncompressed (.sql) backup:
+    cat <PATH-TO-MYSQL-BACKUP.sql.gz> | sudo mysql -D dsmrreader --defaults-file=/etc/mysql/debian.cnf
+    
+    # OR
+    
+    # Restore a compressed (.gz) backup with:
+    zcat <PATH-TO-MYSQL-BACKUP.sql.gz> | sudo mysql -D dsmrreader --defaults-file=/etc/mysql/debian.cnf
+
+
+- Start the application again with ``sudo supervisorctl start all``.
+
+.. note::
+
+    In case the version differs, you can try forcing a deployment reload by: ``sudo su - dsmr`` and then executing ``./post-deploy.sh``.
+
 
 Feature/bug report
 ------------------

docs/installation.rst

@@ -87,6 +87,18 @@ Does Postgres not start due to locales? Try: ``dpkg-reconfigure locales``.  Stil
 
     sudo sudo -u postgres psql -c "alter user dsmrreader with password 'dsmrreader';"
 
+.. note::
+
+    **Optional**: Do you need to restore a **PostgreSQL** database backup as well?
+
+    Restore an uncompressed (``.sql``) backup with::
+    
+        sudo sudo -u postgres psql dsmrreader -f <PATH-TO-POSTGRESQL-BACKUP.sql>
+
+    Or restore a compressed (``.gz``) backup with::
+    
+        zcat <PATH-TO-POSTGRESQL-BACKUP.sql.gz> | sudo sudo -u postgres psql dsmrreader
+
 
 (Option B.) MySQL/MariaDB
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -112,6 +124,18 @@ Install MariaDB. You can also choose to install the closed source MySQL, as they
 
     sudo mysqladmin reload --defaults-file=/etc/mysql/debian.cnf
 
+.. note::
+
+    **Optional**: Do you need to restore a **MySQL** database backup as well?
+
+    Restore an uncompressed (``.sql``) backup with::
+    
+        cat <PATH-TO-MYSQL-BACKUP.sql.gz> | sudo mysql -D dsmrreader --defaults-file=/etc/mysql/debian.cnf
+
+    Or restore a compressed (``.gz``) backup with::
+    
+        zcat <PATH-TO-MYSQL-BACKUP.sql.gz> | sudo mysql -D dsmrreader --defaults-file=/etc/mysql/debian.cnf
+
 
 2. Dependencies
 ---------------