Persistent Storage

.github/workflows/codeql-analysis.yml

@@ -29,7 +29,7 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL

.github/workflows/pkgbuild.yml

@@ -49,7 +49,7 @@ jobs:
     steps:
 
     - name: Acquire sources
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: Build package
       run: |

.github/workflows/tests.yml

@@ -75,7 +75,7 @@ jobs:
     steps:
 
     - name: Acquire sources
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: Install prerequisites (Linux)
       if: runner.os == 'Linux'
@@ -137,7 +137,7 @@ jobs:
         coverage report
 
     - name: Upload coverage data
-      uses: codecov/codecov-action@v3
+      uses: codecov/codecov-action@v4
       with:
         files: ./coverage.xml
         fail_ci_if_error: false

README.md

@@ -534,9 +534,121 @@ aobj.add('foobar://')
 # Send our notification out through our foobar://
 aobj.notify("test")
 ```
-
 You can read more about creating your own custom notifications and/or hooks [here](https://github.com/caronc/apprise/wiki/decorator_notify).
 
+# Persistent Storage
+
+Persistent storage allows Apprise to cache re-occurring actions optionaly to disk. This can greatly reduce the overhead used to send a notification.
+
+There are 3 options Apprise can operate using this:
+1. `AUTO`:  Flush any gathered data for persistent storage on demand.  This option is incredibly light weight.  This is the default behavior for all CLI usage. Content can be manually flushed to disk using this option as well should a developer choose to do so.  The CLI uses this option by default and only writes anything accumulated to disk after all of it's notifications have completed.
+1. `FLUSH`: Flushes any gathered data for persistent storage as often as it is acquired.
+1. `MEMORY`: Only store information in memory, never write to disk.  This is the option one would set if they simply wish to disable Persistent Storage entirely.  By default this is the mode used by the API and is at the developers discretion to enable one of the other options.
+
+## CLI Persistent Storage Commands
+Persistent storage is set to `AUTO` mode by default.
+
+Specifying the keyword `storage` will assume that all subseqent calls are related to the storage subsection of Apprise.
+```bash
+# List all of the occupied space used by Apprise's Persistent Storage:
+apprise storage list
+
+# list is the default option, so the following does the same thing:
+apprise storage
+
+# You can prune all of your storage older then 30 days
+# and not accessed for this period like so:
+apprise storage prune
+
+# You can do a hard reset (and wipe all persistent storage) with:
+apprise storage clean
+
+```
+
+You can also filter your results by adding tags and/or URL Identifiers.  When you get a listing (`apprise storage list`), you may see:
+```
+   # example output of 'apprise storage list':
+   1. f7077a65                                             0.00B    unused
+      - matrixs://abcdef:****@synapse.example12.com/%23general?image=no&mode=off&version=3&msgtype...
+      tags: team
+
+   2. 0e873a46                                            81.10B    active
+      - tgram://W...U//?image=False&detect=yes&silent=no&preview=no&content=before&mdv=v1&format=m...
+      tags: personal
+
+   3. abcd123                                             12.00B    stale
+
+```
+The states are:
+ - `unused`: This plugin has not commited anything to disk for reuse/cache purposes
+ - `active`: This plugin has written content to disk.  Or at the very least, it has prepared a persistent storage location it can write into.
+ - `stale`: The system detected a location where a URL may have possibly written to in the past, but there is nothing linking to it using the URLs provided.  It is likely wasting space or is no longer of any use.
+
+You can use this information to filter your results by specifying _URL ID_ values after your command.  For example:
+```bash
+# The below commands continue with the example already identified above
+# the following would match abcd123 (even though just ab was provided)
+# The output would only list the 'stale' entry above
+apprise storage list ab
+
+# knowing our filter is safe, we could remove it
+# the below command would not obstruct our other to URLs and would only
+# remove our stale one:
+apprise storage clean ab
+
+# Entries can be filtered by tag as well:
+apprise storage list --tag=team
+
+# You can match on multiple URL ID's as well:
+# The followin would actually match the URL ID's of 1. and .2 above
+apprise storage list f 0
+```
+
+For more information on persistent storage, [visit here](https://github.com/caronc/apprise/wiki/persistent_storage).
+
+
+## API Persistent Storage Commands
+By default, no persistent storage is set to be in `MEMORY` mode for those building from within the Apprise API.
+It's at the developers discretion to enable it. But should you choose to do so, it's as easy as including the information in the `AppriseAsset()` object prior to the initialization of your `Apprise()` instance.
+
+For example:
+```python
+from apprise import Apprise
+from apprise import AppriseAsset
+from apprise import PersistentStoreMode
+
+# Prepare a location the persistent storage can write to
+# This immediately assumes you wish to write in AUTO mode
+asset = AppriseAsset(storage_path="/path/to/save/data")
+
+# If you want to be more explicit and set more options, then
+# you may do the following
+asset = AppriseAsset(
+    # Set our storage path directory (minimum requirement to enable it)
+    storage_path="/path/to/save/data",
+
+    # Set the mode... the options are:
+    # 1. PersistentStoreMode.MEMORY
+    #       - disable persistent storage from writing to disk
+    # 2. PersistentStoreMode.AUTO
+    #       - write to disk on demand
+    # 3. PersistentStoreMode.FLUSH
+    #       - write to disk always and often
+    storage_mode=PersistentStoreMode.FLUSH
+
+    # the URL IDs are by default 8 characters in length, there is
+    # really no reason to change this.  You can increase/decrease
+    # it's value here.  Must be > 2; default is 8 if not specified
+    storage_idlen=6,
+)
+
+# Now that we've got our asset, we just work with our Apprise object as we
+# normally do
+aobj = Apprise(asset=asset)
+```
+
+For more information on persistent storage, [visit here](https://github.com/caronc/apprise/wiki/persistent_storage).
+
 # Want To Learn More?
 
 If you're interested in reading more about this and other methods on how to customize your own notifications, please check out the following links:
@@ -545,6 +657,7 @@ If you're interested in reading more about this and other methods on how to cust
 * 🔧 [Troubleshooting](https://github.com/caronc/apprise/wiki/Troubleshooting)
 * ⚙️ [Configuration File Help](https://github.com/caronc/apprise/wiki/config)
 * ⚡ [Create Your Own Custom Notifications](https://github.com/caronc/apprise/wiki/decorator_notify)
+* 💾 [Persistent Storage](https://github.com/caronc/apprise/wiki/persistent_storage)
 * 🌎 [Apprise API/Web Interface](https://github.com/caronc/apprise-api)
 * 🎉 [Showcase](https://github.com/caronc/apprise/wiki/showcase)
 

apprise/__init__.py

@@ -48,16 +48,20 @@
 from .common import CONTENT_INCLUDE_MODES
 from .common import ContentLocation
 from .common import CONTENT_LOCATIONS
+from .common import PersistentStoreMode
+from .common import PERSISTENT_STORE_MODES
 
 from .url import URLBase
 from .url import PrivacyMode
 from .plugins.base import NotifyBase
 from .config.base import ConfigBase
 from .attachment.base import AttachBase
+from . import exception
 
 from .apprise import Apprise
 from .locale import AppriseLocale
 from .asset import AppriseAsset
+from .persistent_store import PersistentStore
 from .apprise_config import AppriseConfig
 from .apprise_attachment import AppriseAttachment
 from .manager_attachment import AttachmentManager
@@ -77,13 +81,18 @@
     # Core
     'Apprise', 'AppriseAsset', 'AppriseConfig', 'AppriseAttachment', 'URLBase',
     'NotifyBase', 'ConfigBase', 'AttachBase', 'AppriseLocale',
+    'PersistentStore',
+
+    # Exceptions
+    'exception',
 
     # Reference
     'NotifyType', 'NotifyImageSize', 'NotifyFormat', 'OverflowMode',
     'NOTIFY_TYPES', 'NOTIFY_IMAGE_SIZES', 'NOTIFY_FORMATS', 'OVERFLOW_MODES',
     'ConfigFormat', 'CONFIG_FORMATS',
     'ContentIncludeMode', 'CONTENT_INCLUDE_MODES',
     'ContentLocation', 'CONTENT_LOCATIONS',
+    'PersistentStoreMode', 'PERSISTENT_STORE_MODES',
     'PrivacyMode',
 
     # Managers

apprise/asset.py

@@ -33,6 +33,7 @@
 from os.path import isfile
 from os.path import abspath
 from .common import NotifyType
+from .common import PersistentStoreMode
 from .manager_plugins import NotificationManager
 
 
@@ -157,6 +158,22 @@ class AppriseAsset:
     # By default, no paths are scanned.
     __plugin_paths = []
 
+    # Optionally set the location of the persistent storage
+    # By default there is no path and thus persistent storage is not used
+    __storage_path = None
+
+    # Optionally define the default salt to apply to all persistent storage
+    # namespace generation (unless over-ridden)
+    __storage_salt = b''
+
+    # Optionally define the namespace length of the directories created by
+    # the storage. If this is set to zero, then the length is pre-determined
+    # by the generator (sha1, md5, sha256, etc)
+    __storage_idlen = 8
+
+    # Set storage to auto
+    __storage_mode = PersistentStoreMode.AUTO
+
     # All internal/system flags are prefixed with an underscore (_)
     # These can only be initialized using Python libraries and are not picked
     # up from (yaml) configuration files (if set)
@@ -171,7 +188,9 @@ class AppriseAsset:
     # A unique identifer we can use to associate our calling source
     _uid = str(uuid4())
 
-    def __init__(self, plugin_paths=None, **kwargs):
+    def __init__(self, plugin_paths=None, storage_path=None,
+                 storage_mode=None, storage_salt=None,
+                 storage_idlen=None, **kwargs):
         """
         Asset Initialization
 
@@ -187,8 +206,49 @@ def __init__(self, plugin_paths=None, **kwargs):
 
         if plugin_paths:
             # Load any decorated modules if defined
+            self.__plugin_paths = plugin_paths
             N_MGR.module_detection(plugin_paths)
 
+        if storage_path:
+            # Define our persistent storage path
+            self.__storage_path = storage_path
+
+        if storage_mode:
+            # Define how our persistent storage behaves
+            self.__storage_mode = storage_mode
+
+        if isinstance(storage_idlen, int):
+            # Define the number of characters utilized from our namespace lengh
+            if storage_idlen < 0:
+                # Unsupported type
+                raise ValueError(
+                    'AppriseAsset storage_idlen(): Value must '
+                    'be an integer and > 0')
+
+            # Store value
+            self.__storage_idlen = storage_idlen
+
+        if storage_salt is not None:
+            # Define the number of characters utilized from our namespace lengh
+
+            if isinstance(storage_salt, bytes):
+                self.__storage_salt = storage_salt
+
+            elif isinstance(storage_salt, str):
+                try:
+                    self.__storage_salt = storage_salt.encode(self.encoding)
+
+                except UnicodeEncodeError:
+                    # Bad data; don't pass it along
+                    raise ValueError(
+                        'AppriseAsset namespace_salt(): '
+                        'Value provided could not be encoded')
+
+            else:  # Unsupported
+                raise ValueError(
+                    'AppriseAsset namespace_salt(): Value provided must be '
+                    'string or bytes object')
+
     def color(self, notify_type, color_type=None):
         """
         Returns an HTML mapped color based on passed in notify type
@@ -356,3 +416,40 @@ def hex_to_int(value):
 
         """
         return int(value.lstrip('#'), 16)
+
+    @property
+    def plugin_paths(self):
+        """
+        Return the plugin paths defined
+        """
+        return self.__plugin_paths
+
+    @property
+    def storage_path(self):
+        """
+        Return the persistent storage path defined
+        """
+        return self.__storage_path
+
+    @property
+    def storage_mode(self):
+        """
+        Return the persistent storage mode defined
+        """
+
+        return self.__storage_mode
+
+    @property
+    def storage_salt(self):
+        """
+        Return the provided namespace salt; this is always of type bytes
+        """
+        return self.__storage_salt
+
+    @property
+    def storage_idlen(self):
+        """
+        Return the persistent storage id length
+        """
+
+        return self.__storage_idlen

apprise/attachment/file.py

@@ -29,6 +29,7 @@
 import re
 import os
 from .base import AttachBase
+from ..utils import path_decode
 from ..common import ContentLocation
 from ..locale import gettext_lazy as _
 
@@ -57,7 +58,10 @@ def __init__(self, path, **kwargs):
 
         # Store path but mark it dirty since we have not performed any
         # verification at this point.
-        self.dirty_path = os.path.expanduser(path)
+        self.dirty_path = path_decode(path)
+
+        # Track our file as it was saved
+        self.__original_path = os.path.normpath(path)
         return
 
     def url(self, privacy=False, *args, **kwargs):
@@ -77,7 +81,7 @@ def url(self, privacy=False, *args, **kwargs):
             params['name'] = self._name
 
         return 'file://{path}{params}'.format(
-            path=self.quote(self.dirty_path),
+            path=self.quote(self.__original_path),
             params='?{}'.format(self.urlencode(params, safe='/'))
             if params else '',
         )