Merge branch 'release-0.5.0'

.pyup.yml

@@ -1,4 +1,4 @@
 update: all
 branch: develop
-schedule: "every week"
+schedule: "every day"
 label_prs: update

.travis.yml

@@ -33,5 +33,6 @@ deploy:
   user: UCCSER
   password:
     secure: 1ImbakdpIPHmO89yAFH5ZApwhCRApI3BtiGjx7sflYWJcMzFydaaS6zo+tU3lY7QHtpWE6HQMjOLweupLgu1od9RJ9BX+4mJ37FHelqjaEGAMVR/e70H4jL/Mn8ImiTwJ14wCnjek/kYrrrMQqDqCLuNvQRb/Q8ipbw+fEBtRlmRgDx2Looik4ehk19iPybpGfb+7mq8rVPk3ZEl7oZp4cMKckqX3IMXX3yNG0tka6M1Q2a41W4N6EPoBgVwcAY2FwzwXEQC+KDjwxVkPviZ+pWbbV/is9spX8SV/BCjnsJWBCNu0x4GOk3atr+R7ZzIs7e7edehy0QG8gzGvhb0qaOyPVWkIvrTJwEEVTfcNGl8dYZar0EuM7GBpP5ttx4IShY/0XfuT3XXl5C50kxuIQocJHjNn0xu6E3x4LheUuDPp3S92zQxNHcOGS9v2syY4Kb3Bxvjlk0HhrPpZ4wo0U93TaB9lAahQsulYS/gbynzYjpphIHLSslK/imQEZNoSz9roKK3Q/JLSQsc2jGdZM93IHWwoB0+uqjDyBLsYRmPaOKOAlew0bGzP4uX3ovMsCexwodTISFjgb/2JOaUkz289lPQ4fK/gz7uVtIDYkHj9oAz+GeT9PlnRFb8U5fiRSJf5OWFt/D149XvXc4c5OES7sgbfK5jvsTfCsVa4L0=
+  distributions: "sdist bdist_wheel"
   on:
     branch: master

README.rst

@@ -6,6 +6,40 @@ Verto is an extension of the Python Markdown package, which allows
 authors to include complex HTML elements with simple text tags in their
 Markdown files.
 
+Basic Usage
+-----------
+
+Verto allows for an author to quickly include images and content and display
+them in a panel (similar to a Bootstrap Collapsible Panel) with the following
+markdown:
+
+.. code-block:: None
+
+  # Example Header
+
+  Example Paragraph
+
+  {panel type="example" title="Example Panel"}
+
+  {image file-path="http://placehold.it/350x150" caption="Example Image"}
+
+  {panel end}
+
+While Verto has many configuration options it can be used immediately
+with little code. For example, if the previous markdown is saved in the file
+called ``example.md`` then the following would convert that file and print the
+output to stdout:
+
+.. code-block:: python
+
+  from verto import Verto
+
+  text = open('example.md', 'r')
+  converter = Verto()
+  result = converter.convert(text)
+  
+  print(result.html_string)
+
 Documentation
 -------------
 

docs/source/changelog.rst

@@ -1,6 +1,21 @@
 Changelog
 #######################################
 
+0.5.0
+=======================================
+
+Fixes:
+
+  - A new more descriptive error when an argument is given and not readable.
+  - Custom HTML string parsing has been implemented, allowing for correct parsing of HTML and XHTML in templates.
+
+Documentation:
+
+  - Basic example in README.
+  - New contributing documentation.
+  - Fixed reference to incorrect file in the image processor documentation.
+  - Added new documentation for implicit processors.
+
 0.4.1
 =======================================
 

docs/source/contributing.rst

@@ -75,12 +75,12 @@ Below is a basic overview of the project structure:
 
   ├── docs/
   ├── verto/
+  |   ├── errors/
   │   ├── html-templates/
   │   ├── VertoExtension.py
   │   ├── Verto.py
   │   ├── processor-info.json
   │   ├── processors/
-  │   │   └── errors/
   │   ├── tests/
   │   └── utils/
   ├── requirements.txt
@@ -90,7 +90,6 @@ The items of interest are:
 
 - ``Verto()`` - The convertor object itself. This is what a user will use to create a Verto converter, and what is used to define a custom processor list, custom html templates and custom Markdown Extensions to use.
 
-
 - ``VertoResult()`` (found in ``Verto.py``) - The object returned by ``Verto()`` containing:
 
   - Converted html string
@@ -99,17 +98,19 @@ The items of interest are:
   - Heading tree
   - Required glossary terms
 
-
 - ``VertoExtension()`` - This is the main class of the project, and inherits the ``Extension`` class from Markdown. It loads all of the processor information, loads the template files and clears and populates the attributes to be returned by the ``VertoResult`` object.
 
-- ``Processors/`` - There is a different processor for each tag. A processor uses it's corresponding description loaded from ``processor-info.json`` to find matches in the text, and uses the given arguments in the matched tag to populate and output it's html template.
+- ``processor-info.json`` - Every processor is listed in this file, and will at least contain a class determining whether it is custom or generic, where custom processors will have a pattern to match it's corresponding tag. Most will also define required and optional parameters, these correspond to arguments in the tag's html template.
+
+- ``processors/`` - There is a different processor for each tag. A processor uses it's corresponding description loaded from ``processor-info.json`` to find matches in the text, and uses the given arguments in the matched tag to populate and output it's html template.
 
 - ``html-templates/`` - The html templates (using the Jinja2 template engine) with variable arguments to be populated by processors.
 
-- ``processor-info.json`` - Every processor is listed in this file, and will at least contain a class determining whether it is custom or generic, where custom processors will have a pattern to match it's corresponding tag. Most will also define required and optional parameters, these correspond to arguments in the tag's html template.
+- ``errors/`` - Contains all the errors exposed by the Verto module. Where an Error is an exception that is caused by user input. New errors should be created in here inheriting from the base ``Error`` class.
 
-- ``tests/`` - explained in the Test Suite section further down the page.
+- ``utils/`` - Contains classes and methods not necessarily unique to Verto that are useful in any sub-module. This includes slugify handlers, html parsers and serialisers, and other utilities. The utilities should be used over external libraries as they are purposely built because of: compatibility reasons, licensing restrictions, and/or unavailability of require features.
 
+- ``tests/`` - explained in the Test Suite section further down the page.
 
 It is important to note that Verto is not just a Markdown Extension, it is a wrapper for Python Markdown. ``VertoExtension`` **is** an extension for Python Markdown. We have created a wrapper because we wanted to not only convert text, but also extract information from the text as it was being converted (recall ``VertoResult()`` listed above).
 
@@ -282,10 +283,12 @@ Each processor should try to be as independent of every other processor as possi
 
 The logic for each processor belongs in the ``processors/`` directory, and there are several other places where processors details need to be listed. These are:
 
-- The processor's relevant information (regex pattern, required parameters etc) should be included in ``processor-info.json``
-- If it should be a default processor, it should be added to the frozenset of ``DEFAULT_PROCESSORS`` in ``Verto.py``
-- The relevant list in ``extendMarkdown()`` in ``VertoExtension.py`` (see `OrderedDict in the Markdown API docs`_ for manipulating processor order)
-- The processor's template should be added to ``html-templates`` using the Jinja2 template engine syntax for variable parameters
+- The processor's relevant information (regex pattern, required parameters etc) should be included in ``processor-info.json``.
+- If it should be a default processor, it should be added to the frozenset of ``DEFAULT_PROCESSORS`` in ``Verto.py``.
+- The relevant list in ``extendMarkdown()`` in ``VertoExtension.py`` (see `OrderedDict in the Markdown API docs`_ for manipulating processor order).
+- The processor's template should be added to ``html-templates`` using the Jinja2 template engine syntax for variable parameters.
+- Any errors should have appropriate classes in the ``errors\`` directory, they should be well described by their class name such that for an expert knows immediately what to do to resolve the issue, otherwise a message should be used to describe the exact causation of the error for a novice.
+
 
 .. _the-test-suite:
 

docs/source/processors/beautify.rst

@@ -0,0 +1,31 @@
+.. _beautify:
+
+Beautify
+#######################################
+
+**Processor name:** ``beautify``
+
+The ``beautify`` processor is a post-processor that tidies and prettifies the HTML to give consistent and predictable output. The processor works by applying the prettify function from the ``beautifulsoup4`` library just before the final output, this means HTML elements will be separated onto individual lines where children are indented by one space. For example given the following document:
+
+.. literalinclude:: ../../../verto/tests/assets/beautify/doc_example_basic_usage.md
+  :language: none
+
+Verto will prettify it into:
+
+.. literalinclude:: ../../../verto/tests/assets/beautify/doc_example_basic_usage_expected.html
+  :language: html
+
+Special Case(s)
+***************************************
+
+For example given the following Markdown:
+
+.. literalinclude:: ../../../verto/tests/assets/beautify/example_inline_code.md
+  :language: none
+
+Verto with ``beautify`` enabled will produce the following html:
+
+.. literalinclude:: ../../../verto/tests/assets/beautify/example_inline_code_expected.html
+  :language: html
+
+Where the ``code`` tag and its contents are unchanged to preserve formatting, this is especially important for whitespace dependent languages.

docs/source/processors/conditional.rst

@@ -3,15 +3,15 @@ Conditional
 
 **Processor name:** ``conditional``
 
+.. danger::
+
+  Conditional blocks require an understanding of Python logical operators and expressions to function properly. The use of this tag requires co-ordination between authors and developers, as the variables used in the condition are expected when the result is rendered in a template engine.
+
 You can include an conditional using the following text tag:
 
 .. literalinclude:: ../../../verto/tests/assets/conditional/doc_example_basic_usage.md
    :language: none
 
-.. note::
-
-  Conditional blocks require an understanding of Python logical operators and expressions to function properly. The use of this tag requires co-ordination between authors and developers, as the variables used in the condition are expected when the result is rendered in a template engine.
-
 Tag Parameters
 ***************************************
 

docs/source/processors/image.rst

@@ -55,7 +55,7 @@ When overriding the HTML for images, the following Jinja2 placeholders are avail
 - ``{{ caption_link }}`` - The URL for the caption link .
 - ``{{ source_link }}`` - The URL for the source .
 
-If the ``file_path`` provided is an relative link, the link is passed through the ``relative-image-link.html`` template.
+If the ``file_path`` provided is a relative link, the link is passed through the ``relative-file-link.html`` template.
 The default HTML for relative images is:
 
 .. literalinclude:: ../../../verto/html-templates/relative-file-link.html

docs/source/processors/index.rst

@@ -29,3 +29,17 @@ The following pages covers how to use the available processors within Markdown t
     scratch
     table-of-contents
     video
+
+Implicit Processors
+#######################################
+
+The following pages cover processors that do not require explicit use when authoring Markdown:
+
+.. toctree::
+    :maxdepth: 1
+
+    beautify
+    jinja
+    remove
+    scratch-compatibility
+    style

docs/source/processors/jinja.rst

@@ -0,0 +1,18 @@
+.. _jinja:
+
+Jinja
+#######################################
+
+**Processor name:** ``jinja``
+
+The ``jinja`` processor is a post-processor that is used to undo HTML escaping on Jinja/Django statements (i.e. ``{% ... %}``) that may be present in the document for further processing of the document after conversion. This processor does not do any sanitizing of the Jinja/Django statements and therefore should not be used on untrusted input without sanitation before or after the Verto conversion. This processor  should be used with the :doc:`conditional` as the default HTML-template produces Jinja statements.
+
+For example the following document with an if statement:
+
+.. literalinclude:: ../../../verto/tests/assets/jinja/doc_example_basic_usage.md
+   :language: html+jinja
+
+Verto will unescape the Jinja/Django statement and produce the following output:
+
+.. literalinclude:: ../../../verto/tests/assets/jinja/doc_example_basic_usage_expected.html
+  :language: html+jinja

docs/source/processors/remove.rst

@@ -0,0 +1,27 @@
+.. _remove:
+
+Remove
+#######################################
+
+**Processor name:** ``remove``
+
+The ``remove`` processor is a post-processor that searches the document for remove HTML-elements (i.e. ``<remove>...</remove>``) and removes them from the document leaving the content unchanged. This is useful when creating HTML-templates as they can be used to add multiple siblings to a parent element that are not valid HTML, allowing the document to be parsed as a valid HTML-document up until their removal.
+
+.. note::
+
+    The ``remove`` processor does not remove the content between the remove element tags, but instead only removes the tag itself.
+
+For example the :doc:`conditional` processors default HTML template, as follows, does not produce valid HTML and so is placed within a remove element so that Verto can add it to the element tree.
+
+.. literalinclude:: ../../../verto/html-templates/conditional.html
+   :language: html+jinja
+
+Therefore a Markdown document like:
+
+.. literalinclude:: ../../../verto/tests/assets/remove/doc_example_basic_usage.md
+   :language: html+jinja
+
+When parsed with Verto will produce the output:
+
+.. literalinclude:: ../../../verto/tests/assets/remove/doc_example_basic_usage_expected.html
+  :language: html+jinja

docs/source/processors/scratch-compatibility.rst

@@ -0,0 +1,26 @@
+.. _scratch-compatibility:
+
+Scratch Compatibility
+#######################################
+
+**Processor name:** ``scratch-compatibility``
+
+The ``scratch-compatibility`` processor is a pre-processor that is enabled by the :doc:`scratch` processor when the ``codehilite`` and ``fenced_code`` extensions are enabled.
+
+When both ``codehilite`` and ``fenced_code`` extensions are enabled the ``fenced_code`` extension modifies the fenced code-blocks by using methods from the ``codehilite`` extension before stashing them to be place in later in the document. The ``scratch-compatibility`` processor is therefore needed to stash the fenced code-blocks before ``fenced_code`` so that they can be processed properly by the :doc:`scratch` processor later.
+
+.. note::
+
+    We consider the ``codehilite`` and ``fenced_code`` extensions a bad way of writing extensions as the output of one dramatically changes depending on if the other is active.
+
+    We believe that an extension like these should produce predictable output and handle compatibility through inputs.
+
+For example if the following Markdown document is processed using both the ``codehilite`` and ``fenced_code`` extensions
+
+.. literalinclude:: ../../../verto/tests/assets/scratch/example_multiple_codeblocks_2.md
+   :language: none
+
+Verto will produce the following output (which is the same as the ``scratch`` processor would expect):
+
+.. literalinclude:: ../../../verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html
+  :language: html

docs/source/processors/scratch.rst

@@ -1,13 +1,17 @@
 Scratch
 #######################################
 
+**Processor name:** ``scratch``
+
+.. danger::
+
+  Scratch blocks require an understanding of the Scratch programming language and how Verto is integrated with other systems. The use of this processor requires co-ordination between authors and developers to achieve the desired functionality.
+
 .. note::
 
     The following examples assume usage of the fenced code extension, by having
     ``markdown.extensions.fenced_code`` in the list of extensions given to Verto.
 
-**Processor name:** ``scratch``
-
 You can include an image of Scratch blocks using
 `Scratch Block Plugin notation`_ using the following notation:
 

docs/source/processors/style.rst

@@ -0,0 +1,45 @@
+.. _style:
+
+Style
+#######################################
+
+**Processor name:** ``style``
+
+The ``style`` processor is a pre-processor that checks that the input Markdown to enforce style rules. These rules include:
+
+  - Processor tags have empty lines before and after.
+  - Processor tags do not share a line with other text.
+
+An example of a valid document follows:
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_valid.md
+   :language: none
+
+Error Example(s)
+**************************************
+
+.. note::
+
+  The examples covered in this section are invalid and will raise errors.
+
+The following examples raise errors because the processor tags do not have empty lines before and after.
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace.md
+   :language: none
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_1.md
+  :language: none
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_2.md
+   :language: none
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_3.md
+  :language: none
+
+The following examples raise errors because the processor tags share a line with other text.
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_solitary.md
+  :language: none
+
+.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_solitary_1.md
+  :language: none

docs/source/processors/table-of-contents.rst

@@ -3,6 +3,10 @@ Table of Contents
 
 **Processor name:** ``table-of-contents``
 
+.. danger::
+
+  The table-of-contents tag currently requires integration with other systems, use of this tag should accompany supervision of a developer.
+
 You can create a placeholder for a web framework (for example: Django) to
 insert a table of contents by using the following tag:
 

requirements.txt

@@ -2,8 +2,8 @@
 markdown==2.6.8
 beautifulsoup4==4.5.3
 Jinja2==2.9.6
-python-slugify==1.2.2
-setuptools
+python-slugify==1.2.3
+setuptools==35.0.0
 
 # Required dependencies for building documentation
 sphinx==1.5.5

verto/__init__.py

@@ -1,4 +1,4 @@
 # flake8: noqa
 from .Verto import Verto
 
-__version__ = '0.4.1'
+__version__ = '0.5.0'