diff --git a/.pyup.yml b/.pyup.yml index 839c262c..01bd557a 100644 --- a/.pyup.yml +++ b/.pyup.yml @@ -1,4 +1,4 @@ update: all branch: develop -schedule: "every week" +schedule: "every day" label_prs: update diff --git a/.travis.yml b/.travis.yml index e4017175..aefbe3ff 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,5 +33,6 @@ deploy: user: UCCSER password: secure: 1ImbakdpIPHmO89yAFH5ZApwhCRApI3BtiGjx7sflYWJcMzFydaaS6zo+tU3lY7QHtpWE6HQMjOLweupLgu1od9RJ9BX+4mJ37FHelqjaEGAMVR/e70H4jL/Mn8ImiTwJ14wCnjek/kYrrrMQqDqCLuNvQRb/Q8ipbw+fEBtRlmRgDx2Looik4ehk19iPybpGfb+7mq8rVPk3ZEl7oZp4cMKckqX3IMXX3yNG0tka6M1Q2a41W4N6EPoBgVwcAY2FwzwXEQC+KDjwxVkPviZ+pWbbV/is9spX8SV/BCjnsJWBCNu0x4GOk3atr+R7ZzIs7e7edehy0QG8gzGvhb0qaOyPVWkIvrTJwEEVTfcNGl8dYZar0EuM7GBpP5ttx4IShY/0XfuT3XXl5C50kxuIQocJHjNn0xu6E3x4LheUuDPp3S92zQxNHcOGS9v2syY4Kb3Bxvjlk0HhrPpZ4wo0U93TaB9lAahQsulYS/gbynzYjpphIHLSslK/imQEZNoSz9roKK3Q/JLSQsc2jGdZM93IHWwoB0+uqjDyBLsYRmPaOKOAlew0bGzP4uX3ovMsCexwodTISFjgb/2JOaUkz289lPQ4fK/gz7uVtIDYkHj9oAz+GeT9PlnRFb8U5fiRSJf5OWFt/D149XvXc4c5OES7sgbfK5jvsTfCsVa4L0= + distributions: "sdist bdist_wheel" on: branch: master diff --git a/README.rst b/README.rst index 69e10bbc..c4c9933e 100644 --- a/README.rst +++ b/README.rst @@ -6,6 +6,40 @@ Verto is an extension of the Python Markdown package, which allows authors to include complex HTML elements with simple text tags in their Markdown files. +Basic Usage +----------- + +Verto allows for an author to quickly include images and content and display +them in a panel (similar to a Bootstrap Collapsible Panel) with the following +markdown: + +.. code-block:: None + + # Example Header + + Example Paragraph + + {panel type="example" title="Example Panel"} + + {image file-path="http://placehold.it/350x150" caption="Example Image"} + + {panel end} + +While Verto has many configuration options it can be used immediately +with little code. For example, if the previous markdown is saved in the file +called ``example.md`` then the following would convert that file and print the +output to stdout: + +.. code-block:: python + + from verto import Verto + + text = open('example.md', 'r') + converter = Verto() + result = converter.convert(text) + + print(result.html_string) + Documentation ------------- diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst index e9e6b101..3bf93c0c 100644 --- a/docs/source/changelog.rst +++ b/docs/source/changelog.rst @@ -1,6 +1,21 @@ Changelog ####################################### +0.5.0 +======================================= + +Fixes: + + - A new more descriptive error when an argument is given and not readable. + - Custom HTML string parsing has been implemented, allowing for correct parsing of HTML and XHTML in templates. + +Documentation: + + - Basic example in README. + - New contributing documentation. + - Fixed reference to incorrect file in the image processor documentation. + - Added new documentation for implicit processors. + 0.4.1 ======================================= diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index d7fb63a3..45336790 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -75,12 +75,12 @@ Below is a basic overview of the project structure: ├── docs/ ├── verto/ + | ├── errors/ │   ├── html-templates/ │   ├── VertoExtension.py │   ├── Verto.py │   ├── processor-info.json │   ├── processors/ - │   │   └── errors/ │   ├── tests/ │   └── utils/ ├── requirements.txt @@ -90,7 +90,6 @@ The items of interest are: - ``Verto()`` - The convertor object itself. This is what a user will use to create a Verto converter, and what is used to define a custom processor list, custom html templates and custom Markdown Extensions to use. - - ``VertoResult()`` (found in ``Verto.py``) - The object returned by ``Verto()`` containing: - Converted html string @@ -99,17 +98,19 @@ The items of interest are: - Heading tree - Required glossary terms - - ``VertoExtension()`` - This is the main class of the project, and inherits the ``Extension`` class from Markdown. It loads all of the processor information, loads the template files and clears and populates the attributes to be returned by the ``VertoResult`` object. -- ``Processors/`` - There is a different processor for each tag. A processor uses it's corresponding description loaded from ``processor-info.json`` to find matches in the text, and uses the given arguments in the matched tag to populate and output it's html template. +- ``processor-info.json`` - Every processor is listed in this file, and will at least contain a class determining whether it is custom or generic, where custom processors will have a pattern to match it's corresponding tag. Most will also define required and optional parameters, these correspond to arguments in the tag's html template. + +- ``processors/`` - There is a different processor for each tag. A processor uses it's corresponding description loaded from ``processor-info.json`` to find matches in the text, and uses the given arguments in the matched tag to populate and output it's html template. - ``html-templates/`` - The html templates (using the Jinja2 template engine) with variable arguments to be populated by processors. -- ``processor-info.json`` - Every processor is listed in this file, and will at least contain a class determining whether it is custom or generic, where custom processors will have a pattern to match it's corresponding tag. Most will also define required and optional parameters, these correspond to arguments in the tag's html template. +- ``errors/`` - Contains all the errors exposed by the Verto module. Where an Error is an exception that is caused by user input. New errors should be created in here inheriting from the base ``Error`` class. -- ``tests/`` - explained in the Test Suite section further down the page. +- ``utils/`` - Contains classes and methods not necessarily unique to Verto that are useful in any sub-module. This includes slugify handlers, html parsers and serialisers, and other utilities. The utilities should be used over external libraries as they are purposely built because of: compatibility reasons, licensing restrictions, and/or unavailability of require features. +- ``tests/`` - explained in the Test Suite section further down the page. It is important to note that Verto is not just a Markdown Extension, it is a wrapper for Python Markdown. ``VertoExtension`` **is** an extension for Python Markdown. We have created a wrapper because we wanted to not only convert text, but also extract information from the text as it was being converted (recall ``VertoResult()`` listed above). @@ -282,10 +283,12 @@ Each processor should try to be as independent of every other processor as possi The logic for each processor belongs in the ``processors/`` directory, and there are several other places where processors details need to be listed. These are: -- The processor's relevant information (regex pattern, required parameters etc) should be included in ``processor-info.json`` -- If it should be a default processor, it should be added to the frozenset of ``DEFAULT_PROCESSORS`` in ``Verto.py`` -- The relevant list in ``extendMarkdown()`` in ``VertoExtension.py`` (see `OrderedDict in the Markdown API docs`_ for manipulating processor order) -- The processor's template should be added to ``html-templates`` using the Jinja2 template engine syntax for variable parameters +- The processor's relevant information (regex pattern, required parameters etc) should be included in ``processor-info.json``. +- If it should be a default processor, it should be added to the frozenset of ``DEFAULT_PROCESSORS`` in ``Verto.py``. +- The relevant list in ``extendMarkdown()`` in ``VertoExtension.py`` (see `OrderedDict in the Markdown API docs`_ for manipulating processor order). +- The processor's template should be added to ``html-templates`` using the Jinja2 template engine syntax for variable parameters. +- Any errors should have appropriate classes in the ``errors\`` directory, they should be well described by their class name such that for an expert knows immediately what to do to resolve the issue, otherwise a message should be used to describe the exact causation of the error for a novice. + .. _the-test-suite: diff --git a/docs/source/processors/beautify.rst b/docs/source/processors/beautify.rst new file mode 100644 index 00000000..a848277c --- /dev/null +++ b/docs/source/processors/beautify.rst @@ -0,0 +1,31 @@ +.. _beautify: + +Beautify +####################################### + +**Processor name:** ``beautify`` + +The ``beautify`` processor is a post-processor that tidies and prettifies the HTML to give consistent and predictable output. The processor works by applying the prettify function from the ``beautifulsoup4`` library just before the final output, this means HTML elements will be separated onto individual lines where children are indented by one space. For example given the following document: + +.. literalinclude:: ../../../verto/tests/assets/beautify/doc_example_basic_usage.md + :language: none + +Verto will prettify it into: + +.. literalinclude:: ../../../verto/tests/assets/beautify/doc_example_basic_usage_expected.html + :language: html + +Special Case(s) +*************************************** + +For example given the following Markdown: + +.. literalinclude:: ../../../verto/tests/assets/beautify/example_inline_code.md + :language: none + +Verto with ``beautify`` enabled will produce the following html: + +.. literalinclude:: ../../../verto/tests/assets/beautify/example_inline_code_expected.html + :language: html + +Where the ``code`` tag and its contents are unchanged to preserve formatting, this is especially important for whitespace dependent languages. diff --git a/docs/source/processors/conditional.rst b/docs/source/processors/conditional.rst index d7ac94d1..cf5ae3c9 100644 --- a/docs/source/processors/conditional.rst +++ b/docs/source/processors/conditional.rst @@ -3,15 +3,15 @@ Conditional **Processor name:** ``conditional`` +.. danger:: + + Conditional blocks require an understanding of Python logical operators and expressions to function properly. The use of this tag requires co-ordination between authors and developers, as the variables used in the condition are expected when the result is rendered in a template engine. + You can include an conditional using the following text tag: .. literalinclude:: ../../../verto/tests/assets/conditional/doc_example_basic_usage.md :language: none -.. note:: - - Conditional blocks require an understanding of Python logical operators and expressions to function properly. The use of this tag requires co-ordination between authors and developers, as the variables used in the condition are expected when the result is rendered in a template engine. - Tag Parameters *************************************** diff --git a/docs/source/processors/image.rst b/docs/source/processors/image.rst index 560de5a5..2ac1eac7 100644 --- a/docs/source/processors/image.rst +++ b/docs/source/processors/image.rst @@ -55,7 +55,7 @@ When overriding the HTML for images, the following Jinja2 placeholders are avail - ``{{ caption_link }}`` - The URL for the caption link . - ``{{ source_link }}`` - The URL for the source . -If the ``file_path`` provided is an relative link, the link is passed through the ``relative-image-link.html`` template. +If the ``file_path`` provided is a relative link, the link is passed through the ``relative-file-link.html`` template. The default HTML for relative images is: .. literalinclude:: ../../../verto/html-templates/relative-file-link.html diff --git a/docs/source/processors/index.rst b/docs/source/processors/index.rst index 3c4dc920..ccca2b10 100644 --- a/docs/source/processors/index.rst +++ b/docs/source/processors/index.rst @@ -29,3 +29,17 @@ The following pages covers how to use the available processors within Markdown t scratch table-of-contents video + +Implicit Processors +####################################### + +The following pages cover processors that do not require explicit use when authoring Markdown: + +.. toctree:: + :maxdepth: 1 + + beautify + jinja + remove + scratch-compatibility + style diff --git a/docs/source/processors/jinja.rst b/docs/source/processors/jinja.rst new file mode 100644 index 00000000..bcc1ec3d --- /dev/null +++ b/docs/source/processors/jinja.rst @@ -0,0 +1,18 @@ +.. _jinja: + +Jinja +####################################### + +**Processor name:** ``jinja`` + +The ``jinja`` processor is a post-processor that is used to undo HTML escaping on Jinja/Django statements (i.e. ``{% ... %}``) that may be present in the document for further processing of the document after conversion. This processor does not do any sanitizing of the Jinja/Django statements and therefore should not be used on untrusted input without sanitation before or after the Verto conversion. This processor should be used with the :doc:`conditional` as the default HTML-template produces Jinja statements. + +For example the following document with an if statement: + +.. literalinclude:: ../../../verto/tests/assets/jinja/doc_example_basic_usage.md + :language: html+jinja + +Verto will unescape the Jinja/Django statement and produce the following output: + +.. literalinclude:: ../../../verto/tests/assets/jinja/doc_example_basic_usage_expected.html + :language: html+jinja diff --git a/docs/source/processors/remove.rst b/docs/source/processors/remove.rst new file mode 100644 index 00000000..78f345e9 --- /dev/null +++ b/docs/source/processors/remove.rst @@ -0,0 +1,27 @@ +.. _remove: + +Remove +####################################### + +**Processor name:** ``remove`` + +The ``remove`` processor is a post-processor that searches the document for remove HTML-elements (i.e. ``...``) and removes them from the document leaving the content unchanged. This is useful when creating HTML-templates as they can be used to add multiple siblings to a parent element that are not valid HTML, allowing the document to be parsed as a valid HTML-document up until their removal. + +.. note:: + + The ``remove`` processor does not remove the content between the remove element tags, but instead only removes the tag itself. + +For example the :doc:`conditional` processors default HTML template, as follows, does not produce valid HTML and so is placed within a remove element so that Verto can add it to the element tree. + +.. literalinclude:: ../../../verto/html-templates/conditional.html + :language: html+jinja + +Therefore a Markdown document like: + +.. literalinclude:: ../../../verto/tests/assets/remove/doc_example_basic_usage.md + :language: html+jinja + +When parsed with Verto will produce the output: + +.. literalinclude:: ../../../verto/tests/assets/remove/doc_example_basic_usage_expected.html + :language: html+jinja diff --git a/docs/source/processors/scratch-compatibility.rst b/docs/source/processors/scratch-compatibility.rst new file mode 100644 index 00000000..0071f921 --- /dev/null +++ b/docs/source/processors/scratch-compatibility.rst @@ -0,0 +1,26 @@ +.. _scratch-compatibility: + +Scratch Compatibility +####################################### + +**Processor name:** ``scratch-compatibility`` + +The ``scratch-compatibility`` processor is a pre-processor that is enabled by the :doc:`scratch` processor when the ``codehilite`` and ``fenced_code`` extensions are enabled. + +When both ``codehilite`` and ``fenced_code`` extensions are enabled the ``fenced_code`` extension modifies the fenced code-blocks by using methods from the ``codehilite`` extension before stashing them to be place in later in the document. The ``scratch-compatibility`` processor is therefore needed to stash the fenced code-blocks before ``fenced_code`` so that they can be processed properly by the :doc:`scratch` processor later. + +.. note:: + + We consider the ``codehilite`` and ``fenced_code`` extensions a bad way of writing extensions as the output of one dramatically changes depending on if the other is active. + + We believe that an extension like these should produce predictable output and handle compatibility through inputs. + +For example if the following Markdown document is processed using both the ``codehilite`` and ``fenced_code`` extensions + +.. literalinclude:: ../../../verto/tests/assets/scratch/example_multiple_codeblocks_2.md + :language: none + +Verto will produce the following output (which is the same as the ``scratch`` processor would expect): + +.. literalinclude:: ../../../verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html + :language: html diff --git a/docs/source/processors/scratch.rst b/docs/source/processors/scratch.rst index 687b205c..f71ea6bc 100644 --- a/docs/source/processors/scratch.rst +++ b/docs/source/processors/scratch.rst @@ -1,13 +1,17 @@ Scratch ####################################### +**Processor name:** ``scratch`` + +.. danger:: + + Scratch blocks require an understanding of the Scratch programming language and how Verto is integrated with other systems. The use of this processor requires co-ordination between authors and developers to achieve the desired functionality. + .. note:: The following examples assume usage of the fenced code extension, by having ``markdown.extensions.fenced_code`` in the list of extensions given to Verto. -**Processor name:** ``scratch`` - You can include an image of Scratch blocks using `Scratch Block Plugin notation`_ using the following notation: diff --git a/docs/source/processors/style.rst b/docs/source/processors/style.rst new file mode 100644 index 00000000..273c8521 --- /dev/null +++ b/docs/source/processors/style.rst @@ -0,0 +1,45 @@ +.. _style: + +Style +####################################### + +**Processor name:** ``style`` + +The ``style`` processor is a pre-processor that checks that the input Markdown to enforce style rules. These rules include: + + - Processor tags have empty lines before and after. + - Processor tags do not share a line with other text. + +An example of a valid document follows: + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_valid.md + :language: none + +Error Example(s) +************************************** + +.. note:: + + The examples covered in this section are invalid and will raise errors. + +The following examples raise errors because the processor tags do not have empty lines before and after. + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace.md + :language: none + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_1.md + :language: none + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_2.md + :language: none + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_whitespace_3.md + :language: none + +The following examples raise errors because the processor tags share a line with other text. + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_solitary.md + :language: none + +.. literalinclude:: ../../../verto/tests/assets/style/doc_example_block_solitary_1.md + :language: none diff --git a/docs/source/processors/table-of-contents.rst b/docs/source/processors/table-of-contents.rst index 64597c5e..47df4d04 100644 --- a/docs/source/processors/table-of-contents.rst +++ b/docs/source/processors/table-of-contents.rst @@ -3,6 +3,10 @@ Table of Contents **Processor name:** ``table-of-contents`` +.. danger:: + + The table-of-contents tag currently requires integration with other systems, use of this tag should accompany supervision of a developer. + You can create a placeholder for a web framework (for example: Django) to insert a table of contents by using the following tag: diff --git a/requirements.txt b/requirements.txt index fea37b6f..99af28db 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,8 +2,8 @@ markdown==2.6.8 beautifulsoup4==4.5.3 Jinja2==2.9.6 -python-slugify==1.2.2 -setuptools +python-slugify==1.2.3 +setuptools==35.0.0 # Required dependencies for building documentation sphinx==1.5.5 diff --git a/verto/__init__.py b/verto/__init__.py index 382a79cf..71b8e834 100644 --- a/verto/__init__.py +++ b/verto/__init__.py @@ -1,4 +1,4 @@ # flake8: noqa from .Verto import Verto -__version__ = '0.4.1' +__version__ = '0.5.0' diff --git a/verto/errors/ArgumentDefinitionError.py b/verto/errors/ArgumentDefinitionError.py new file mode 100644 index 00000000..11bce068 --- /dev/null +++ b/verto/errors/ArgumentDefinitionError.py @@ -0,0 +1,17 @@ +from verto.errors.Error import Error + + +class ArgumentDefinitionError(Error): + '''Exception raised when an argument exists but is not readable. + The most likely scenario is that an author has used single quotes + instead of double quotes. + + Attributes: + argument: the argument that was at error + message: explanation of why error was thrown + ''' + + def __init__(self, argument, message): + super().__init__(message) + self.argument = argument + self.message = message diff --git a/verto/processors/errors/ArgumentMissingError.py b/verto/errors/ArgumentMissingError.py similarity index 91% rename from verto/processors/errors/ArgumentMissingError.py rename to verto/errors/ArgumentMissingError.py index 26b57d73..ef26de1d 100644 --- a/verto/processors/errors/ArgumentMissingError.py +++ b/verto/errors/ArgumentMissingError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class ArgumentMissingError(Error): diff --git a/verto/processors/errors/ArgumentValueError.py b/verto/errors/ArgumentValueError.py similarity index 91% rename from verto/processors/errors/ArgumentValueError.py rename to verto/errors/ArgumentValueError.py index e976eb60..aa2c709a 100644 --- a/verto/processors/errors/ArgumentValueError.py +++ b/verto/errors/ArgumentValueError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class ArgumentValueError(Error): diff --git a/verto/processors/errors/Error.py b/verto/errors/Error.py similarity index 100% rename from verto/processors/errors/Error.py rename to verto/errors/Error.py diff --git a/verto/errors/HtmlParseError.py b/verto/errors/HtmlParseError.py new file mode 100644 index 00000000..df893729 --- /dev/null +++ b/verto/errors/HtmlParseError.py @@ -0,0 +1,18 @@ +from verto.errors.Error import Error + + +class HtmlParseError(Error): + '''Exception raised when parsing HTML text and an + error is found. + + Attributes: + line_num: The line number the error occurred on + offset: The position in the line the error occurred + message: explanation of why error was thrown + ''' + + def __init__(self, line_num, offset, message): + super().__init__(message) + self.line_num = line_num + self.offset = offset + self.message = message diff --git a/verto/processors/errors/NoSourceLinkError.py b/verto/errors/NoSourceLinkError.py similarity index 89% rename from verto/processors/errors/NoSourceLinkError.py rename to verto/errors/NoSourceLinkError.py index e2c1106f..8ef589e2 100644 --- a/verto/processors/errors/NoSourceLinkError.py +++ b/verto/errors/NoSourceLinkError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class NoSourceLinkError(Error): diff --git a/verto/processors/errors/NoVideoIdentifierError.py b/verto/errors/NoVideoIdentifierError.py similarity index 89% rename from verto/processors/errors/NoVideoIdentifierError.py rename to verto/errors/NoVideoIdentifierError.py index 7fd73a65..162dc5ff 100644 --- a/verto/processors/errors/NoVideoIdentifierError.py +++ b/verto/errors/NoVideoIdentifierError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class NoVideoIdentifierError(Error): diff --git a/verto/processors/errors/StyleError.py b/verto/errors/StyleError.py similarity index 58% rename from verto/processors/errors/StyleError.py rename to verto/errors/StyleError.py index 75f1e48a..4953bb40 100644 --- a/verto/processors/errors/StyleError.py +++ b/verto/errors/StyleError.py @@ -1,13 +1,13 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class StyleError(Error): '''Exception raised when a Style rule is broken. Attributes: - rule -- rule which was broken - lines -- lines where the style rule was broken - message -- explanation of why error was thrown + line_nums: the line numbers the rule as broken on + lines: lines where the style rule was broken + message: explanation of why error was thrown ''' def __init__(self, line_nums, lines, message): diff --git a/verto/processors/errors/TagNotMatchedError.py b/verto/errors/TagNotMatchedError.py similarity index 89% rename from verto/processors/errors/TagNotMatchedError.py rename to verto/errors/TagNotMatchedError.py index b3b44f54..95a03f03 100644 --- a/verto/processors/errors/TagNotMatchedError.py +++ b/verto/errors/TagNotMatchedError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class TagNotMatchedError(Error): diff --git a/verto/processors/errors/UnsupportedVideoPlayerError.py b/verto/errors/UnsupportedVideoPlayerError.py similarity index 89% rename from verto/processors/errors/UnsupportedVideoPlayerError.py rename to verto/errors/UnsupportedVideoPlayerError.py index bd6982e7..3c257270 100644 --- a/verto/processors/errors/UnsupportedVideoPlayerError.py +++ b/verto/errors/UnsupportedVideoPlayerError.py @@ -1,4 +1,4 @@ -from verto.processors.errors.Error import Error +from verto.errors.Error import Error class UnsupportedVideoPlayerError(Error): diff --git a/verto/processors/errors/__init__.py b/verto/errors/__init__.py similarity index 100% rename from verto/processors/errors/__init__.py rename to verto/errors/__init__.py diff --git a/verto/html-templates/image.html b/verto/html-templates/image.html index eb85e6e3..adfbb35d 100644 --- a/verto/html-templates/image.html +++ b/verto/html-templates/image.html @@ -2,8 +2,7 @@ + class="{% if alignment == 'left' %}left-align{% elif alignment =='center' %}center-align{% elif alignment =='right' %}right-align{% endif %}"> {%- if caption and caption_link -%}

{{ caption }}

{%- elif caption -%} diff --git a/verto/html-templates/interactive.html b/verto/html-templates/interactive.html index 11c6d5df..a5154713 100644 --- a/verto/html-templates/interactive.html +++ b/verto/html-templates/interactive.html @@ -1,22 +1,22 @@ - {% if type == 'in-page' -%} - - {{ "{% include 'interactive/" }}{{ name }}{{ "/index.html' %}" }} - - {% elif type == 'iframe' -%} - - {% elif type == 'whole-page' -%} - - -
- {% if text -%} - {{ text }} - {% else -%} - Click to load {{ name }} - {% endif -%} -
-
- {%- endif -%} +{%- if type == 'in-page' -%} + + {{ "{% include 'interactive/" }}{{ name }}{{ "/index.html' %}" }} + +{% elif type == 'iframe' -%} + +{% elif type == 'whole-page' -%} + + +
+ {% if text -%} + {{ text }} + {% else -%} + Click to load {{ name }} + {% endif -%} +
+
+{%- endif -%} diff --git a/verto/processors/ConditionalProcessor.py b/verto/processors/ConditionalProcessor.py index de966efb..e02e87d9 100644 --- a/verto/processors/ConditionalProcessor.py +++ b/verto/processors/ConditionalProcessor.py @@ -1,6 +1,8 @@ from markdown.blockprocessors import BlockProcessor -from verto.processors.errors.TagNotMatchedError import TagNotMatchedError +from verto.errors.TagNotMatchedError import TagNotMatchedError from verto.processors.utils import etree, parse_arguments, parse_flag, blocks_to_string +from verto.utils.HtmlParser import HtmlParser +from verto.utils.HtmlSerializer import HtmlSerializer from collections import OrderedDict import re @@ -108,8 +110,9 @@ def run(self, parent, blocks): # Render template and compile into an element html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) def get_content(self, blocks): ''' Recursively parses blocks into an element tree, returning @@ -185,5 +188,5 @@ def parse_blocks(self, blocks): # Convert parsed element tree back into html text for rendering content = '' for child in content_tree: - content += etree.tostring(child, encoding='unicode', method='html') + content += HtmlSerializer.tostring(child) return content diff --git a/verto/processors/GenericContainerBlockProcessor.py b/verto/processors/GenericContainerBlockProcessor.py index 8c718b25..ed79c5e8 100644 --- a/verto/processors/GenericContainerBlockProcessor.py +++ b/verto/processors/GenericContainerBlockProcessor.py @@ -1,7 +1,9 @@ from markdown.blockprocessors import BlockProcessor -from verto.processors.errors.TagNotMatchedError import TagNotMatchedError -from verto.processors.errors.ArgumentValueError import ArgumentValueError +from verto.errors.TagNotMatchedError import TagNotMatchedError +from verto.errors.ArgumentValueError import ArgumentValueError from verto.processors.utils import etree, parse_arguments, process_parameters, blocks_to_string +from verto.utils.HtmlParser import HtmlParser +from verto.utils.HtmlSerializer import HtmlSerializer import re @@ -97,7 +99,7 @@ def run(self, parent, blocks): content = '' for child in content_tree: - content += etree.tostring(child, encoding='unicode', method='html') + '\n' + content += HtmlSerializer.tostring(child) + '\n' if content.strip() == '': message = 'content cannot be blank.' @@ -107,5 +109,6 @@ def run(self, parent, blocks): context = self.process_parameters(self.processor, self.template_parameters, argument_values) html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) diff --git a/verto/processors/GenericTagBlockProcessor.py b/verto/processors/GenericTagBlockProcessor.py index c8a9aeb1..dda305fb 100644 --- a/verto/processors/GenericTagBlockProcessor.py +++ b/verto/processors/GenericTagBlockProcessor.py @@ -1,5 +1,6 @@ from markdown.blockprocessors import BlockProcessor -from verto.processors.utils import etree, parse_arguments, process_parameters +from verto.processors.utils import parse_arguments, process_parameters +from verto.utils.HtmlParser import HtmlParser import re @@ -59,5 +60,6 @@ def run(self, parent, blocks): context = self.process_parameters(self.processor, self.template_parameters, argument_values) html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) diff --git a/verto/processors/GlossaryLinkPattern.py b/verto/processors/GlossaryLinkPattern.py index 0c94dff7..b9b25b56 100644 --- a/verto/processors/GlossaryLinkPattern.py +++ b/verto/processors/GlossaryLinkPattern.py @@ -1,5 +1,6 @@ from markdown.inlinepatterns import Pattern -from verto.processors.utils import etree, parse_arguments +from verto.processors.utils import parse_arguments +from verto.utils.HtmlParser import HtmlParser import re @@ -56,6 +57,6 @@ def handleMatch(self, match): context['id'] = identifier html_string = self.template.render(context) - node = etree.fromstring(html_string) - - return node + parser = HtmlParser() + parser.feed(html_string).close() + return parser.get_root() diff --git a/verto/processors/HeadingBlockProcessor.py b/verto/processors/HeadingBlockProcessor.py index 9be1fad4..ddd2ef9d 100644 --- a/verto/processors/HeadingBlockProcessor.py +++ b/verto/processors/HeadingBlockProcessor.py @@ -1,6 +1,6 @@ from markdown.blockprocessors import BlockProcessor -from markdown.util import etree from verto.utils.HeadingNode import DynamicHeadingNode +from verto.utils.HtmlParser import HtmlParser import re @@ -76,8 +76,10 @@ def run(self, parent, blocks): context['level_{0}'.format(i + 1)] = level_val html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) + self.add_to_heading_tree(heading, heading_slug, level) def add_to_heading_tree(self, heading, heading_slug, level): diff --git a/verto/processors/ImageBlockProcessor.py b/verto/processors/ImageBlockProcessor.py index e4a3fef4..6720724f 100644 --- a/verto/processors/ImageBlockProcessor.py +++ b/verto/processors/ImageBlockProcessor.py @@ -1,5 +1,6 @@ from verto.processors.GenericTagBlockProcessor import GenericTagBlockProcessor -from verto.processors.utils import etree, parse_arguments +from verto.processors.utils import parse_arguments +from verto.utils.HtmlParser import HtmlParser import re @@ -73,5 +74,6 @@ def run(self, parent, blocks): context['hover_text'] = argument_values.get('hover-text', None) html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) diff --git a/verto/processors/InteractiveBlockProcessor.py b/verto/processors/InteractiveBlockProcessor.py index fb0fd5c5..71ec87b3 100644 --- a/verto/processors/InteractiveBlockProcessor.py +++ b/verto/processors/InteractiveBlockProcessor.py @@ -1,5 +1,6 @@ from verto.processors.GenericTagBlockProcessor import GenericTagBlockProcessor -from verto.processors.utils import etree, parse_arguments +from verto.processors.utils import parse_arguments +from verto.utils.HtmlParser import HtmlParser import re @@ -84,5 +85,6 @@ def run(self, parent, blocks): context['file_path'] = file_path html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) diff --git a/verto/processors/RelativeLinkPattern.py b/verto/processors/RelativeLinkPattern.py index 79a7e43a..865eadc7 100644 --- a/verto/processors/RelativeLinkPattern.py +++ b/verto/processors/RelativeLinkPattern.py @@ -1,5 +1,5 @@ +from verto.utils.HtmlParser import HtmlParser from markdown.inlinepatterns import Pattern -from markdown.util import etree from html import escape import re @@ -41,6 +41,6 @@ def handleMatch(self, match): context['text'] = match.group('link_text') html_string = self.template.render(context) - node = etree.fromstring(html_string) - - return node + parser = HtmlParser() + parser.feed(html_string).close() + return parser.get_root() diff --git a/verto/processors/ScratchTreeprocessor.py b/verto/processors/ScratchTreeprocessor.py index f5024342..363b4642 100644 --- a/verto/processors/ScratchTreeprocessor.py +++ b/verto/processors/ScratchTreeprocessor.py @@ -1,5 +1,7 @@ from markdown.treeprocessors import Treeprocessor from verto.processors.utils import etree +from verto.utils.HtmlParser import HtmlParser +from verto.utils.HtmlSerializer import HtmlSerializer from collections import namedtuple from functools import reduce from hashlib import sha256 @@ -50,14 +52,15 @@ def run(self, root): html_string, safe = self.markdown.htmlStash.rawHtmlBlocks[i] node = None try: - node = etree.fromstring(html_string) + parser = HtmlParser() + node = parser.feed(html_string).close().get_root() except etree.ParseError: pass if node is None: continue self.process_html(node) - html_string = etree.tostring(node, encoding='unicode', method='html') + html_string = HtmlSerializer.tostring(node) self.markdown.htmlStash.rawHtmlBlocks[i] = html_string, safe def process_html(self, node): @@ -92,7 +95,8 @@ def process_html(self, node): images.append(content_hash) html_string = self.template.render({'images': images}) - new_node = etree.fromstring(html_string) + parser = HtmlParser() + new_node = parser.feed(html_string).close().get_root() node.tag = 'remove' node.text = '' diff --git a/verto/processors/StylePreprocessor.py b/verto/processors/StylePreprocessor.py index 29899227..ef893568 100644 --- a/verto/processors/StylePreprocessor.py +++ b/verto/processors/StylePreprocessor.py @@ -1,4 +1,4 @@ -from verto.processors.errors.StyleError import StyleError +from verto.errors.StyleError import StyleError from markdown.preprocessors import Preprocessor import re diff --git a/verto/processors/VideoBlockProcessor.py b/verto/processors/VideoBlockProcessor.py index 2cabe09c..7706388c 100644 --- a/verto/processors/VideoBlockProcessor.py +++ b/verto/processors/VideoBlockProcessor.py @@ -1,7 +1,8 @@ from verto.processors.GenericTagBlockProcessor import GenericTagBlockProcessor -from verto.processors.errors.NoVideoIdentifierError import NoVideoIdentifierError -from verto.processors.errors.UnsupportedVideoPlayerError import UnsupportedVideoPlayerError -from verto.processors.utils import etree, parse_arguments +from verto.errors.NoVideoIdentifierError import NoVideoIdentifierError +from verto.errors.UnsupportedVideoPlayerError import UnsupportedVideoPlayerError +from verto.processors.utils import parse_arguments +from verto.utils.HtmlParser import HtmlParser import re @@ -74,8 +75,9 @@ def run(self, parent, blocks): context['video_url'] = self.vimeo_template.render(context) html_string = self.template.render(context) - node = etree.fromstring(html_string) - parent.append(node) + parser = HtmlParser() + parser.feed(html_string).close() + parent.append(parser.get_root()) def extract_video_identifier(self, video_url): '''Extracts an identifier and service from a video url. diff --git a/verto/processors/utils.py b/verto/processors/utils.py index 77b4a787..3999124d 100644 --- a/verto/processors/utils.py +++ b/verto/processors/utils.py @@ -1,8 +1,9 @@ import re from markdown.util import etree # noqa: F401 from collections import OrderedDict, defaultdict -from verto.processors.errors.ArgumentMissingError import ArgumentMissingError -from verto.processors.errors.ArgumentValueError import ArgumentValueError +from verto.errors.ArgumentDefinitionError import ArgumentDefinitionError +from verto.errors.ArgumentMissingError import ArgumentMissingError +from verto.errors.ArgumentValueError import ArgumentValueError def parse_argument(argument_key, arguments, default=None): @@ -15,7 +16,16 @@ def parse_argument(argument_key, arguments, default=None): Returns: Value of an argument as a string if found, otherwise None. ''' - result = re.search(r'(^|\s+){}="([^"]*("(?<=\\")[^"]*)*)"'.format(argument_key), arguments) + is_argument = re.search(r'(^|\s+){}='.format(argument_key), arguments) + if not is_argument: + return default + + result = re.match(r'(^|\s+){}="([^"]*("(?<=\\")[^"]*)*)"'.format(argument_key), arguments[is_argument.start():]) + + if is_argument and result is None: + msg = "Argument found but value not contained in double quotes." + raise ArgumentDefinitionError(argument_key, msg) + if result: argument_value = result.group(2).replace(r'\"', r'"') else: diff --git a/verto/tests/BeautifyTest.py b/verto/tests/BeautifyTest.py index 39bb579c..74a25a4d 100644 --- a/verto/tests/BeautifyTest.py +++ b/verto/tests/BeautifyTest.py @@ -1,8 +1,5 @@ import markdown -from unittest.mock import Mock - from verto.VertoExtension import VertoExtension -from verto.processors.CommentPreprocessor import CommentPreprocessor from verto.tests.ProcessorTest import ProcessorTest class BeautifyTest(ProcessorTest): @@ -16,6 +13,15 @@ def __init__(self, *args, **kwargs): ProcessorTest.__init__(self, *args, **kwargs) self.processor_name = 'beautify' + def test_doc_example_basic(self): + '''Checks that basic usecase works as expected. + ''' + test_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage.md') + + converted_test_string = markdown.markdown(test_string, extensions=[self.verto_extension]) + expected_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage_expected.html', strip=True) + self.assertEqual(expected_string, converted_test_string) + def test_example_inline_code(self): '''Tests to see that inline code formatting is unchanged. ''' diff --git a/verto/tests/ConditionalTest.py b/verto/tests/ConditionalTest.py index 554ecc56..36d491ea 100644 --- a/verto/tests/ConditionalTest.py +++ b/verto/tests/ConditionalTest.py @@ -3,7 +3,7 @@ from verto.VertoExtension import VertoExtension from verto.processors.ConditionalProcessor import ConditionalProcessor -from verto.processors.errors.TagNotMatchedError import TagNotMatchedError +from verto.errors.TagNotMatchedError import TagNotMatchedError from verto.tests.ProcessorTest import ProcessorTest diff --git a/verto/tests/FrameTest.py b/verto/tests/FrameTest.py index c2625b4e..79fefb0c 100644 --- a/verto/tests/FrameTest.py +++ b/verto/tests/FrameTest.py @@ -3,7 +3,8 @@ from verto.VertoExtension import VertoExtension from verto.processors.GenericTagBlockProcessor import GenericTagBlockProcessor -from verto.processors.errors.ArgumentMissingError import ArgumentMissingError +from verto.errors.ArgumentMissingError import ArgumentMissingError +from verto.errors.ArgumentDefinitionError import ArgumentDefinitionError from verto.tests.ProcessorTest import ProcessorTest class FrameTest(ProcessorTest): @@ -35,6 +36,17 @@ def test_example_no_link(self): with self.assertRaises(ArgumentMissingError): markdown.markdown(test_string, extensions=[self.verto_extension]) + def test_example_single_quote_argument_error(self): + '''Tests that single quotes as an argument raise the + ArgumentDefinitionError. This is a test that affects all + processors of the generic type (and any that use the + parse_argument function in utils). + ''' + test_string = self.read_test_file(self.processor_name, 'example_single_quote_argument_error.md') + + with self.assertRaises(ArgumentDefinitionError): + converted_test_string = markdown.markdown(test_string, extensions=[self.verto_extension]) + #~ # Doc Tests #~ diff --git a/verto/tests/HtmlParserTest.py b/verto/tests/HtmlParserTest.py new file mode 100644 index 00000000..2c94ebae --- /dev/null +++ b/verto/tests/HtmlParserTest.py @@ -0,0 +1,209 @@ +from verto.tests.BaseTest import BaseTest +from verto.utils.HtmlParser import HtmlParser +from verto.utils.HtmlSerializer import HtmlSerializer +from verto.errors.HtmlParseError import HtmlParseError +from markdown.util import etree + + +class HtmlParserTest(BaseTest): + '''Tests that the HtmlParser and HtmlSerializer can be used to + take in an produce the same HTML string. + ''' + + def __init__(self, *args, **kwargs): + '''Setup asset file directory. + ''' + super().__init__(*args, **kwargs) + self.test_type = "html-parser" + + def read_test_file(self, filename): + '''Returns a string for a given file. + + Args: + filename: The filename of the file found in the asset + directory. + Returns: + A string of the given file. + ''' + return super().read_test_file(self.test_type, filename, True) + + # ~ + # Valid Examples + # ~ + + def test_example_basic_usage(self): + '''Checks that the expected usecase works. + ''' + input_text = self.read_test_file('example_basic_usage.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + self.assertEquals('html', root.tag) + + elements = list(root) + self.assertEquals(1, len(elements)) + self.assertEquals('body', elements[0].tag) + + elements = list(elements[0]) # Open Body + self.assertEquals(3, len(elements)) + self.assertEquals('h1', elements[0].tag) + self.assertEquals('p', elements[1].tag) + self.assertEquals('div', elements[2].tag) + + elements = list(elements[2]) # Open Div + self.assertEquals(2, len(elements)) + self.assertEquals('img', elements[0].tag) + self.assertEquals('a', elements[1].tag) + + img = elements[0] + self.assertEquals('Example text.', img.get('alt')) + self.assertEquals('example.com/example.jpg', img.get('src')) + + a = elements[1] + self.assertEquals('https://www.example.com', a.get('href')) + + root_string = HtmlSerializer.tostring(root) + self.assertEquals(input_text, root_string) + + def test_example_simple_void_tag(self): + '''Checks that a simple (unclosed) void tag is created without + error. + ''' + input_text = self.read_test_file('example_simple_void_tag.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + + self.assertEquals('img', root.tag) + self.assertEquals('Example text.', root.get('alt')) + self.assertEquals('example.com/example.jpg', root.get('src')) + + root_string = HtmlSerializer.tostring(root) + self.assertEquals(input_text, root_string) + + def test_example_simple_closed_void_tag(self): + '''Checks that a simple void tag with closing '\' is created + without error. + ''' + input_text = self.read_test_file('example_simple_closed_void_tag.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + + self.assertEquals('img', root.tag) + self.assertEquals('Example text.', root.get('alt')) + self.assertEquals('example.com/example.jpg', root.get('src')) + + root_string = HtmlSerializer.tostring(root) + expected_text = self.read_test_file('example_simple_void_tag.html') + self.assertEquals(expected_text, root_string) + + def test_example_comment(self): + '''Checks that comments are added unchanged. + ''' + input_text = self.read_test_file('example_comment.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + + self.assertEquals(etree.Comment, root.tag) + + root_string = HtmlSerializer.tostring(root) + self.assertEquals(input_text, root_string) + + def test_example_comment_ie(self): + '''Checks that ie comments are added unchanged. + ''' + input_text = self.read_test_file('example_comment_ie.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + + self.assertEquals(etree.Comment, root.tag) + + root_string = HtmlSerializer.tostring(root) + self.assertEquals(input_text, root_string) + + def test_example_data_and_subelements(self): + '''Checks that data and subelements work together. + ''' + input_text = self.read_test_file('example_data_and_subelements.html') + parser = HtmlParser() + parser.feed(input_text).close() + root = parser.get_root() + self.assertEquals('html', root.tag) + + elements = list(root) + self.assertEquals(1, len(elements)) + self.assertEquals('body', elements[0].tag) + + elements = list(elements[0]) # Open Body + self.assertEquals(2, len(elements)) + self.assertEquals('h1', elements[0].tag) + self.assertEquals('p', elements[1].tag) + + elements = list(elements[1]) # Open p + self.assertEquals(3, len(elements)) + self.assertEquals('em', elements[0].tag) + self.assertEquals('b', elements[1].tag) + self.assertEquals('a', elements[2].tag) + + root_string = HtmlSerializer.tostring(root) + self.assertEquals(input_text, root_string) + + # ~ + # Invalid Examples + # ~ + + def test_example_access_root_before_feed_error(self): + '''Checks that the AttributeError is raised is the root element + is accessed before it is created. + ''' + parser = HtmlParser() + with self.assertRaises(AttributeError): + parser.get_root() + + def test_example_multiple_roots_error(self): + '''Checks that when multiple roots are detected that an exception + is raised. + ''' + input_text = self.read_test_file('example_multiple_roots_error.html') + parser = HtmlParser() + with self.assertRaises(HtmlParseError): + parser.feed(input_text).close() + + def test_example_lone_end_tag_error(self): + '''Checks that lone end tags cause an exception to be raised. + ''' + input_text = self.read_test_file('example_lone_end_tag_error.html') + parser = HtmlParser() + with self.assertRaises(HtmlParseError): + parser.feed(input_text).close() + + def test_example_missing_end_tag_error(self): + '''Checks that elements (that need to be closed) cause an + exception to be raised. + ''' + input_text = self.read_test_file('example_missing_end_tag_error.html') + parser = HtmlParser() + with self.assertRaises(HtmlParseError): + parser.feed(input_text).close() + + def test_example_missing_end_tag_implicit_error(self): + '''Checks that elements (that need to be closed) cause an + exception to be raised, when they are implicitly closed + by an outer closing tag. + ''' + input_text = self.read_test_file('example_missing_end_tag_implicit_error.html') + parser = HtmlParser() + with self.assertRaises(HtmlParseError): + parser.feed(input_text).close() + + def test_example_data_without_tags_error(self): + '''Checks that data without a root tag causes an exception to + be raised. + ''' + input_text = self.read_test_file('example_data_without_tags_error.html') + parser = HtmlParser() + with self.assertRaises(HtmlParseError): + parser.feed(input_text).close() diff --git a/verto/tests/ImageTest.py b/verto/tests/ImageTest.py index 07017cb5..b1004feb 100644 --- a/verto/tests/ImageTest.py +++ b/verto/tests/ImageTest.py @@ -4,8 +4,8 @@ from verto.VertoExtension import VertoExtension from verto.processors.ImageBlockProcessor import ImageBlockProcessor -from verto.processors.errors.ArgumentMissingError import ArgumentMissingError -from verto.processors.errors.ArgumentValueError import ArgumentValueError +from verto.errors.ArgumentMissingError import ArgumentMissingError +from verto.errors.ArgumentValueError import ArgumentValueError from verto.tests.ProcessorTest import ProcessorTest class ImageTest(ProcessorTest): diff --git a/verto/tests/JinjaTest.py b/verto/tests/JinjaTest.py new file mode 100644 index 00000000..79482658 --- /dev/null +++ b/verto/tests/JinjaTest.py @@ -0,0 +1,23 @@ +import markdown +from verto.VertoExtension import VertoExtension +from verto.tests.ProcessorTest import ProcessorTest + +class JinjaTest(ProcessorTest): + '''The major concern with beautifying is that preformatted tags and + code blocks are unchanged. The tests here cover these cases. + ''' + + def __init__(self, *args, **kwargs): + '''Set processor name in class for file names. + ''' + ProcessorTest.__init__(self, *args, **kwargs) + self.processor_name = 'jinja' + + def test_doc_example_basic(self): + '''Checks that basic usecase works as expected. + ''' + test_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage.md') + + converted_test_string = markdown.markdown(test_string, extensions=[self.verto_extension]) + expected_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage_expected.html', strip=True) + self.assertEqual(expected_string, converted_test_string) diff --git a/verto/tests/PanelTest.py b/verto/tests/PanelTest.py index 313a292a..018949a8 100644 --- a/verto/tests/PanelTest.py +++ b/verto/tests/PanelTest.py @@ -3,8 +3,8 @@ from verto.VertoExtension import VertoExtension from verto.processors.GenericContainerBlockProcessor import GenericContainerBlockProcessor -from verto.processors.errors.TagNotMatchedError import TagNotMatchedError -from verto.processors.errors.ArgumentValueError import ArgumentValueError +from verto.errors.TagNotMatchedError import TagNotMatchedError +from verto.errors.ArgumentValueError import ArgumentValueError from verto.tests.ProcessorTest import ProcessorTest class PanelTest(ProcessorTest): @@ -140,6 +140,20 @@ def test_missing_tag_inner(self): self.assertRaises(TagNotMatchedError, lambda x: markdown.markdown(x, extensions=[self.verto_extension]), test_string) + def test_contains_inner_image(self): + '''Tests that other processors within a panel + still renders correctly. + ''' + verto_extension = VertoExtension([self.processor_name, 'image'], {}) + test_string = self.read_test_file(self.processor_name, 'contains_inner_image.md') + blocks = self.to_blocks(test_string) + + self.assertListEqual([True, False, False, False, True], [self.block_processor.test(blocks, block) for block in blocks], msg='"{}"'.format(test_string)) + + converted_test_string = markdown.markdown(test_string, extensions=[verto_extension]) + expected_string = self.read_test_file(self.processor_name, 'contains_inner_image_expected.html', strip=True) + self.assertEqual(expected_string, converted_test_string) + #~ # Doc Tests #~ diff --git a/verto/tests/RemoveTest.py b/verto/tests/RemoveTest.py new file mode 100644 index 00000000..a91cdead --- /dev/null +++ b/verto/tests/RemoveTest.py @@ -0,0 +1,23 @@ +import markdown +from verto.VertoExtension import VertoExtension +from verto.tests.ProcessorTest import ProcessorTest + +class RemoveTest(ProcessorTest): + '''The major concern with beautifying is that preformatted tags and + code blocks are unchanged. The tests here cover these cases. + ''' + + def __init__(self, *args, **kwargs): + '''Set processor name in class for file names. + ''' + ProcessorTest.__init__(self, *args, **kwargs) + self.processor_name = 'remove' + + def test_doc_example_basic(self): + '''Checks that basic usecase works as expected. + ''' + test_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage.md') + + converted_test_string = markdown.markdown(test_string, extensions=[self.verto_extension]) + expected_string = self.read_test_file(self.processor_name, 'doc_example_basic_usage_expected.html', strip=True) + self.assertEqual(expected_string, converted_test_string) diff --git a/verto/tests/StyleTest.py b/verto/tests/StyleTest.py index 1a4878f0..2424de00 100644 --- a/verto/tests/StyleTest.py +++ b/verto/tests/StyleTest.py @@ -3,7 +3,7 @@ from verto.VertoExtension import VertoExtension from verto.processors.StylePreprocessor import StylePreprocessor -from verto.processors.errors.StyleError import StyleError +from verto.errors.StyleError import StyleError from verto.tests.ProcessorTest import ProcessorTest class StyleTest(ProcessorTest): diff --git a/verto/tests/VideoTest.py b/verto/tests/VideoTest.py index 239e8afb..87171b68 100644 --- a/verto/tests/VideoTest.py +++ b/verto/tests/VideoTest.py @@ -3,9 +3,9 @@ from verto.VertoExtension import VertoExtension from verto.processors.VideoBlockProcessor import VideoBlockProcessor -from verto.processors.errors.NoSourceLinkError import NoSourceLinkError -from verto.processors.errors.NoVideoIdentifierError import NoVideoIdentifierError -from verto.processors.errors.UnsupportedVideoPlayerError import UnsupportedVideoPlayerError +from verto.errors.NoSourceLinkError import NoSourceLinkError +from verto.errors.NoVideoIdentifierError import NoVideoIdentifierError +from verto.errors.UnsupportedVideoPlayerError import UnsupportedVideoPlayerError from verto.tests.ProcessorTest import ProcessorTest diff --git a/verto/tests/assets/beautify/doc_example_basic_usage.md b/verto/tests/assets/beautify/doc_example_basic_usage.md new file mode 100644 index 00000000..2471485d --- /dev/null +++ b/verto/tests/assets/beautify/doc_example_basic_usage.md @@ -0,0 +1,3 @@ +

Example Title

+

Example paragraph.

+

Example paragraph in a div.

diff --git a/verto/tests/assets/beautify/doc_example_basic_usage_expected.html b/verto/tests/assets/beautify/doc_example_basic_usage_expected.html new file mode 100644 index 00000000..c766c660 --- /dev/null +++ b/verto/tests/assets/beautify/doc_example_basic_usage_expected.html @@ -0,0 +1,11 @@ +

+ Example Title +

+

+ Example paragraph. +

+
+

+ Example paragraph in a div. +

+
diff --git a/verto/tests/assets/html-parser/example_basic_usage.html b/verto/tests/assets/html-parser/example_basic_usage.html new file mode 100644 index 00000000..b9b942d3 --- /dev/null +++ b/verto/tests/assets/html-parser/example_basic_usage.html @@ -0,0 +1,10 @@ + + +

Example Heading

+

Example paragraph

+
+ Example text. + Example link. +
+ + diff --git a/verto/tests/assets/html-parser/example_comment.html b/verto/tests/assets/html-parser/example_comment.html new file mode 100644 index 00000000..dd67fa55 --- /dev/null +++ b/verto/tests/assets/html-parser/example_comment.html @@ -0,0 +1 @@ + diff --git a/verto/tests/assets/html-parser/example_comment_ie.html b/verto/tests/assets/html-parser/example_comment_ie.html new file mode 100644 index 00000000..d8b3f703 --- /dev/null +++ b/verto/tests/assets/html-parser/example_comment_ie.html @@ -0,0 +1,3 @@ + diff --git a/verto/tests/assets/html-parser/example_data_and_subelements.html b/verto/tests/assets/html-parser/example_data_and_subelements.html new file mode 100644 index 00000000..d4cee624 --- /dev/null +++ b/verto/tests/assets/html-parser/example_data_and_subelements.html @@ -0,0 +1,6 @@ + + +

Example Heading

+

Example paragraph with Emphasis and Bold stylised text. Finally a link to an Example website.

+ + diff --git a/verto/tests/assets/html-parser/example_data_without_tags_error.html b/verto/tests/assets/html-parser/example_data_without_tags_error.html new file mode 100644 index 00000000..eab3465a --- /dev/null +++ b/verto/tests/assets/html-parser/example_data_without_tags_error.html @@ -0,0 +1 @@ +Lipsum lorem diff --git a/verto/tests/assets/html-parser/example_lone_end_tag_error.html b/verto/tests/assets/html-parser/example_lone_end_tag_error.html new file mode 100644 index 00000000..489525dd --- /dev/null +++ b/verto/tests/assets/html-parser/example_lone_end_tag_error.html @@ -0,0 +1,4 @@ +
+

Example paragraph

+ +
diff --git a/verto/tests/assets/html-parser/example_missing_end_tag_error.html b/verto/tests/assets/html-parser/example_missing_end_tag_error.html new file mode 100644 index 00000000..fd34070f --- /dev/null +++ b/verto/tests/assets/html-parser/example_missing_end_tag_error.html @@ -0,0 +1,2 @@ +
+

Example paragraph.

diff --git a/verto/tests/assets/html-parser/example_missing_end_tag_implicit_error.html b/verto/tests/assets/html-parser/example_missing_end_tag_implicit_error.html new file mode 100644 index 00000000..f9ba210f --- /dev/null +++ b/verto/tests/assets/html-parser/example_missing_end_tag_implicit_error.html @@ -0,0 +1,3 @@ +
+ Example link. +
diff --git a/verto/tests/assets/html-parser/example_multiple_roots_error.html b/verto/tests/assets/html-parser/example_multiple_roots_error.html new file mode 100644 index 00000000..4de6b6d5 --- /dev/null +++ b/verto/tests/assets/html-parser/example_multiple_roots_error.html @@ -0,0 +1,6 @@ +
+

Example heading

+
+
+

Example paragraph.

+
diff --git a/verto/tests/assets/html-parser/example_simple_closed_void_tag.html b/verto/tests/assets/html-parser/example_simple_closed_void_tag.html new file mode 100644 index 00000000..13fd1e6c --- /dev/null +++ b/verto/tests/assets/html-parser/example_simple_closed_void_tag.html @@ -0,0 +1 @@ +Example text. diff --git a/verto/tests/assets/html-parser/example_simple_void_tag.html b/verto/tests/assets/html-parser/example_simple_void_tag.html new file mode 100644 index 00000000..64c0aafe --- /dev/null +++ b/verto/tests/assets/html-parser/example_simple_void_tag.html @@ -0,0 +1 @@ +Example text. diff --git a/verto/tests/assets/iframe/example_single_quote_argument_error.md b/verto/tests/assets/iframe/example_single_quote_argument_error.md new file mode 100644 index 00000000..8f999ae8 --- /dev/null +++ b/verto/tests/assets/iframe/example_single_quote_argument_error.md @@ -0,0 +1 @@ +{iframe link='http://www.google.com'} diff --git a/verto/tests/assets/jinja/doc_example_basic_usage.md b/verto/tests/assets/jinja/doc_example_basic_usage.md new file mode 100644 index 00000000..1d9f8022 --- /dev/null +++ b/verto/tests/assets/jinja/doc_example_basic_usage.md @@ -0,0 +1,7 @@ +
+ {% if thing < object %} +

+ A paragraph explaining the < operation. +

+ {% endif %} +
diff --git a/verto/tests/assets/jinja/doc_example_basic_usage_expected.html b/verto/tests/assets/jinja/doc_example_basic_usage_expected.html new file mode 100644 index 00000000..2af4bcef --- /dev/null +++ b/verto/tests/assets/jinja/doc_example_basic_usage_expected.html @@ -0,0 +1,7 @@ +
+ {% if thing < object %} +

+ A paragraph explaining the < operation. +

+ {% endif %} +
diff --git a/verto/tests/assets/panel/contains_inner_image.md b/verto/tests/assets/panel/contains_inner_image.md new file mode 100644 index 00000000..d96ee30f --- /dev/null +++ b/verto/tests/assets/panel/contains_inner_image.md @@ -0,0 +1,9 @@ +{panel type="extra-for-experts" title="Lorem ipsum" subtitle="Nunc non accumsan" expanded="always"} + +Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc non accumsan libero, et suscipit velit. Phasellus quis lorem eget lacus rutrum lobortis. Integer tincidunt convallis arcu a porttitor. Ut urna sapien, egestas mollis lectus a, suscipit placerat ante. Vestibulum id congue tellus. Nullam dapibus felis eu ligula mattis, vel consectetur nibh aliquet. Maecenas nec elit orci. Curabitur ut massa maximus, ultrices diam at, porttitor erat. Praesent eu purus vitae ligula elementum iaculis in non eros. Quisque eu aliquet dolor, ut pharetra lacus. + +{image file-path="http://placehold.it/350x150" caption="Placeholder image" source="https://placehold.it/" hover-text="This is hover text" alignment="left"} + +Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc non accumsan libero, et suscipit velit. Phasellus quis lorem eget lacus rutrum lobortis. Integer tincidunt convallis arcu a porttitor. Ut urna sapien, egestas mollis lectus a, suscipit placerat ante. Vestibulum id congue tellus. Nullam dapibus felis eu ligula mattis, vel consectetur nibh aliquet. Maecenas nec elit orci. Curabitur ut massa maximus, ultrices diam at, porttitor erat. Praesent eu purus vitae ligula elementum iaculis in non eros. Quisque eu aliquet dolor, ut pharetra lacus. + +{panel end} diff --git a/verto/tests/assets/panel/contains_inner_image_expected.html b/verto/tests/assets/panel/contains_inner_image_expected.html new file mode 100644 index 00000000..25d4f9d3 --- /dev/null +++ b/verto/tests/assets/panel/contains_inner_image_expected.html @@ -0,0 +1,27 @@ +
+
+ + Lorem ipsum: + + Nunc non accumsan +
+
+

+ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc non accumsan libero, et suscipit velit. Phasellus quis lorem eget lacus rutrum lobortis. Integer tincidunt convallis arcu a porttitor. Ut urna sapien, egestas mollis lectus a, suscipit placerat ante. Vestibulum id congue tellus. Nullam dapibus felis eu ligula mattis, vel consectetur nibh aliquet. Maecenas nec elit orci. Curabitur ut massa maximus, ultrices diam at, porttitor erat. Praesent eu purus vitae ligula elementum iaculis in non eros. Quisque eu aliquet dolor, ut pharetra lacus. +

+
+ +

+ Placeholder image +

+

+ + Source + +

+
+

+ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc non accumsan libero, et suscipit velit. Phasellus quis lorem eget lacus rutrum lobortis. Integer tincidunt convallis arcu a porttitor. Ut urna sapien, egestas mollis lectus a, suscipit placerat ante. Vestibulum id congue tellus. Nullam dapibus felis eu ligula mattis, vel consectetur nibh aliquet. Maecenas nec elit orci. Curabitur ut massa maximus, ultrices diam at, porttitor erat. Praesent eu purus vitae ligula elementum iaculis in non eros. Quisque eu aliquet dolor, ut pharetra lacus. +

+
+
diff --git a/verto/tests/assets/remove/doc_example_basic_usage.md b/verto/tests/assets/remove/doc_example_basic_usage.md new file mode 100644 index 00000000..ee260009 --- /dev/null +++ b/verto/tests/assets/remove/doc_example_basic_usage.md @@ -0,0 +1,11 @@ +
+ +

+ Content in here. +

+ {{ Django Variable }} +

+ Something about the variable. +

+
+
diff --git a/verto/tests/assets/remove/doc_example_basic_usage_expected.html b/verto/tests/assets/remove/doc_example_basic_usage_expected.html new file mode 100644 index 00000000..a2c7abc5 --- /dev/null +++ b/verto/tests/assets/remove/doc_example_basic_usage_expected.html @@ -0,0 +1,9 @@ +
+

+ Content in here. +

+ {{ Django Variable }} +

+ Something about the variable. +

+
diff --git a/verto/tests/assets/scratch/example_multiple_codeblocks.md b/verto/tests/assets/scratch/example_multiple_codeblocks.md index 02119a3a..cc0f1e7f 100644 --- a/verto/tests/assets/scratch/example_multiple_codeblocks.md +++ b/verto/tests/assets/scratch/example_multiple_codeblocks.md @@ -1,4 +1,4 @@ -Scratch is great for kids you can great simple code like: +Scratch is great for kids you can create simple code like: ```scratch when flag clicked diff --git a/verto/tests/assets/scratch/example_multiple_codeblocks_2.md b/verto/tests/assets/scratch/example_multiple_codeblocks_2.md index 8379007c..f5caf30f 100644 --- a/verto/tests/assets/scratch/example_multiple_codeblocks_2.md +++ b/verto/tests/assets/scratch/example_multiple_codeblocks_2.md @@ -1,4 +1,4 @@ -Scratch is great for kids you can great simple code like: +Scratch is great for kids you can create simple code like: ```scratch when flag clicked diff --git a/verto/tests/assets/scratch/example_multiple_codeblocks_expected.html b/verto/tests/assets/scratch/example_multiple_codeblocks_expected.html index 91fd4b54..1508f19e 100644 --- a/verto/tests/assets/scratch/example_multiple_codeblocks_expected.html +++ b/verto/tests/assets/scratch/example_multiple_codeblocks_expected.html @@ -1,5 +1,5 @@

- Scratch is great for kids you can great simple code like: + Scratch is great for kids you can create simple code like:

diff --git a/verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html b/verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html index 3e3fa15b..9afff2de 100644 --- a/verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html +++ b/verto/tests/assets/scratch/example_multiple_codeblocks_expected_2.html @@ -1,5 +1,5 @@

- Scratch is great for kids you can great simple code like: + Scratch is great for kids you can create simple code like:

diff --git a/verto/tests/start_tests.py b/verto/tests/start_tests.py index ddc2e2ec..6f1c2707 100644 --- a/verto/tests/start_tests.py +++ b/verto/tests/start_tests.py @@ -21,6 +21,9 @@ from verto.tests.TableOfContentsTest import TableOfContentsTest from verto.tests.ScratchTest import ScratchTest from verto.tests.BeautifyTest import BeautifyTest +from verto.tests.HtmlParserTest import HtmlParserTest +from verto.tests.JinjaTest import JinjaTest +from verto.tests.RemoveTest import RemoveTest def parse_args(): '''Parses the arguments for running the test suite, these are @@ -76,7 +79,10 @@ def unit_suite(): unittest.makeSuite(FrameTest), unittest.makeSuite(TableOfContentsTest), unittest.makeSuite(ScratchTest), - unittest.makeSuite(BeautifyTest) + unittest.makeSuite(BeautifyTest), + unittest.makeSuite(HtmlParserTest), + unittest.makeSuite(JinjaTest), + unittest.makeSuite(RemoveTest) )) if __name__ == '__main__': diff --git a/verto/utils/HtmlParser.py b/verto/utils/HtmlParser.py new file mode 100644 index 00000000..a296f3ac --- /dev/null +++ b/verto/utils/HtmlParser.py @@ -0,0 +1,228 @@ +import html.parser +from verto.errors.HtmlParseError import HtmlParseError +from markdown.util import etree + + +class HtmlParser(html.parser.HTMLParser): + ''' Used to convert an HTML string into an ElementTree. Since this + is not defaultly supported by fromstring (XML only). + ''' + + VOID_ELEMENTS = [ + 'area', 'base', 'br', 'command', 'embed', 'hr', 'img', 'input', + 'link', 'input', 'link', 'meta', 'param', 'source' + ] + + OPTIONALLY_CLOSE_ELEMENTS = [ + 'body', 'colgroup', 'dd', 'dt', 'head', 'html', 'li', 'optgroup', + 'option', 'p', 'tbody', 'td', 'tfoot', 'th', 'thead', 'tr' + ] + + def __init__(self, *args, **kwargs): + '''Create a new parser. + ''' + super().__init__(convert_charrefs=True, *args, **kwargs) + self.root = None + self.closed = False + self.stack = [] + + def get_root(self): + '''Gets the root element after parsing. + + Returns: + An etree Element of the root node. + Raises: + Error: If no root has been found or the parser + has not been closed yet. + ''' + if self.root is None or not self.closed: + raise AttributeError("Operations out of order: root accessed before created.") + return self.root + + def feed(self, data): + '''Feed some text to the parser. + + Args: + data: The text to feed to the parser. + Returns: + Itself. + ''' + super().feed(data) + return self + + def close(self): + '''Force processing of all buffered data as if it were + followed by an end-of-file mark. + + Returns: + Itself. + ''' + for element in self.stack: + if element.tag not in HtmlParser.OPTIONALLY_CLOSE_ELEMENTS: + line, pos = self.getpos() + msg = "Trying to implicitly close a normal element ({}).".format(element.tag) + raise HtmlParseError(line, pos, msg) + self.closed = True + super().close() + return self + + def reset(self): + '''Reset the instance. Loses all unprocessed data. + ''' + self.root = None + self.closed = False + self.stack = [] + super().reset() + + def add_element(self, element): + '''Adds an element to the element tree. + + Args: + element: An etree Element to append to the element tree. + ''' + if self.root is None and len(self.stack) <= 0: + self.root = element + elif self.root is not None and len(self.stack) <= 0: + line, pos = self.getpos() + raise HtmlParseError(line, pos, "Secondary root node found.") + else: + self.stack[-1].append(element) + + if element.tag not in HtmlParser.VOID_ELEMENTS and element.tag != etree.Comment: + self.stack.append(element) + + def handle_starttag(self, tag, attrs): + '''This method is called to handle the start of a tag + (e.g. `
`). + + Args: + tag: The name of the tag (converted to lowercase). + attrs: A list of tuples (name, value) where the name is + converted to lowercase and qoutes on the value have + been removed. + ''' + element = etree.Element(tag, dict(attrs)) + self.add_element(element) + + def handle_endtag(self, tag): + '''This method is called to handle the end tag of an + element (e.g. `
`). + + Args: + tag: The name of the tag (converted to lowercase). + ''' + if tag not in HtmlParser.VOID_ELEMENTS: + found = False + while not found and len(self.stack) > 0: + element = self.stack.pop() + if element.tag == tag: + found = True + elif element.tag not in HtmlParser.OPTIONALLY_CLOSE_ELEMENTS: + line, pos = self.getpos() + raise HtmlParseError(line, pos, "Trying to implicitly close a normal element.") + + if not found: + line, pos = self.getpos() + raise HtmlParseError(line, pos, "Trying to close an unopened element.") + + def handle_startendtag(self, tag, attrs): + '''Similar to handle_starttag(), but called when the parser + encounters an XHTML-style empty tag (). + + Args: + tag: The name of the tag (converted to lowercase). + attrs: A list of tuples (name, value) where the name is + converted to lowercase and qoutes on the value have + been removed. + ''' + self.handle_starttag(tag, attrs) + self.handle_endtag(tag) + + def handle_data(self, data): + '''This method is called to process arbitrary data (e.g. + text nodes and the content of and + ). + + Args: + data: The content between the tags. + ''' + if len(self.stack) <= 0: + if data.strip() != '': + line, pos = self.getpos() + raise HtmlParseError(line, pos, "Data outside of the HTML tree.") + else: + sibling = list(self.stack[-1])[-1] if list(self.stack[-1]) else None + if sibling is not None: + sibling.tail = (sibling.tail or '') + data + else: + self.stack[-1].text = (self.stack[-1].text or '') + data + + def handle_comment(self, data): + '''This method is called when a comment is encountered + (e.g. ). + + Note: + The content of Internet Explorer conditional comments + (condcoms) will also be sent to this method. + Args: + data: The string of the comment. + ''' + element = etree.Comment(data) + self.add_element(element) + + def handle_entityref(self, name): + '''This method is called to process a named character reference + of the form &name; (e.g. >). + + Note: + This function should never be called, since the HTMLParser + is initialized with convert_charrefs as True. + Args: + name: The string of entity with ampersand and semicolon + removed. + ''' + super().handle_entityref(name) + + def handle_charref(self, name): + '''This method is called to process decimal and hexadecimal + numeric character references of the form `&#NNN;` and + `&#xNNN;`. + + Note: + This function should never be called, since the HTMLParser + is initialized with convert_charrefs as True. + Args: + name: The string of the decimal or hexadecimal of the char. + ''' + super().handle_charref(name) + + def unknown_decl(self, data): + '''This method is called when an unrecognized declaration is read by the parser. + + Args: + data: The entire contents of the declaration inside + the `<[!...]>` markup. + ''' + raise NotImplementedError("Unknown declarations are not supported.") + + def handle_decl(self, decl): + '''This method is called to handle an HTML doctype declaration + (e.g. ). + + Args: + decl: The entire contents of the declaration inside + the `` markup. + ''' + raise NotImplementedError("HTML declarations are not supported.") + + def handle_pi(self, data): + '''Method called when a processing instruction is encountered. + (e.g. ``). + + Note: + An XHTML processing instruction using the trailing '?' will + cause the '?' to be included in data. + Args: + data: The entire processing instruction as a string. + ''' + raise NotImplementedError("Processing instructions are not supported.") diff --git a/verto/utils/HtmlSerializer.py b/verto/utils/HtmlSerializer.py new file mode 100644 index 00000000..f542cb19 --- /dev/null +++ b/verto/utils/HtmlSerializer.py @@ -0,0 +1,29 @@ +from markdown.util import etree +import re + + +class HtmlSerializer(object): + '''Converts an element tree object which is HTML into + a string. + ''' + + COMMENT_PATTERN = r'' + + @staticmethod + def tostring(root): + '''Converts an etree into a string. + + Args: + root: An Element from the ElementTree library. + Returns: + A string of the serialized HTML tree. + ''' + string = etree.tostring(root, encoding='unicode', method='html') + + def unescape_comment(matchobj): + return r''.format( + matchobj.group('start_condition'), + matchobj.group('content'), + matchobj.group('end_condition')) + string = re.sub(HtmlSerializer.COMMENT_PATTERN, unescape_comment, string, flags=re.DOTALL) + return string