Ruleset inheritance dependencies (#65)

ryan95f · web-flow · commit a160c662b1ff · 2023-04-29T16:51:04.000+01:00
* Updated the example import.ys to have an inherited ruleset to test the inhertiance feature. Updated the grammar to include ruleset inheritance. Implemented detection by reusing the dependency manager and added a basic placeholder during the resolving process

* Fixed pylint violation with the use of .keys() in the has_cycle method

* Updated the resolve_ruleset_inheritance function to have an improved implementation that accounts for the structure of dependencies

* Minor styling improvements to the resolve_ruleset_inheritance. Updated docstrings to account for the new exception. Added some additional test cases to confirm the inheritance management system is working

* Added an additional test to the resolve_ruleset_inheritance to account for rulesets that do not have any inheritance requirements

* Modified the import example to have a better example. Updated the test files to include additional cases such as a full parse of the inheritance schema

* updated the setup.py version to 0.4.1 and updated the changelog

* Added python version 3.11 to the test github workflow in the python version matrix

* Updated the schema components documentation with an ruleset inheritance docs
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -28,7 +28,7 @@ jobs:
     runs-on: ubuntu-20.04
     strategy:
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10", "3.11"]
       max-parallel: 2
       fail-fast: true
     steps:
diff --git a/changelog.md b/changelog.md
@@ -60,3 +60,9 @@
 * Added namespaces to the import statements with the `as` keyword
 * Improvements to the loading of rulesets to allow for schema files to be less restrictive in the order resources are defined.
 * Improvements to the grammar file to include new terminals and remove duplicate constructs
+
+## v0.4.1 (29th April 2023)
+
+* Added ruleset inheritance to the Yamlator schema syntax
+* Added dependency management to the ruleset inheritance process
+* Added Python 3.11 to the `test` Github Workflow
diff --git a/docs/schema_components.md b/docs/schema_components.md
@@ -19,6 +19,9 @@ Below are the various components that can be used to construct a schema with Yam
   * [Union Type](#union-type)
 * [Importing schemas](#importing-schemas)
   * [Schema import paths](#schema-import-paths)
+* [Ruleset Inheritance](#ruleset-inheritance)
+  * [Strict ruleset inheritance](#strict-ruleset-inheritance)
+  * [Inheritance from imported schemas](#inheritance-from-imported-schemas)
 
 ## Schema
 
@@ -476,3 +479,91 @@ main/
 The `apis` will be fetched from the `web` directory and `common.ys` is located in the same directory location as the `main.ys` file. An full example of importing schemas can be found in [example folder](../example/imports/).
 
 __NOTE__: If an import cycle is detected, Yamlator will exit with a non-zero status code.
+
+## Ruleset Inheritance
+
+In Yamlator version `0.4.1`, rulesets can now inherit other rulesets to reduce duplicating rules. A ruleset can inherit other rulesets from the current schema or from schemas that have been imported.
+
+__NOTE__: A ruleset can only inherit a single ruleset. Multi-inheritance is currently __NOT__ supported.
+
+An example of ruleset inheritance where the `Employee` ruleset will inherit the rules from the `Person` ruleset:
+
+```text
+ruleset Person {
+    first_name str
+    surname str
+}
+
+ruleset Employee(Person) {
+    employee_id str
+}
+```
+
+The `Employee` ruleset will now contain a total of 3 rules, 2 from the `Person` ruleset and 1 rule from its own block.
+
+If a child ruleset has a rule with the same name as the inherited ruleset, then the child rule will override the parent. For example, given the following example:
+
+```text
+ruleset Versions {
+    version str
+    kind str
+}
+
+ruleset Foo(Versions) {
+    version int
+}
+```
+
+In this case, the `Foo` ruleset will only have 2 rules. The `kind` rule will be inherited but the `version` rule in `Foo` will override the exiting rule in `Versions`.
+
+### Strict Ruleset Inheritance
+
+If the parent ruleset is a strict ruleset, strict mode will __NOT__ be inherited by the child ruleset. In order for the child ruleset to use strict mode, it must specify it is a strict ruleset in its definition. For example:
+
+```text
+strict ruleset Foo {
+    bar str
+    baz int
+}
+
+ruleset Request1(Foo) {
+    id str
+}
+
+strict ruleset Request2(Foo) {
+    id str
+}
+```
+
+In the above case, the ruleset `Request1` will not enforce strict mode. However `Request2` will enforce strict mode since it has been defined as a strict ruleset.
+
+### Inheritance from imported schemas
+
+Rulesets can be inherited by rulesets from other files that have been imported with or without a namespace. For example:
+
+The given ruleset is located in a file called `common.ys`
+
+```text
+ruleset Project {
+    id str
+    version str
+    kind str
+}
+```
+
+This ruleset can then be imported with or without a namespace and be inherited:
+
+```text
+import Project from "common.ys" as common
+import Project from "common.ys"
+
+ruleset AwesomeProject1(common.Project) {
+    ...
+}
+
+ruleset AwesomeProject2(Project) {
+    ...
+}
+```
+
+__NOTE__: If a inheritance cycle is detected, Yamlator will exit with a non-zero status code.
diff --git a/example/imports/common.ys b/example/imports/common.ys
@@ -1,7 +1,9 @@
 import Project from "../strict_mode/strict.ys"
 
-enum Values {
+enum Envs {
     TEST = 1
+    DEV = 2
+    PROD = 3
 }
 
 enum Status {
diff --git a/example/imports/import.yaml b/example/imports/import.yaml
@@ -1,3 +1,5 @@
 project:
   version: 1
-  name: test
+  name: my-awesome-project
+  details:
+    author: Foo Bar
diff --git a/example/imports/import.ys b/example/imports/import.ys
@@ -1,12 +1,17 @@
-import Values, Status, ProjectDetails from "common.ys" as core
+import Envs, Status, ProjectDetails from "common.ys" as core
 
 schema {
     project Project
-    value core.Values
+    env core.Envs
 }
 
-strict ruleset Project {
+strict ruleset Project(Versions) {
     status core.Status
     apis list(str)
     details core.ProjectDetails
+}
+
+ruleset Versions {
+    version str
+    kind str
 }
diff --git a/setup.py b/setup.py
@@ -2,7 +2,7 @@
 
 import setuptools
 
-VERSION = '0.4.0'
+VERSION = '0.4.1'
 PACKAGE_NAME = 'yamlator'
 DESCRIPTION = 'Yamlator is a CLI tool that allows a YAML file to be validated using a lightweight schema language'  # nopep8
 
diff --git a/tests/cmd/constants.py b/tests/cmd/constants.py
@@ -17,6 +17,7 @@
 VALID_SCHEMA = f'{_BASE_VALID_PATH}/valid.ys'
 VALID_KEYLESS_DIRECTIVE_SCHEMA = f'{_BASE_VALID_PATH}/keyless_directive.ys'
 VALID_KEYLESS_RULES_SCHEMA = f'{_BASE_VALID_PATH}/keyless_and_standard_rules.ys'
+VALID_INHERITANCE_SCHEMA = f'{_BASE_VALID_PATH}/inheritance.ys'
 
 # These files don't exists and are used to force specific errors in the tests
 NONE_PATH = None
diff --git a/tests/files/valid/inheritance.ys b/tests/files/valid/inheritance.ys
@@ -5,9 +5,21 @@ import Status, User from "base.ys"
 schema {
     employees list(Employee)
     status Status
+    projects list(Project)
 }
 
 ruleset Employee (base.Employee) {
     department str
     salary float
 }
+
+ruleset Versions {
+    version str
+    kind str
+}
+
+ruleset Project(Versions) {
+    description str
+    cost float
+    timeline str
+}
diff --git a/tests/parser/loaders/test_resolve_ruleset_inheritance.py b/tests/parser/loaders/test_resolve_ruleset_inheritance.py
@@ -10,6 +10,7 @@
 from yamlator.types import SchemaTypes
 from yamlator.types import YamlatorRuleset
 from yamlator.exceptions import ConstructNotFoundError
+from yamlator.exceptions import CycleDependencyError
 from yamlator.parser.loaders import resolve_ruleset_inheritance
 
 
@@ -27,7 +28,21 @@ class TestResolveRulesetInheritance(unittest.TestCase):
                 is_strict=False,
                 parent=RuleType(SchemaTypes.RULESET, lookup='Bar')
             )
-        }, ConstructNotFoundError)
+        }, ConstructNotFoundError),
+        ('with_cycle_ruleset', {
+            'Foo': YamlatorRuleset(
+                name='Foo',
+                rules=[],
+                is_strict=False,
+                parent=RuleType(SchemaTypes.RULESET, lookup='Bar')
+            ),
+            'Bar': YamlatorRuleset(
+                name='Bar',
+                rules=[],
+                is_strict=False,
+                parent=RuleType(SchemaTypes.RULESET, lookup='Foo')
+            )
+        }, CycleDependencyError)
     ])
     def test_resolve_ruleset_raises_error(self, name: str, rulesets: Any,
                                           expected_exception: Exception):
@@ -37,7 +52,37 @@ def test_resolve_ruleset_raises_error(self, name: str, rulesets: Any,
         with self.assertRaises(expected_exception):
             resolve_ruleset_inheritance(rulesets)
 
-    def test_resolve_ruleset_inheritance(self):
+    def test_resolve_ruleset_inheritance_without_ruleset_inheritance(self):
+        rulesets = {
+            'Foo': YamlatorRuleset(
+                name='Foo',
+                rules=[
+                    Rule('name', RuleType(SchemaTypes.STR), True),
+                    Rule('age', RuleType(SchemaTypes.INT), False)
+                ],
+                is_strict=False,
+            ),
+            'Bar': YamlatorRuleset(
+                name='Bar',
+                rules=[
+                    Rule('first_name', RuleType(SchemaTypes.STR), True),
+                    Rule('last_name', RuleType(SchemaTypes.STR), True)
+                ]
+            )
+        }
+
+        expected_foo_rule_count = 2
+        expected_bar_rule_count = 2
+
+        updated_rules = resolve_ruleset_inheritance(rulesets)
+
+        actual_foo_rule_count = len(updated_rules['Foo'].rules)
+        actual_bar_rule_count = len(updated_rules['Bar'].rules)
+
+        self.assertEqual(expected_foo_rule_count, actual_foo_rule_count)
+        self.assertEqual(expected_bar_rule_count, actual_bar_rule_count)
+
+    def test_resolve_ruleset_inheritance_with_simple_inheritance(self):
         rulesets = {
             'Foo': YamlatorRuleset(
                 name='Foo',
@@ -129,6 +174,96 @@ def test_resolve_ruleset_inheritance_without_any_parents(self):
         self.assertEqual(expected_ruleset_count, actual_ruleset_count)
         self.assertEqual(expected_foo_rule_count, actual_foo_rule_count)
 
+    def test_resolve_ruleset_inheritance_with_cascading_inheritance(self):
+        rulesets = {
+            'Foo': YamlatorRuleset(
+                name='Foo',
+                rules=[
+                    Rule('message', RuleType(SchemaTypes.STR), True),
+                    Rule('name', RuleType(SchemaTypes.STR), True),
+                ],
+                is_strict=False,
+                parent=RuleType(SchemaTypes.RULESET, lookup='Bar')
+            ),
+            'Bar': YamlatorRuleset(
+                name='Bar',
+                rules=[
+                    Rule('number', RuleType(SchemaTypes.FLOAT), True),
+                ],
+                parent=RuleType(SchemaTypes.RULESET, lookup='Faux')
+            ),
+            'Faux': YamlatorRuleset(
+                name='Faux',
+                rules=[
+                    Rule('version', RuleType(SchemaTypes.STR), True),
+                    Rule('type', RuleType(SchemaTypes.STR), True),
+                ],
+                is_strict=False
+            )
+        }
+
+        expected_foo_rule_count = 5
+        expected_bar_rule_count = 3
+
+        updated_rules = resolve_ruleset_inheritance(rulesets)
+
+        actual_foo_rule_count = len(updated_rules['Foo'].rules)
+        actual_bar_rule_count = len(updated_rules['Bar'].rules)
+
+        self.assertEqual(expected_foo_rule_count, actual_foo_rule_count)
+        self.assertEqual(expected_bar_rule_count, actual_bar_rule_count)
+
+    def test_resolve_ruleset_inheritance_multiple_objects_same_dependency(self):
+        rulesets = {
+            'Foo': YamlatorRuleset(
+                name='Foo',
+                rules=[
+                    Rule('message', RuleType(SchemaTypes.STR), True),
+                    Rule('name', RuleType(SchemaTypes.STR), True),
+                ],
+                is_strict=False,
+                parent=RuleType(SchemaTypes.RULESET, lookup='Faux')
+            ),
+            'Bar': YamlatorRuleset(
+                name='Bar',
+                rules=[
+                    Rule('number', RuleType(SchemaTypes.FLOAT), True),
+                ],
+                parent=RuleType(SchemaTypes.RULESET, lookup='Faux')
+            ),
+            'Baz': YamlatorRuleset(
+                name='Baz',
+                rules=[
+                    Rule('name', RuleType(SchemaTypes.STR), True),
+                    Rule('item', RuleType(SchemaTypes.STR), False),
+                    Rule('cost', RuleType(SchemaTypes.FLOAT), False),
+                ],
+                parent=RuleType(SchemaTypes.RULESET, lookup='Bar')
+            ),
+            'Faux': YamlatorRuleset(
+                name='Faux',
+                rules=[
+                    Rule('version', RuleType(SchemaTypes.STR), True),
+                    Rule('type', RuleType(SchemaTypes.STR), True),
+                ],
+                is_strict=False
+            )
+        }
+
+        expected_foo_rule_count = 4
+        expected_bar_rule_count = 3
+        expected_baz_rule_count = 6
+
+        updated_rules = resolve_ruleset_inheritance(rulesets)
+
+        actual_foo_rule_count = len(updated_rules['Foo'].rules)
+        actual_bar_rule_count = len(updated_rules['Bar'].rules)
+        actual_baz_rule_count = len(updated_rules['Baz'].rules)
+
+        self.assertEqual(expected_foo_rule_count, actual_foo_rule_count)
+        self.assertEqual(expected_bar_rule_count, actual_bar_rule_count)
+        self.assertEqual(expected_baz_rule_count, actual_baz_rule_count)
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/tests/parser/test_parse_schema.py b/tests/parser/test_parse_schema.py
@@ -46,6 +46,8 @@ def test_parse_with_empty_text(self):
             constants.VALID_KEYLESS_DIRECTIVE_SCHEMA, 1),
         ('with_keyless_schema_and_rules',
             constants.VALID_KEYLESS_RULES_SCHEMA, 2),
+        ('with_inheritance',
+            constants.VALID_INHERITANCE_SCHEMA, 3),
     ])
     def test_parse_with_valid_content(self, name: str, schema_path: str,
                                       expected_schema_rule_count: int):
diff --git a/yamlator/grammar/grammar.lark b/yamlator/grammar/grammar.lark
@@ -17,7 +17,7 @@ import_statement: "import" imported_types "from" IMPORT_STATEMENT_PATH ("as" NAM
 imported_types: (CONTAINER_TYPE_NAME ("," CONTAINER_TYPE_NAME)*)
 
 schema_entry: STRICT_KEYWORD? "schema" "{" rule+ "}"
-ruleset: STRICT_KEYWORD? "ruleset" CONTAINER_TYPE_NAME "{" rule+ "}"
+ruleset: STRICT_KEYWORD? "ruleset" CONTAINER_TYPE_NAME (ruleset_parent)? "{" rule+ "}"
 ruleset_parent: "(" container_type ")"
 
 // Enum constructs
diff --git a/yamlator/parser/core.py b/yamlator/parser/core.py
@@ -210,6 +210,8 @@ def container_type(self, tokens: Iterator[Token]) -> RuleType:
         name = tokens[0]
         if len(tokens) > 1:
             name = f'{tokens[0]}.{tokens[1]}'
+        else:
+            name = name.value
 
         schema_type = self.seen_constructs.get(name, SchemaTypes.UNKNOWN)
         rule_type = RuleType(schema_type=schema_type, lookup=name)
diff --git a/yamlator/parser/dependency.py b/yamlator/parser/dependency.py
diff --git a/yamlator/parser/loaders.py b/yamlator/parser/loaders.py

Original file line number	Diff line number	Diff line change
`@@ -1,7 +1,9 @@`
`1`	`1`	`import Project from "../strict_mode/strict.ys"`
`2`	`2`
`3`		`-enum Values {`
	`3`	`+enum Envs {`
`4`	`4`	`TEST = 1`
	`5`	`+ DEV = 2`
	`6`	`+ PROD = 3`
`5`	`7`	`}`
`6`	`8`
`7`	`9`	`enum Status {`