Merge pull request #34 from ConfD-Developer/micnovak/delete_tests

delete in stream telemetry - in demo + test

data/demo.xml

@@ -3,6 +3,10 @@
   <subscription>
     <STREAM>
       <changes>
+        <element>
+          <path>/gnmi-tools/top-for-delete</path>
+          <del/>
+        </element>
         <element>
           <path>/interfaces/interface[name=if_5]/type</path>
           <val>fastEther</val>
@@ -37,8 +41,12 @@
           <path>/interfaces-state/interface[name=state_if_6]/type</path>
           <val>gigabitEthernet</val>
         </element>
+        <element>
+          <path>/interfaces/interface[name=if_6]/type</path>
+          <del/>
+        </element>
         <element>send</element>
       </changes>
     </STREAM>
   </subscription>
-</demo>
+</demo>

docs/ConfD_gNMI_adapter.adoc

@@ -23,25 +23,26 @@ endif::[]
 :Author:    Michal Novák
 :email:     micnovak@cisco.com
 :URL:       https://www.tail-f.com/
-:Date:      2023-01-24
-:Revision:  0.3.0
+:Date:      2023-12-12
+:Revision:  0.4.0
 
 == Version history
 
 [options="header", cols="1s,10,^2s,2e"]
 |======
 | Document version     | Notes                                                  | Date        | Author
 | 0.0.1           | Initial document version.                       | 2021-02-09      | {author} {email}
-| 0.1.0          | Run options updated, added seq. diagrams for Subscribe operation.                      |  2021-03-23     | {author} {email}
+| 0.1.0          | Run options updated, added sequence diagrams for `Subscribe` operation.                      |  2021-03-23     | {author} {email}
 | 0.2.0   | External data provider description, fixes, documentation update.                      | 2021-04-27      | {author} {email}
+| 0.3.0   | Update command line options, added Encoding description                          | 2023-01-24      | {author} {email}
 | {revision}   | Update command line options, added Encoding description            | {date}      | {author} {email}
 |======
 
 toc::[]
 
 == Introduction
 
-https://www.tail-f.com/management-agent/[ConfD] is configuration management agent supporting various standard and proprietary northbound interfaces like:
+https://www.tail-f.com/management-agent/[ConfD] is a configuration management agent supporting various standard and proprietary northbound interfaces like:
 
 * https://tools.ietf.org/html/rfc6241[NETCONF]
 * https://tools.ietf.org/html/rfc8040[RESTCONF]
@@ -51,29 +52,29 @@ https://www.tail-f.com/management-agent/[ConfD] is configuration management agen
 * CLI
 
 https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md[gNMI] is another popular north bound interface, which is not implemented by ConfD.
-In this demo project we will implement gNMI Adapter over existing ConfD interfaces to make (at least partial) gNMI support.
+In this project, we will implement gNMI Adapter over existing ConfD interfaces to make (at least partial) gNMI support.
 In the beginning, we will provide basic functionality, for most common operations, later on we will add more.
-This demo focuses on functionality and simplicity, we have chosen Python as implementation language.
+The project focuses on functionality and simplicity, we have chosen Python as implementation language.
 
-We use general approach, so the demo can be adapted for other management agents and tools as well.
+We use the general approach, so the project can be adapted for other management agents and tools as well.
 
-This demo is still work in progress, see <<Limitations and TODOs>> section.
+This project is still work in progress, see <<Limitations and TODOs>> section.
 
 === Copy/Paste and Output blocks
 
-In this note you can find script and code examples, that can be directly pasted into the shell or CLI terminal. We will use following block style for the copy/paste ready text:
+In this note, you can find script and code examples that can be directly pasted into the shell or CLI terminal. We will use the following block style for the copy/paste ready text:
 
 [source,shell,role="acopy"]
 ----
 pip install grpcio-tools
 ----
 
 NOTE: make sure all commands have executed - confirm last command with kbd:[ENTER], if needed.
-If viewed on https://github.com[GitHub], you may find following
+If viewed on https://github.com[GitHub], you may find the following
 browser https://github.com/zenorocha/codecopy[extension] useful (out-of-the-box *copy to clipboard* button).
 
 The output of the shell CLI commands or file content will be displayed
-with following block style:
+with the following block style:
 
 .[.small]_output_
 [.output]
@@ -104,7 +105,7 @@ Options:
 
 We expect https://www.python.org/[Python3] to be installed. The `python` and `pip` commands are from Python3 environment. If not, use `python3` or `pip3` instead (or use e.g. `sudo apt-get install python-is-python3`)
 
-TIP: For package installation and development, you may consider creating https://docs.python.org/3/tutorial/venv.html[python virtual environment].
+TIP: For package installation and development, you may consider creating a https://docs.python.org/3/tutorial/venv.html[python virtual environment].
 
 === gRPC Python tools
 
@@ -126,7 +127,7 @@ pip install --upgrade grpcio-tools
 
 === Pytest
 
-For automated tests we will use https://www.pytest.org/[pytest] framework.
+For automated tests, we will use a https://www.pytest.org/[pytest] framework.
 If you want to run tests, use `pip` to install it.
 
 .Installation
@@ -141,7 +142,7 @@ pip install pytest
 pip install --upgrade pytest
 ----
 
-NOTE: `pytest` may be available also as package in your distribution (e.g. `apt-get install python3-pytest`). We still recommend to use `pip` to get the latest version.
+NOTE: `pytest` is often available also as package in your distribution (e.g. `apt-get install python3-pytest`). We still recommend to use `pip` to get the latest version.
 
 === ConfD
 
@@ -153,7 +154,7 @@ Install https://www.tail-f.com/management-agent/[ConfD Premium] or https://www.t
 source ${CONFD_DIR}/confdrc
 ----
 
-TIP: See https://info.tail-f.com/confd-evaluation-kick-start-guide[ConfD Kick Start Guide] for additional information.
+TIP: See https://info.tail-f.com/confd-evaluation-kick-start-guide[ConfD Kick-Start Guide] for additional information.
 
 === Build environment
 
@@ -182,7 +183,7 @@ service gNMI {
 
 NOTE: The interface itself looks relatively simple, but the `Request` and `Response` messages may be complex. `Subscribe` method has many variants. More details can be found in the https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification[gNMI Specification].
 
-== Building gNMI Adapter demo
+== Building the gNMI Adapter
 
 === gNMI python binding
 
@@ -206,7 +207,7 @@ https://tools.ietf.org/html/rfc8343[`ietf-interfaces.yang`] and its dependencies
 
 NOTE: The used datamodel and initial configuration is used for demonstration in this note. The gNMI Adapter can run against any other ConfD instance with different data model. In this case, paths and values will be different. See examples with ConfD example application <<c_stats, `5-c_stats`>> and <<iter-c, `iter_c`>>.
 
-== Running gNMI Adapter demo
+== Running the gNMI Adapter
 
 Before running the adapter, we need to make sure gNMI python binding is created.
 
@@ -216,11 +217,11 @@ Before running the adapter, we need to make sure gNMI python binding is created.
 make clean all
 ----
 
-The adapter can be run in _demo_ and _api_ mode.
+The adapter can be run in _demo_, _confd_ and _nso_ mode.
 
-In the _demo_ mode it does not require running ConfD, it partly emulates `ietf-interfaces.yang` data model and initial configuration. This mode is useful for testing, development, etc.
+In the _demo_ mode it does not require running ConfD or NSO, it partly emulates `ietf-interfaces.yang` data model and initial configuration. This mode is useful for testing, development, etc.
 
-In case we want to run adapter against ConfD (_api_ mode), we can use `Makefile` `start` target to start ConfD with initial demo configuration.
+In case we want to  test adapter against ConfD  (_confd_  mode), we can use `Makefile` `start` target to start ConfD with initial configuration.
 
 .rebuild everything and start ConfD and load demo configuration
 [source, shell, role="acopy"]
@@ -450,6 +451,14 @@ For operational data: +
 `confd_cmd -o -fr -c "set /interfaces-state/interface{state_if_8}/type gigabitEthernet"`
 
 
+NOTE: You can also test `STREAM` subscription in demo mode by passing configuration file. +
+Start the server: +
+`./src/confd_gnmi_server.py -t demo --logging=info --cfg=./data/demo.xml` +
+ +
+And check with the client: +
+`./src/confd_gnmi_client.py -o subscribe -s STREAM --read-count=10 --prefix /ietf-interfaces:interfaces --path interface[name=if_5]/name --path interface[name=if_5]/type`
+
+
 .subscribe for `list` entry
 [source, shell, role="acopy"]
 ----
@@ -1054,7 +1063,7 @@ TIP: To list-only tests, use  `./test.sh --collect-only -q  tests/`
 
 === Limitations and TODOs
 
-The implementation of the adapter (still in early phase) is demo reference implementation that shows how to add gNMI support to existing ConfD interfaces.
+The implementation of the adapter is a reference implementation that shows how to add gNMI support to existing ConfD and NSO interfaces.
 It can be extended according to deployment requirements.
 This not all gNMI functionality are currently supported. They may be added in the future.
 
@@ -1094,7 +1103,7 @@ aggregation should not be used for Subscriptions by default)
 
 == Conclusion
 
-gNMI Adapter Demo can provide initial gNMI functionality to ConfD.
+gNMI Adapter can provide initial gNMI functionality to ConfD and NSO.
 
 == References
 

gnmi-tools.xml

@@ -54,5 +54,25 @@
             <type>opticalTransport</type>
             <admin-state>In-Service</admin-state>
         </double-key-list>
+        <top-for-delete>
+            <top-list>
+                <name>n1</name>
+            </top-list>
+            <top-list>
+                <name>n2</name>
+                <empty-leaf/>
+            </top-list>
+            <top-list>
+                <name>n3</name>
+                <pres/>
+            </top-list>
+            <top-list>
+                <name>n4</name>
+                <down>
+                    <str-leaf>test</str-leaf>
+                    <int-leaf>23</int-leaf>
+                </down>
+            </top-list>
+        </top-for-delete>
     </gnmi-tools>
 </config>

gnmi-tools.yang

@@ -1,4 +1,5 @@
 module gnmi-tools {
+
   namespace "http://cisco.com/ns/yang/gnmi-tools";
   prefix gnmi-tools;
 
@@ -17,6 +18,14 @@ module gnmi-tools {
     }
   }
 
+  grouping top-list {
+    list top-list {
+      key name;
+      leaf name {type string;}
+      uses combo;
+    }
+  }
+
   container gnmi-tools {
     container top {
       uses combo;
@@ -27,18 +36,18 @@ module gnmi-tools {
       uses combo;
     }
 
-    list top-list {
-      key name;
-      leaf name {type string;}
-      uses combo;
-    }
+    uses top-list;
 
     list double-key-list {
         key "name type";
         leaf name {type string;}
         leaf type {type string;}
         leaf admin-state {type string;}
     }
+
+    container top-for-delete {
+      uses top-list;
+    }
   }
 
 }

src/confd_gnmi_adapter.py

@@ -125,11 +125,10 @@ def add_path_for_monitoring(self, path, prefix):
             pass
 
         @abstractmethod
-        def get_monitored_changes(self) -> List:
+        def get_subscription_notifications(self) -> list[gnmi_pb2.Notification]:
             """
             Get gNMI subscription updates for changed values
-            :return: gNMI update array
-            #TODO should we also return delete array
+            :return: gNMI Notification array
             """
             pass
 
@@ -261,15 +260,6 @@ def changes(self):
             log.debug("<== responses=%s", responses)
             return responses
 
-        def get_subscription_notifications(self):
-            update = self.get_monitored_changes()
-            notif = gnmi_pb2.Notification(timestamp=get_timestamp_ns(),
-                                          prefix=self.subscription_list.prefix,
-                                          update=update,
-                                          delete=[],
-                                          atomic=False)
-            return [notif]
-
         def _get_next_sample_interval_and_subscriptions(self,
                                                         first_sample_time: int):
             interval = None

src/confd_gnmi_api_adapter.py

@@ -114,7 +114,7 @@ def __init__(self, adapter, subscription_list):
             self.stop_pipe = None
             self.subpoint_paths = {}
 
-        def get_subscription_notifications(self):
+        def get_subscription_notifications(self) -> list[gnmi_pb2.Notification]:
             return [gnmi_pb2.Notification(timestamp=get_timestamp_ns(),
                                           prefix=prefix,
                                           update=updates,
@@ -146,9 +146,6 @@ def _get_subscription_notifications(self):
                         yield prefix, updates, deletes
                 self.change_db = []
 
-        def get_monitored_changes(self):
-            raise NotImplementedError
-
         def get_sample(self, path, prefix, allow_aggregation=False,
                        start_change_processing=False):
             log.debug("==>")

src/confd_gnmi_client.py

@@ -18,7 +18,7 @@
 from confd_gnmi_common import HOST, PORT, make_xpath_path, VERSION, \
     common_optparse_options, common_optparse_process, make_gnmi_path, \
     datatype_str_to_int, subscription_mode_str_to_int, \
-    encoding_int_to_str, encoding_str_to_int, get_time_string
+    encoding_int_to_str, encoding_str_to_int, get_time_string, get_timestamp_ns
 
 from gnmi_pb2_grpc import gNMIStub
 
@@ -147,18 +147,21 @@ def print_notification(n):
         pfx_str = make_xpath_path(gnmi_prefix=n.prefix)
         print("timestamp {} prefix {} atomic {}".format(
             get_time_string(n.timestamp), pfx_str, n.atomic))
-        print("Updates:")
-        for u in n.update:
-            if u.val.json_val:
-                value = json.loads(u.val.json_val)
-            elif u.val.json_ietf_val:
-                value = json.loads(u.val.json_ietf_val)
-            else:
-                value = str(u.val)
-            print("path: {} value {}".format(pfx_str + make_xpath_path(u.path),
-                                             value))
-        for dpath in n.delete:
-            print("path deleted: {}".format(pfx_str + make_xpath_path(dpath)))
+        if n.update:
+            print("Updates:")
+            for u in n.update:
+                if u.val.json_val:
+                    value = json.loads(u.val.json_val)
+                elif u.val.json_ietf_val:
+                    value = json.loads(u.val.json_ietf_val)
+                else:
+                    value = str(u.val)
+                print("path: {} value {}".format(pfx_str + make_xpath_path(u.path),
+                                                 value))
+        if n.delete:
+            print("Deletes:")
+            for dpath in n.delete:
+                print("path deleted: {}".format(pfx_str + make_xpath_path(dpath)))
 
     @staticmethod
     def read_subscribe_responses(responses, read_count=-1):
@@ -169,7 +172,10 @@ def read_subscribe_responses(responses, read_count=-1):
                 log.info("******* Subscription received response=%s read_count=%i",
                          response, read_count)
                 print("subscribe - response read_count={}".format(read_count))
-                ConfDgNMIClient.print_notification(response.update)
+                if response.sync_response:
+                    print(f"timestamp {get_time_string(get_timestamp_ns())} Sync_response")
+                else:
+                    ConfDgNMIClient.print_notification(response.update)
                 num_read += 1
                 if read_count > 0:
                     read_count -= 1