Release v4.0

Author: arobenko

CMakeLists.txt

@@ -11,7 +11,7 @@ if (NOT ASCIIDOCTOR_PDF_EXE)
     message(FATAL_ERROR "asciidoctor-pdf is not found")
 endif ()
 
-set (VERSION "v3.1.3")
+set (VERSION "v4.0")
 set (MAIN_FILE "${PROJECT_SOURCE_DIR}/commsdsl_spec.adoc")
 
 add_custom_target("pdf" ALL

aliases/aliases.adoc

@@ -129,8 +129,8 @@ the generated code as a comment for the relevant access functions.
 ----
 
 === More on Aliases in <message>-es ===
-When a new <<messages-messages, &lt;message&gt; >> is defined it can
-copy all the fields from other already defined <<messages-messages, &lt;message&gt; >>
+When a new <<messages-messages, &lt;message&gt; >> is defined, it can
+copy all the fields from other already defined <<messages-messages, &lt;message&gt; >> or <<fields-bundle, &lt;bundle&gt; >>
 (using **copyFieldsFrom** <<intro-properties, property>>).
 By default all the <<aliases-aliases, &lt;alias&gt; >> definitions are also copied.
 It is possible to modify this default behavior by using **copyFieldsAliases** 
@@ -154,7 +154,7 @@ It is possible to modify this default behavior by using **copyFieldsAliases**
 Similar to <<messages-messages, &lt;message&gt; >>-es 
 <<interfaces-interfaces, &lt;interface&gt; >> can also use **copyFieldsFrom** 
 <<intro-properties, property>> to copy its field from some other 
-<<interfaces-interfaces, &lt;interface&gt; >> definition and have all
+<<interfaces-interfaces, &lt;interface&gt; >> or <<fields-bundle, &lt;bundle&gt; >> definition and have all
 the aliases copied by default. The control of such default copying behavior
 is also done by using **copyFieldsAliases** 
 <<intro-properties, property>> with <<intro-boolean, boolean>> value.

appendix/checksum.adoc

@@ -8,7 +8,7 @@ for detailed description.
 |===
 |Property Name|Allowed Type / Value|DSL Version|Required|Default Value ^.^|Description
 
-|**alg**|"sum", "crc-ccitt", "crc-16", "crc-32", "custom"|1|yes||Checksum calculation algorithm.
+|**alg**|"sum", "xor", "crc-ccitt", "crc-16", "crc-32", "custom"|1|yes||Checksum calculation algorithm.
 |**algName**|<<intro-names, name>> string|1|no (unless **alg** is **custom**)||Name of the custom algorithm. Applicable only if **alg="custom"**.
 |**from**|<<intro-names, name>> string|1|yes (only if **until** is not specified)||Name of the frame layer, from which the checksum calculation starts.
 |**until**|<<intro-names, name>> string|1|yes (only if **from** is not specified)||Name of the frame layer, until (and including) which the checksum calculation is executed.

appendix/enum.adoc

@@ -16,6 +16,7 @@ for detailed description.
 |**hexAssign**|<<intro-boolean, bool>>|1|no|false|Force generated code to assign enum values using hexadecimal numbers.
 |**nonUniqueAllowed**|<<intro-boolean, bool>>|1|no|false|Allow non unique **&lt;validValue&gt;**-es.
 |**validCheckVersion**|<<intro-boolean, bool>>|1|no|false|Take into account protocol version when generating code for field's value validity check.
+|**availableLengthLimit**|<<intro-boolean, bool>>|4|no|false|Allow having less bytes in the buffer than required by the **type** when performing (de)serialization.
 |===
 
 ==== Properties of &lt;validValue&gt; Child Element of &lt;enum&gt; Field ====

appendix/fields.adoc

@@ -2,7 +2,7 @@
 === Common Properties of Fields ===
 Refer to <<fields-common, Common Properties of Fields>> chapter for detailed description. 
 
-[cols="^.^28,^.^10,^.^8,^.^8,^.^10,36", options="header"]
+[cols="^.^27,^.^11,^.^8,^.^8,^.^10,36", options="header"]
 |===
 |Property Name|Allowed Type / Value|DSL Version|Required|Default Value ^.^|Description
 
@@ -24,4 +24,10 @@ Applicable only to members of the <<messages-messages, &lt;message&gt; >> or <<f
 |**customizable**|<<intro-boolean, bool>>|1|no|false|Mark the field to allow compile time customization regardless of code generator's level of customization.
 |**semanticType**|"none", "messageId", "version", "length"|1|no|none|Specify semantic type of the field. It allows code generator to generate special code for special cases. Value "length" was introduced in **v2** of this specification.
 |**forceGen**|<<intro-boolean, bool>>|3|no|false|Force generation of field's code regardless of it's being referenced or not.
+|**readOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **read** operation.
+|**writeOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **write** operation.
+|**refreshOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **refresh** operation.
+|**lengthOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **length** retrieval.
+|**validOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **valid** retrieval.
+|**nameOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **name** retrieval.
 |===

appendix/int.adoc

@@ -26,6 +26,8 @@ for detailed description.
 |**displayOffset**|<<intro-numeric, numeric>>|1|no|0|Indicates to GUI analysis tools to add specified offset value to a field's value when displaying it.
 |**nonUniqueSpecialsAllowed**|<<intro-boolean, bool>>|2|no|false|Allow non unique **&lt;special&gt;**-s.
 |**displaySpecials**|<<intro-boolean, bool>>|2|no|true|Control displaying **&lt;special&gt;** values in analysis tools.
+|**defaultValidValue**|<<intro-numeric, numeric>> or <<intro-names, name>>|4|no||Convenience bunding of the **defaultValue** and **validValue** into a single property.
+|**availableLengthLimit**|<<intro-boolean, bool>>|4|no|false|Allow having less bytes in the buffer than required by the **type** when performing (de)serialization.
 |===
 
 ==== Properties of &lt;special&gt; Child Element of &lt;int&gt; Field ====

appendix/interface.adoc

@@ -9,8 +9,8 @@ for detailed description.
 
 |**name**|<<intro-names, name>> string|1|yes||Name of the interface.
 |**description**|string|1|no||Human readable description of the interface.
-|**copyFieldsFrom**|<<intro-references, reference>> string|1|no||Interface definition from which fields need to be copied.
-|**copyFieldsAliases**|<<intro-boolean, bool>>|3|no|true|Control copy of the defined <<aliases-aliases, aliases>> when **copyFieldsFrom** property is used to copy fields from the other <<interfaces-interfaces, &lt;interface&gt; >>.
+|**copyFieldsFrom**|<<intro-references, reference>> string|1|no||Interface or bundle field definition from which fields need to be copied.
+|**copyFieldsAliases**|<<intro-boolean, bool>>|3|no|true|Control copy of the defined <<aliases-aliases, aliases>> when **copyFieldsFrom** property is used to copy fields from the other <<interfaces-interfaces, &lt;interface&gt; >> or <<fields-bundle, &lt;bundle&gt; >>.
 |===
 
 

appendix/message.adoc

@@ -3,23 +3,30 @@
 Refer to <<messages-messages, Messages>> chapter
 for detailed description. 
 
-[cols="^.^28,^.^10,^.^8,^.^8,^.^10,36", options="header"]
+[cols="^.^27,^.^11,^.^8,^.^8,^.^10,36", options="header"]
 |===
 |Property Name|Allowed Type / Value|DSL Version|Required|Default Value ^.^|Description
 
 |**name**|<<intro-names, name>> string|1|yes||Name of the message.
 |**id**|<<intro-numeric, numeric>>|1|yes||Numeric ID of the message.
 |**description**|string|1|no||Human readable description of the message.
 |**displayName**|string|1|no||Name of the message to display. If empty, the code generator must use value of property **name** instead.
-|**copyFieldsFrom**|<<intro-references, reference>> string|1|no||Message definition from which fields need to be copied.
+|**copyFieldsFrom**|<<intro-references, reference>> string|1|no||Message or bundle field definition from which fields need to be copied.
 |**order**|<<intro-numeric, numeric>>|1|no|0|Relative order of the messages with the same **id**.
 |**sinceVersion**|<<intro-numeric, unsigned>>|1|no|0|Version of the protocol in which message was introduced.
 |**deprecated**|<<intro-numeric, unsigned>>|1|no|max unsigned|Version of the protocol in which message was deprecated. +
 Must be greater than value of **sinceVersion**.
 |**removed**|<<intro-boolean, bool>>|1|no|false|Indicates whether deprecated message has been removed from being supported.
 |**sender**|"both", "client", "server"|1|no|both|Endpoint that sends the message.
 |**customizable**|<<intro-boolean, bool>>|1|no|false|Mark the message to allow compile time customization regardless of code generator's level of customization.
-|**copyFieldsAliases**|<<intro-boolean, bool>>|3|no|true|Control copy of the defined <<aliases-aliases, aliases>> when **copyFieldsFrom** property is used to copy fields from the other <<messages-messages, &lt;message&gt; >>.
+|**copyFieldsAliases**|<<intro-boolean, bool>>|3|no|true|Control copy of the defined <<aliases-aliases, aliases>> when **copyFieldsFrom** property is used to copy fields from the other <<messages-messages, &lt;message&gt; >> or <<fields-bundle, &lt;bundle&gt; >>.
+|**validateMinLength**|<<intro-numeric, numeric>>|4|no||Specify expected minimal serialization length of the message fields.
+|**readOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **read** operation.
+|**writeOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **write** operation.
+|**refreshOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **refresh** operation.
+|**lengthOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **length** retrieval.
+|**validOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **valid** retrieval.
+|**nameOverride**|"any", "replace", "extend", "none"|4|no|any|Control the overriding code for **name** retrieval.
 |===
 
 Extra child XML elements allowed:

commsdsl_spec.adoc

@@ -1,6 +1,6 @@
-= CommsDSL Specification v3.1.3
+= CommsDSL Specification v4.0
 Alex Robenko
-v3.1.3, 2020-10
+v4.0, 2022-05
 :doctype: article
 :toc: left
 :source-highlighter: rouge

fields/common.adoc

@@ -4,6 +4,7 @@ Every field is different, and defines its own properties and/or other aspects.
 However, there are common properties, that are applicable to every field. 
 They are summarized below.
 
+[[fields-common-name]]
 ==== Name ====
 Every field must define it's <<intro-names, name>>, which is expected to be 
 used by a code generator when defining a relevant class. The name is defined
@@ -18,6 +19,7 @@ using **name** <<intro-properties, property>>.
 </schema>
 ----
 
+[[fields-common-description]]
 ==== Description ====
 It is possible to provide a description of the field about what it is and
 how it is expected to be used. This description is only for documentation
@@ -39,6 +41,7 @@ generated class. The <<intro-properties, property>> is **description**.
 </schema>
 ----
 
+[[fields-common-reusing-other-fields]]
 ==== Reusing Other Fields ====
 Sometimes two different fields are very similar, but differ in one particular
 aspect. **CommsDSL** allows copying all the properties from previously defined
@@ -65,6 +68,7 @@ The **reuse** is allowed only from the field of the same **kind**. For example,
 it is **NOT** allowed for <<fields-enum, &lt;enum&gt; >> field to reuse definition of
 <<fields-int, &lt;int&gt; >>, only other <<fields-enum, &lt;enum&gt; >>.
 
+[[fields-common-forcing-generation]]
 ==== Forcing Generation ====
 By default the code generator is expected not to generate code for fields that
 are not <<intro-references, referenced>> by any other field to reduce amount of 
@@ -82,6 +86,7 @@ has been introduced with <<intro-boolean, boolean>> value.
 </schema>
 ----
 
+[[fields-common-display-properties]]
 ==== Display Properties ====
 **CommsDSL** supports generation of not only field's serialization and
 value access functionality, but also of various GUI protocol visualization, debugging and
@@ -141,6 +146,7 @@ value.
 </schema>
 ----
 
+[[fields-common-versioning]]
 ==== Versioning ====
 **CommsDSL** allows providing an information in what version the field was added
 to a particular message, as well as in what version it was deprecated, and whether
@@ -180,6 +186,7 @@ should be ignored by the code generator. It is allowed when field is a member
 of a <<messages-messages, &lt;message&gt; >> or a 
 <<fields-bundle, &lt;bundle&gt; >> field.
 
+[[fields-common-failing-read-of-the-field-on-invalid-value]]
 ==== Failing Read of the Field on Invalid Value ====
 Some fields may specify what values are considered to be valid, and there may
 be a need to fail the **read** operation in case the received value is invalid.
@@ -217,6 +224,7 @@ operation doesn't fail. The **order** property of the message specifies in
 what order the messages with the same ID must be read. It described in more
 detail in <<messages-messages, Messages>> chapter.
 
+[[fields-common-pseudo-fields]]
 ==== Pseudo Fields ====
 Sometimes there may be a need to have "psuedo" fields, which are implemented
 using proper field abstration, and are handled as
@@ -235,6 +243,7 @@ value.
 </schema>
 ----
 
+[[fields-common-customizable-fields]]
 ==== Customizable Fields ====
 The code generator is expected to allow some level of compile time customization of the 
 generated code, such as choosing different data structures and/or adding/replacing
@@ -254,6 +263,7 @@ customizable at compile time.
 </schema>
 ----
 
+[[fields-common-semantic-type]]
 ==== Semantic Type ====
 Sometimes code generator may generate a bit different (or better) code for fields that are
 used for some particular purpose. To specify such purpose use **semanticType**
@@ -289,4 +299,44 @@ was introduced in **v2.0** of **CommsDSL** specification.
 </schema>
 ----
 
+[[fields-common-ensuring-overriding-code-injection]]
+==== Ensuring Overriding Code Injection  ====
+In some cases the **CommsDSL** may be insufficient to express the the protocol definition in correct way.
+To handle such cases the code generator is expected to allow injection of the custom / overriding code
+to fix / replace the default implementation produced by the code generator. 
+
+The code generator is expected to generate the default code for the following operations:
+
+* **read** - Reading the field's value from the input buffer
+* **write** - Writing the field's value to the output buffer
+* **refresh** - Update the field's value to a consistent state. Note, that in some composite fields 
+like <<fields-bundle, &lt;bundle&gt; >>, values of internal members may depend on the values of other 
+members creating the situation where the whole state may be inconsistent / incorrect. Invoking the **refresh** operation is 
+expected to update the values of inner fields to bring the field into a consistent / correct state.
+* **length** - Returning the serialization length of the field.
+* **valid** - Returning whether the field's value is valid.
+* **name** - Returning the name of the field.
+
+To ensure visibility and/or help with forcing the injection of the correcting code, 
+the **CommsDSL** provides the following <<intro-properties, properties>>.
+
+* **readOverride** - Control the overriding code injection for the **read** operation.
+* **writeOverride** - Control the overriding code injection for the **write** operation.
+* **refreshOverride** - Control the overriding code injection for the **refresh** operation.
+* **lengthOverride** - Control the overriding code injection for the **length** operation.
+* **validOverride** - Control the overriding code injection for the **valid** operation.
+* **nameOverride** - Control the overriding code injection for the **name** operation.
+
+The value of the <<intro-properties, properties>> above can be one of the following:
+
+* **any** (default) - Inject the overriding code if available, use the code generated by default if not.
+* **replace** - The default code code is incorrect, replace it with the provided one. The code generator is expected
+to report a failure in case the overriding code is unavailable.
+* **extend** - The default code is still incorrect, but the code generator is expected to 
+produce it (with some minor renaming to avoid names clashing) to allow reuse and extension in the overriding code. The code generator is expected
+to report a failure in case the overriding code is unavailable.
+* **none** - The code generator is expected to ignore the overriding code if it's available and 
+produce the one generated by default instead.
+
+
 Use <<appendix-fields, properties table>> for future references.

fields/data.adoc

@@ -13,8 +13,11 @@ as well as extra properties and elements described below.
 </schema>
 ----
 Such definition of the **&lt;data&gt;** does **NOT** have any limit on
-the length of the data, and will consume all the available bytes in the 
-input buffer.
+the length of the data, and will consume all the __available__ bytes in the 
+input buffer. Note, that the __available__ bytes may be bound by the 
+<<frames-size, &lt;size&gt; >> layer of the <<frames-frames, &lt;frame&gt; >> or 
+other means. It does NOT necessarily mean that all the bytes received from the I/O link
+will be consumed.
 
 ==== Default Value ====
 The default value of the **&lt;data&gt;** field when constructed can be specified

fields/enum.adoc

@@ -142,6 +142,29 @@ use **length** property to specify custom serialization length.
 <<fields-enum-underlying-type, underlying type>> (**intvar** and **uintvar**), 
 it means **maximum** allowed length.
 
+Some protocols allow (de)serialization of the **&lt;enum&gt;** field value having less bytes
+in the buffer than is required by its <<fields-enum-underlying-type, underlying type>> serialization. 
+Usually the length of such an **&lt;enum&gt;**
+field is specified externally, by the preceding "length" field in TLV (type-length-value) triplet
+(explained in detail in <<fields-variant, &lt;variant&gt; Field>> section). In order to allow code
+generator to handle such cases use **availableLengthLimit** <<intro-properties, property>>
+with <<intro-boolean, boolean>> value.
+[source,xml]
+----
+<?xml version="1.0" encoding="UTF-8"?>
+<schema name="MyProtocol" endian="big">
+    <fields>
+        <enum name="SomeEnumField" type="uint32" availableLengthLimit="true">
+            <validValue name="Val1" val="0x0" />
+            <validValue name="Val2" val="0x0a0b" />
+            <validValue name="Val3" val="0xffffff" />
+        </enum>
+    </fields>
+</schema>
+----
+Note, that the code generator is responsible to allow external limiting serialization length for such fields
+before **write** operation takes place.
+
 ==== Length in Bits ====
 **&lt;enum&gt;** field can be a member of <<fields-bitfield, &lt;bitfield&gt; >> field.
 In this case the serialization length may be specified in bits using **bitLength**