Merge pull request #48 from M0r13n/iterative-decoding

Iterative decoding

CHANGELOG.txt

@@ -1,6 +1,24 @@
 ====================
  pyais CHANGELOG
 ====================
+-------------------------------------------------------------------------------
+ Version 2.0.0-alpha  6 Feb 2022
+-------------------------------------------------------------------------------
+
+* WARNING: The v2 release will introduce breaking changes
+* Introduces the possibility to encode messages
+* decoding has been rewritten and implements an iterative decoding approach
+* The following fields were renamed:
+    * message_fragments -> frag_cnt
+    * fragment_number -> frag_num
+    * message_id -> seq_id
+    * type -> msg_type
+    * shiptype -> ship_type
+* `msg.decode()` does not return a `pyais.messages.AISMessage` instance anymore
+  * instead an instance of `pyais.messages.MessageTypeX` is returned, where `X` is the type of the message (1-27)
+* in v1 you called `decoded.content` to get the decoded message as a dictionary - this is now `decoded.asdict()`
+
+
 -------------------------------------------------------------------------------
  Version 1.6.2  2 May 2021
 -------------------------------------------------------------------------------

Makefile

@@ -14,4 +14,11 @@ check-build:
 type-check:
 	mypy ./pyais
 
+clean:
+	rm -rf .mypy_cache
+	rm -rf build
+	rm -rf dist
+	rm coverage.xml
+	rm .coverage
+
 test: run_tests flake type-check

README.md

@@ -30,6 +30,30 @@ the [AIS standard](https://en.wikipedia.org/wiki/Automatic_identification_system
 I open to any form of idea to further improve this library. If you have an idea or a feature request - just open an
 issue. :-)
 
+# Migrating from v1 to v2
+
+Version 2.0.0 of pyais had some breaking changes that were needed to improve the lib. While I tried to keep those
+breaking changes to a minimum, there are a few places that got changed.
+
+* `msg.decode()` does not return a `pyais.messages.AISMessage` instance anymore
+  * instead an instance of `pyais.messages.MessageTypeX` is returned, where `X` is the type of the message (1-27)
+* in v1 you called `decoded.content` to get the decoded message as a dictionary - this is now `decoded.asdict()`
+
+### Typical example in v1
+
+```py
+message = NMEAMessage(b"!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
+decoded = message.decode()
+data = decoded.content
+```
+
+### Typical example in v2
+```py
+message = NMEAMessage(b"!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
+decoded = message.decode()
+data = decoded.asdict()
+```
+
 # Installation
 
 The project is available at Pypi:
@@ -40,58 +64,77 @@ $ pip install pyais
 
 # Usage
 
-Using this module is easy. If you want to parse a file, that contains AIS messages, just copy the following code and
-replace `filename` with your desired filename.
-
-```python
-from pyais import FileReaderStream
+U Decode a single part AIS message using `decode()`::
 
-filename = "sample.ais"
+```py
+from pyais import decode
 
-for msg in FileReaderStream(filename):
-    decoded_message = msg.decode()
-    ais_content = decoded_message.content
+decoded = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+print(decoded)
 ```
 
-It is possible to directly convert messages into JSON.
+The `decode()` functions accepts a list of arguments: One argument for every part of a multipart message::
+
+```py
+from pyais import decode
 
-```python
-from pyais import TCPStream
+parts = [
+    b"!AIVDM,2,1,4,A,55O0W7`00001L@gCWGA2uItLth@DqtL5@F22220j1h742t0Ht0000000,0*08",
+    b"!AIVDM,2,2,4,A,000000000000000,2*20",
+]
 
-for msg in TCPStream('ais.exploratorium.edu'):
-    json_data = msg.decode().to_json()
+# Decode a multipart message using decode
+decoded = decode(*parts)
+print(decoded)
 ```
 
-You can also parse a single message encoded as bytes or from a string:
+Also the `decode()` function accepts either strings or bytes::
 
-```python
-from pyais import NMEAMessage, decode_msg
+```py
+from pyais import decode
 
-message = NMEAMessage(b"!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
-message = NMEAMessage.from_string("!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
+decoded_b = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+decoded_s = decode("!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+assert decoded_b == decoded_s
+```
 
-# or newer 
+Decode the message into a dictionary::
 
-msg = decode_msg("!AIVDM,1,1,,A,403Ovl@000Htt<tSF0l4Q@100`Pq,0*28")
-msg = decode_msg(b"!AIVDM,1,1,,A,403Ovl@000Htt<tSF0l4Q@100`Pq,0*28")
+```py
+from pyais import decode
 
+decoded = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+as_dict = decoded.asdict()
+print(as_dict)
 ```
 
-See the example folder for more examples.
-
-Another common use case is the reception of messages via UDP. This lib comes with an `UDPStream` class that enables just
-that. This stream class also handles out-of-order delivery of messages, which can occur when using UDP.
+Read a file::
 
 ```py
-from pyais.stream import UDPStream
+from pyais.stream import FileReaderStream
 
-host = "127.0.0.1"
-port = 55555
+filename = "sample.ais"
 
-for msg in UDPStream(host, port):
-    msg.decode()
-    # do something with it
+for msg in FileReaderStream(filename):
+    decoded = msg.decode()
+    print(decoded)
+```
+
+Decode a stream of messages (e.g. a list or generator)::
 
+```py
+from pyais import IterMessages
+
+fake_stream = [
+    b"!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23",
+    b"!AIVDM,1,1,,A,133sVfPP00PD>hRMDH@jNOvN20S8,0*7F",
+    b"!AIVDM,1,1,,B,100h00PP0@PHFV`Mg5gTH?vNPUIp,0*3B",
+    b"!AIVDM,1,1,,B,13eaJF0P00Qd388Eew6aagvH85Ip,0*45",
+    b"!AIVDM,1,1,,A,14eGrSPP00ncMJTO5C6aBwvP2D0?,0*7A",
+    b"!AIVDM,1,1,,A,15MrVH0000KH<:V:NtBLoqFP2H9:,0*2F",
+]
+for msg in IterMessages(fake_stream):
+    print(msg.decode())
 ```
 
 ## Encode
@@ -139,10 +182,10 @@ encoded = encode_dict(data, radio_channel="B", talker_id="AIVDM")[0]
 It is also possible to create messages directly and pass them to `encode_payload`.
 
 ```py
-from pyais.encode import MessageType5, encode_payload
+from pyais.encode import MessageType5, encode_msg
 
 payload = MessageType5.create(mmsi="123", shipname="Titanic", callsign="TITANIC", destination="New York")
-encoded = encode_payload(payload)
+encoded = encode_msg(payload)
 print(encoded)
 ```
 

docs/examples/encode.rst

@@ -2,9 +2,24 @@
 Encode AIS messages
 #############################
 
-It is also possible to encode messages using pyais.
-Currently, this library supports creating NMEA formatted AIS type messages from type 1 to type 10. Support for other types
-is planned.
+The full specification of the AIVDM/AIVDO protocol is out of the scope of this example.
+For a good overview of the AIVDM/AIVDO Sentence Layer please refer to this project: https://gpsd.gitlab.io/gpsd/AIVDM.html#_aivdmaivdo_sentence_layer
+
+But you should keep the following things in mind:
+
+- AIS messages are part of a two layer protocol
+- the outer layer is the NMEA 0183 data exchange format
+- the actual AIS message is part of the NMEA 0183’s 82-character payload
+- because some AIS messages are larger than 82 characters they need to be split across several fragments
+- there are 27 different types of AIS messages which differ in terms of fields
+
+Now to the actual encoding of messages: It is possible to encode a dictionary of values into an AIS message.
+To do so, you need some values that you want to encode. The keys need to match the interface of the actual message.
+You can call `.fields()` on any message class, to get glimpse on the available fields for each message type.
+Unknown keys in the dict are simply omitted by pyais. Most keys have default values and do not need to
+be passed explicitly. Only the keys `type` and `mmsi` are always required
+
+For the following example, let's assume that we want to create a type 1 AIS message.
 
 Examples
 ----------

docs/examples/file.rst

@@ -6,17 +6,15 @@ Reading and parsing files
 Examples
 --------
 
-Parse a file::
+The following example shows how to read and parse AIS messages from a file::
 
     from pyais.stream import FileReaderStream
 
     filename = "sample.ais"
 
     for msg in FileReaderStream(filename):
-        decoded_message = msg.decode()
-        ais_content = decoded_message.content
-        # Do something with the ais message
-
+        decoded = msg.decode()
+        print(decoded)
 
 Please note, that by default the following lines are ignored:
 

docs/examples/single.rst

@@ -14,41 +14,65 @@ References
 Examples
 --------
 
-The newest version of Pyais introduced a more convinenient method to decode messages: `decode_msg`::
+Decode a single part AIS message using `decode()`::
 
-    from pyais import decode_msg
-    decode_msg(b"!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
-    # => {'type': 1, 'repeat': 0, 'mmsi': '366053209', 'status': <NavigationStatus.RestrictedManoeuverability: 3>, 'turn': 0, 'speed': 0.0, 'accuracy': 0, 'lon': -122.34161833333333, 'lat': 37.80211833333333, 'course': 219.3, 'heading': 1, 'second': 59, 'maneuver': <ManeuverIndicator.NotAvailable: 0>, 'raim': 0, 'radio': 2281}
+    from pyais import decode
+    decoded = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+    print(decoded)
 
-    # or
-    decode_msg("!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
-    # => {'type': 1, 'repeat': 0, 'mmsi': '366053209', 'status': <NavigationStatus.RestrictedManoeuverability: 3>, 'turn': 0, 'speed': 0.0, 'accuracy': 0, 'lon': -122.34161833333333, 'lat': 37.80211833333333, 'course': 219.3, 'heading': 1, 'second': 59, 'maneuver': <ManeuverIndicator.NotAvailable: 0>, 'raim': 0, 'radio': 2281}
+The `decode()` functions accepts a list of arguments: One argument for every part of a multipart message::
 
+    from pyais import decode
 
-    # or decode a multiline message
-    decode_msg(
-            b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07',
-            b'!AIVDM,2,2,1,A,F@V@00000000000,2*35',
-        )
-    # => {'type': 5, 'repeat': 0, 'mmsi': '210035000', 'ais_version': 0, 'imo': 9514755, 'callsign': '5BXT2', 'shipname': 'NORDIC HAMBURG', 'shiptype': <ShipType.Cargo_HazardousCategory_A: 71>, 'to_bow': 142, 'to_stern': 10, 'to_port': 11, 'to_starboard': 11, 'epfd': <EpfdType.GPS: 1>, 'month': 7, 'day': 20, 'hour': 5, 'minute': 0, 'draught': 7.1, 'destination': 'CTT-LAYBY', 'dte': 0}
+    parts = [
+        b"!AIVDM,2,1,4,A,55O0W7`00001L@gCWGA2uItLth@DqtL5@F22220j1h742t0Ht0000000,0*08",
+        b"!AIVDM,2,2,4,A,000000000000000,2*20",
+    ]
 
+    # Decode a multipart message using decode
+    decoded = decode(*parts)
+    print(decoded)
 
-.. warning::
 
-   **Please note**, that `decode_msg` is only meant to decode a single message.
-   You **can not** use it to decode multiple messages at once.
-   But it supports multiline messages
+Also the `decode()` function accepts either strings or bytes::
 
+    decoded_b = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+    decoded_s = decode("!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+    assert decoded_b == decoded_s
 
-Decode a single message (bytes)::
+Decode the message into a dictionary::
 
-    message = NMEAMessage(b"!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
-    print(message.decode())
-    # => {'type': 1, 'repeat': 0, 'mmsi': '366053209', 'status': <NavigationStatus.RestrictedManoeuverability: 3>, 'turn': 0, 'speed': 0.0, 'accuracy': 0, 'lon': -122.34161833333333, 'lat': 37.80211833333333, 'course': 219.3, 'heading': 1, 'second': 59, 'maneuver': <ManeuverIndicator.NotAvailable: 0>, 'raim': 0, 'radio': 2281}
+    decoded = decode(b"!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+    as_dict = decoded.asdict()
+    print(as_dict)
 
+Decode the message into a serialized JSON string::
 
-Decode a single message (str)::
+    decoded = decode("!AIVDM,1,1,,B,15NG6V0P01G?cFhE`R2IU?wn28R>,0*05")
+    json = decoded.to_json()
+    print(json)
 
-    message = NMEAMessage.from_string("!AIVDM,1,1,,B,15M67FC000G?ufbE`FepT@3n00Sa,0*5C")
-    print(message.decode())
-    # => {'type': 1, 'repeat': 0, 'mmsi': '366053209', 'status': <NavigationStatus.RestrictedManoeuverability: 3>, 'turn': 0, 'speed': 0.0, 'accuracy': 0, 'lon': -122.34161833333333, 'lat': 37.80211833333333, 'course': 219.3, 'heading': 1, 'second': 59, 'maneuver': <ManeuverIndicator.NotAvailable: 0>, 'raim': 0, 'radio': 2281}
+Read a file::
+
+    from pyais.stream import FileReaderStream
+
+    filename = "sample.ais"
+
+    for msg in FileReaderStream(filename):
+        decoded = msg.decode()
+        print(decoded)
+
+Decode a stream of messages (e.g. a list or generator)::
+
+    from pyais import IterMessages
+
+    fake_stream = [
+        b"!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23",
+        b"!AIVDM,1,1,,A,133sVfPP00PD>hRMDH@jNOvN20S8,0*7F",
+        b"!AIVDM,1,1,,B,100h00PP0@PHFV`Mg5gTH?vNPUIp,0*3B",
+        b"!AIVDM,1,1,,B,13eaJF0P00Qd388Eew6aagvH85Ip,0*45",
+        b"!AIVDM,1,1,,A,14eGrSPP00ncMJTO5C6aBwvP2D0?,0*7A",
+        b"!AIVDM,1,1,,A,15MrVH0000KH<:V:NtBLoqFP2H9:,0*2F",
+    ]
+    for msg in IterMessages(fake_stream):
+        print(msg.decode())

docs/examples/sockets.rst

@@ -8,27 +8,27 @@ Examples
 
 Connect to a TCP socket::
 
-    from pyais.stream import TCPStream
+    from pyais.stream import TCPConnection
 
-    url = 'ais.exploratorium.edu'
-    port = 80
+    url = '127.0.0.1'
+    port = 12346
 
-    for msg in TCPStream(url, port=80):
+    for msg in TCPConnection(url, port=port):
         decoded_message = msg.decode()
-        ais_content = decoded_message.content
+        ais_content = decoded_message
         print(ais_content)
-        # Do something with the ais message
+        # Do something with the AIS message
 
 
-Connect to a UDP socket::
+Open to a UDP socket::
 
-    from pyais.stream import UDPStream
+    from pyais.stream import UDPReceiver
 
     host = "127.0.0.1"
-    port = 55555
+    port = 12346
 
-    for msg in UDPStream(host, port):
-        msg.decode()
+    for msg in UDPReceiver(host, port):
+        print(msg.decode())
         # do something with it
 
 The UDP stream handles out of order delivery of messages. By default it keeps the last up to 10.000 messages in memory to search for multiline messages.