Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

MicBoucinha · 2025-07-18T10:03:29Z

Important

We started working on this PR by merging the contributions made in PR#1

Refactoring & Project Structure

Refactored enums, class members, and function naming for consistency with official Harp naming conventions
Changed namespace to harp and sub-namespace reorganization:
- harp.communication for communication-related modules
- harp.protocol for core protocol definitions
- harp.devices for device-specific implementations (using PEP420 for implicit namespace packages)

Features & Enhancements

Updated Harp protocol support with missing features
- Added support for device serial number and missing registers
- Improved message handling (arrays, timestamps, error detection) for different payloads
Updated currently known Harp devices names
Added context manager support to Device
Added static factory method HarpMessage.create
Changed dump file convention to be at the Device level instead of message level
Removed ReadXXXHarpMessage and WriteXXXHarpMessage classes and ReadXXX and WriteXXX static methods from HarpMessage and moved them to Device
Removed existing behavior driver because of new harp.behavior namespace package

Documentation

Added MkDocs documentation for API, and project structure, and per-device documentation
Use numpy style convention for docstrings documentation
Added docstrings to all public methods and classes
Moved and updated existing examples to docs folder
Created GitHub Action to generate documentation (manually triggered)

Documentation here: https://fchampalimaud.github.io/pyharp

Testing

Updated some tests as needed

Technical

Convert project to uv

Contributors

Major contributions by Luís Teixeira and José Grilo

harp.devices

Automatically generated python package for each device and respective documentation
Added harp.devices namespace package for device-specific implementations

Repository here: https://github.com/fchampalimaud/harp.devices

As mentioned in issue #2, we were really interested in developing the Python Harp ecosystem further.

Our starting point was PR #1, which already had pushed pyharp forward (e.g., by implementing a way to deal with the device's events), and try to build on top of that. At first, we mainly focused on completing what was missing regarding the protocol (e.g., add a generic way to deal with the different payload types) and the interaction with a generic Harp device (e.g., add methods for the missing common registers).

Our goal was to have a simple enough but versatile pyharp interface and we based our rationale to be as similar as possible as the Bonsai.Harp implementation. One way to achieve that was to get inspiration from the CreateMessage Bonsai node to implement the HarpMessage.create static method. Additionally, we added methods for the Device class that allow the users to read/write from/to registers that have different payload types.

Another thing we tried to focus on was documentation. It surely can be improved but, at the moment, we documented every function and class of the project in order to facilitate the API usage and adapted/added some minimal examples that should help the user get started with the pyharp package.

Beyond being possible to connect and interact with a generic Harp device, we also wanted to generate device-specific interfaces. On the one hand, we would like that each Harp device had its own package, so that if, for example, a user just wants to use Harp Behavior, it doesn't need to import a package that contains the interfaces for every board. This would also help when there's individual updates on each device. On the other hand, we would like that every package (including pyharp "core") had the same feeling, in the sense the user would get the implicit idea that every package is part of the same ecosystem. In order to achieve that, we organized the project directories according to PEP420, in which the global ecosystem has the following structure:

harp/
├── communication/
├── data (harp-python content hopefully)
├── protocol/
└── devices/
    ├── behavior/
    ├── device_a/
    ├── ...
    └── device_z/

We also wanted to have a way to generate the device-specific packages automatically, so that we could easily add new devices and keep the documentation up-to-date. For that, we used reflex-generator to generate the device-specific packages based on the device's YAML. Currently, the generated packages are available in the harp.devices repository mentioned above, but we believe that the Python interface should reside in each device's repository. PR#86 to reflex-generator has been submitted to support the Python package generation.

Finally, we abandoned the pyharp name for the package because, unfortunately, someone registered a package in PyPI with the same name. The package name is currently harp-protocol. At the same time, and mostly to keep coherence with the namespace naming convention used on C#/Bonsai side, we decided to change the global namespace to harp.

Known limitations of the PEP420 usage

To allow the usage of PEP420, we cannot have members under harp or harp.devices that are not namespace packages. This means that we cannot have a __init__.py file under those directories. As a consequence, we cannot have any code directly in those paths, which means that we cannot have any imports there.

This also means that the harp-python package cannot currently be installed in the same venv until some changes happen there. Our suggestion at the time was to place it's contents under harp.data and then import it from there. This would allow the harp-python package to be installed in the same venv as harp-protocol and the individual harp devices. We also submitted a PR to the harp-python repository to reflect this change in PR#52.

With these new PRs, we aim to expand the current features of the Python implementation and make it more accessible for new users, helping them adopt and engage with the HARP ecosystem. We also see value in consolidating the Python implementation within the official harp-tech repository, and we’re more than happy to help maintain it. This would help avoid fragmented development efforts and lead to a more unified and robust solution for the community. We will continue iterating on these developments and commit new features to this PR while we work toward a consensus on merging it into the harp-tech repository.

Dev/editable mode

Added ReadS32 and refactored WriteS32 to handle array inputs

bruno-f-cruz

Thanks for taking a stab at this. I left some first pass feedback.
I think we should discuss a bit the architecture of the device API synchronously during one of the meetings. I dont have a specific issue with it, but I am just afraid it may become hard to maintain if each register has its own specific read/write method with a name given without any sort of consistency.

bruno-f-cruz · 2025-10-01T14:14:31Z

.github/workflows/build-docs.yml

+      - name: Clone harp.devices repository
+        uses: actions/checkout@v4
+        with:
+          repository: fchampalimaud/harp.devices


I am not sure the package repository should be coupled with the generation of docs for the devices.

In the long term, we could consider to separate the docs generation from the packages in different repos. The current solution is not set in stone, but we would like to keep the documentation for the generic device and for the specific devices together.

bruno-f-cruz · 2025-10-01T14:14:58Z

.gitignore

.vscode should be added to the list

The reason we keep the .vscode directory in the repo is because it has the extensions (and respective settings) we actively use for development and that we would like for contributors to use as well. Do you have an alternative suggestion regarding this matter?

I would not include it at all since vscode is not required for development and even if people use vscode they may use different extensions. Anything that you want contributors to follow should be enforced via standard metadata files and linting (e.g. pyproject.toml and github actions)

bruno-f-cruz · 2025-10-01T14:19:42Z

LICENSE

This should be changed to harp-tech, similar to what happens with the other software packages

We can change it when the move of the repo to harp-tech is imminent.

10-harp.rules

bruno-f-cruz · 2025-10-01T15:41:12Z

LICENSE

@@ -1,6 +1,7 @@
 MIT License

 Copyright (c) 2020 OEPS & Filipe Carvalho


We should consider making this standard with the copyright in the other software packages.

https://github.com/harp-tech/harp-python/blob/3d97d0b800533e4821896f744e52569c6413c6f7/LICENSE#L3

Sure, same as answered previously,

We can change it when the move of the repo to harp-tech is imminent.

src/harp-serial/harp/serial/device.py

bruno-f-cruz · 2025-10-01T16:59:47Z

src/harp-serial/harp/serial/device.py

+        # open file if it is defined
+        if self._dump_file_path is not None:
+            self._dump_file = open(self._dump_file_path, "ab")


Not sure I follow how you would want to use this.
I can see cases where users may want to skip querying the device for metadata, but this suggests that a valid use-case is to cache it?

We have to look at this more carefully. We can discuss it further.

bruno-f-cruz · 2025-10-01T17:05:11Z

src/harp-serial/harp/serial/device.py

+        reply = self.send(HarpMessage.create(MessageType.READ, address, PayloadType.U8))
+        return OperationMode(reply.payload & OperationCtrl.OP_MODE)
+
+    def dump_registers(self) -> list:


I think we should discuss a bit this API synchronously during one of the meetings. I dont dislike this but it has the potential to grow a bit out of control.

This was only for this specific register where we don't know yet how to constrain the user's input regarding the OP_MODE to be of type OperationCtrl. In terms of usage, for this particular register, having the separated method made more sense. But we should discuss it further.

bruno-f-cruz · 2025-10-01T17:07:09Z

.pre-commit-config.yaml

@@ -0,0 +1,22 @@
+repos:


I would suggest turning this into an action instead of a pre-commit.

Why an action only?
I might be missing something, but personally, I prefer to have checks locally before committing when possible.
We can discuss this further.

Pre-commits are something we need to rely on people installing. GitHub actions are something we can enforce during PRs.
Pre-commits can be bypassed locally, github actions can't
I can't verify the output of a pre-commit hook that runs in your PC, but I can verify the log of a GitHub action.

You are right that doesnt need to be an action "only", but if we are maintaining one thing, let it be the github action.

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

…o feature/pr_review_changes

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

bruno-f-cruz

A few more notes to go over during the SRM meeting

bruno-f-cruz · 2025-10-29T17:41:53Z

.gitignore

I would not include it at all since vscode is not required for development and even if people use vscode they may use different extensions. Anything that you want contributors to follow should be enforced via standard metadata files and linting (e.g. pyproject.toml and github actions)

bruno-f-cruz · 2025-10-29T17:43:18Z

.gitignore

+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule


I am wondering where this .gitignore is coming from. It seems quite opinionated. Is there a reason to exclude so much stuff? Are you really expecting people to use pdm, celery and flask to develop this library?

bruno-f-cruz · 2025-10-29T17:46:43Z

.pre-commit-config.yaml

@@ -0,0 +1,22 @@
+repos:


Pre-commits are something we need to rely on people installing. GitHub actions are something we can enforce during PRs.
Pre-commits can be bypassed locally, github actions can't
I can't verify the output of a pre-commit hook that runs in your PC, but I can verify the log of a GitHub action.

You are right that doesnt need to be an action "only", but if we are maintaining one thing, let it be the github action.

bruno-f-cruz · 2025-10-29T17:48:48Z

mkdocs.yml

+            show_source: false
+            extensions:
+              - griffe_fieldz
+  - git-committers:


I thought you had to list each contribuitor individually, but it seems the extension uses the repo metadata to do it for you? Ignore this if that is indeed the case.

bruno-f-cruz · 2025-10-29T17:50:11Z

src/harp-protocol/harp/protocol/base.py

@@ -0,0 +1,280 @@
+from datetime import datetime
+from enum import IntEnum, IntFlag
+


Maybe a simple:

__PROTOCOL_VERSION__ = 1.15

bruno-f-cruz · 2025-10-29T18:01:51Z

src/harp-serial/harp/serial/device.py

@@ -0,0 +1,1347 @@
+from __future__ import annotations  # for type hints (PEP 563)


Remove comment

bruno-f-cruz · 2025-10-29T18:04:07Z

src/harp-serial/harp/serial/device.py

+        The serial number of the device
+    """
+
+    WHO_AM_I: int


My worry is that you are representing the same thing in two different places. Seems super error prone if we ever update.

bruno-f-cruz · 2025-10-29T18:06:19Z

src/harp-serial/harp/serial/device.py

+    TIMESTAMP_OFFSET: int
+
+    _ser: HarpSerial
+    _dump_file_path: Optional[Path]


Is this actually being used anywhere?

I like the idea of having a logger that makes everything log in the standard format. But this should be decoupled from the device class.
Maybe lets remove this implementation for now?

bruno-f-cruz · 2025-10-29T18:09:31Z

src/harp-serial/harp/serial/device.py

+
+        return reply is not None
+
+    def reset_device(


We should consider removing these syntactic sugar methods and just create message constructors for the core registers instead. Something closer to what we have in the csharp api.

In other words:

reset_message = ResetMessage(...) device.send(reset_message)

Seems a bit more scalable in cases where the protocol changes.

bruno-f-cruz · 2025-10-29T18:11:01Z

src/harp-serial/harp/serial/device.py

+        if self._dump_file:
+            self._dump_file.write(reply)
+
+    def get_events(self) -> list[HarpMessage]:


This is grabing not only events but also other messages in the queue, no?

bruno-f-cruz · 2025-10-30T17:38:18Z

src/harp-serial/harp/serial/device.py

+    def load(self) -> None:
+        """
+        Loads the data stored in the device's common registers.
+        """


class CoreRegisters # This should be generated from the core.yml in theory WHOAMI = RegConfiguration for reg in CoreRegisters: device.read(reg)

bruno-f-cruz · 2025-10-30T17:40:31Z

src/harp-serial/harp/serial/device.py

+        if self._dump_file_path is not None:
+            self._dump_file = open(self._dump_file_path, "ab")
+
+    def disconnect(self) -> None:


refactor to match serial and call it close()

bruno-f-cruz · 2025-10-30T17:57:43Z

src/harp-serial/harp/serial/device.py

+
+        return reply
+
+    def send(


# Device knows how to read and write Register types class Device: def write(Register): # Register can be core or application specific def read(Register): def request/send/command(HarpMessage) class HarpBehavior(Device): ... # Registers have a reg and know how to parse/compose payloads from value class RegisterPayload: _reg = 32 def make_payload(...) # The API thus looks like: behavior.write(DigitalOutputSetPayload(value)) -> DigitalOutpuSetPayload behavior.write(RegisterPayload<T>) -> RegisterPayload<T>

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Poofjunior and others added 30 commits April 13, 2022 17:02

can now pip instal in editable mode

be0c7a0

updating readme

ef9936e

fixing readme bug

69e11f0

adding udev rules

4fff7ba

device_names is now a default dict; examples now also work on linux

d974ee4

fixing pdf link; tweaking serial port to use ftdi rule

fbdacbd

Update README.md

33d8cc8

PEP 563 return type hints

0191b23

grand messages.py refactor

6a73a99

more message.py refactor

4108f6b

adding harp rules

e3457bb

adding simple behavior io stuff

455de81

adding prelim behavior driver

4e5e58f

adding a string representation

0e47767

added __str__ and active mode script

43629c0

printing DI event changes works

edfc361

updating how payload is parsed

d5d835a

adding enabling of events

3a122a3

adding inWaiting to reads

2fc5c58

read dumped registers

3f7dfba

expose more core features

401bd36

Merge pull request #1 from AllenNeuralDynamics/dev/editable_mode

ac57cb3

Dev/editable mode

add WriteU32 and WriteS32

c2bba09

fix signed error in WriteS32

fb54476

add debug level logging.

e7900d4

Added ReadS32 and refactored WriteS32 to handle array inputs

1b1e833

Merge pull request harp-tech#2 from jessy-liao/master

325769c

Added ReadS32 and refactored WriteS32 to handle array inputs

Add threaded serial port

368e346

clean up formatting changes

2381e8f

add event_count method and update test

9e30545

bruno-f-cruz requested a review from glopesdev October 1, 2025 14:11

bruno-f-cruz requested changes Oct 1, 2025

View reviewed changes

MicBoucinha and others added 23 commits October 8, 2025 15:50

Remove unnecessary udev rules file

d7a6a63

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor HARP_VERSION to CORE_VERSION

d454897

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update documentation

4162bff

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor base.py enums for consistency and clarity

6e714ec

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor HarpMessage to consolidate message creation

9a440b2

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Remove HarpMessage.create static method

ed8fca5

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor HarpMessage to remove ReplyHarpMessage

78cce0e

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor timeout variables

6f97c92

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Rename HarpTimeoutError to HarpTimeoutException and add context to it

1f24190

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update comment

36a8363

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Add HarpMessage context to Harp exceptions

e97480c

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Remove constraint when creating HarpMessage

8e84b3a

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Change build system

aa564c7

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Remove device_names

3d8d10d

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Handle potential serial connection errors

ccdef2a

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update minimum Python version

c24c51f

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Fix: write events to dump file

e1025d5

Merge remote-tracking branch 'origin/feature/dump_events_to_file' int…

5676d38

…o feature/pr_review_changes

Refactor to improve message reading and timeout handling in Device

1fbfb2d

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Separate docs from dev dependencies

5f62673

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Bump version of packages

c61184b

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update documentation

7523b47

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Merge branch 'feature/pr_review_changes' into develop

ede6aa4

bruno-f-cruz requested changes Oct 29, 2025

View reviewed changes

bruno-f-cruz reviewed Oct 30, 2025

View reviewed changes

Update docs

6f5cde3

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

		@@ -1,6 +1,7 @@
		MIT License

		Copyright (c) 2020 OEPS & Filipe Carvalho

		@@ -0,0 +1,280 @@
		from datetime import datetime
		from enum import IntEnum, IntFlag

		@@ -0,0 +1,1347 @@
		from __future__ import annotations # for type hints (PEP 563)

Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

Are you sure you want to change the base?

Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

Uh oh!

Conversation

MicBoucinha commented Jul 18, 2025

Refactoring & Project Structure

Features & Enhancements

Documentation

Testing

Technical

Contributors

harp.devices

Known limitations of the PEP420 usage

Uh oh!

bruno-f-cruz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bruno-f-cruz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bruno-f-cruz Oct 30, 2025 •

edited

Loading