feat: Improve onyxsdk-pen integration #1452

zinstack625 · 2025-03-19T23:06:37Z

Updated onyxsdk dependencies

Makes stuff work on some newer devices. Don't really know why, but before that, stylus input resulted in full refreshes, as if there were no support for that altogether. This was observed both on Play Store and on local builds. May, but should not break other devices support

Reworked OnyxsdkPenArea.kt

Removed explicit drawing
Much of the stuff done in drawPreview had absolutely no effects. Whole magic seems to happen in TouchHelper. Removing that manual drawing, again, may break stuff on other devices, but most of the logic remained nonetheless. In case of breakages, should be trivial to bring that stuff back, or, if my intuition is correct, further simplify this class. Would really like to hear some feedback on that removed code if possible
Established message passing between OnyxsdkPenArea and Dart
In order to modify state of the PlatformView, message passing was required. It is used to update stroke parameters from Flutter UI toolbar to the PlatformView. I see the use as "refreshing" the creationParams. Might be useful in the future, both in of itself and as a precedent for other usecases
Mapped used tools to those in onyxsdk-pen
This in effect resulted in directly-refreshed preview looking very similar to the underlying image. Most of the tools are practically indistinguishable and are only insignificantly changed in shape (underlying image is less detailed) and, in some cases significantly in color during full refresh (seen on attached video). Exception being the pencil (seen on attached video), but it's still pretty close. Don't think it can get closer without some legally questionable voodoo magic with onyxsdk

Dart side

Most are due to preparation of tool parameters to Kotlin side, some are due to canvas being unaware of the tool being used. I tried to make significant changes as close to the actual use as possible, but may have introduced a couple of unnecessary abstractions and translations along the way. Could possibly be done better, but I'm inexperienced in Flutter and Dart. Hopefully the changes are not disruptive

Seems to be doing good on Note Air 4C, but would be good if someone could test on other devices. This does not seem to introduce any slowdowns, does not touch any significant logic and does not introduce any UI changes

20250320.webm

Great project BTW

codecov · 2025-03-19T23:45:38Z

Codecov Report

❌ Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 48.13%. Comparing base (5534312) to head (5911175).

Files with missing lines	Patch %	Lines
lib/components/canvas/canvas.dart	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1452      +/-   ##
==========================================
+ Coverage   47.99%   48.13%   +0.13%     
==========================================
  Files         118      118              
  Lines        8723     8746      +23     
==========================================
+ Hits         4187     4210      +23     
  Misses       4536     4536

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fe-Ti · 2025-03-20T20:32:10Z

Hi,

On my NoteAir 2 there is a huge lag (approx. 30s) after I lift my pen and before the actual stroke is rendered from scribble.

I tried to fix this issue and came up with this solution (same for eraser):

        override fun onBeginRawDrawing(b: Boolean, touchPoint: TouchPoint) {
            // begin of stylus data
            touchHelper.setRawDrawingRenderEnabled(true)
            reset()
        }

        override fun onEndRawDrawing(b: Boolean, touchPoint: TouchPoint) {
            // end of stylus data
            touchHelper.setRawDrawingRenderEnabled(false)
            reset()
            scheduleRefresh()
        }

With such toggling the lag disappeared and expirience became closer to proprietary note app.

Also there is an issue with releasing resources of touchHelper. For some reason the dispose() function is not called if I exit an app via navigation buttons/gestures (e.g. going to browser or launcher). So the pen continues to scribble all over the screen :)

zinstack625 · 2025-03-21T14:08:24Z

On my NoteAir 2 there is a huge lag (approx. 30s) after I lift my pen and before the actual stroke is rendered from scribble

Regarding this lag, this should be handled in scheduleRefresh(). Setting it too low may result in display refreshing too often, which in itself may be undesirable(?). 1 sec should be reasonable, but 30 secs is definitely unintended behavior. Perhaps forcing a refresh with EpdController is better than hoping it would on itself...

Also there is an issue with releasing resources of touchHelper. For some reason the dispose() function is not called if I exit an app via navigation buttons/gestures (e.g. going to browser or launcher). So the pen continues to scribble all over the screen :)

Can reproduce, dispose() is indeed not called on app minimization, and is even skipped on app close (however view is still destroyed and this sideeffect is gone). No method from PlatformView is called on this action. I guess the best option is to figure whether canvas is shown from Dart side (is there such a callback somewhere?) and pass it to Kotlin...

override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "updateStroke") {
        val params = call.arguments<Map<String, Any>?>()
        updateStroke(params)
        result.success(null)
    } else if (call.method == "disableDraw") {
        touchHelper.setRawDrawingEnabled(false)
    } else if (call.method == "enableDraw") {
        touchHelper.setRawDrawingEnabled(true)
    } else {
        result.notImplemented()
    }
}

Fe-Ti · 2025-03-21T15:11:13Z

I guess the best option is to figure whether canvas is shown from Dart side (is there such a callback somewhere?) and pass it to Kotlin...

In such way overriding dispose() in _OnyxSdkPenAreaState may help

  @override
  void dispose() {
    channel.invokeMethod('disableDraw', creationParams).catchError((e) {});
  }

I'm not sure if it doesn't break anything in onyxsdk package interface.

Iey4iej3 · 2025-04-05T08:09:29Z

I am unable to compile and test, but did you encounter #1459 before the fix?

zinstack625 · 2025-04-06T15:28:51Z

I am unable to compile and test, but did you encounter #1459 before the fix?

Kinda, in some circumstances, not really sure, might just be my writing. If it is there, it will probably still be there, I did not mess with the code that gets the stylus data that's used after the refresh. I however can not test anything before 800e450, stuff just straight up does not work for me until that (I have a note air 4c, which is probably too new for version used before that)

Iey4iej3 · 2025-04-07T06:38:11Z

Kinda, in some circumstances, not really sure, might just be my writing.

The most obvious strange thing is that, the top dot in "i" is thickened into a bullet. It does not look like this in the preview, but becomes this after a refresh.

Iey4iej3 · 2025-04-21T10:05:49Z

I am unable to compile and test, but did you encounter #1459 before the fix?

Kinda, in some circumstances, not really sure, might just be my writing. If it is there, it will probably still be there, I did not mess with the code that gets the stylus data that's used after the refresh. I however can not test anything before 800e450, stuff just straight up does not work for me until that (I have a note air 4c, which is probably too new for version used before that)

Actually, your patch seems to fix #1459. At least, it is much better. To clarify, it is not fixed by only upgrading onyxsdk as in 800e450. It is unclear why this patch fixes the issue. Thank you very much!

adil192

Really impressive, thank you! Just requesting some tiny changes

adil192 · 2025-05-12T08:18:03Z

packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt

+    enum class StrokeStyle(val Id: Int) {
+        FountainPen(0),
+        Pen(1),
+        Brush(2),
+        Pencil(3),
+        Marker(4)
+    }


We should define this enum in the dart package too as OnyxStrokeStyle

adil192 · 2025-05-12T08:18:41Z

lib/components/canvas/canvas.dart

+  int toolToOnyx(Tool currentTool) {
+    if (placeholder) return 5;
+    if (currentTool is Pencil) {
+      return 3;
+    } else if (currentTool is Highlighter) {
+      return 4;
+    } else if (currentTool is Eraser) {
+      return 1;
+    } else if (currentTool is Select) {
+      return 1;
+    } else if (currentTool is LaserPointer) {
+      return 1;
+    } else if (currentTool is Pen) {
+      if (currentTool.isPressureEnabled()) {
+        return 2;
+      } else {
+        return 1;
+      }
+    } else {
+      return 5;
+    }
+  }
+


This can then use the enum's values instead of ints

adil192 · 2025-05-12T08:19:37Z

lib/components/canvas/canvas.dart

+  int getColor() {
+      if (currentTool is Pen) {
+        return (currentTool as Pen).color.toARGB32();
+      } else {
+        return Colors.black.toARGB32();
+      }
+  }
+  double getWidth() {
+      if (currentTool is Pen) {
+        double baseSize = (currentTool as Pen).getSize() * currentScale;
+        if ((currentTool as Pen).isPressureEnabled()) {
+          return baseSize;
+        } else {
+          return baseSize * 2;
+        }
+      } else {
+        return 3;
+      }
+  }


I think these methods would be best named getOnyxTool, getOnyxColor, getOnyxWidth so their purpose is more clear

adil192 · 2025-05-12T08:20:31Z

lib/data/tools/pen.dart

+  bool isPressureEnabled() {
+    return pressureEnabled;
+  }
+  double getSize() {
+    return options.size;
+  }
+


These seem like redundant getters, we can just use the underlying fields themselves

packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt

adil192 · 2025-05-12T08:27:54Z

packages/onyxsdk_pen/lib/onyxsdk_pen_area.dart

  /// is still writing, which will make the screen get stuck in a half-drawn
  /// state.
  final Duration refreshDelay;
+  final int strokeStyle;


This should be an enum arg too like in kotlin: e.g. strokeStyle: OnyxStrokeStyle.fountainPen

Iey4iej3 · 2025-07-18T14:03:16Z

I hope that somebody could take up to improve this patch.

jdkruzr · 2025-07-22T00:03:21Z

hey there -- it sounds like this Onyx SDK work is more or less complete but hasn't been merged yet? or am I missing something?

Iey4iej3 · 2025-07-22T14:06:39Z

There are some comments above on coding. I suspect that these improvements of coding could be done by AI. I do not know how to do that though.

jdkruzr · 2025-07-23T00:04:43Z

has anyone else noticed that when you include this stuff it breaks the Text control?

zinstack625 · 2025-07-24T16:12:31Z

Sorry I've been out of touch, been busy with lots of stuff. I'll try looking into all the stuff soon

This is a bit janky, and sometimes fails to disable the overlay if user is fast enough. However, in situations where it does correctly work, it's a nice QoL, and when it doesn't, nothing actually breaks

zinstack625 · 2025-07-27T23:02:48Z

Okay, seems like dart code is more tolerable now. While I was at it, I also made a small "feature" in the overlay to be able to select "disabled" tool from dart, which should make some stuff like erasing and editing text a bit more tolerable.

has anyone else noticed that when you include this stuff it breaks the Text control?

With 5911175 editing text is better. Typing on keyboard with a stylus however still initiates immediate draw, which blocks the screen refresh. At 2AM today I'm out of ideas both with how it is possible and how to fix that besides disabling the overlay altogether in those cases.

With small batch of ~~hopelessly banging the head against the table~~ experiments all I could do is break stuff. I'll try throwing some more stuff at a wall in the general area of https://github.com/zinstack625/saber/blob/main/packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt#L125 this week and maybe something will happen...

jdkruzr · 2025-07-27T23:16:26Z

I've been hacking on it with Claude Code; I may have him take a crack at fixing this tonight as well. Really appreciate you diving back into this, man.

…

On Sun, Jul 27, 2025, 6:03 PM Антон Климанов ***@***.***> wrote: *zinstack625* left a comment (saber-notes/saber#1452) <#1452 (comment)> Okay, seems like dart code is more tolerable now. While I was at it, I also made a small "feature" in the overlay to be able to select "disabled" tool from dart, which *should* make some stuff like erasing and editing text a bit more tolerable. has anyone else noticed that when you include this stuff it breaks the Text control? With 5911175 <5911175> editing text is better. Typing on keyboard with a stylus however still initiates immediate draw, which blocks the screen refresh. At 2AM today I'm out of ideas both with how it is possible and how to fix that besides disabling the overlay altogether in those cases. With small batch of hopelessly banging the head against the table experiments all I could do is break stuff. I'll try throwing some more stuff at a wall in the general area of https://github.com/zinstack625/saber/blob/main/packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt#L125 this week and maybe something will happen... — Reply to this email directly, view it on GitHub <#1452 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEW5VXRZ4FFJQVLU7TA7Y33KVLC5AVCNFSM6AAAAABZMB7TR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMRUG43TQNZRGM> . You are receiving this because you commented.Message ID: ***@***.***>

jdkruzr · 2025-07-28T21:38:46Z

Okay, seems like dart code is more tolerable now. While I was at it, I also made a small "feature" in the overlay to be able to select "disabled" tool from dart, which should make some stuff like erasing and editing text a bit more tolerable.

has anyone else noticed that when you include this stuff it breaks the Text control?

With 5911175 editing text is better. Typing on keyboard with a stylus however still initiates immediate draw, which blocks the screen refresh. At 2AM today I'm out of ideas both with how it is possible and how to fix that besides disabling the overlay altogether in those cases.

With small batch of ~~hopelessly banging the head against the table~~ experiments all I could do is break stuff. I'll try throwing some more stuff at a wall in the general area of https://github.com/zinstack625/saber/blob/main/packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt#L125 this week and maybe something will happen...

did you fix this? I just built and ran it on my Tab Ultra C Pro and it seems fine -- I can get in and out of the text tool without it crashing or losing the Onyx SDK support -- with the exception of very quick strokes sometimes resulting in big filled circle "bubbles" -- especially things like the middle stroke of a capital H. it seems to be a worse problem right after you open or create a note.

also -- can you explain why it still first draws according to how your pen writes and then after a delay seems to re-render using some other visualization library? is it not possible to just keep the first set of strokes? I feel like Flutter is doing something unwanted of its own accord here but I don't know enough yet to diagnose what it is.

I also wonder if there's maybe a good reason to have a toggle in the settings menu to turn Onyx support on and off, even just from a code path separation point of view.

github-actions bot mentioned this pull request Mar 21, 2025

Top Issues Dashboard #1196

Open

zinstack625 force-pushed the main branch from 556ccfe to a9c8748 Compare March 21, 2025 15:33

This comment was marked as off-topic.

Sign in to view

Iey4iej3 mentioned this pull request Apr 21, 2025

Onyx Boox: strokes become strange after refresh #1459

Open

2 tasks

Iey4iej3 mentioned this pull request May 9, 2025

Onyx Device: Strokes go over Toolbar and icons #1465

Open

2 tasks

adil192 requested changes May 12, 2025

View reviewed changes

zinstack625 added 11 commits July 27, 2025 22:11

Chore: Update onyxsdk

cb122f5

Feat: Implement sdk strokes in onyx preview

505d830

Feat: pass selected tool to preview [1/3]

03abd19

Feat: pass selected tool to preview [2/3]

34d323f

Feat: pass selected tool to preview [3/3]

78514d1

fix: fixup tests

3333f4c

fix: make linter happy

1a097af

fix: force display refresh on stylus lift

7e676e1

fix: disable drawing on app hide

0c01bb4

chore: prettify dart tool parametrization in canvas

e907f51

chore: remove redundant getters

7636a45

Feat: disable onyx-sdk-pen overlay on eraser

4fa29f7

This is a bit janky, and sometimes fails to disable the overlay if user is fast enough. However, in situations where it does correctly work, it's a nice QoL, and when it doesn't, nothing actually breaks

zinstack625 force-pushed the main branch from a9c8748 to 4fa29f7 Compare July 27, 2025 22:03

chore: figure out used tool without reflection

5911175

adil192 mentioned this pull request Oct 19, 2025

Onyx SDK may be removed due to 16KB page size support requirements #1565

Open

Uh oh!

feat: Improve onyxsdk-pen integration #1452

Are you sure you want to change the base?

feat: Improve onyxsdk-pen integration #1452

Uh oh!

Conversation

zinstack625 commented Mar 19, 2025

Updated onyxsdk dependencies

Reworked OnyxsdkPenArea.kt

Dart side

Uh oh!

codecov bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Fe-Ti commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zinstack625 commented Mar 21, 2025

Uh oh!

Fe-Ti commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Iey4iej3 commented Apr 5, 2025

Uh oh!

zinstack625 commented Apr 6, 2025

Uh oh!

Iey4iej3 commented Apr 7, 2025

Uh oh!

This comment was marked as off-topic.

Iey4iej3 commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adil192 left a comment

Choose a reason for hiding this comment

Uh oh!

adil192 May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adil192 May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adil192 May 12, 2025

Choose a reason for hiding this comment

Uh oh!

adil192 May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adil192 May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Iey4iej3 commented Jul 18, 2025

Uh oh!

jdkruzr commented Jul 22, 2025

Uh oh!

Iey4iej3 commented Jul 22, 2025

Uh oh!

jdkruzr commented Jul 23, 2025

Uh oh!

zinstack625 commented Jul 24, 2025

Uh oh!

zinstack625 commented Jul 27, 2025

Uh oh!

jdkruzr commented Jul 27, 2025 via email

Uh oh!

jdkruzr commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Mar 19, 2025 •

edited

Loading

Fe-Ti commented Mar 20, 2025 •

edited

Loading

Fe-Ti commented Mar 21, 2025 •

edited

Loading

Iey4iej3 commented Apr 21, 2025 •

edited

Loading

adil192 May 12, 2025 •

edited

Loading

adil192 May 12, 2025 •

edited

Loading

adil192 May 12, 2025 •

edited

Loading

jdkruzr commented Jul 28, 2025 •

edited

Loading