Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Oculus Quest compatibility/Keyboard not shown/Enter+Del not working #1832

Closed
Schlumie opened this issue Jan 22, 2024 · 19 comments
Closed

Oculus Quest compatibility/Keyboard not shown/Enter+Del not working #1832

Schlumie opened this issue Jan 22, 2024 · 19 comments
Labels
Milestone

Comments

@Schlumie
Copy link

Describe the bug
I tried noVNC on Oculus Quest 3 build in Browser. Its really nice and easy to work on a remote machines. But the keyboard symbol is missing... so i commented out the :root.noVNC_connected #noVNC_mobile_buttons css. Show/Hide keyboard is functional now but is not displayed. Maybe a force flag in settings to enable keyboard icon?
Another Problem is that everythings working, also german umlauts, BUT enter and delete keys doing nothing/are ignored... Any hints?

To Reproduce
Steps to reproduce the behavior:

  1. Run noVNC on Linux/Ubuntu, example with wayland gnome-vnc:
    git clone https://github.com/novnc/noVNC.git
    grdctl vnc enable
    systemctl --user start gnome-remote-desktop
    websockify -D --web=noVNC/ --cert=novnc.pem 6080 localhost:5900

  2. Open Standard Browser in Oculus Quest 3 (OS V60)

  3. Goto URL of noVNC Server.... https://xxxx:6080

  4. Connect

Expected behavior
Keyboard symbol should be visible.
Enter+Delete Key should be working.

Client (please complete the following information):

  • OS: Oculus Quest 3 V60
  • Browser: Default/Intern

Server (please complete the following information):

  • noVNC version: master as of 2024-01-22
  • VNC server: gnome-remote-desktop
  • WebSocket proxy: websockify

Additional context
Enter+Delete key not working only testable if keyboard symbol forced via commenting: :root.noVNC_connected #noVNC_mobile_buttons

@CendioOssman
Copy link
Member

There are two issues here, which I would guess are unrelated. The non-functional keys might be a server issue.

Could you test with TigerVNC's Xvnc/vncserver? It has a known robust keyboard handling.

I'm not familiar with the Oculus interface. Does it automatically pop up a keyboard whenever a text field is activated? I know that with SteamVR you need to click a keyboard button in the VR interface.

What does this page give you from the Oculus web browser:

https://patrickhlauke.github.io/touch/pointer-hover-any-pointer-any-hover/

@Schlumie
Copy link
Author

Schlumie commented Jan 22, 2024

Screenshot_20240122_155704_Meta Quest
Screenshot_20240122_153921_Meta Quest
TigerVNC same problem no Enter/Del key. Screenshot for the pointer test....
Is there a keyboard debugmode where all keyevents gets logged? Or how can I help to dial this down?

@CendioOssman
Copy link
Member

Yes, if you open noVNC's settings, you can change the log level to debug. At that point, you'll get keyboard events in the browser's console log.

@CendioOssman
Copy link
Member

@samhed, what do you make of that capability list?

@Schlumie
Copy link
Author

Screenshot_20240122_161445_Meta Quest
I tried this event testpage and pressed key "a" then "Enter" then "Del".
I will try to get debug consolelog. Have to figure out this is possible on Oculus...

@CendioOssman
Copy link
Member

This is a good test site otherwise:

https://domeventviewer.com/key-event-viewer.html

It doesn't work exactly like noVNC, but similar.

@Schlumie
Copy link
Author

Schlumie commented Jan 22, 2024

connecting to ws://xxxxxxxx:6080/websockify
rfb.js:1385 Server ProtocolVersion: 003.008
ui.js:1241 Server asked for credentials
credentials @ ui.js:1241
(anonymous) @ eventtarget.js:32
dispatchEvent @ eventtarget.js:32
_negotiateStdVNCAuth @ rfb.js:1662
_negotiateAuthentication @ rfb.js:1972
_initMsg @ rfb.js:2167
_handleMessage @ rfb.js:987
_recvMessage @ websock.js:360
vnc.html:1 The resource http://xxxx:6080/app/images/info.svg was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
rfb.js:2079 Screen: 800x600, bpp: 32, depth: 32, bigEndian: 0, trueColor: 255, redMax: 255, greenMax: 255, blueMax: 255, redShift: 16, greenShift: 8, blueShift: 0
display.js:201 Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently
resize @ display.js:201
_resize @ rfb.js:2870
_handleExtendedDesktopSize @ rfb.js:2826
_handleRect @ rfb.js:2568
_framebufferUpdate @ rfb.js:2531
_normalMsg @ rfb.js:2448
_handleMessage @ rfb.js:977
_recvMessage @ websock.js:360
rfb.js:483 Sending keysym (down): 108
rfb.js:483 Sending keysym (up): 108
rfb.js:483 Sending keysym (down): 115
rfb.js:483 Sending keysym (up): 115

No event for Enter/Del...

Also tried: https://domeventviewer.com/key-event-viewer.html and typed l + [Enter]+ [Del]
Bildschirmfoto vom 2024-01-22 16-43-19

@samhed
Copy link
Member

samhed commented Jan 22, 2024

Regarding showing the virtual keyboard button; unfortunately it seems those pointer/hover capabilities doesn't help whatsoever. We rely on those media queries to detect if we have a touch device or not. Perhaps we need a way to overwrite our touch detection..

@Schlumie
Copy link
Author

Thats one part which I can manually overrule/solve. Why Enter/Del is not working is still a mystery. I can try to add a keyboardsymbol for enter and delete like Ctrl+Alt+Del as workaround...

@CendioOssman
Copy link
Member

It seems to be behaving like an ancient web browser. We should still have handling for that, so I'm not sure why noVNC isn't picking up the keys. Perhaps their browser has more bugs.

The browser log is incomplete, though. Make sure there aren't any filters active in the browser. The debug output seems to be entirely missing.

@Schlumie
Copy link
Author

Schlumie commented Jan 23, 2024

The event testsites are picking up the events, dont know if its done the same way. Theres is no consolefilter set, but Inspect is done via Chrome/Remote Inspect. Theres no other way to get the console.
Bildschirmfoto vom 2024-01-23 10-06-46
Bildschirmfoto vom 2024-01-23 10-15-18

@CendioOssman
Copy link
Member

The log level filter is over on the right-hand side:

image

Please check that as well. All levels should be checked there.

@Schlumie
Copy link
Author

Schlumie commented Jan 23, 2024

Console with all levels enabled:
bt.test.at-1706002494090.log
PS: I added a enter/del symbol at the A submenu. At least I can now send those keys via icon.

@CendioOssman
Copy link
Member

Great. Now, the log shows something for the Enter and Backspace keys. So we are getting something from the browser. We're just not able to make fully sense out of it. Let me have a look at the code and see if I can figure out what goes wrong.

@CendioOssman
Copy link
Member

Ah, I think it is because the browser has incorrectly implemented KeyboardEvent.key, rather than omitting it completely.

Could you try this change:

diff --git a/core/input/util.js b/core/input/util.js
index 58f84e55..36b69817 100644
--- a/core/input/util.js
+++ b/core/input/util.js
@@ -67,7 +67,7 @@ export function getKeycode(evt) {
 // Get 'KeyboardEvent.key', handling legacy browsers
 export function getKey(evt) {
     // Are we getting a proper key value?
-    if (evt.key !== undefined) {
+    if ((evt.key !== undefined) && (evt.key !== 'Unidentified')) {
         // Mozilla isn't fully in sync with the spec yet
         switch (evt.key) {
             case 'OS': return 'Meta';

@Schlumie
Copy link
Author

Its working! Thanks a lot! Only change I still need is to force show the keyboard symbol. Maybe add a checkbox in the settings as long theres no good mediaquerys for that?

@CendioOssman
Copy link
Member

Its working! Thanks a lot!

Great. I've committed it as ab2fd41.

Note that you should really report this issue to Meta. The major browsers have supported modern key events for almost ten years now, so we might remove support for legacy stuff eventually. Oculus should really get in sync with the big players.

Only change I still need is to force show the keyboard symbol. Maybe add a checkbox in the settings as long theres no good mediaquerys for that?

Yeah, it sounds like we need an override for when we're not getting good detection of the environment.

@Schlumie
Copy link
Author

I reported it to Meta.

@LMattsson
Copy link

Separated the issue with the virtual keyboard button not showing to issue #1837. Since everything seems to be sorted here I'll go ahead and close this.

@LMattsson LMattsson added the bug label Feb 9, 2024
@LMattsson LMattsson added this to the v1.5.0 milestone Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants