 |
 |
ResidentMIDIKeyboard for Android is a virtual MIDI keyboard application for Android. It supports MIDI 1.0 and 2.0 (June 2023 Updates).
It works both as (1) a MIDI client keyboard that can connect to MIDI output devices, as well as (2) a virtual MIDI keyboard that DAWs can connect to it to receive MIDI input events.
It supports "Expression Mode": instead of switching keys on the dragged loations, we can send pitchbend messages for horizontal dragging, per-note pitchbend messags for vertical dragging, and polyphonic (key) pressures for pressure changes on the device.
(Note: per-note pitchbend works only on MIDI2 mode, and even on MIDI2 mode there is not really a MIDI device that supports per-note pitchbend, as far as @atsushieno knows as of July 2023.)
There are 3 ways to use the app (actually two for users; one for app developers):
- Using the app: it barely makes sense, but launch the main activity that comes with the keyboard.
- As a System Alert Window: you need to approve the UI overlay permission (it is usually dangerous as an app could show deceitful UI like a password box for your bank while you are running a banking app, so Android system explicitly asks the user to identify that the app is not malicious), use it anytime e.g. over your Home Launcher, or over any DAW.
- As a SurfaceControlViewHost: it is not for user, but for developers. We provide a public GUI service
org.androidaudioplugin.residentmidikeyboard.MidiKeyboardViewService.
The app is primarily intended to meet all these conditions:
- a MIDI client that works anywhere System Alert Window works
- a virtual MIDI input keyboard that DAWs can connect to and receive MIDI inputs
- supports both MIDI 1.0 and 2.0
- makes use of mobile features such as multi-touches and follows mobile design principles
- supports UMP packets over MIDI 1.0 on top of hacky "implicit agreement" between this app and the UMP device
My primary reasoning for this app is that I do not want to embed a MIDI functionality test MIDI features on our AAP (Audio Plugins For Android) plugin manager UI.
When the app is launched, it shows a foreground nofitication that contains "Show", "Hide" and "Switch to App" links. When you perform "Show" command, the System Alert Window keyboard should show up.
Note that you have to explicitly permit UI overlay on Settings in prior.
SurfaceControlViewHost lets developers (not end users) achieve "embedded remote UI" which works like RemoteViews but with much less limitation. Basically anything you can render on a SurfaceView can be implemented on the service side, and UI interaction is possible. ResidentMidiKeyboard uses Jetpack Compose, which can be also transferred.
To embed our MIDI keyboard, you do not have to add reference to our library; you only have to install ResidentMidiKeyboard app, and write some client code that binds org.androidaudioplugin.residentmidikeyboard.MidiKeyboardViewService just like ordinary Android Service.
This service receives Message i.e. any Intent with certain key-value bundles. The required items are:
| Key | Type | Role |
|---|---|---|
| "opcode" | Int | 0 for connecting, 1 for disconnecting |
| "hostToken" | Binder | the host token: get from SurfaceView.hostToken |
| "displayId" | Int | the display ID: get from SurfaceView.display.displayId |
| "width" | Int | initial width e.g. from SurfaceView.width |
| "height" | Int | initial height e.g. from SurfaceView.height |
The MainActivity comes with an example embedded SurfaceView with makes use of MidiKeyboardSurfaceControlClient class. (You can reuse it too.)
It is highly encouraged to ensure that you use it as an optional add-in feature; it is always possible to embed the control your own app by just using ComposeAudioControls library (compose-audio-controls-midi module).
Note that you will have to ensure that you allocate necessary space for expanded DropDownMenu e.g. by making container scrollable. It is one of the common Jetpack Compose issues (or tricks) regarding how to use it appropriately on a custom View.
The latest RMK supports MIDI 2.0 UMP protocol on Android 13 or later. The MIDI device port list shows both those for MIDI 1.0 and 2.0 protocols.
For UMP device ports, UMP is the native choice to send the messages and it should work out of the box. The rest of this section does NOT apply to this native mode.
For traditional MIDI 1.0 ports, we can still send UMPs so that IF the output device actually accepts UMPs then we can communicate with UMPs - there is a checkbox option that enables MIDI 2.0. Once you check it, it will send any MIDI messages as MIDI 2.0 UMP messages instead of MIDI 1.0 bytes stream. Note that most of your MIDI output devices do not support MIDI 2.0 as of July 2023.
AAP (Audio Plugins For Android) MidiDeviceServices support these MIDI 2.0 UMPs and this MIDI2 mode works with them. You can install those AAP instrument plugins via AAP APK Installer.
(The latest AAP also supports MidiUmpDeviceService on Android 15 or later. This "UMP over MIDI 1.0 ports" feature is for earlier Android versions.)
When you check "MIDI2" box, it sends UMP "Stream Configuration Request" message that asks the destination to switch to MIDI 2.0 protocol. Let me emphasize: it sends a UMP message. Note that the recipient should be already capable of receiving UMP, which usually is not the case for most of Android MIDI API devices out there.
(The latest MIDI 2.0 specification probably expects that the MIDI 2.0 output port ("input port" in Android MIDI API wording) should be distinct from that of MIDI 1.0 output port, but there is no such Android MIDI API that makes it work as such yet. So, everything is basically based on a hackable assumption on Android platform up to Android 14.)
Also note that since we do NOT really pair the output device with its expected input devices for bidirectional messaging, the expected reply (Stream Configuration Notification message) that would be sent by the recipient is ignored. If the recipient fails if no input port exists, then the device is not feasible for us (I should probably state, we are not feasible for them).
ResidentMIDIKeyboard offers a "nominal" MIDI output port (or input port in Android MIDI API wording) so that a valid MIDI 2.0 client that conforms to June 2023 Updates specification can send its UMP Stream Configuration Notification message to the port in reply to our Stream Configuration Request message (note that ResidentMIDIKeyboard sends it without expecting replies, as described above, regardless of the connection state, at least for now).
We submodule compose-audio-controls for convenience, as we most likely depend on unpublished features that are only on main branch without release tags. Our project (as in settings.gradle.kts) references those modules in the git submodule directly, instead of depending on mavenLocal().
The same could go for ktmidi, but it is so far referenced in packages.
ResidentMidiKeyboard code is released under the MIT license.
It contains an application icon that is a derived work of Material Design 3 Icons which are released under the Apache V2 License.
ResidentMidiKeyboard makes use of jeziellago/compose-markdown which is released under the MIT license.