Support Android and IOS #29

thewh1teagle · 2024-10-21T03:32:38Z

Currently, sherpa-rs supports Windows, Linux, and macOS.

It would be beneficial to add support for Android and iOS as well. At first glance, it seems feasible to link to prebuilt sherpa-onnx libraries. However, it's unclear whether we can easily cross-compile from any platform to any other, and whether Android should use jniLibs or if the approach is more similar to how Linux builds are handled.

ghjdegithub · 2024-10-30T06:27:53Z

@thewh1teagle
I've used flutter_rust_bridge and it's easy to call rust lib
https://cjycode.com/flutter_rust_bridge/

As long as you can successfully run iOS and Android after calling sherpa-rs under this framework, then you are basically successful.

thewh1teagle · 2024-11-23T22:50:17Z

@ghjdegithub

Thanks! I'm wondering how we actually provide a way first to call this library from java / flutter / swift / kotlin.
I mean, I guess we need first the sherpa-onnx android/ios pre compiled libs. should we link them then just like with windows/linux/macos? how it's different?

For Android we might need to support:

armv7-linux-androideabi (32 bit arm)
aarch64-linux-android (64 bit arm)

Plus the NDK

For IOS:

aarch64-apple-ios

csukuangfj · 2024-11-24T09:22:15Z

By the way, sherpa-onnx also supports HarmonyOS from Huawei

thewh1teagle · 2024-11-24T21:46:54Z

By the way, sherpa-onnx also supports HarmonyOS from Huawei

How can we use sherpa C API in Android / IOS? does sherpa-onnx have prebuilt .so files for Android or .dylib for IOS? (if that's valid)

csukuangfj · 2024-11-24T23:28:20Z

yes, we use jni

ghjdegithub · 2024-11-26T09:24:20Z

By the way, sherpa-onnx also supports HarmonyOS from Huawei

How can we use sherpa C API in Android / IOS? does sherpa-onnx have prebuilt .so files for Android or .dylib for IOS? (if that's valid)

sherpa-onnx has always been there, I've run through both android and ios

thewh1teagle · 2024-11-28T01:55:56Z

Some research I did, since I haven't worked with mobile, reveals that Android and iOS cannot directly use native libraries. Instead, Android requires a JNI wrapper library (which is native but wrapped), and iOS also requires a wrapper, though XCode can generate it automatically from a .dylib or .framework for Swift/ObjC.

This means we need to expose a C API from sherpa-rs. However, this seems unnecessary since sherpa-onnx already does this. Instead, I think we should add an option to compile sherpa-rs with the precompiled or source code of sherpa-onnx for Android/iOS. The user can then write their own native wrapper for the specific use case.

https://gist.github.com/thewh1teagle/794add8801e7c0923042ff877b179ee5

ghjdegithub · 2024-11-28T05:54:45Z

Some research I did, since I haven't worked with mobile, reveals that Android and iOS cannot directly use native libraries. Instead, Android requires a JNI wrapper library (which is native but wrapped), and iOS also requires a wrapper, though XCode can generate it automatically from a .dylib or .framework for Swift/ObjC.

This means we need to expose a C API from sherpa-rs. However, this seems unnecessary since sherpa-onnx already does this. Instead, I think we should add an option to compile sherpa-rs with the precompiled or source code of sherpa-onnx for Android/iOS. The user can then write their own native wrapper for the specific use case.

There's no need at all, there's no need to use java ffi and swift ffi to call the C APIs exposed by rust.
And are you sure someone needs to call the C api that is exposed after rust wraps the C++ library?

thewh1teagle · 2024-11-28T16:15:30Z

There's no need at all, there's no need to use java ffi and swift ffi to call the C APIs exposed by rust.

Do you have/know some project that expose Rust to swift/java? Also we don't have currently C API exposed by Rust

And are you sure someone needs to call the C api that is exposed after rust wraps the C++ library?

There's no way to call Rust directly it needs to be exposed as C API (or maybe with some library that makes it easier)

thewh1teagle · 2024-11-28T16:36:38Z

There's no need at all, there's no need to use java ffi and swift ffi to call the C APIs exposed by rust.

See in sherpa-onnx repo, there's complete folder just for JNI exposing
https://github.com/k2-fsa/sherpa-onnx/tree/f3f896146268bf748cf58ad717e82d89d1a913d1/sherpa-onnx/jni

ghjdegithub · 2024-11-28T23:28:23Z

jni is java's ffi 😅😅

ghjdegithub · 2024-11-29T00:59:25Z

I think the essence of making a rust wrapper is to output the ub behavior brought by C++, and provide safe rust code, rather than simply taking the c api and using it directly, so the sub-essence is still unsafe, so the necessity of exposing a c api is questionable. And the security package I wrote hasn't avoided some UB yet, I also need to modify it, add some lifecycle and encapsulation, and put an end to the UB behavior brought by the C API.

oddpxl · 2024-12-19T01:09:43Z

1+

Would be great to compile for Android & IOS... I'm using Tauri to compile for Mac and Android - without sherpa-rs support I suppose I have to take the Webassembly route... ..not ideal

thewh1teagle · 2024-12-19T01:17:33Z

Would be great to compile for Android & IOS... I'm using Tauri to compile for Mac and Android - without sherpa-rs support I suppose I have to take the Webassembly route... ..not ideal

JNI for Java and other platforms is not planned for sherpa-rs since sherpa-onnx already created JNI,
The good news is that afaik Tauri doesn't relied on JNI for calling Rust logic. so we can just compile this library to Linux aarch64 (x64) / armv7 (x32) and you can use it directly in Tauri.

thewh1teagle · 2024-12-19T01:19:23Z

There's no need at all, there's no need to use java ffi and swift ffi to call the C APIs exposed by rust.

By the way that's incorrect. You most have Java FFI / Swift FFI to call C libraries in mobile in most cases.
Sometimes the platforms know to create the FFI code directly but afaik for Java you most create it manually just like sherpa-onnx did.

oddpxl · 2024-12-19T12:53:03Z

The good news is that afaik Tauri doesn't relied on JNI for calling Rust logic. so we can just compile this library to Linux aarch64 (x64) / armv7 (x32) and you can use it directly in Tauri.

Ah !

Excellent !

..only just now got Tauri to compile with an embedded onnx + VAD processing a file...

..next is trying streaming audio...then try compile for Android... Excited now :)

Thanks for the feedback. Much appreciated.

oddpxl · 2024-12-19T20:28:48Z

@thewh1teagle

I've got audio captured ( in the webview ) being VAD processed ( Rust side ) using Tauri on Mac...

..but get this error when I try to build for Android ( see below )

Any suggestions ?

CMake Error at cmake/onnxruntime.cmake:123 (message): Only support Linux, macOS, and Windows at present. Will support other OSes later

csukuangfj · 2024-12-19T21:24:08Z

could you post complete logs?

oddpxl · 2024-12-20T00:31:25Z

@csukuangfj See below :)

I'm definitely out of my comfort zone here so please just let me know if I missed something ?

I use Tauri 2.0 - capturing audio in the webview and send that rust side to VAD process, works fine on Mac...

..but when I compile for Android it's a no go...

`> tauri "info"

[✔] Environment
- OS: Mac OS 14.5.0 arm64 (X64)
✔ Xcode Command Line Tools: installed
✔ rustc: 1.83.0 (90b35a623 2024-11-26)
✔ cargo: 1.83.0 (5ffbef321 2024-10-29)
✔ rustup: 1.27.1 (54dd3d00f 2024-04-24)
✔ Rust toolchain: stable-aarch64-apple-darwin (default)
- node: 22.9.0
- pnpm: 8.15.5
- npm: 10.8.3

[-] Packages
- tauri 🦀: 2.1.1
- tauri-build 🦀: 2.0.3
- wry 🦀: 0.47.2
- tao 🦀: 0.30.8
- tauri-cli 🦀: 1.5.11
- @tauri-apps/api : 2.1.1
- @tauri-apps/cli : 2.1.0

[-] Plugins
- tauri-plugin-barcode-scanner 🦀: 2.2.0
- @tauri-apps/plugin-barcode-scanner : 2.0.0-beta.2 (outdated, latest: 2.2.0)
- tauri-plugin-shell 🦀: 2.2.0
- @tauri-apps/plugin-shell : 2.0.0-beta.2 (outdated, latest: 2.2.0)
- tauri-plugin-store 🦀: 2.2.0
- @tauri-apps/plugin-store : 2.2.0

[-] App
- build-type: bundle
- CSP: unset
- frontendDist: ../dist
- devUrl: http://localhost:1420/
- framework: SolidJS
- bundler: Vite`

`

pnpm tauri android build

warning: sherpa-rs-sys@0.5.1: Failed to download binaries. fallback to manual build.

error: failed to run custom build command for sherpa-rs-sys v0.5.1

Caused by:
process didn't exit successfully: /src-tauri/target/release/build/sherpa-rs-sys-71c8eb8408c49e3c/build-script-build (exit status: 101)
--- stdout
cargo:rerun-if-changed=wrapper.h
cargo:rerun-if-changed=./sherpa-onnx
cargo:rerun-if-changed=dist.txt
cargo::rerun-if-env-changed=SHERPA_BUILD_SHARED_LIBS
cargo::rerun-if-env-changed=CMAKE_BUILD_PARALLEL_LEVEL
cargo::rerun-if-env-changed=CMAKE_VERBOSE
cargo::rerun-if-env-changed=SHERPA_LIB_PATH
cargo::rerun-if-env-changed=SHERPA_STATIC_CRT
cargo::rerun-if-env-changed=SHERPA_LIB_PROFILE
cargo::rerun-if-env-changed=BUILD_DEBUG
cargo:rerun-if-env-changed=TARGET
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-linux-android
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
cargo:rerun-if-changed=wrapper.h
cargo:rerun-if-changed=./sherpa-onnx/sherpa-onnx/c-api/c-api.h
cargo:rerun-if-changed=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include/stdint.h
cargo:warning=Failed to download binaries. fallback to manual build.
CMAKE_TOOLCHAIN_FILE_aarch64-linux-android = None
CMAKE_TOOLCHAIN_FILE_aarch64_linux_android = None
TARGET_CMAKE_TOOLCHAIN_FILE = None
CMAKE_TOOLCHAIN_FILE = None
CMAKE_GENERATOR_aarch64-linux-android = None
CMAKE_GENERATOR_aarch64_linux_android = None
TARGET_CMAKE_GENERATOR = None
CMAKE_GENERATOR = None
CMAKE_PREFIX_PATH_aarch64-linux-android = None
CMAKE_PREFIX_PATH_aarch64_linux_android = None
TARGET_CMAKE_PREFIX_PATH = None
CMAKE_PREFIX_PATH = None
CMAKE_aarch64-linux-android = None
CMAKE_aarch64_linux_android = None
TARGET_CMAKE = None
CMAKE = None
running: cd "/src-tauri/target/aarch64-linux-android/release/build/sherpa-rs-sys-06fcd8216bd0c551/out/build" && CMAKE_PREFIX_PATH="" LC_ALL="C" "cmake" "/src-tauri/target/aarch64-linux-android/release/build/sherpa-rs-sys-06fcd8216bd0c551/out/sherpa-onnx" "-DSHERPA_ONNX_ENABLE_C_API=ON" "-DSHERPA_ONNX_ENABLE_BINARY=OFF" "-DBUILD_SHARED_LIBS=ON" "-DSHERPA_ONNX_ENABLE_WEBSOCKET=OFF" "-DSHERPA_ONNX_ENABLE_TTS=OFF" "-DSHERPA_ONNX_BUILD_C_API_EXAMPLES=OFF" "-DSHERPA_ONNX_ENABLE_TTS=ON" "-DCMAKE_SYSTEM_NAME=Android" "-DCMAKE_SYSTEM_PROCESSOR=aarch64" "-DCMAKE_INSTALL_PREFIX=/src-tauri/target/aarch64-linux-android/release/build/sherpa-rs-sys-06fcd8216bd0c551/out" "-DCMAKE_C_FLAGS= -DANDROID -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/Users/xxx/Library/Android/sdk/ndk/27.2.12479018/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang" "-DCMAKE_CXX_FLAGS= -DANDROID -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/Users/xxx/Library/Android/sdk/ndk/27.2.12479018/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang++" "-DCMAKE_ASM_FLAGS= -DANDROID -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/Users/xxx/Library/Android/sdk/ndk/27.2.12479018/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang" "-DCMAKE_BUILD_TYPE=Release"
-- Configuring incomplete, errors occurred!

--- stderr
CMake Error at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/Platform/Android-Determine.cmake:217 (message):
Android: Neither the NDK or a standalone toolchain was found.
Call Stack (most recent call first):
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineSystem.cmake:190 (include)
CMakeLists.txt:8 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
thread 'main' panicked at /Users/xxx/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.52/src/lib.rs:1115:5:

command did not execute successfully, got: exit status: 1

build script failed, must exit now
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
Failed to run cargo build: command ["cargo", "build", "--package", "tauri-mobile-neo", "--manifest-path", "/src-tauri/Cargo.toml", "--target", "aarch64-linux-android", "--features", "tauri/custom-protocol tauri/rustls-tls", "--lib", "--release"] exited with code 101 Error Failed to run cargo build: command ["cargo", "build", "--package", "tauri-mobile-neo", "--manifest-path", "/src-tauri/Cargo.toml", "--target", "aarch64-linux-android", "--features", "tauri/custom-protocol tauri/rustls-tls", "--lib", "--release"] exited with code 101
ELIFECYCLE Command failed with exit code 1.

`

I then tried a bunch of things to make it build - it got messy fast - but I believe I made it use NDK / CMake etc - and I ultimately got this error...

CMake Error at cmake/onnxruntime.cmake:123 (message): Only support Linux, macOS, and Windows at present. Will support other OSes later

Also tried to make it use the android binaries here ( with no luck )

https://github.com/k2-fsa/sherpa-onnx/releases

Out of ideas :)

Would be real annoying having to use webassembly in the webview haha painful 🗡️

csukuangfj · 2024-12-20T03:39:21Z

but I believe I made it use NDK / CMake etc

Glad to hear you have managed to do that.

and I ultimately got this error.

Please do the following to resolve it.

Download onnxruntime pre-built libs for Android.

# You can put it anywhere you like. We use /tmp below as an example.

cd /tmp

onnxruntime_version=1.17.1
mkdir $onnxruntime_version
cd $onnxruntime_version

wget -c -q https://github.com/csukuangfj/onnxruntime-libs/releases/download/v${onnxruntime_version}/onnxruntime-android-${onnxruntime_version}.zip

unzip onnxruntime-android-${onnxruntime_version}.zip
rm onnxruntime-android-${onnxruntime_version}.zip

If you want to build sherpa-rs for Android arm64-v8a, please use

export SHERPA_ONNXRUNTIME_LIB_DIR=/tmp/$onnxruntime_version/jni/arm64-v8a/
export SHERPA_ONNXRUNTIME_INCLUDE_DIR=/tmp/$onnxruntime_version/headers/

# Run your existing command as before

Similarly, if you want to build sherpa-rs for Android armeabi-v7a, please use

export SHERPA_ONNXRUNTIME_LIB_DIR=/tmp/$onnxruntime_version/jni/armeabi-v7a/
export SHERPA_ONNXRUNTIME_INCLUDE_DIR=/tmp/$onnxruntime_version/headers/

# Run your existing command as before

I hope you have got the idea and know how to build for Android x86_64 and x86
That's it! You can find more about how to build sherpa-onnx for Android at

thewh1teagle · 2024-12-20T12:33:53Z

Out of ideas :)

Would be real annoying having to use webassembly in the webview haha painful 🗡️

Once k2-fsa/sherpa-onnx#1634 resolved it will be easier and fast.
Currently the only way is to compile from source but even then we need few modifications to sherpa-rs that I do in #52

csukuangfj · 2024-12-20T12:39:13Z

Please refer to.the bash.scripts mentioned.in the previous comment. Just enable the option to build C API then your issue is fixed!

thewh1teagle · 2024-12-20T12:50:21Z

Please refer to.the bash.scripts mentioned.in the previous comment. Just enable the option to build C API then your issue is fixed!

The use of precompiled Sherpa C API libraries is helpful, as I'm trying to avoid building them myself or requiring others to do so.

oddpxl · 2024-12-20T14:05:39Z

@csukuangfj All make sense...thanks for the guidance ! ..probably should wait for @thewh1teagle as I'm clearly messing up my build process trying to make it work :) ( need to learn more )

Amazing work on sherpa + rs thanks guys

..can't wait for this to work on Android !!

thewh1teagle · 2024-12-24T17:07:31Z

I added support for Android and IOS (in main, didn't released yet)
Also compiled it and ran it successfully on aarch64-linux-android!

csukuangfj · 2024-12-25T03:38:57Z

Is there an Android APK so that users can try it?

thewh1teagle · 2024-12-25T07:43:04Z

Is there an Android APK so that users can try it?

I tested it with adb.
I'll add example android app. see #59

Does it possible to load model from the memory? I have issues with the filesystem.
Also, is there some very small punctuation model? the model I use is sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2 but it's very large file for android

thewh1teagle · 2024-12-25T08:06:58Z

Released v0.6.0 with Android and IOS support

thewh1teagle added the feature New feature or request label Nov 23, 2024

thewh1teagle mentioned this issue Dec 20, 2024

Add release for android C API k2-fsa/sherpa-onnx#1634

Closed

thewh1teagle mentioned this issue Dec 24, 2024

Trying to build to android target, but failing #11

Closed

thewh1teagle closed this as completed Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Android and IOS #29

Support Android and IOS #29

thewh1teagle commented Oct 21, 2024

ghjdegithub commented Oct 30, 2024 •

edited

Loading

thewh1teagle commented Nov 23, 2024 •

edited

Loading

csukuangfj commented Nov 24, 2024

thewh1teagle commented Nov 24, 2024

csukuangfj commented Nov 24, 2024

ghjdegithub commented Nov 26, 2024

thewh1teagle commented Nov 28, 2024 •

edited

Loading

ghjdegithub commented Nov 28, 2024

thewh1teagle commented Nov 28, 2024

thewh1teagle commented Nov 28, 2024

ghjdegithub commented Nov 28, 2024

ghjdegithub commented Nov 29, 2024

oddpxl commented Dec 19, 2024 •

edited

Loading

thewh1teagle commented Dec 19, 2024

thewh1teagle commented Dec 19, 2024

oddpxl commented Dec 19, 2024 •

edited

Loading

oddpxl commented Dec 19, 2024 •

edited

Loading

csukuangfj commented Dec 19, 2024

oddpxl commented Dec 20, 2024 •

edited

Loading

csukuangfj commented Dec 20, 2024

thewh1teagle commented Dec 20, 2024

csukuangfj commented Dec 20, 2024

thewh1teagle commented Dec 20, 2024 •

edited

Loading

oddpxl commented Dec 20, 2024 •

edited

Loading

thewh1teagle commented Dec 24, 2024 •

edited

Loading

csukuangfj commented Dec 25, 2024

thewh1teagle commented Dec 25, 2024

thewh1teagle commented Dec 25, 2024

Support Android and IOS #29

Support Android and IOS #29

Comments

thewh1teagle commented Oct 21, 2024

ghjdegithub commented Oct 30, 2024 • edited Loading

thewh1teagle commented Nov 23, 2024 • edited Loading

csukuangfj commented Nov 24, 2024

thewh1teagle commented Nov 24, 2024

csukuangfj commented Nov 24, 2024

ghjdegithub commented Nov 26, 2024

thewh1teagle commented Nov 28, 2024 • edited Loading

ghjdegithub commented Nov 28, 2024

thewh1teagle commented Nov 28, 2024

thewh1teagle commented Nov 28, 2024

ghjdegithub commented Nov 28, 2024

ghjdegithub commented Nov 29, 2024

oddpxl commented Dec 19, 2024 • edited Loading

thewh1teagle commented Dec 19, 2024

thewh1teagle commented Dec 19, 2024

oddpxl commented Dec 19, 2024 • edited Loading

oddpxl commented Dec 19, 2024 • edited Loading

csukuangfj commented Dec 19, 2024

oddpxl commented Dec 20, 2024 • edited Loading

csukuangfj commented Dec 20, 2024

thewh1teagle commented Dec 20, 2024

csukuangfj commented Dec 20, 2024

thewh1teagle commented Dec 20, 2024 • edited Loading

oddpxl commented Dec 20, 2024 • edited Loading

thewh1teagle commented Dec 24, 2024 • edited Loading

csukuangfj commented Dec 25, 2024

thewh1teagle commented Dec 25, 2024

thewh1teagle commented Dec 25, 2024

ghjdegithub commented Oct 30, 2024 •

edited

Loading

thewh1teagle commented Nov 23, 2024 •

edited

Loading

thewh1teagle commented Nov 28, 2024 •

edited

Loading

oddpxl commented Dec 19, 2024 •

edited

Loading

oddpxl commented Dec 19, 2024 •

edited

Loading

oddpxl commented Dec 19, 2024 •

edited

Loading

oddpxl commented Dec 20, 2024 •

edited

Loading

thewh1teagle commented Dec 20, 2024 •

edited

Loading

oddpxl commented Dec 20, 2024 •

edited

Loading

thewh1teagle commented Dec 24, 2024 •

edited

Loading