Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README #236

Merged
merged 11 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 114 additions & 77 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,56 +12,81 @@

[![crates.io](https://img.shields.io/crates/v/livekit.svg)](https://crates.io/crates/livekit)
[![livekit docs.rs](https://img.shields.io/docsrs/livekit)](https://docs.rs/livekit/latest/)
[![Builds](https://github.com/livekit/client-sdk-native/actions/workflows/builds.yml/badge.svg?branch=main)](https://github.com/livekit/client-sdk-native/actions/workflows/builds.yml)
[![Tests](https://github.com/livekit/client-sdk-native/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/livekit/client-sdk-native/actions/workflows/tests.yml)

⚠️ Warning

> This SDK is currently in Developer Preview mode and not ready for production use. There will be bugs and APIs may change during this period.
>
> We welcome and appreciate any feedback or contributions. You can create issues here or chat live with us in the #rust-developer-preview channel within the [LiveKit Community Slack](https://livekit.io/join-slack).
[![Builds](https://github.com/livekit/rust-sdks/actions/workflows/builds.yml/badge.svg?branch=main)](https://github.com/livekit/rust-sdks/actions/workflows/builds.yml)
[![Tests](https://github.com/livekit/rust-sdks/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/livekit/rust-sdks/actions/workflows/tests.yml)

## Features

- [x] Receiving tracks
- [x] Cross-platform ( currently tested on Windows & MacOS )
- [x] Data channels
- [x] Publishing tracks
- [x] Data channels
- [x] Simulcast
- [ ] SVC codecs (AV1/VP9)
- [ ] Adaptive Streaming
- [ ] Dynacast
- [x] Simulcast
- [ ] Hardware video enc/dec
- [ ] NvEnc for Windows
- [x] Hardware video enc/dec
- [x] VideoToolbox for MacOS/iOS
- Supported Platforms
- [x] Windows
- [x] MacOS
- [x] Linux
- [x] iOS
- [x] Android

## Crates

- `livekit-core`: LiveKit protocol implementation
- `livekit-utils`: Shared utilities between our crates
- `livekit-ffi`: Bindings for other languages. Uses `livekit-core`.
- `libwebrtc`: Safe Rust bindings to libwebrtc
- `webrtc-sys`: Unsafe bindings to libwebrtc
- `livekit-api`: Server APIs and auth token generation
- `livekit`: LiveKit real-time SDK
- `livekit-ffi`: Internal crate, used to generate bindings for other languages

## Motivation and Design Goals
## Getting started

LiveKit aims to provide an open source, end-to-end WebRTC stack that works everywhere. We have two goals in mind with this SDK:
Currently, Tokio is required to use this SDK, however we plan to make the async executor runtime agnostic.

1. Build a standalone, cross-platform LiveKit client SDK for Rustaceans.
2. Build a common core for other platform-specific SDKs (e.g. Unity, Unreal, iOS, Android)
## Using Server API

Regarding (2), we've already developed a number of [client SDKs](https://github.com/livekit?q=client-sdk&type=all) for several platforms and encountered a few challenges in the process:
### Generating an access token

- There's a significant amount of business/control logic in our signaling protocol and WebRTC. Currently, this logic needs to be implemented in every new platform we support.
- Interactions with media devices and encoding/decoding are specific to each platform and framework.
- For multi-platform frameworks (e.g. Unity, Flutter, React Native), the aforementioned tasks proved to be extremely painful.
```rust
use livekit_api::access_token;
use std::env;

fn create_token() -> Result<String, access_token::AccessTokenError> {
let api_key = env::var("LIVEKIT_API_KEY").expect("LIVEKIT_API_KEY is not set");
let api_secret = env::var("LIVEKIT_API_SECRET").expect("LIVEKIT_API_SECRET is not set");

let token = access_token::AccessToken::with_api_key(&api_key, &api_secret)
.with_identity("rust-bot")
.with_name("Rust Bot")
.with_grants(access_token::VideoGrants {
room_join: true,
room: "my-room".to_string(),
..Default::default()
})
.to_jwt();
return token
}
```

Thus, we posited a Rust SDK, something we wanted build anyway, encapsulating all our business logic and platform-specific APIs into a clean set of abstractions, could also serve as the foundation for our other SDKs!
### Creating a room with RoomService API

We'll first use it as a basis for our Unity SDK (under development), but over time, it will power our other SDKs, as well.
```rust
use livekit_api::services::room::{CreateRoomOptions, RoomClient};

## Getting started
#[tokio::main]
async fn main() {
let room_service = RoomClient::new("http://localhost:7880").unwrap();

Currently, Tokio is required to use this SDK, however we plan to make the async executor runtime agnostic.
let room = room_service
.create_room("my_room", CreateRoomOptions::default())
.await
.unwrap();

println!("Created room: {:?}", room);
}
```

## Using Real-time SDK

### Connect to a Room and listen for events:

Expand All @@ -70,76 +95,88 @@ use livekit::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
let (room, mut room_events) = Room::connect(&url, &token).await?;

while let Some(event) = room_events.recv().await {
match event {
RoomEvent::TrackSubscribed { track, publication, participant } => {
// ...
}
_ => {}
}
}

Ok(())
let (room, mut room_events) = Room::connect(&url, &token).await?;

while let Some(event) = room_events.recv().await {
match event {
RoomEvent::TrackSubscribed { track, publication, participant } => {
// ...
}
_ => {}
}
}

Ok(())
}
```

### Receive video frames of a subscribed track

```rust
...
use futures::StreamExt; // this trait is required for iterating on audio & video frames
use livekit::prelude::*;

match event {
RoomEvent::TrackSubscribed { track, publication, participant } => {
if let RemoteTrackHandle::Video(video_track) => {
let rtc_track = video_track.rtc_track();
rtc_track.on_frame(Box::new(move |frame, buffer| {
// Just received a video frame!
// The buffer is YuvEncoded, you can decode it to ABGR by using our yuv_helper
// See the basic_room example for the conversion
});
} else {
// Audio Track..
}
}
_ => {}
RoomEvent::TrackSubscribed { track, publication, participant } => {
match track {
RemoteTrack::Audio(audio_track) => {
let rtc_track = audio_track.rtc_track();
let mut audio_stream = NativeAudioStream::new(rtc_track);
tokio::spawn(async move {
// Receive the audio frames in a new task
while let Some(audio_frame) = audio_stream.next().await {
log::info!("received audio frame - {audio_frame:#?}");
}
});
},
RemoteTrack::Video(video_track) => {
let rtc_track = video_track.rtc_track();
let mut video_stream = NativeVideoStream::new(rtc_track);
tokio::spawn(async move {
// Receive the video frames in a new task
while let Some(video_frame) = video_stream.next().await {
log::info!("received video frame - {video_frame:#?}");
}
});
},
}
},
_ => {}
}
```

## Examples

We made a [basic room demo](https://github.com/livekit/client-sdk-native/tree/main/examples/basic_room) leveraging all the current SDK features. Videos are rendered using wgpu and egui.

![](https://github.com/livekit/client-sdk-rust/blob/main/examples/images/simple-room-demo.gif)
![](https://github.com/livekit/rust-sdks/blob/main/examples/images/simple-room-demo.gif)

## FAQ
- [basic room](https://github.com/livekit/rust-sdks/tree/main/examples/basic_room): simple example connecting to a room.
- [wgpu_room](https://github.com/livekit/rust-sdks/tree/main/examples/wgpu_room): complete example app with video rendering using wgpu and egui.
- [mobile](https://github.com/livekit/rust-sdks/tree/main/examples/mobile): mobile app targeting iOS and Android
- [play_from_disk](https://github.com/livekit/rust-sdks/tree/main/examples/play_from_disk): publish audio from a wav file
- [save_to_disk](https://github.com/livekit/rust-sdks/tree/main/examples/save_to_disk): save received audio to a wav file

### Do you plan to offer a C/C++ SDK?

Yes! In fact, we also plan to release an SDK for C++ in the coming months. It, like our other platform-specific SDKs, will use the Rust SDK. 🙂

### Did you consider C/C++ as your common core?
## Motivation and Design Goals

Yes. We chose Rust over C++ for a few reasons:
LiveKit aims to provide an open source, end-to-end WebRTC stack that works everywhere. We have two goals in mind with this SDK:

- Rust's ownership model and thread-safety leads to fewer crashes/issues.
- Rust's build system requires less configuration and is easier to work with.
- While we love C/C++, it's a bit nicer to write code in Rust.
- Rust has a rich ecosystem of tools (e.g. websockets, async executor).
- Having the WebAssembly target will be useful down the road, C++ has Emscripten but it's a bit harder to set up and doesn't yet have WebRTC support.
1. Build a standalone, cross-platform LiveKit client SDK for Rustaceans.
2. Build a common core for other platform-specific SDKs (e.g. Unity, Unreal, iOS, Android)

### Did you look at [Arcas](https://github.com/arcas-io/libwebrtc) for libwebrtc bindings?
Regarding (2), we've already developed a number of [client SDKs](https://github.com/livekit?q=client-sdk&type=all) for several platforms and encountered a few challenges in the process:

Yes. Our build system is inspired by LBL's work! Given that some of our logic (e.g. hardware decoder code) is in C++ and that we may need to bridge more/different things than Arcas, we decided it was better to have our own bindings for full control.
- There's a significant amount of business/control logic in our signaling protocol and WebRTC. Currently, this logic needs to be implemented in every new platform we support.
- Interactions with media devices and encoding/decoding are specific to each platform and framework.
- For multi-platform frameworks (e.g. Unity, Flutter, React Native), the aforementioned tasks proved to be extremely painful.

### Did you consider using [webrtc.rs](https://webrtc.rs/) instead of libwebrtc?
Yes! As webrtc.rs matures, we'll eventually migrate to a pure Rust stack. For now, we chose libwebrtc for a few reasons:
Thus, we posited a Rust SDK, something we wanted build anyway, encapsulating all our business logic and platform-specific APIs into a clean set of abstractions, could also serve as the foundation for our other SDKs!

- Chrome's adoption and usage means libwebrtc is thoroughly battle-tested.
- webrtc.rs is ported from Pion (which [our SFU](https://github.com/livekit/livekit) is built on) and a better fit for server-side use.
- libwebrtc currently supports more features like encoding/decoding and includes platform-specific code for dealing with media devices.
We'll first use it as a basis for our Unity SDK (under development), but over time, it will power our other SDKs, as well.

<!--BEGIN_REPO_NAV-->

<br/><table>

<thead><tr><th colspan="2">LiveKit Ecosystem</th></tr></thead>
<tbody>
<tr><td>Client SDKs</td><td><a href="https://github.com/livekit/components-js">Components</a> · <a href="https://github.com/livekit/client-sdk-js">JavaScript</a> · <a href="https://github.com/livekit/client-sdk-swift">iOS/macOS</a> · <a href="https://github.com/livekit/client-sdk-android">Android</a> · <a href="https://github.com/livekit/client-sdk-flutter">Flutter</a> · <a href="https://github.com/livekit/client-sdk-react-native">React Native</a> · <b>Rust</b> · <a href="https://github.com/livekit/client-sdk-python">Python</a> · <a href="https://github.com/livekit/client-sdk-unity-web">Unity (web)</a> · <a href="https://github.com/livekit/client-sdk-unity">Unity (beta)</a></td></tr><tr></tr>
Expand Down
10 changes: 10 additions & 0 deletions examples/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions examples/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ members = [
"save_to_disk",
"wgpu_room",
"webhooks",
"api",
]
10 changes: 10 additions & 0 deletions examples/api/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[package]
name = "api"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1", features = ["full", "parking_lot"] }
livekit-api = { path = "../../livekit-api", version = "0.2.0", features = ["native-tls"] }
futures = "0.3"

13 changes: 13 additions & 0 deletions examples/api/src/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
use livekit_api::services::room::{CreateRoomOptions, RoomClient};

#[tokio::main]
async fn main() {
let room_service = RoomClient::new("http://localhost:7880").unwrap();

let room = room_service
.create_room("my_room", CreateRoomOptions::default())
.await
.unwrap();

println!("Created room: {:?}", room);
}
1 change: 1 addition & 0 deletions examples/basic_room/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ edition = "2021"
tokio = { version = "1", features = ["full"] }
env_logger = "0.10"
livekit = { path = "../../livekit", version = "0.2.0", features = ["native-tls"]}
livekit-api = { path = "../../livekit-api", version = "0.2.0"}
log = "0.4"
16 changes: 15 additions & 1 deletion examples/basic_room/src/main.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use livekit_api::access_token;
use livekit::prelude::*;
use std::env;

Expand All @@ -9,7 +10,20 @@ async fn main() {
env_logger::init();

let url = env::var("LIVEKIT_URL").expect("LIVEKIT_URL is not set");
let token = env::var("LIVEKIT_TOKEN").expect("LIVEKIT_TOKEN is not set");
let api_key = env::var("LIVEKIT_API_KEY").expect("LIVEKIT_API_KEY is not set");
let api_secret = env::var("LIVEKIT_API_SECRET").expect("LIVEKIT_API_SECRET is not set");

let token = access_token::AccessToken::with_api_key(&api_key, &api_secret)
.with_identity("rust-bot")
.with_name("Rust Bot")
.with_grants(access_token::VideoGrants {
room_join: true,
room: "my-room".to_string(),
..Default::default()
})
.to_jwt()
.unwrap();


let (room, mut rx) = Room::connect(&url, &token, RoomOptions::default())
.await
Expand Down
2 changes: 1 addition & 1 deletion libwebrtc/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ edition = "2021"
homepage = "https://livekit.io"
license = "Apache-2.0"
description = "Livekit safe bindings to libwebrtc"
repository = "https://github.com/livekit/client-sdk-rust"
repository = "https://github.com/livekit/rust-sdks"

[dependencies]
livekit-protocol = { path = "../livekit-protocol", version = "0.2.0" }
Expand Down
2 changes: 1 addition & 1 deletion livekit-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.2.0"
license = "Apache-2.0"
description = "Rust Server SDK for LiveKit"
edition = "2021"
repository = "https://github.com/livekit/client-sdk-rust"
repository = "https://github.com/livekit/rust-sdks"

[features]
# By default ws TLS is not enabled
Expand Down
4 changes: 2 additions & 2 deletions livekit-ffi/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ name = "livekit-ffi"
version = "0.3.13"
edition = "2021"
license = "Apache-2.0"
description = "LiveKit interface to easily use the Rust SDK on other languages"
repository = "https://github.com/livekit/client-sdk-rust"
description = "FFI interface for bindings in other languages"
repository = "https://github.com/livekit/rust-sdks"

[features]
default = ["rustls-tls-native-roots"]
Expand Down
2 changes: 1 addition & 1 deletion livekit-protocol/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.2.0"
edition = "2021"
license = "Apache-2.0"
description = "Livekit protocol and utilities for the Rust SDK"
repository = "https://github.com/livekit/client-sdk-rust"
repository = "https://github.com/livekit/rust-sdks"

[dependencies]
tokio = { version = "1", features = ["full"] }
Expand Down
2 changes: 1 addition & 1 deletion livekit/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.2.0"
edition = "2021"
license = "Apache-2.0"
description = "Rust Client SDK for LiveKit"
repository = "https://github.com/livekit/client-sdk-rust"
repository = "https://github.com/livekit/rust-sdks"

[features]
# By default ws TLS is not enabled
Expand Down
Loading