Skip to content

Session API integration #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Session API integration #29

wants to merge 21 commits into from
+189 −1,337

Conversation

@pblazej
Copy link
Contributor

@pblazej pblazej commented Aug 25, 2025 •
edited
Loading

  • Extracts many parts of AppViewModel into Session
  • Simplifies internal dependencies, etc. so that the app is mostly a wrapper around (extracted) Session - ready to be consumed by UI

Needs SDK+components bump

@pblazej pblazej mentioned this pull request Aug 25, 2025
Closed
@pblazej pblazej force-pushed the blaze/agent-abstraction branch 2 times, most recently from 449ffa3 to 8182fad Compare August 27, 2025 07:21
@pblazej
Copy link
Contributor Author

pblazej commented Sep 3, 2025

As for known issues: the visualizer for avatars probably won't work in the "minimal" case, due to wrong order of fallbacks, will fix this soon - let's focus on the public APIs.

@pblazej pblazej requested review from bcherry and lukasIO September 3, 2025 11:58
1egoman
1egoman reviewed Sep 3, 2025
View reviewed changes
Copy link

@1egoman 1egoman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some notes - I think some of the discussion on the web / react pull request also applies here too, and we just need to come to some consistent decisions both implementations can adopt.

@pblazej
Copy link
Contributor Author

pblazej commented Sep 4, 2025

@1egoman thank you for digging into that (again) 😄 IMO the areas of improvement are still the same (not that many of them). Let it serve as our bullet points for the meeting.

@pblazej
Copy link
Contributor Author

pblazej commented Sep 10, 2025

@lukasIO @1egoman recent improvements and notable mentions:

  • moved Message creation, split userInput from userTranscript
  • removed Features in favor of app-level FeatureSwitches
  • exposed _room
  • added CredentialsProvider, let's try to finalize that in the web pr first 🤞
  • moved all media/track related stuff to what's called DeviceSwitcher for now (better name LocalMedia ?)
  • added agentName "handling" - that should be enough to "wait for this particular agent"

@pblazej
Copy link
Contributor Author

pblazej commented Sep 11, 2025 •
edited
Loading

@lukasIO re: separate "media/devices hook" - I updated the minimal example above 🫴

  MinimalExample()
                .environmentObject(DeviceSwitcher(room: room))
                .environmentObject(AgentSession(credentials: Sandbox(id: "your-id"), room: room))

The key is both must point to the same Room (created externally, no way to make it implicit). Also, you gotta think about initialization (cannot instantiate prop with another prop basically).

The only solution it to make them coupled, so that the Room is passed implicitly like that d8a2e04. I think it's fairly easy to understand and harder to break.

@pblazej pblazej force-pushed the blaze/agent-abstraction branch from d8a2e04 to 25ce2cf Compare September 18, 2025 12:18
@pblazej pblazej changed the title [Draft] Agent abstraction [WIP] Agent API integration Sep 18, 2025
@pblazej pblazej mentioned this pull request Sep 18, 2025
Merged
bcherry
bcherry reviewed Sep 22, 2025
View reviewed changes
Copy link
Contributor

@bcherry bcherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a really nice simplification

bcherry
bcherry reviewed Sep 22, 2025
View reviewed changes
VoiceAgent/VoiceAgentApp.swift Outdated
private static let sandboxId = Bundle.main.object(forInfoDictionaryKey: "LiveKitSandboxId") as! String
@StateObject private var conversation = Conversation(credentials: CachingCredentialsProvider(SandboxTokenServer(id: Self.sandboxId)),
// agentName: ...
room: Room(roomOptions: RoomOptions(defaultScreenShareCaptureOptions: ScreenShareCaptureOptions(useBroadcastExtension: true))))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should include some notes about alternatives and also update the readme instructions to match the changes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, let's try to merge the pyramid of PRs first, I need to bump sdk+components anyway here 👍

@pblazej pblazej changed the title [WIP] Agent API integration [WIP] Session API integration Oct 16, 2025
@pblazej pblazej force-pushed the blaze/agent-abstraction branch from 904bbb0 to b329d36 Compare October 23, 2025 10:01
@pblazej pblazej force-pushed the blaze/agent-abstraction branch from 1c08e99 to ea30032 Compare October 27, 2025 09:17
@pblazej pblazej changed the title [WIP] Session API integration Session API integration Oct 27, 2025
@pblazej pblazej marked this pull request as ready for review October 27, 2025 09:26
@pblazej pblazej requested a review from hiroshihorie October 27, 2025 09:26
pblazej added a commit to livekit/client-sdk-swift that referenced this pull request Oct 27, 2025
@pblazej
Session API (#789)
845aee2 
Adds 3 basic building blocks for simple(r) agent experiences:
- `Session` - connection, pre-connect, agent dispatch, agent filtering
(e.g. by name), all agents, messages (broadcasted and aggregated for
now)
- `Agent` - wrapper around `Participant`, knows its tracks and internal
state
- `LocalMedia` - (unrelated) helper to deal with local tracks in SwiftUI

Example: livekit-examples/agent-starter-swift#29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bcherry bcherry bcherry left review comments

@1egoman 1egoman 1egoman left review comments

@lukasIO lukasIO Awaiting requested review from lukasIO

@hiroshihorie hiroshihorie Awaiting requested review from hiroshihorie

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pblazej @bcherry @1egoman @lukasIO