Intergration with pipecat framework

[wizd]

Summary

Pipecat is open source framework for voice and multimodal conversational AI. link

Motivation

PipeCat supports multiple WebRTC services modularly; however, there are currently no open-source providers available. Therefore, PipeCat serves as an excellent option for individuals looking to establish their own WebRTC solutions.

Describe alternatives you've considered

Pipecat + Pion will create a full stack of opensource webrtc solution.

Additional context

Real-Time Voice and Video Inference (RTVI)

[dsa]

Check out:
https://github.com/livekit/livekit
https://github.com/livekit/agents

It’s built on Pion, and fully open source including the transport.

[kwindla]

In case it's useful, there's a Pipecat LiveKit transport:

https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/services/livekit.py

Example:

https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/29-livekit-audio-chat.py

Separately, though, it would be really fun to build a little hackable WebRTC server designed from the ground up for realtime AI.

I've been thinking about this a little bit and would love to work on it if other people who are interested. Pion and Pipecat could fit together really nicely for this. I've written chunks of three or four SFUs (depending on how you count) over the years, and I think that the requirements/architecture for voice+vision AI are different from the things that shaped SFU design up until now.

[wizd]

In case it's useful, there's a Pipecat LiveKit transport:

https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/transports/services/livekit.py

Example:

https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/29-livekit-audio-chat.py

Separately, though, it would be really fun to build a little hackable WebRTC server designed from the ground up for realtime AI.

I've been thinking about this a little bit and would love to work on it if other people who are interested. Pion and Pipecat could fit together really nicely for this. I've written chunks of three or four SFUs (depending on how you count) over the years, and I think that the requirements/architecture for voice+vision AI are different from the things that shaped SFU design up until now.

I couldn't agree more. Next-gen agent ecosystems are set to run on WebRTC networks, and I believe we can work together to develop an improved server architecture for these systems.