Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(azure): Realtime API support #1287

Merged
merged 3 commits into from
Jan 29, 2025

Conversation

deyaaeldeen
Copy link
Contributor

Support Azure Realtime API

In order to do so, this PR adds an azure factory method to OpenAIRealtimeWebSocket and OpenAIRealtimeWS classes that can handle asynchronous operations required during WebSocket connection setup.

Key Changes

1. New azure Method

  • Previously, the WebSocket connection was established in the class constructor, which could not handle async operations like fetching Azure AD tokens.
  • The new azure method enables asynchronous fetching of the Azure AD token and then pass it to the class constructor.

2. New _getAzureADToken Method in AzureOpenAI

  • Adds a _getAzureADToken method in the AzureOpenAI class to provide refreshed Azure AD tokens.

@deyaaeldeen deyaaeldeen force-pushed the feat/azure-rt-factory branch from e90657b to f2e649f Compare January 24, 2025 19:07
Copy link
Collaborator

@RobertCraigie RobertCraigie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@@ -9,7 +9,7 @@ async function main() {
rt.send({
type: 'session.update',
session: {
modalities: ['foo'] as any,
modalities: ['text'],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops thanks for fixing!

Comment on lines 86 to 92
if (azureCheck) {
if (this.url.searchParams.get('Authorization') !== null) {
this.url.searchParams.set('Authorization', '<REDACTED>');
} else {
this.url.searchParams.set('api-key', '<REDACTED>');
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why do we need to do this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We try not to expose secrets and this.url is exposed on the client. Follows the same spirit of #1218

@@ -26,6 +26,7 @@ export class OpenAIRealtimeWebSocket extends OpenAIRealtimeEmitter {
props: {
model: string;
dangerouslyAllowBrowser?: boolean;
onUrl?: (url: URL) => void;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Suggested change
onUrl?: (url: URL) => void;
onURL?: (url: URL) => void;

also would be nice to mention / mark this as internal-only

Suggested change
onUrl?: (url: URL) => void;
/**
* Callback to mutate the URL, needed for Azure.
* @internal
*/
onURL?: (url: URL) => void;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9a73f6f

this.url = buildRealtimeURL(client, props.model);
props.onUrl?.(this.url);

const azureCheck = isAzure(client);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional suggestion: I would personally find this easier to read if isAzure(client) was just inlined instead of extracted to a variable like this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9a73f6f

@@ -1,6 +1,7 @@
import { RealtimeClientEvent, RealtimeServerEvent, ErrorEvent } from '../../resources/beta/realtime/realtime';
import { EventEmitter } from '../../lib/EventEmitter';
import { OpenAIError } from '../../error';
import OpenAI, { AzureOpenAI } from 'openai';
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Suggested change
import OpenAI, { AzureOpenAI } from 'openai';
import OpenAI, { AzureOpenAI } from '../../index';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9a73f6f

@@ -509,6 +509,26 @@ const result = await openai.chat.completions.create({
console.log(result.choices[0]!.message?.content);
```

### Realtime API
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @kwhinnery-openai I suspect you'll want to do some wordsmithing here

Copy link
Contributor

@kwhinnery-openai kwhinnery-openai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will follow on with some copy edits, but no blocking feedback

@RobertCraigie RobertCraigie changed the title [Azure] Support Realtime API feat(azure): Realtime API support Jan 29, 2025
@RobertCraigie RobertCraigie merged commit fe090c0 into openai:next Jan 29, 2025
3 checks passed
@stainless-app stainless-app bot mentioned this pull request Jan 29, 2025
stainless-app bot pushed a commit that referenced this pull request Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants