Skip to content

Fix stop reason for a few providers#468

Closed
narengogi wants to merge 37 commits intoPortkey-AI:mainfrom
narengogi:unsync-anthropic-stop-reason
Closed

Fix stop reason for a few providers#468
narengogi wants to merge 37 commits intoPortkey-AI:mainfrom
narengogi:unsync-anthropic-stop-reason

Conversation

@narengogi
Copy link
Member

@narengogi narengogi commented Jul 20, 2024

Title:

  • Transform Stop Reason from Provider specific enums to OpenAI Response Schema compliant ones

Description:

  • I'm using Enums instead of string types to be explicit, it adds a few lines of code which would have been simply overlapping types if we used string types instead, I find this more readable
  • Did not make any of the existing response types non nullable

TO-DO

  • At some of the places, stop reason is hardcoded to be length/stop, I did not change those as part of this PR.
  • Types and Utils can be extracted from inside the providers complete.ts and chatComplete.ts files to make the code more readable

Motivation:

  • Portkey Gateway currently does not transform the stop_reason at a lot of places and it fails with strictly typed languages and fails validations.

Related Issues:

Also thanks to @unsync for his initial PR on this. I've extended it to all the providers

@narengogi narengogi force-pushed the unsync-anthropic-stop-reason branch from cb64d75 to 23e49b7 Compare July 20, 2024 11:23
@narengogi narengogi force-pushed the unsync-anthropic-stop-reason branch from 23e49b7 to 9214e19 Compare July 20, 2024 11:45
@narengogi narengogi force-pushed the unsync-anthropic-stop-reason branch from 2ba8c98 to fa1751a Compare July 23, 2024 11:48
@narengogi narengogi marked this pull request as ready for review July 24, 2024 12:03
@narengogi
Copy link
Member Author

@VisargD for your review

@roh26it roh26it requested a review from VisargD July 24, 2024 12:59
@VisargD
Copy link
Collaborator

VisargD commented Jul 26, 2024

Hey! This PR is not currently handling stop reason transformation for streaming calls.

@narengogi
Copy link
Member Author

provider chat chat streaming completion completion streaming
Ai21 N/A N/A
Anthropic
Anyscale
Azure OpenAI N/A N/A
Cohere
DeepInfra N/A N/A
Fireworks
Google
Groq N/A N/A
Lingyi N/A N/A
Mistral
MonsterAPI N/A N/A N/A
Moonshot N/A N/A N/A
Novita
Ollama [ ] [ ] [ ] [ ]
Openrouter Not possible, they return different stop reason depending on deployed model/provider
Palm N/A N/A
Perplexity N/A N/A
Predibase Not possible, they return different stop reason depending on deployed model/provider
Reka N/A N/A N/A
Together
Workers
Zhipu N/A N/A

@narengogi
Copy link
Member Author

@VisargD please review this, I'll have to make changes again if new providers are added

@narengogi
Copy link
Member Author

closing this for now as discussed with @VisargD and @ayush-portkey
Stems from decision not to transform any of the json values, and only stick to transforming keys.

Transforming stop reason Not Transforming
✅ Compatible with OpenAI SDK, even when type validation is applied (zod, pydantic, etc) Will break with OpenAI SDK if type validation applied
Context loss ✅ No Context loss
Added maintenance ✅ No maintenance

Illustrated example on why doing this would mean context loss:
VERTEX_AI provider returns following types with key finishReason

export enum finishReason {
  FINISH_REASON_UNSPECIFIED = 'FINISH_REASON_UNSPECIFIED',
  STOP = 'STOP',
  MAX_TOKENS = 'MAX_TOKENS',
  SAFETY = 'SAFETY',
  RECITATION = 'RECITATION',
  OTHER = 'OTHER',
  BLOCKLIST = 'BLOCKLIST',
  PROHIBITED_CONTENT = 'PROHIBITED_CONTENT',
  SPII = 'SPII',
}

since OPENAI only has 4 finish_reason enums, we map the above keys to one of the 4.

const transformVertexGeminiChatStopReason = (
  stopReason?: VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON | string
): OPEN_AI_CHAT_COMPLETION_FINISH_REASON => {
  switch (stopReason) {
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.STOP:
      return OPEN_AI_CHAT_COMPLETION_FINISH_REASON.stop;
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.SAFETY:
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.RECITATION:
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.BLOCKLIST:
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.PROHIBITED_CONTENT:
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.SPII:
      return OPEN_AI_CHAT_COMPLETION_FINISH_REASON.content_filter;
    case VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON.MAX_TOKENS:
      return OPEN_AI_CHAT_COMPLETION_FINISH_REASON.length;
    default:
      return OPEN_AI_CHAT_COMPLETION_FINISH_REASON.stop;
  }
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants