feat(oracle): Add rerank and guardrails support#1540
feat(oracle): Add rerank and guardrails support#1540fede-kamel wants to merge 10 commits intoPortkey-AI:mainfrom
Conversation
… tests Oracle GenAI Provider Enhancements: - Add embeddings support (embed.ts) with Cohere model compatibility - Add model configuration system (modelConfig.ts) for model-specific behaviors - Add model handler classes for different families (Meta, Gemini, GPT, Grok, Cohere) - Support reasoningContent for GPT-OSS reasoning models - Handle content in multiple formats (array, string, reasoningContent) - Add oracleEndpointId to Options type for dedicated serving mode Testing: - Add comprehensive unit tests (61 tests passing) - Add integration tests for chat completions and embeddings - Add run-tests.sh script for easy test execution with OCI profiles Models tested: - Google Gemini 2.5 (flash, flash-lite, pro) - OpenAI GPT-OSS (20b, 120b) - Meta Llama Maverick - Cohere embeddings (v4.0, v3.0 variants)
- Add usesMaxCompletionTokens() helper for GPT-5+ model detection - Update chatRequest transform to use maxCompletionTokens for GPT-5 models - Add model configs for openai.gpt-5 and openai.gpt-4o families - Add OracleStreamChunk type definition for streaming responses - Set appropriate minTokens (500) for GPT-5 reasoning models
- Test multiple tool calls in a single response (weather, time, stock) - Test parallel tool calls with streaming - Verify models can call 2+ tools simultaneously
Implement /v1/rerank endpoint for Oracle GenAI, enabling document relevance ranking for RAG pipelines. - Add rerankHandler for standard /v1/rerank route - Add Oracle rerank request/response transforms - Map to Oracle's /actions/rerankText API - Support top_n, return_documents, max_chunks_per_doc params The rerank endpoint type was already defined in Portkey's endpointStrings, making this a safe addition that extends the existing API surface.
- Add unit tests for OracleRerankConfig transformations - Add unit tests for OracleRerankResponseTransform - Add integration test scaffold for /v1/rerank endpoint - Update test resources with rerank constants and templates - Add Oracle-specific test configuration for rerank Unit tests verify: - Query to input transformation - Document array transformations (string and object formats) - ServingMode configuration with ON_DEMAND default - CompartmentId extraction from provider options - Parameter mappings (top_n, return_documents, max_chunks_per_doc) - Response transformation for success and error cases
Integration test for Oracle GenAI rerank endpoint that validates: - Request parameter transformations (query, documents, servingMode) - Response transformations (results, scores, documents) - Error response handling Note: Cohere rerank models (cohere.rerank-v3.5, etc.) were retired for ON_DEMAND serving in OCI GenAI as of May 2025. Live API tests require a DEDICATED AI cluster with RERANK_COHERE shape. Run with: npx tsx src/tests/oracle-rerank.integration.test.ts
Only Cohere rerank models are supported in Oracle GenAI: - cohere.rerank-v3.5 - cohere.rerank-multilingual-v3.1 - cohere.rerank-english-v3.1 The validateRerankModel function throws an error if: - Model is undefined or empty - Model doesn't start with "cohere.rerank" This prevents confusing 404 errors when using unsupported models.
As of Feb 2026, OCI GenAI rerank models are only available via DEDICATED AI clusters. ON_DEMAND serving has been retired. Changes: - Add documentation about DEDICATED cluster requirement - Support oracleEndpointId for DEDICATED mode - Add tests for DEDICATED mode with endpointId To use rerank: 1. Create a dedicated AI cluster with RERANK_COHERE shape 2. Set oracleServingMode: 'DEDICATED' 3. Set oracleEndpointId to your endpoint OCID
- Add guardrails endpoint for content moderation, PII detection, and prompt injection - Transform OpenAI moderation format to/from Oracle guardrails API - Add getProxyEndpoint to handle /moderations proxy requests - Add listModels utility for model discovery - Include comprehensive unit tests (13 tests)
- Fix response interface to use camelCase keys matching actual OCI API (contentModeration, personallyIdentifiableInformation, promptInjection) - Update hasPII check to use Array.isArray() for proper detection - Update unit tests to match real API response format - Add integration tests that call real OCI GenAI Guardrails API (skipped by default, run with OCI_PROFILE=<profile>)
fede-kamel
changed the title
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive rerank and guardrails (content moderation) support to the Oracle GenAI provider, building upon the foundation established in PR #1537. The implementation provides OpenAI-compatible interfaces for Oracle's GenAI services.
Changes:
- Adds rerank endpoint mapping Cohere's API format to OCI GenAI
/actions/rerankText - Implements guardrails for content moderation, PII detection, and prompt injection detection
- Enhances chat completion with improved tool call handling and multimodal content support
Reviewed changes
Copilot reviewed 38 out of 39 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/providers/oracle/rerank.ts |
Implements rerank endpoint with Cohere model validation |
src/providers/oracle/guardrails.ts |
Implements content moderation and PII detection |
src/providers/oracle/listModels.ts |
Utility for listing available OCI GenAI models |
src/providers/oracle/modelConfig.ts |
Centralized model configuration and capabilities |
src/providers/oracle/handlers/ |
Model-specific handlers for different families |
src/providers/oracle/chatComplete.ts |
Enhanced with tool calls, multimodal content, and GPT-5 support |
src/providers/oracle/embed.ts |
Embeddings implementation with serving mode support |
src/handlers/rerankHandler.ts |
New handler for rerank endpoint |
src/index.ts |
Registers /v1/rerank route |
src/utils/env.ts |
Fixes module loading to use synchronous require |
src/types/requestBody.ts |
Adds Oracle-specific types and rerank parameters |
| Test files | Comprehensive unit and integration tests for all new features |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export const createOracleHeaders = (oracleConfig: OracleConfig) => { | ||
| return { | ||
| 'x-portkey-provider': 'oracle', | ||
| 'Content-Type': 'application/json', | ||
| 'x-portkey-oracle-tenancy': oracleConfig.tenancy, | ||
| 'x-portkey-oracle-user': oracleConfig.user, | ||
| 'x-portkey-oracle-fingerprint': oracleConfig.fingerprint, | ||
| 'x-portkey-oracle-private-key': oracleConfig.privateKey, | ||
| 'x-portkey-oracle-region': oracleConfig.region, | ||
| 'x-portkey-oracle-compartment-id': oracleConfig.compartmentId, | ||
| ...(oracleConfig.servingMode && { | ||
| 'x-portkey-oracle-serving-mode': oracleConfig.servingMode, | ||
| }), | ||
| }; |
There was a problem hiding this comment.
The createOracleHeaders function does not include the oracleEndpointId header (x-portkey-oracle-endpoint-id) which is required for DEDICATED mode rerank operations. This means the rerank tests won't be able to pass the endpoint ID through headers even if it's set in the OracleConfig. Add support for the endpoint ID header similar to how servingMode is handled.
| privateKey: string; | ||
| region: string; | ||
| compartmentId: string; | ||
| servingMode?: string; |
There was a problem hiding this comment.
The OracleConfig interface is missing the endpointId field which is required for DEDICATED mode operations (rerank, potentially embeddings). This field should be added to the interface and populated from environment variables similar to other Oracle configuration fields.
| servingMode?: string; | |
| servingMode?: string; | |
| endpointId?: string; |
| async function loadNodeModules() { | ||
| if (isNodeInstance && !modulesLoaded) { | ||
| path = await import('path'); | ||
| fs = await import('fs'); | ||
| modulesLoaded = true; | ||
| } | ||
| } |
There was a problem hiding this comment.
The async loadNodeModules function is defined but never called or used. It's redundant since the modules are loaded synchronously below it. This unused function should be removed to avoid confusion.
| if (servingMode === 'DEDICATED') { | ||
| return { | ||
| servingType: 'DEDICATED', | ||
| endpointId: provider.oracleEndpointId, | ||
| }; | ||
| } | ||
| return { | ||
| servingType: 'ON_DEMAND', | ||
| modelId: params.model, | ||
| }; |
There was a problem hiding this comment.
When using DEDICATED serving mode without an endpointId, the code should either throw an error or provide a clear warning. Currently, it silently falls back to using modelId, which may result in an API error. This could be confusing for users who explicitly set DEDICATED mode but forget to provide the endpointId. Consider adding validation to ensure endpointId is present when DEDICATED mode is used.
| const servingType = providerOptions.oracleServingMode || 'ON_DEMAND'; | ||
|
|
||
| // For DEDICATED mode, use endpointId instead of modelId | ||
| if (servingType === 'DEDICATED' && providerOptions.oracleEndpointId) { | ||
| return { | ||
| servingType: 'DEDICATED', | ||
| endpointId: providerOptions.oracleEndpointId, | ||
| }; | ||
| } | ||
|
|
||
| return { | ||
| servingType, | ||
| modelId: params.model, | ||
| }; |
There was a problem hiding this comment.
Similar to the embed.ts issue, when DEDICATED serving mode is used without an endpointId, the code silently falls back to using modelId. This could cause confusing API errors. Consider adding validation to ensure endpointId is provided when DEDICATED mode is explicitly set, or document this fallback behavior clearly.
| const flagged: boolean = | ||
| overallScore > 0.5 || promptInjectionScore > 0.5 || hasPII; |
There was a problem hiding this comment.
The threshold of 0.5 for determining if content is flagged is hardcoded and not configurable. Different use cases may require different sensitivity levels. Consider making this threshold configurable through request parameters or provider options, allowing users to adjust the sensitivity of the moderation system based on their requirements.
|
@narengogi let's start reviewing #1537 better - this one comesafter |
|
Rebased check done: branch is already up to date with Validation run:
Re-pinging for review. |
|
Retested everything end-to-end and we are good to go here. Latest validation:
All good from my side for merge. |
|
@narengogi quick ping when you have a moment: everything has been rebased and retested end-to-end, and this is ready from my side. Let’s make the OCI <> Portkey integration great together 🚀 |
|
Friendly follow-up on this one when you have a moment: @narengogi @VisargD\n\nWould appreciate a review. Thank you! |
Summary
Adds two new endpoints for Oracle GenAI:
New Features
Rerank Endpoint
/actions/rerankTextcohere.rerank-v3.5,cohere.rerank-multilingual-v3.1,cohere.rerank-english-v3.1Guardrails (Content Moderation)
/v1/moderationsformat to OCI/actions/applyGuardrailsModel Listing
/v1/modelsformatTesting
Unit Tests
rerank.test.ts- 26 testsguardrails.test.ts- 13 testsIntegration Tests (Real API)
guardrails.integration.test.ts- 4 tests against real OCI GenAI APIoracle-rerank.integration.test.ts- Transform validation testsRun integration tests:
Files Changed (new in this PR)
src/providers/oracle/rerank.tssrc/providers/oracle/rerank.test.tssrc/providers/oracle/guardrails.tssrc/providers/oracle/guardrails.test.tssrc/providers/oracle/guardrails.integration.test.tssrc/providers/oracle/listModels.ts