feat:Add API endpoint for grouped examples with associated runs #184
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
WalkthroughThis pull request introduces a new API endpoint for fetching examples with associated runs, grouped by a specified field. It updates the API documentation and enhances the query schema with additional parameters such as offset, limit, preview, and sort options. New schemas for grouping the examples and handling the response have been added alongside a new method in the dataset views to support these changes. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant API as Endpoint Handler
participant DB as Database
Client->>API: GET /api/v1/datasets/{dataset_id}/group/runs
API->>DB: Query grouped examples using QueryGroupedExamplesWithRuns
DB-->>API: Return grouped examples data
API-->>Client: Respond with GroupedExamplesWithRunsResponse
Possibly related PRs
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
coderabbitai
bot
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
src/libs/LangSmith/openapi.yaml (4)
3321-3362: New Endpoint Definition for Grouped Runs
The new/api/v1/datasets/{dataset_id}/group/runsendpoint is well-structured with clearly defined tags, summary, parameters, request body, responses, and security requirements. The description indicates that the endpoint handles associated run fetching based on session IDs; ensure that the documentation (or code comments elsewhere) clarifies exactly how grouping is determined and which field from the request body drives the grouping.
16046-16069: Introduction of GroupExampleRunsByField and GroupedExamplesWithRunsResponse
The new enumGroupExampleRunsByFieldwith valuesrun_metadataandexample_metadatais clear and concise. TheGroupedExamplesWithRunsResponseschema appropriately mandates agroupsproperty, which is an array ofExampleWithRunsGroupobjects. If possible, consider adding example values or additional descriptive documentation to further assist API consumers in understanding the expected data.
18288-18305: Enhancement of Query Parameters for Grouping
The added pagination and preview parameters (offset,limit, andpreview) effectively extend query capabilities. To maintain consistency with the integer type, consider using integer literals (e.g., use1instead of1.0) in theminimumandmaximumconstraints, even though JSON Schema may accept decimals for integers. This small refinement can improve clarity in schema definitions.
18369-18411: Definition of QueryGroupedExamplesWithRuns Schema
The new schemaQueryGroupedExamplesWithRunsconsolidates several key parameters needed by the endpoint, includingsession_ids, pagination options (offset,limit,preview),group_by,metadata_key, andper_group_limit. All required fields are appropriately specified. Similar to the previous comment, consider using whole number literals for numeric boundaries (e.g.,0instead of0.0and1instead of1.0) for enhanced clarity. Ensure that the documentation clearly communicates the purpose of each field, especially for new API consumers who may depend on these details when constructing queries.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (22)
src/libs/LangSmith/Generated/JsonConverters.GroupExampleRunsByField.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/JsonConverters.GroupExampleRunsByFieldNullable.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/JsonSerializerContext.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/JsonSerializerContextTypes.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.DatasetsClient.ReadExamplesWithRuns.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.DatasetsClient.ReadExamplesWithRunsGrouped.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.IDatasetsClient.ReadExamplesWithRuns.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.IDatasetsClient.ReadExamplesWithRunsGrouped.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.IPublicClient.ReadSharedDatasetExamplesWithRuns.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroup.Json.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroup.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroupFeedbackStats.Json.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroupFeedbackStats.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroupGroupKey.Json.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.ExampleWithRunsGroupGroupKey.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.GroupExampleRunsByField.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.GroupedExamplesWithRunsResponse.Json.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.GroupedExamplesWithRunsResponse.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.QueryExampleSchemaWithRuns.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.QueryGroupedExamplesWithRuns.Json.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.Models.QueryGroupedExamplesWithRuns.g.csis excluded by!**/generated/**src/libs/LangSmith/Generated/LangSmith.PublicClient.ReadSharedDatasetExamplesWithRuns.g.csis excluded by!**/generated/**
📒 Files selected for processing (1)
src/libs/LangSmith/openapi.yaml(6 hunks)
🔇 Additional comments (1)
src/libs/LangSmith/openapi.yaml (1)
18314-18321: Integration of Sort Parameters
The introduction of thesort_paramsproperty—referencing theSortParamsForRunsComparisonViewschema—enhances the query's flexibility for sorting results. Please ensure that the referenced schema is defined comprehensively elsewhere in your OpenAPI document so that it meets the sorting requirements for this endpoint.
| ExampleWithRunsGroup: | ||
| title: ExampleWithRunsGroup | ||
| required: | ||
| - filter | ||
| - group_key | ||
| - examples | ||
| type: object | ||
| properties: | ||
| filter: | ||
| title: Filter | ||
| type: string | ||
| count: | ||
| title: Count | ||
| type: integer | ||
| nullable: true | ||
| total_tokens: | ||
| title: Total Tokens | ||
| type: integer | ||
| nullable: true | ||
| total_cost: | ||
| title: Total Cost | ||
| type: string | ||
| nullable: true | ||
| min_start_time: | ||
| title: Min Start Time | ||
| type: string | ||
| format: date-time | ||
| nullable: true | ||
| max_start_time: | ||
| title: Max Start Time | ||
| type: string | ||
| format: date-time | ||
| nullable: true | ||
| latency_p50: | ||
| title: Latency P50 | ||
| type: number | ||
| nullable: true | ||
| latency_p99: | ||
| title: Latency P99 | ||
| type: number | ||
| nullable: true | ||
| feedback_stats: | ||
| title: Feedback Stats | ||
| type: object | ||
| nullable: true | ||
| group_key: | ||
| title: Group Key | ||
| examples: | ||
| title: Examples | ||
| anyOf: | ||
| - type: array | ||
| items: | ||
| $ref: '#/components/schemas/ExampleWithRuns' | ||
| - type: array | ||
| items: | ||
| $ref: '#/components/schemas/ExampleWithRunsCH' | ||
| prompt_tokens: | ||
| title: Prompt Tokens | ||
| type: integer | ||
| nullable: true | ||
| completion_tokens: | ||
| title: Completion Tokens | ||
| type: integer | ||
| nullable: true | ||
| prompt_cost: | ||
| title: Prompt Cost | ||
| type: string | ||
| nullable: true | ||
| completion_cost: | ||
| title: Completion Cost | ||
| type: string | ||
| nullable: true | ||
| error_rate: | ||
| title: Error Rate | ||
| type: number | ||
| nullable: true | ||
| description: Group of examples with runs. |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Addition of ExampleWithRunsGroup Schema
The newly added ExampleWithRunsGroup schema is comprehensive and covers important properties such as filtering, counts, tokens, cost, and timing information. However, the group_key property is missing a type definition. It is required according to the schema’s required list, so please add its type (for example, "type": "string") to enforce schema consistency.
Summary by CodeRabbit