Skip to content

Add local inference service for task summarization #219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add local inference service for task summarization #219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
24a3253
Add local inference service with summarize command
Nov 17, 2025
5445b73
add claude settings
Nov 17, 2025
ec38067
Fix gosec G602 slice bounds warning
Nov 17, 2025
c578df0
Add async inference loading with CATNIP_INFERENCE env flag
Nov 18, 2025
1cfccbe
Fix corepack pnpm installation with explicit versions
Nov 18, 2025
7f12371
Merge branch 'main' into fix/inference-bos-token
vanpelt Nov 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .claude/settings.local.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
"Bash(git commit:*)",
"WebFetch(domain:expo.dev)",
"WebSearch",
"Bash(find:*)",
"WebFetch(domain:github.com)",
"WebFetch(domain:raw.githubusercontent.com)"
"Skill(superpowers:brainstorming)",
"Bash(python3:*)"
],
Expand Down
134 changes: 134 additions & 0 deletions container/docs/docs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1315,6 +1315,78 @@ const docTemplate = `{
}
}
},
"/v1/inference/status": {
"get": {
"description": "Check if local inference is available and get service information",
"produces": [
"application/json"
],
"tags": [
"inference"
],
"summary": "Get inference service status",
"responses": {
"200": {
"description": "Inference service status",
"schema": {
"$ref": "#/definitions/internal_handlers.InferenceStatusResponse"
}
}
}
}
},
"/v1/inference/summarize": {
"post": {
"description": "Generate a short task summary and git branch name using local GGUF model",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"inference"
],
"summary": "Summarize task and generate branch name",
"parameters": [
{
"description": "Summarization request",
"name": "request",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/internal_handlers.SummarizeRequest"
}
}
],
"responses": {
"200": {
"description": "Successfully generated summary and branch name",
"schema": {
"$ref": "#/definitions/internal_handlers.SummarizeResponse"
}
},
"400": {
"description": "Invalid request",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
},
"500": {
"description": "Inference error",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
},
"503": {
"description": "Inference not available on this platform",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
}
}
}
},
"/v1/notifications": {
"post": {
"description": "Sends a notification event to all connected SSE clients, including the TUI app which can display native macOS notifications",
Expand Down Expand Up @@ -1846,6 +1918,10 @@ const docTemplate = `{
}
},
"definitions": {
"fiber.Map": {
"type": "object",
"additionalProperties": true
},
"github_com_vanpelt_catnip_internal_models.ClaudeActivityState": {
"type": "string",
"enum": [
Expand Down Expand Up @@ -3187,6 +3263,37 @@ const docTemplate = `{
}
}
},
"internal_handlers.InferenceStatusResponse": {
"description": "Status of the local inference service",
"type": "object",
"properties": {
"architecture": {
"description": "Architecture (amd64, arm64)",
"type": "string",
"example": "arm64"
},
"available": {
"description": "Whether inference is available on this platform",
"type": "boolean",
"example": true
},
"error": {
"description": "Error message if initialization failed",
"type": "string",
"example": "model not found"
},
"modelPath": {
"description": "Model path if loaded",
"type": "string",
"example": "/Users/user/.catnip/models/gemma3-270m-summarizer-Q4_K_M.gguf"
},
"platform": {
"description": "Platform name (darwin, linux, windows)",
"type": "string",
"example": "darwin"
}
}
},
"internal_handlers.NotificationPayload": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -3225,6 +3332,33 @@ const docTemplate = `{
"$ref": "#/definitions/internal_handlers.ActiveSessionInfo"
}
},
"internal_handlers.SummarizeRequest": {
"description": "Request to summarize a task and generate a branch name",
"type": "object",
"properties": {
"prompt": {
"description": "Task description or code changes to summarize",
"type": "string",
"example": "Add user authentication with OAuth2"
}
}
},
"internal_handlers.SummarizeResponse": {
"description": "Response containing task summary and suggested branch name",
"type": "object",
"properties": {
"branchName": {
"description": "Git branch name in kebab-case with category prefix",
"type": "string",
"example": "feat/add-user-auth"
},
"summary": {
"description": "2-4 word summary in Title Case",
"type": "string",
"example": "Add User Auth"
}
}
},
"internal_handlers.UploadResponse": {
"description": "Response containing upload status and file location",
"type": "object",
Expand Down
134 changes: 134 additions & 0 deletions container/docs/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1312,6 +1312,78 @@
}
}
},
"/v1/inference/status": {
"get": {
"description": "Check if local inference is available and get service information",
"produces": [
"application/json"
],
"tags": [
"inference"
],
"summary": "Get inference service status",
"responses": {
"200": {
"description": "Inference service status",
"schema": {
"$ref": "#/definitions/internal_handlers.InferenceStatusResponse"
}
}
}
}
},
"/v1/inference/summarize": {
"post": {
"description": "Generate a short task summary and git branch name using local GGUF model",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"inference"
],
"summary": "Summarize task and generate branch name",
"parameters": [
{
"description": "Summarization request",
"name": "request",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/internal_handlers.SummarizeRequest"
}
}
],
"responses": {
"200": {
"description": "Successfully generated summary and branch name",
"schema": {
"$ref": "#/definitions/internal_handlers.SummarizeResponse"
}
},
"400": {
"description": "Invalid request",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
},
"500": {
"description": "Inference error",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
},
"503": {
"description": "Inference not available on this platform",
"schema": {
"$ref": "#/definitions/fiber.Map"
}
}
}
}
},
"/v1/notifications": {
"post": {
"description": "Sends a notification event to all connected SSE clients, including the TUI app which can display native macOS notifications",
Expand Down Expand Up @@ -1843,6 +1915,10 @@
}
},
"definitions": {
"fiber.Map": {
"type": "object",
"additionalProperties": true
},
"github_com_vanpelt_catnip_internal_models.ClaudeActivityState": {
"type": "string",
"enum": [
Expand Down Expand Up @@ -3184,6 +3260,37 @@
}
}
},
"internal_handlers.InferenceStatusResponse": {
"description": "Status of the local inference service",
"type": "object",
"properties": {
"architecture": {
"description": "Architecture (amd64, arm64)",
"type": "string",
"example": "arm64"
},
"available": {
"description": "Whether inference is available on this platform",
"type": "boolean",
"example": true
},
"error": {
"description": "Error message if initialization failed",
"type": "string",
"example": "model not found"
},
"modelPath": {
"description": "Model path if loaded",
"type": "string",
"example": "/Users/user/.catnip/models/gemma3-270m-summarizer-Q4_K_M.gguf"
},
"platform": {
"description": "Platform name (darwin, linux, windows)",
"type": "string",
"example": "darwin"
}
}
},
"internal_handlers.NotificationPayload": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -3222,6 +3329,33 @@
"$ref": "#/definitions/internal_handlers.ActiveSessionInfo"
}
},
"internal_handlers.SummarizeRequest": {
"description": "Request to summarize a task and generate a branch name",
"type": "object",
"properties": {
"prompt": {
"description": "Task description or code changes to summarize",
"type": "string",
"example": "Add user authentication with OAuth2"
}
}
},
"internal_handlers.SummarizeResponse": {
"description": "Response containing task summary and suggested branch name",
"type": "object",
"properties": {
"branchName": {
"description": "Git branch name in kebab-case with category prefix",
"type": "string",
"example": "feat/add-user-auth"
},
"summary": {
"description": "2-4 word summary in Title Case",
"type": "string",
"example": "Add User Auth"
}
}
},
"internal_handlers.UploadResponse": {
"description": "Response containing upload status and file location",
"type": "object",
Expand Down
Loading