-
Notifications
You must be signed in to change notification settings - Fork 266
Description
Add ModelsLab Engine for Multi-Modal AI Generation Support
Analysis
GitHub Agentic Workflows currently supports 4 engines (Copilot, Claude, Codex, Gemini) focused primarily on text generation and coding tasks. There is no engine that provides comprehensive multi-modal AI generation capabilities including image, video, audio, and 3D content generation.
ModelsLab offers a unified API platform with 200+ AI models covering:
- Text Generation: GPT-4, Claude, Llama, Mistral, and specialized models
- Image Generation: FLUX, SDXL, community models, background removal, upscaling
- Video Generation: Text-to-video, image-to-video, video editing
- Audio/Voice: TTS, voice cloning, music generation
- 3D Generation: Text-to-3D, image-to-3D
- Creative Tools: Interior design, face enhancement, uncensored chat
Adding ModelsLab as the 5th engine will enable agentic workflows to generate rich multimedia content, expanding beyond text-only automation to full creative and technical content generation.
Implementation Plan
Please implement the following changes to add ModelsLab engine support:
1. Create ModelsLab Engine (pkg/workflow/modelslab_engine.go)
Create a new ModelsLab engine following the established patterns from copilot_engine.go and claude_engine.go:
package workflow
import (
"fmt"
"maps"
"strconv"
"strings"
"time"
"github.com/github/gh-aw/pkg/constants"
"github.com/github/gh-aw/pkg/logger"
)
var modelslabLog = logger.New("workflow:modelslab_engine")
// ModelsLabEngine represents the ModelsLab AI platform agentic engine
type ModelsLabEngine struct {
BaseEngine
}
func NewModelsLabEngine() *ModelsLabEngine {
return &ModelsLabEngine{
BaseEngine: BaseEngine{
id: "modelslab",
displayName: "ModelsLab",
description: "Uses ModelsLab API with 200+ models for text, image, video, audio, and 3D generation",
experimental: false,
supportsToolsAllowlist: true, // ModelsLab supports tool allowlisting
supportsMaxTurns: false, // ModelsLab does not support max-turns feature
supportsMaxContinuations: true, // ModelsLab supports multi-turn generation
supportsWebFetch: false, // ModelsLab does not have built-in web-fetch
supportsWebSearch: false, // ModelsLab does not have built-in web-search
supportsFirewall: true, // ModelsLab supports network firewalling via AWF
supportsPlugins: false, // ModelsLab does not support plugin installation
supportsLLMGateway: false, // ModelsLab does not support LLM gateway
},
}
}Engine Capabilities Configuration:
supportsToolsAllowlist: true- ModelsLab can restrict available toolssupportsMaxTurns: false- ModelsLab API doesn't have native max-turns supportsupportsWebFetch/WebSearch: false- ModelsLab doesn't have built-in web capabilitiessupportsFirewall: true- ModelsLab can work with AWF for network restrictionssupportsLLMGateway: false- ModelsLab doesn't provide LLM gateway functionality
2. Implement Required Interface Methods
Add the following methods to modelslab_engine.go:
// GetModelEnvVarName returns the native environment variable for ModelsLab API key
func (e *ModelsLabEngine) GetModelEnvVarName() string {
return "MODELSLAB_API_KEY"
}
// GetRequiredSecretNames returns the secrets required by ModelsLab engine
func (e *ModelsLabEngine) GetRequiredSecretNames(workflowData *WorkflowData) []string {
secrets := []string{"MODELSLAB_API_KEY"}
// Add MCP gateway API key if MCP servers are present
if HasMCPServers(workflowData) {
secrets = append(secrets, "MCP_GATEWAY_API_KEY")
}
// Add safe-inputs secret names
if IsSafeInputsEnabled(workflowData.SafeInputs, workflowData) {
safeInputsSecrets := collectSafeInputsSecrets(workflowData.SafeInputs)
for varName := range safeInputsSecrets {
secrets = append(secrets, varName)
}
}
return secrets
}
// GetSecretValidationStep returns secret validation for ModelsLab API key
func (e *ModelsLabEngine) GetSecretValidationStep(workflowData *WorkflowData) GitHubActionStep {
if workflowData.EngineConfig != nil && workflowData.EngineConfig.Command != "" {
modelslabLog.Printf("Skipping secret validation step: custom command specified (%s)", workflowData.EngineConfig.Command)
return GitHubActionStep{}
}
return GenerateMultiSecretValidationStep(
[]string{"MODELSLAB_API_KEY"},
"ModelsLab",
"https://github.github.com/gh-aw/reference/engines/#modelslab",
getEngineEnvOverrides(workflowData),
)
}3. Add Installation Steps Method
Implement GetInstallationSteps to install ModelsLab CLI or dependencies:
func (e *ModelsLabEngine) GetInstallationSteps(workflowData *WorkflowData) []GitHubActionStep {
modelslabLog.Printf("Generating installation steps for ModelsLab engine: workflow=%s", workflowData.Name)
// Skip installation if custom command is specified
if workflowData.EngineConfig != nil && workflowData.EngineConfig.Command != "" {
modelslabLog.Printf("Skipping installation steps: custom command specified (%s)", workflowData.EngineConfig.Command)
return []GitHubActionStep{}
}
var steps []GitHubActionStep
// Define engine configuration
config := EngineInstallConfig{
Secrets: []string{"MODELSLAB_API_KEY"},
DocsURL: "https://github.github.com/gh-aw/reference/engines/#modelslab",
NpmPackage: "modelslab-sdk",
Version: "latest",
Name: "ModelsLab",
CliName: "modelslab",
InstallStepName: "Install ModelsLab SDK",
}
// Add Node.js setup step first
npmSteps := GenerateNpmInstallSteps(
config.NpmPackage,
config.Version,
config.InstallStepName,
config.CliName,
true, // Include Node.js setup
)
if len(npmSteps) > 0 {
steps = append(steps, npmSteps[0]) // Setup Node.js step
}
// Add AWF installation if firewall is enabled
if isFirewallEnabled(workflowData) {
firewallConfig := getFirewallConfig(workflowData)
agentConfig := getAgentConfig(workflowData)
var awfVersion string
if firewallConfig != nil {
awfVersion = firewallConfig.Version
}
awfInstall := generateAWFInstallationStep(awfVersion, agentConfig)
if len(awfInstall) > 0 {
steps = append(steps, awfInstall)
}
}
// Add ModelsLab SDK installation
if len(npmSteps) > 1 {
steps = append(steps, npmSteps[1:]...)
}
return steps
}4. Add Execution Steps Method
Implement GetExecutionSteps for running ModelsLab workflows:
func (e *ModelsLabEngine) GetExecutionSteps(workflowData *WorkflowData, logFile string) []GitHubActionStep {
modelslabLog.Printf("Generating execution steps for ModelsLab engine: workflow=%s, firewall=%v", workflowData.Name, isFirewallEnabled(workflowData))
var steps []GitHubActionStep
// Build modelslab CLI arguments
var modelslabArgs []string
// Add API key via environment variable (secure)
// ModelsLab API key is passed via MODELSLAB_API_KEY env var
// Add MCP configuration if MCP servers are present
if HasMCPServers(workflowData) {
modelslabLog.Print("Adding MCP configuration")
modelslabArgs = append(modelslabArgs, "--mcp-config", "/tmp/gh-aw/mcp-config/mcp-servers.json")
}
// Add allowed tools configuration
allowedTools := e.computeAllowedModelsLabToolsString(workflowData.Tools, workflowData.SafeOutputs, workflowData.CacheMemoryConfig)
if allowedTools != "" {
modelslabArgs = append(modelslabArgs, "--allowed-tools", allowedTools)
}
// Add debug logging
modelslabArgs = append(modelslabArgs, "--debug-file", logFile)
modelslabArgs = append(modelslabArgs, "--verbose")
// Add output format for structured output
modelslabArgs = append(modelslabArgs, "--output-format", "json")
// Build command
commandName := "modelslab"
if workflowData.EngineConfig != nil && workflowData.EngineConfig.Command != "" {
commandName = workflowData.EngineConfig.Command
modelslabLog.Printf("Using custom command: %s", commandName)
}
commandParts := []string{commandName}
commandParts = append(commandParts, modelslabArgs...)
commandParts = append(commandParts, "\"$(cat /tmp/gh-aw/aw-prompts/prompt.txt)\"")
modelslabCommand := shellJoinArgs(commandParts)
// Build environment variables
env := map[string]string{
"MODELSLAB_API_KEY": "${{ secrets.MODELSLAB_API_KEY }}",
"GH_AW_PROMPT": "/tmp/gh-aw/aw-prompts/prompt.txt",
"GITHUB_WORKSPACE": "${{ github.workspace }}",
}
// Add MCP configuration env if present
if HasMCPServers(workflowData) {
env["GH_AW_MCP_CONFIG"] = "/tmp/gh-aw/mcp-config/mcp-servers.json"
}
// Add safe outputs configuration
applySafeOutputEnvToMap(env, workflowData)
// Add custom environment variables from engine config
if workflowData.EngineConfig != nil && len(workflowData.EngineConfig.Env) > 0 {
maps.Copy(env, workflowData.EngineConfig.Env)
}
// Build execution command with AWF if firewall enabled
var command string
if isFirewallEnabled(workflowData) {
// Get allowed domains for ModelsLab API
allowedDomains := []string{
"modelslab.com",
"api.modelslab.com",
"cdn.modelslab.com",
}
// Add network permissions from workflow
allowedDomains = append(allowedDomains, getNetworkAllowedDomains(workflowData.NetworkPermissions)...)
command = BuildAWFCommand(AWFCommandConfig{
EngineName: "modelslab",
EngineCommand: modelslabCommand,
LogFile: logFile,
WorkflowData: workflowData,
UsesTTY: false, // ModelsLab doesn't require TTY
UsesAPIProxy: false, // ModelsLab doesn't use LLM gateway
AllowedDomains: allowedDomains,
})
} else {
command = fmt.Sprintf(`set -o pipefail
%s 2>&1 | tee -a %s`, modelslabCommand, logFile)
}
// Generate execution step
stepName := "Execute ModelsLab Engine"
var stepLines []string
stepLines = append(stepLines, " - name: "+stepName)
stepLines = append(stepLines, " id: agentic_execution")
// Add timeout
if workflowData.TimeoutMinutes != "" {
timeoutValue := strings.TrimPrefix(workflowData.TimeoutMinutes, "timeout-minutes: ")
stepLines = append(stepLines, " timeout-minutes: "+timeoutValue)
} else {
stepLines = append(stepLines, fmt.Sprintf(" timeout-minutes: %d", int(constants.DefaultAgenticWorkflowTimeout/time.Minute)))
}
// Filter environment variables for security
allowedSecrets := e.GetRequiredSecretNames(workflowData)
filteredEnv := FilterEnvForSecrets(env, allowedSecrets)
// Format step with command and environment
stepLines = FormatStepWithCommandAndEnv(stepLines, command, filteredEnv)
steps = append(steps, GitHubActionStep(stepLines))
return steps
}5. Add Tool Configuration Methods
Implement methods for ModelsLab tool configuration:
// computeAllowedModelsLabToolsString computes the allowed tools string for ModelsLab CLI
func (e *ModelsLabEngine) computeAllowedModelsLabToolsString(tools map[string]any, safeOutputs map[string]any, cacheMemoryConfig *CacheMemoryConfig) string {
var allowedTools []string
// Add basic tools
allowedTools = append(allowedTools, "text_generation")
allowedTools = append(allowedTools, "image_generation")
allowedTools = append(allowedTools, "video_generation")
allowedTools = append(allowedTools, "audio_generation")
allowedTools = append(allowedTools, "3d_generation")
// Add MCP tools if present
for toolName := range tools {
if strings.HasPrefix(toolName, "mcp__") {
allowedTools = append(allowedTools, toolName)
}
}
return strings.Join(allowedTools, ",")
}
// GetLogParserScriptId returns the JavaScript parser for ModelsLab logs
func (e *ModelsLabEngine) GetLogParserScriptId() string {
return "parse_modelslab_log"
}6. Add Constants (pkg/constants/constants.go)
Add ModelsLab-specific constants following existing patterns:
// ModelsLab engine constants
const (
// ModelsLabAPIKeyEnvVar is the environment variable for ModelsLab API key
ModelsLabAPIKeyEnvVar = "MODELSLAB_API_KEY"
// DefaultModelsLabVersion is the default version of ModelsLab SDK to install
DefaultModelsLabVersion = "latest"
// ModelsLab model environment variables
EnvVarModelAgentModelsLab = "GH_AW_MODEL_AGENT_MODELSLAB"
EnvVarModelDetectionModelsLab = "GH_AW_MODEL_DETECTION_MODELSLAB"
)7. Register Engine (pkg/workflow/agentic_engine.go)
Update the NewEngineRegistry function to register ModelsLab engine:
func NewEngineRegistry() *EngineRegistry {
agenticEngineLog.Print("Creating new engine registry")
registry := &EngineRegistry{
engines: make(map[string]CodingAgentEngine),
}
// Register built-in engines
registry.Register(NewClaudeEngine())
registry.Register(NewCodexEngine())
registry.Register(NewCopilotEngine())
registry.Register(NewGeminiEngine())
registry.Register(NewModelsLabEngine()) // Add ModelsLab engine
agenticEngineLog.Printf("Registered %d engines", len(registry.engines))
return registry
}8. Add Error Patterns (pkg/workflow/modelslab_engine_tools.go)
Create error pattern detection for ModelsLab following the established pattern:
package workflow
// GetErrorPatterns returns error patterns for ModelsLab engine log analysis
func (e *ModelsLabEngine) GetErrorPatterns() []ErrorPattern {
return []ErrorPattern{
{
Pattern: "API key invalid",
Category: "authentication",
Severity: "high",
Description: "ModelsLab API key is invalid or expired",
},
{
Pattern: "Rate limit exceeded",
Category: "rate_limit",
Severity: "medium",
Description: "ModelsLab API rate limit exceeded",
},
{
Pattern: "Model not found",
Category: "model_error",
Severity: "high",
Description: "Specified model is not available in ModelsLab",
},
}
}9. Add Log Parsing (pkg/workflow/modelslab_logs.go)
Implement log parsing for ModelsLab following the established patterns:
package workflow
import (
"encoding/json"
"strings"
)
// ParseLogMetrics extracts metrics from ModelsLab log content
func (e *ModelsLabEngine) ParseLogMetrics(logContent string, verbose bool) LogMetrics {
metrics := LogMetrics{}
lines := strings.Split(logContent, "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" {
continue
}
// Parse JSON output from ModelsLab
var logEntry map[string]interface{}
if err := json.Unmarshal([]byte(line), &logEntry); err == nil {
if msgType, ok := logEntry["type"].(string); ok {
switch msgType {
case "generation_start":
metrics.TotalRequests++
case "generation_complete":
metrics.SuccessfulRequests++
case "error":
metrics.FailedRequests++
}
}
}
}
return metrics
}10. Add Tests (pkg/workflow/modelslab_engine_test.go)
Create comprehensive tests for ModelsLab engine following existing test patterns:
package workflow
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestModelsLabEngine_Creation(t *testing.T) {
engine := NewModelsLabEngine()
assert.Equal(t, "modelslab", engine.GetID())
assert.Equal(t, "ModelsLab", engine.GetDisplayName())
assert.Contains(t, engine.GetDescription(), "200+ models")
assert.False(t, engine.IsExperimental())
}
func TestModelsLabEngine_Capabilities(t *testing.T) {
engine := NewModelsLabEngine()
assert.True(t, engine.SupportsToolsAllowlist())
assert.False(t, engine.SupportsMaxTurns())
assert.True(t, engine.SupportsMaxContinuations())
assert.False(t, engine.SupportsWebFetch())
assert.False(t, engine.SupportsWebSearch())
assert.True(t, engine.SupportsFirewall())
assert.False(t, engine.SupportsPlugins())
assert.Equal(t, -1, engine.SupportsLLMGateway())
}
func TestModelsLabEngine_RequiredSecrets(t *testing.T) {
engine := NewModelsLabEngine()
workflowData := &WorkflowData{
Name: "test-workflow",
}
secrets := engine.GetRequiredSecretNames(workflowData)
require.Len(t, secrets, 1)
assert.Contains(t, secrets, "MODELSLAB_API_KEY")
}
func TestModelsLabEngine_SecretValidation(t *testing.T) {
engine := NewModelsLabEngine()
workflowData := &WorkflowData{
Name: "test-workflow",
}
step := engine.GetSecretValidationStep(workflowData)
require.NotEmpty(t, step)
stepStr := strings.Join(step, "\n")
assert.Contains(t, stepStr, "MODELSLAB_API_KEY")
assert.Contains(t, stepStr, "ModelsLab")
}
func TestModelsLabEngine_InstallationSteps(t *testing.T) {
engine := NewModelsLabEngine()
workflowData := &WorkflowData{
Name: "test-workflow",
}
steps := engine.GetInstallationSteps(workflowData)
require.NotEmpty(t, steps)
// Should include Node.js setup
firstStep := strings.Join(steps[0], "\n")
assert.Contains(t, firstStep, "Setup Node.js")
// Should include ModelsLab SDK installation
found := false
for _, step := range steps {
stepStr := strings.Join(step, "\n")
if strings.Contains(stepStr, "modelslab-sdk") {
found = true
break
}
}
assert.True(t, found, "Should include ModelsLab SDK installation step")
}11. Update Documentation (docs/src/content/docs/reference/engines.md)
Add ModelsLab engine documentation following existing engine documentation patterns:
## ModelsLab
The ModelsLab engine provides access to 200+ AI models for comprehensive multi-modal content generation.
### Setup
1. Get your API key from [ModelsLab](https://modelslab.com/api-keys)
2. Add `MODELSLAB_API_KEY` to your repository secrets
3. Use `engine: modelslab` in your workflow
### Capabilities
- **Text Generation**: GPT-4, Claude, Llama, Mistral, and specialized models
- **Image Generation**: FLUX, SDXL, community models, background removal, upscaling
- **Video Generation**: Text-to-video, image-to-video, video editing
- **Audio/Voice**: TTS, voice cloning, music generation
- **3D Generation**: Text-to-3D, image-to-3D
- **Creative Tools**: Interior design, face enhancement, uncensored chat
### Configuration
```yaml
engine: modelslab
model: gpt-4 # or any supported modelExamples
Create images:
engine: modelslab
prompt: Generate a logo for my startupGenerate videos:
engine: modelslab
prompt: Create a 10-second video of a cat playing
### 12. **Update Engine Selection** (`pkg/cli/engine_selection.go`)
If there's engine selection logic, update it to include ModelsLab:
```go
var SupportedEngines = []string{
"copilot",
"claude",
"codex",
"gemini",
"modelslab", // Add ModelsLab to supported engines
}
13. Follow Validation Architecture (pkg/workflow/modelslab_validation.go)
Add ModelsLab-specific validation following the validation architecture:
package workflow
import "fmt"
// validateModelsLabConfig validates ModelsLab-specific configuration
func validateModelsLabConfig(workflowData *WorkflowData) error {
if workflowData.Engine != "modelslab" {
return nil
}
// Validate API key is configured
if workflowData.EngineConfig != nil {
if _, exists := workflowData.EngineConfig.Env["MODELSLAB_API_KEY"]; !exists {
// Check if using custom command that might handle auth differently
if workflowData.EngineConfig.Command == "" {
return fmt.Errorf("MODELSLAB_API_KEY environment variable not configured. Expected secret reference like ${{ secrets.MODELSLAB_API_KEY }}. Example: env: { MODELSLAB_API_KEY: \"${{ secrets.MODELSLAB_API_KEY }}\" }")
}
}
}
return nil
}14. Add MCP Configuration Support
Implement MCP configuration if ModelsLab supports MCP servers:
// RenderMCPConfig renders MCP configuration for ModelsLab
func (e *ModelsLabEngine) RenderMCPConfig(yaml *strings.Builder, tools map[string]any, mcpTools []string, workflowData *WorkflowData) error {
// ModelsLab uses standard MCP configuration
// Implementation similar to Claude engine
return nil
}15. Add Firewall Domain Configuration
Add ModelsLab domains to firewall allowed domains:
// GetModelsLabAllowedDomains returns domains that ModelsLab engine needs access to
func GetModelsLabAllowedDomains() []string {
return []string{
"modelslab.com",
"api.modelslab.com",
"cdn.modelslab.com",
"storage.modelslab.com", // For generated content
}
}Technical Implementation Guidelines
Error Message Format
All validation errors must follow: [what's wrong]. [what's expected]. [example]
Example:
return fmt.Errorf("MODELSLAB_API_KEY not configured. Expected secret reference in engine.env. Example: env: { MODELSLAB_API_KEY: \"${{ secrets.MODELSLAB_API_KEY }}\" }")Console Output
Use styled console functions from pkg/console:
import "github.com/github/gh-aw/pkg/console"
fmt.Println(console.FormatSuccessMessage("ModelsLab engine configured successfully"))
fmt.Println(console.FormatInfoMessage("Installing ModelsLab SDK..."))
fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))File Organization
Follow the established engine patterns:
- Main engine file:
modelslab_engine.go - Tools configuration:
modelslab_engine_tools.go - Log parsing:
modelslab_logs.go - Validation:
modelslab_validation.go - Tests:
modelslab_engine_test.go
File Path Security
Use fileutil.ValidateAbsolutePath for all file operations:
import "github.com/github/gh-aw/pkg/fileutil"
cleanPath, err := fileutil.ValidateAbsolutePath(userInputPath)
if err != nil {
return fmt.Errorf("invalid path: %w", err)
}Quality Assurance
Required Checks
- Run
make agent-finishbefore completion - All methods follow existing engine patterns exactly
- Error messages use the standard format template
- Tests cover all public methods and error conditions
- Documentation follows established engine documentation format
- Console output uses styled formatting functions
Testing Strategy
- Unit tests for all engine methods
- Integration tests for installation and execution
- Error handling tests for invalid configurations
- MCP configuration tests if applicable
- Firewall domain tests
Breaking Changes
This is not a breaking change as it:
- Only adds a new engine option
- Does not modify existing engine behavior
- Does not change CLI commands or flags
- Does not alter JSON output structure
- Maintains backward compatibility
Documentation Updates
- Add ModelsLab section to engines documentation
- Include setup instructions with API key configuration
- Provide examples for different content types (text, image, video, audio)
- Document capabilities and limitations
- Add troubleshooting section for common issues
Implementation Success Criteria:
✅ ModelsLab engine can be selected with engine: modelslab
✅ Installation steps properly install ModelsLab SDK
✅ Execution steps run ModelsLab CLI with proper authentication
✅ Secret validation ensures MODELSLAB_API_KEY is configured
✅ MCP integration works if MCP servers are defined
✅ Firewall integration allows ModelsLab API domains
✅ Error handling provides clear, actionable error messages
✅ All tests pass and follow existing test patterns
✅ Documentation is complete and follows existing format
This implementation will expand GitHub Agentic Workflows from text-only automation to comprehensive multi-modal AI content generation, enabling workflows to create images, videos, audio, and 3D content alongside traditional coding tasks.