`/v1/images/*` support?

[mord0d]

There's a fork with server support (but it's not fully compatible with OpenAI specs). Any chance it could be supported?

[mostlygeek]

You can try it with the /upstream/:modelname endpoint. This is just a pass through HTTP proxy.

[mord0d]

You can try it with the /upstream/:modelname endpoint. This is just a pass through HTTP proxy.

I saw this in README, thanks.

But I'm thinking about /v1/images/* endpoints (at least /v1/images/generations). Any plans for this?
There's not much open-source implementations that follows OpenAI spec for images generation though.

// I'm currently trying to make stable-diffusion.cpp match the spec, but my knowledge of C++ isn't suffice.

[mostlygeek]

If there was a working open source server to test with, adding the endpoint would be fairly trivial. Let me know if/when you have one working.

[mord0d]

Let me know if/when you have one working.

I've found this repo, but it depends on AUTOMATIC1111/stable-diffusion-webui which doesn't work for me (I'm on FreeBSD, their automatic installation process just doesn't work, and manual installation isn't simple: many required Python modules can't be built for that platform).

---

// I've changed the title to make it more general.

[jhemmond]

Why not do this instead? I have it working and takes a few mins to setup.

Use Koboldcpp as backend as it supports Image Gen. Configure it as another model in llama-swap.yaml file. Use Vulkan/Cuda to accelerate the image gen process. Here's my config for reference. You can download compatible models from civit.ai

This will open stable diffusion web interface on the port of your choice. It's Automatic1111 compatible so you can pair it with OpenWeb UI interface if you like to.

models:
  "Image Generator":
    proxy: "http://127.0.0.1:7860/v1/"
    cmd: >
      \path\to\kbold executable\koboldcpp-nocuda.exe --sdmodel   \path\to\model\realisticVisionV60B1_v51HyperVAE.safetensors --host 127.0.0.1 --port 7860 --usevulkan --gpulayers 99
    ttl: 300
    checkEndpoint: /sdapi/v1/sd-models

[mord0d]

Why not do this instead?

@jhemmond

There's a lot of clients for LLMs, because almost every server provides an OpenAI-compatible API (see below).
And why there's none for SDs? Because we have a lot of incompatible Stable Diffusion APIs, and none of them even stable.

Only OpenAI made a spec, not the best one, but it became a de facto standard. Not because it's good but because it's stable and have good documentation.

We don't need an all-in-one solution for LLM+SD, do one thing and do it well.

With llama-swap, we could create an alternative to ChatGPT. Maybe not as powerful as ChatGPT itself, but private, uncensored, free (not absolutely free, we still have to pay electricity bills and buy new hardware, but still) and no one will ever shut it down or block.

Well, about koboldcpp…
It's a good all-in-one solution, but koboldcpp uses some Python requirements which isn't available for FreeBSD.

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity.

[github-actions]

This issue was closed because it has been inactive for 2 weeks since being marked as stale.

[gooofy]

We just tried this with a simple python based wrapper for ComfyUI as the server. The addition for generations support in llama-swap is indeed trivial:

diff --git a/proxy/proxymanager.go b/proxy/proxymanager.go
index 39295e0..ea5d1a4 100644
--- a/proxy/proxymanager.go
+++ b/proxy/proxymanager.go
@@ -259,6 +259,9 @@ func (pm *ProxyManager) setupGinEngine() {
        pm.ginEngine.POST("/v1/audio/speech", pm.proxyOAIHandler)
        pm.ginEngine.POST("/v1/audio/transcriptions", pm.proxyOAIPostFormHandler)
 
+       // Support image generation endpoint
+       pm.ginEngine.POST("/v1/images/generations", pm.proxyOAIHandler)
+
        pm.ginEngine.GET("/v1/models", pm.listModelsHandler)
 
        // in proxymanager_loghandlers.go

just in case anybody wants to try this, here is the (vibe-coded) image generation server:

#!/usr/bin/env python3
import json
import base64
import time
import uuid
from typing import Optional
import requests
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

###########################################################################
# Configuration
###########################################################################

COMFY_URL = "http://127.0.0.1:8188"
WORKFLOW_PATH = "workflows/openai_api_server.json"   # <-- update if needed
HOST = "127.0.0.1"
PORT = 8085

###########################################################################
# Load workflow at startup
###########################################################################

#with open(WORKFLOW_PATH, "r") as f:
#    BASE_WORKFLOW = json.load(f)

###########################################################################
# Helpers
###########################################################################

def convert_ui_to_api_format(ui_workflow: dict) -> dict:
    """
    Convert ComfyUI UI workflow format to API format.
    UI format has 'nodes' array, API format is a dict of node_id: node_data
    """
    api_workflow = {}
    
    # Map of known node types to their widget input names
    WIDGET_INPUT_NAMES = {
        "CLIPTextEncode": ["text"],
        "EmptyLatentImage": ["width", "height", "batch_size"],
        "KSamplerSelect": ["sampler_name"],
        "BasicScheduler": ["scheduler", "steps", "denoise"],
        "VAELoader": ["vae_name"],
        "DualCLIPLoader": ["clip_name1", "clip_name2", "type", "device"],
        "UNETLoader": ["unet_name", "weight_dtype"],
        "SaveImage": ["filename_prefix"],
        "RandomNoise": ["noise_seed", "control_after_generate"],
    }
    
    for node in ui_workflow.get("nodes", []):
        node_id = str(node["id"])
        node_type = node["type"]
        
        # Build inputs from the node's input connections and widgets
        inputs = {}
        widget_values = node.get("widgets_values", [])
        widget_idx = 0
        
        # Process input connections from the "inputs" array
        if "inputs" in node:
            for input_def in node["inputs"]:
                input_name = input_def["name"]
                
                if input_def.get("link") is not None:
                    # This input is connected - find the source node
                    link_id = input_def["link"]
                    
                    # Find the link in the links array
                    for link in ui_workflow.get("links", []):
                        if link[0] == link_id:
                            # link format: [link_id, source_node, source_slot, target_node, target_slot, type]
                            source_node_id = str(link[1])
                            source_slot = link[2]
                            inputs[input_name] = [source_node_id, source_slot]
                            break
        
        # Now add widget values as inputs
        # Try to use known widget names for this node type
        if node_type in WIDGET_INPUT_NAMES:
            widget_names = WIDGET_INPUT_NAMES[node_type]
            for i, widget_name in enumerate(widget_names):
                if i < len(widget_values):
                    inputs[widget_name] = widget_values[i]
        elif widget_values:
            # For unknown node types, try generic approach
            # Some nodes just need the raw widget values
            inputs["_widget_values"] = widget_values
        
        api_workflow[node_id] = {
            "class_type": node_type,
            "inputs": inputs
        }
    
    return api_workflow


def set_prompt(workflow: dict, prompt: str):
    """
    Locate CLIPTextEncode node in the workflow dict (API format) and replace the prompt text.
    """
    for node_id, node in workflow.items():
        if node["class_type"] == "CLIPTextEncode":
            # In API format, the text prompt should be in inputs
            if "widgets_values" in node["inputs"]:
                node["inputs"]["widgets_values"][0] = prompt
            else:
                # Try to find the 'text' input
                node["inputs"]["text"] = prompt
    return workflow


def set_size(workflow: dict, width: int, height: int):
    """
    Modify EmptyLatentImage node to change output resolution dynamically (API format).
    """
    for node_id, node in workflow.items():
        if node["class_type"] == "EmptyLatentImage":
            if "widgets_values" in node["inputs"]:
                node["inputs"]["widgets_values"][0] = width
                node["inputs"]["widgets_values"][1] = height
            else:
                node["inputs"]["width"] = width
                node["inputs"]["height"] = height
    return workflow


def comfy_queue_prompt(workflow):
    """Send the workflow to ComfyUI's /prompt endpoint."""
    payload = {"prompt": workflow}
    r = requests.post(f"{COMFY_URL}/prompt", json=payload)
    r.raise_for_status()
    return r.json()


def comfy_get_image(prompt_id):
    """Poll ComfyUI until the image is generated."""
    for _ in range(120):  # 120 seconds timeout
        time.sleep(1)
        hist = requests.get(f"{COMFY_URL}/history/{prompt_id}").json()

        if prompt_id in hist:
            data = hist[prompt_id]
            if "outputs" in data:
                for node_id, output in data["outputs"].items():
                    if "images" in output:
                        img = output["images"][0]
                        filename = img["filename"]
                        subfolder = img["subfolder"]
                        filetype = img.get("type", "output")

                        # Download the image binary
                        img_bytes = requests.get(
                            f"{COMFY_URL}/view?filename={filename}"
                            f"&subfolder={subfolder}&type={filetype}"
                        ).content

                        return img_bytes

    raise RuntimeError("Timed out waiting for ComfyUI image")

###########################################################################
# FastAPI OpenAI-Compatible Endpoint
###########################################################################

class OpenAIImageRequest(BaseModel):
    prompt: str
    size: Optional[str] = "1024x1024"
    model: Optional[str] = "flux-dev"
    n: Optional[int] = 1

app = FastAPI()

@app.post("/v1/images/generations")
def generate_image(req: OpenAIImageRequest):
    logging.info(f"Image generation request received for prompt: '{req.prompt}' with size: {req.size}")

    # Clone the static workflow
    logging.info("Cloning static workflow.")
    workflow = json.loads(json.dumps(BASE_WORKFLOW))

    # Update prompt text
    logging.info(f"Setting prompt to: '{req.prompt}'")
    workflow = set_prompt(workflow, req.prompt)

    # Update width/height dynamically
    try:
        size_str = req.size if req.size else "1024x1024"
        w, h = map(int, size_str.lower().split("x"))
        logging.info(f"Setting image size to: {w}x{h}")
        workflow = set_size(workflow, w, h)
    except:
        logging.info("Failed to parse size or size not provided, keeping original size in workflow.")
        pass  # fallback: keep original size in workflow

    # Queue prompt in ComfyUI
    logging.info("Queueing prompt in ComfyUI.")
    job = comfy_queue_prompt(workflow)
    prompt_id = job["prompt_id"]
    logging.info(f"Prompt queued with ID: {prompt_id}")

    # Get generated image
    logging.info(f"Waiting for image with prompt ID: {prompt_id}")
    img = comfy_get_image(prompt_id)
    logging.info("Image generated successfully.")

    # Return OpenAI-compatible response
    return {
        "created": int(time.time()),
        "data": [{
            "b64_json": base64.b64encode(img).decode("utf-8")
        }]
    }

@app.get("/health")
def health_check():
    return {"status": "ok"}

###########################################################################
# Standalone entry point with embedded Uvicorn
###########################################################################

if __name__ == "__main__":
    import argparse
    import sys

    parser = argparse.ArgumentParser(description="OpenAI-compatible FLUX server for ComfyUI")
    parser.add_argument("--host", type=str, default=HOST, help="Host address to bind")
    parser.add_argument("--port", type=int, default=PORT, help="Port to listen on")
    parser.add_argument("--workflow", type=str, default=WORKFLOW_PATH, help="Path to workflow JSON file")

    args = parser.parse_args()

    # Override globals with command-line args
    HOST = args.host
    PORT = args.port
    WORKFLOW_PATH = args.workflow

    # Reload workflow if overridden
    with open(WORKFLOW_PATH, "r") as f:
        ui_workflow = json.load(f)
    
    # Convert UI format to API format
    BASE_WORKFLOW = convert_ui_to_api_format(ui_workflow)

    print(f"Starting OpenAI-compatible FLUX server on http://{HOST}:{PORT}")
    print(f"Using workflow: {WORKFLOW_PATH}")
    print(f"Talking to ComfyUI at: {COMFY_URL}")
    print(f"Converted workflow from UI format to API format ({len(BASE_WORKFLOW)} nodes)")

    # Use dynamic module name for safety
    module_name = sys.modules[__name__].__name__

    uvicorn.run(
        f"{module_name}:app",
        host=HOST,
        port=PORT,
        reload=False,
        workers=1
    )

[xezpeleta]

If there was a working open source server to test with, adding the endpoint would be fairly trivial. Let me know if/when you have one working.

@mord0d I know this issue is closed, but I was looking for this feature and I stumbled upon with this project:

https://github.com/matatonic/openedai-images-flux

In my opinion an interesting use case for llama-swap is to build an openroute alternative based on local models. Not just for LLM's but any other OpenAI API compatible services: text generation, image generation, audio generation, etc.

By the way, good job @mostlygeek ! Thanks for everything

[xezpeleta]

By the way, it looks that stable-diffusion.cpp is also working on supporting OpenAI API

leejet/stable-diffusion.cpp#1037
https://github.com/leejet/stable-diffusion.cpp/tree/server