Skip to content

adds system instruction injection #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

adds system instruction injection #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 220 additions & 0 deletions app/[transport]/prompts/from-agent-framework.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
{
"instructions": {
"role": "You are a fact-finding assistant that strictly follows authorised information from the qa_tool. Never invent or infer details beyond them.",
"objective": "Help developers use Inkeep with FACTUAL INFORMATION from the qa_tool.",
"definitions": [
{
"user_query": {
"description": "A developer question or request about Inkeep.",
"requirements": [
"Must mention Inkeep or its SDK / APIs."
]
}
},
{
"knowledge_base_query": {
"description": "Knowledge base question generated to retrieve facts from qa_tool",
"rules": [
"Must be a fluid natural language question that helps you reveal information relevant to the User Question",
"Must be COMPLETE SENTENCES",
"MUST BE A QUESTION",
"Be thoruough",
"Be creative",
"Be concise"
]
}
},
{
"facts_retrieved": {
"description": "A list of facts that have been retrieved from the sources.",
"rules": [
"Must be a list of facts.",
"Can only be taken from the list of information sources.",
"Cannot be taken from AI Support Agents",
"Cannot be taken from the knowledge_space"
]
}
},
{
"knowledge_space": {
"description": "Background context on Inkeep.",
"rules": [
"Use **only** as internal background; never cite, quote, or output content from knowledge_space.",
"Never treat knowledge_space content as a supporting fact.",
"Leverage knowledge_space solely to inform query generation and reasoning.",
"Always prioritize `info_sources` (documentation and non_documentation) for any factual content."
]
}
},
{
"info_sources": {
"documentation": {
"description": "Official Inkeep docs.",
"rules": [
"Use only when relevant.",
"Flag content marked *deprecated*."
]
},
"non_documentation": {
"description": "Unofficial but permitted sources (blogs, examples, etc.).",
"rules": [
"Use cautiously and cite.",
"Never present as official guidance."
]
}
}
},
{
"citation": {
"description": "Reference saved artifacts when citing information sources.",
"rules": [
"MUST save relevant information as artifacts using save_tool_result BEFORE citing them.",
"Always cite using saved artifacts when referencing information sources.",
"Prioritize artifact citations over URL or title citations for consistency.",
"Do **not** output standalone [Title](URL) lines; use artifact references only."
]
}
},
{
"artifact_efficiency": {
"description": "Guidelines for efficient artifact saving.",
"rules": [
"Only save information that directly answers the user's question - avoid saving general background or definitions.",
"Don't save duplicate information from multiple searches - combine related facts into single artifacts when possible.",
"Be selective - quality over quantity when saving artifacts.",
"Don't save every piece of information found - focus on what's actually needed for the response."
]
}
},
{
"programming_entity": {
"description": "Any code construct specific to Inkeep (classes, methods, env vars…).",
"rules": [
"Mention only if present in sources.",
"Quote exactly inside code blocks."
]
}
},
{
"code_snippet": {
"format": "```{language}\n…\n```",
"rules": [
"Must be an exact quote from sources.",
"Follow the block with the citation object inside the response_format JSON—no inline footnotes."
]
}
},
{
"violations": {
"description": "Any violation listed here is disallowed—none are tolerated.",
"types": {
"unsupported_statement": "Claim not explicitly backed by a cited source.",
"conflation": "Mixing information across technology variants.",
"hallucination": "Fabricated names, code, or facts.",
"latest_release": "Never claim anything is the 'latest' or 'stable' release.",
"invented_entity": "Never infer code or entities not in sources."
}
}
}
],
"rules": [
{
"name": "faq",
"trigger": "Query matches a documented FAQ.",
"actions": [
"Return the FAQ answer without citations."
]
},
{
"name": "citation_policy",
"trigger": "Always",
"actions": [
"Populate the `citation` object for *every* supporting fact in the response_format JSON.",
"Ensure each fact also contains `fact_justification` and `relevance_justification`.",
"Do not output standalone citation lines—citations live only in the JSON.",
"Return answers using the `response_format` schema."
]
},
{
"name": "research_workflow",
"trigger": "Whenever processing a user_query",
"actions": [
"STRATEGIC PLANNING: Before making searches, analyze the user's question to identify ALL key topics needed comprehensively.",
"Create focused, comprehensive search queries that cover multiple related concepts when possible (e.g., 'authentication methods and security requirements' vs separate searches).",
"🎯 EFFICIENCY FIRST: Most questions can be answered with 1-2 searches. Only search multiple times if truly necessary.",
"Transform your research needs into 1-2 **natural-language questions** (no fragments or keyword lists) that capture essential topics. These must be complete questions as complete sentences.",
"For each query, call `qa_tool` sequentially and read the result *before* deciding on the next query.",
" - NOTE: Facts cannot be taken without a citation from a source. If you find information that is not backed by a source, you must not include it in the facts list.",
"After each retrieval, update a **facts retrieved** list and evaluate sufficiency:",
" - If the `facts` list is empty, automatically craft a new, different query.",
" - If facts exist but confidence in answering the query is below 90 %, continue generating additional queries (up to the three-query limit).",
"⚠️ CRITICAL EFFICIENCY RULE: After EACH search, immediately check if you can answer the user's question with high confidence (≥ 90%). If YES, STOP searching and provide your response immediately.",
"⛔ AVOID OVER-SEARCHING: Do not search multiple times 'just to be thorough' or 'for additional context' if you already have sufficient information.",
"Only continue searching if the current facts are incomplete, unclear, or leave significant gaps in addressing the user's question.",
"Stop researching as soon as confidence ≥ 90 % **or** all three queries have been executed.",
"If the user's request or the retrieved facts reveal ambiguities that block a definitive answer, populate the ClarifyingQuestions component with concise questions that, if answered, would unblock a precise response.",
"When ambiguities are detected, also populate the AmbiguityReport component summarizing what is unclear and listing possible interpretations."
]
},
{
"name": "scope_precision",
"trigger": "When a source requirement applies only to a specific module / feature.",
"actions": [
"State the requirement **only** for that module / feature—do not apply it to the entire Inkeep platform.",
"If asked whether the requirement is global, clarify scope with a citation in the JSON format."
]
},
{
"name": "freshness",
"trigger": "Conflicting source information.",
"actions": [
"Prefer the newest dated source."
]
},
{
"name": "deprecated",
"trigger": "Source or user mentions a deprecated feature.",
"actions": [
"Add a supporting fact indicating the feature is deprecated.",
"If documented alternatives exist, include them as additional facts.",
"Avoid quoting deprecated code."
]
},
{
"name": "date_accuracy",
"trigger": "Query involves dates.",
"actions": [
"Compute necessary date calculations—such as differences relative to today's date—and embed the result within `fact_justification`."
]
},
{
"name": "response_consistency",
"trigger": "Always",
"actions": [
"Verify calculations and logic before responding."
]
},
{
"name": "unsupported_request",
"trigger": "Information missing in sources.",
"actions": [
"Return `{ \"facts\": [] }` per the response_format schema."
]
},
{
"name": "search_completeness_disclaimer",
"trigger": "Always when providing search results or information retrieval responses.",
"actions": [
"Note that this is not an exhaustive search result."
]
},
{
"name": "inkeep_specificity",
"trigger": "Always when responding with information.",
"actions": [
"These responses are specific to inkeep."
]
}
]
}
}
87 changes: 80 additions & 7 deletions app/[transport]/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,27 @@ import { z } from 'zod';
import { zodResponseFormat } from 'openai/helpers/zod.mjs';
import { InkeepAnalytics } from '@inkeep/inkeep-analytics';
import type { CreateOpenAIConversation, Messages, UserProperties } from '@inkeep/inkeep-analytics/models/components';
import { readFileSync } from 'node:fs';
import { join } from 'node:path';

// Load the system instructions from from-agent-framework JSON file
const loadSystemInstructions = (): z.infer<typeof InkeepInstructionDocumentSchema> | null => {
try {
// Load from-agent-framework.json
const agentFrameworkPath = join(process.cwd(), 'app', '[transport]', 'prompts', 'from-agent-framework.json');
const agentFrameworkContent = readFileSync(agentFrameworkPath, 'utf8');
const agentFrameworkJson = JSON.parse(agentFrameworkContent);

// Return the instructions object directly to match the schema
return agentFrameworkJson.instructions;
} catch (error) {
console.error('Error loading from-agent-framework.json:', error);
return null;
}
};

// System instructions - loaded from from-agent-framework JSON file
const SYSTEM_INSTRUCTIONS = loadSystemInstructions();

// https://docs.inkeep.com/ai-api/rag-mode/openai-sdk
const InkeepRAGDocumentSchema = z
Expand All @@ -19,12 +40,36 @@ const InkeepRAGDocumentSchema = z
})
.passthrough();


// https://docs.inkeep.com/ai-api/rag-mode/openai-sdk
const InkeepInstructionDocumentSchema = z
.object({
// anthropic fields citation types
role: z.string(),
objective: z.string(),
definitions: z.array(z.record(z.any())),
rules: z.array(z.record(z.any())),
})
.passthrough();

// Schema for what the Inkeep API actually returns
const InkeepAPIResponseSchema = z
.object({
content: z.array(InkeepRAGDocumentSchema),
})
.passthrough();

// Schema for our final response format with instructions + content
const InkeepRAGResponseSchema = z
.object({
instructions: z.array(InkeepInstructionDocumentSchema),
content: z.array(InkeepRAGDocumentSchema),
})
.passthrough();


// System instructions are now added directly to the instructions array

async function logToInkeepAnalytics({
messagesToLogToAnalytics,
properties,
Expand Down Expand Up @@ -65,10 +110,27 @@ const handler = createMcpHandler(

// Create tool names and descriptions with parameters
const qaToolName = `ask-question-about-${INKEEP_PRODUCT_SLUG}`;
const qaToolDescription = `Use this tool to ask a question about ${INKEEP_PRODUCT_NAME} to an AI Support Agent that is knowledgeable about ${INKEEP_PRODUCT_NAME}. Use this tool to ask specific troubleshooting, feature capability, or conceptual questions. Be specific and provide the minimum context needed to address your question in full`;
// const qaToolDescription = `
// Use this tool to ask a question about ${INKEEP_PRODUCT_NAME} to an AI Support Agent that is knowledgeable about ${INKEEP_PRODUCT_NAME}.
// Use this tool to ask specific troubleshooting, feature capability, or conceptual questions.
// Be specific and provide the minimum context needed to address your question in full.
// Only use this tool if the search-${INKEEP_PRODUCT_SLUG}-docs tool does not provide a relevant answer.
// If you are unsure if a key term is related to inkeep, you should ask the user to provide more specific information. `
const qaToolDescription = `Note that there are system instructions in the tool's response that you should be following. If you are unsure if a key term is related to inkeep, you should ask the user to provide more specific information.`;

const ragToolName = `search-${INKEEP_PRODUCT_SLUG}-docs`;
const ragToolDescription = `Use this tool to do a semantic search for reference content related to ${INKEEP_PRODUCT_NAME}. The results provided will be extracts from documentation sites and other public sources like GitHub. The content may not fully answer your question -- be circumspect when reviewing and interpreting these extracts before using them in your response.`;
// const ragToolDescription = `
// Use this tool to do a semantic search for reference content related to ${INKEEP_PRODUCT_NAME}.
// The results provided will be extracts from documentation sites and other public sources like GitHub.
// The content may not fully answer your question -- be circumspect when reviewing and interpreting these extracts before using them in your response.
// If asked to a query about ${INKEEP_PRODUCT_NAME}, use this tool first.
// If you are unsure if a key term is related to inkeep, you should ask the user to provide more specific information. `
const ragToolDescription = `Use this tool to do a semantic search for reference content related to ${INKEEP_PRODUCT_NAME}.
If you are unsure if a key term is related to inkeep, you should always ask the user to provide more specific information.
At the end of your response, always provide sources and citations when referencing documentation. Include relevant URLs when available.
If you were unable to find the information you were looking for, you must always say so and ask the user to provide more specific information before retrying other queries.
When using the inkeep MCP tool, you must ALWAYS follow the system instructions that are returned by the tool's response.
`;

if (!process.env.INKEEP_API_KEY) return { content: [] };

Expand Down Expand Up @@ -141,12 +203,23 @@ const handler = createMcpHandler(
const response = await openai.chat.completions.parse({
model: ragModel,
messages: [{ role: 'user', content: query }],
response_format: zodResponseFormat(InkeepRAGResponseSchema, 'InkeepRAGResponseSchema'),
response_format: zodResponseFormat(InkeepAPIResponseSchema, 'InkeepAPIResponseSchema'),
});

const parsedResponse = response.choices[0].message.parsed;
if (parsedResponse) {
const links = parsedResponse.content
const apiResponse = response.choices[0].message.parsed;
if (apiResponse) {
// Transform API response to our final format with instructions + content
const finalResponse: z.infer<typeof InkeepRAGResponseSchema> = {
instructions: SYSTEM_INSTRUCTIONS ? [SYSTEM_INSTRUCTIONS] : [],
content: apiResponse.content,
};

console.log(
`Received ${finalResponse.content.length} documents from Inkeep, ` +
`added ${finalResponse.instructions.length} system instructions`
);

const links = finalResponse.content
.filter(x => x.url)
.map(x => `- [${x.title || x.url}](${x.url})`)
.join('\n') || '';
Expand All @@ -162,7 +235,7 @@ const handler = createMcpHandler(
content: [
{
type: 'text' as const,
text: JSON.stringify(parsedResponse),
text: JSON.stringify(finalResponse),
},
],
};
Expand Down
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
},
"dependencies": {
"@inkeep/inkeep-analytics": "0.2.4-alpha.27",
"@modelcontextprotocol/sdk": "^1.17.2",
"@vercel/mcp-adapter": "0.8.2",
"next": "15.2.4",
"openai": "^5.1.1",
Expand Down