From 03c1cf6906288335d2194832081f180747df9dd3 Mon Sep 17 00:00:00 2001 From: Tanushree Sharma Date: Fri, 19 Sep 2025 12:32:36 -0700 Subject: [PATCH 1/4] prompt management guide --- src/docs.json | 1 + .../manage-prompts-programmatically.mdx | 4 +- src/langsmith/manage-prompts.mdx | 25 ++++---- .../prompt-management-best-practices.mdx | 58 +++++++++++++++++++ 4 files changed, 74 insertions(+), 14 deletions(-) create mode 100644 src/langsmith/prompt-management-best-practices.mdx diff --git a/src/docs.json b/src/docs.json index ebc023be8..559ebdf46 100644 --- a/src/docs.json +++ b/src/docs.json @@ -911,6 +911,7 @@ "pages": [ "langsmith/prompt-engineering", "langsmith/prompt-engineering-concepts", + "langsmith/prompt-management-best-practices", { "group": "Create and update prompts", "pages": [ diff --git a/src/langsmith/manage-prompts-programmatically.mdx b/src/langsmith/manage-prompts-programmatically.mdx index 4a7d5c01d..380b4a227 100644 --- a/src/langsmith/manage-prompts-programmatically.mdx +++ b/src/langsmith/manage-prompts-programmatically.mdx @@ -5,9 +5,7 @@ sidebarTitle: Manage prompts programmatically You can use the LangSmith Python and TypeScript SDK to manage prompts programmatically. - -Previously this functionality lived in the `langchainhub` package which is now deprecated. All functionality going forward will live in the `langsmith` package. - +We do not reccomend pulling prompts from the Prompt Hub in your production code on every request. Please see our guide on [best practices for using the prompt hub](/langsmith/prompt-management-best-practices) for more information. ## Install packages diff --git a/src/langsmith/manage-prompts.mdx b/src/langsmith/manage-prompts.mdx index b462ff6eb..34a273788 100644 --- a/src/langsmith/manage-prompts.mdx +++ b/src/langsmith/manage-prompts.mdx @@ -65,15 +65,20 @@ Some common use cases of this include: Navigate to the **Prompts** section in the left-hand sidebar or from the application homepage. In the top right corner, click on the `+ Webhook` button. -Add a webhook URL and any required headers. +1. Add a webhook URL +2. Select the event(s) you want to trigger the webhook for: + - On each commit to a prompt + - When a new tag is created for a prompt + - When a tag is moved for a prompt +3. Add any required headers. +4. Test out your webhook by clicking **Send test notification**. This will send a test notification to the webhook URL you provided with a sample payload. +4. Click **Create Webhook**. -You can only configure one webhook per workspace. If you want to configure multiple per workspace or set up a different webhook for each prompt, let us know in the [LangChain Forum](https://forum.langchain.com/). +Currently, you can only configure one webhook per workspace. If you want to configure multiple per workspace or set up a different webhook for each prompt, let us know in the [LangChain Forum](https://forum.langchain.com/). -To test out your webhook, click the **Send test notification** button. This will send a test notification to the webhook URL you provided with a sample payload. - -The sample payload is a JSON object with the following fields: +### Webhook Payload - `prompt_id`: The ID of the prompt that was committed. - `prompt_name`: The name of the prompt that was committed. @@ -82,13 +87,12 @@ The sample payload is a JSON object with the following fields: - `created_by`: The author of the commit. - `manifest`: The manifest of the prompt. -### Trigger the webhook - -Commit to a prompt to trigger the webhook you've configured. -#### Use the Playground +### Trigger the webhook +Webhooks are triggered on prompt changes in the UI or via the API. -If you do this in the Playground, you'll be prompted to deselect the webhooks you'd like to avoid triggering. +#### In the UI +If you have a webhook configured, when you create a prompt commit in the Playground, you will see the Trigger Webhook options. You can deselect the webhook if you don't want to trigger it on that specific commit. ![](/langsmith/images/commit-prompt-playground.png) @@ -111,4 +115,3 @@ Here you'll find all of the publicly listed prompts in the LangChain Hub. You ca To view prompts tied to your workspace, visit the **Prompts** tab in the sidebar. ![](/langsmith/images/prompts-tab.png) - diff --git a/src/langsmith/prompt-management-best-practices.mdx b/src/langsmith/prompt-management-best-practices.mdx new file mode 100644 index 000000000..7dac9c955 --- /dev/null +++ b/src/langsmith/prompt-management-best-practices.mdx @@ -0,0 +1,58 @@ +--- +title: Prompt Hub best practices +sidebarTitle: Prompt Hub best practices +--- + +Prompts are a fundamental component of your LLM application's logic. A minor change can significantly change an LLM's response or tool selection, making lifecycle management important. LangSmith provides tools to manage prompts across development, testing, and production environments: + +- **Prompt Hub**: A central store for managing prompts. This is a especially useful source of truth for your prompts when you are collaborating with a team to iterate on them. +- **Playground**: Test and experiment with prompts and models. + +This guide shows you how to build a workflow for developing, testing, and deploying prompts with LangSmith. + +## Develop and iterate on prompts in the playground + +Prompt development is an iterative, experimental process. Using the [Playground](/prompt_engineering/how_to_guides#playground), you can: +- **Test with different inputs** and **compare the results** of different prompts and LLMs side-by-side. +- **Refine your prompt** using the embedded [Prompt Canvas](/prompt_engineering/how_to_guides/prompt_canvas) to have an LLM improve your prompt. +- **Prototype tools, set structured outputs** and test the interaction within the Playground. +- **[Run evaluations](/langsmith/run-evaluation-from-prompt-playground)** over your prompt in the Playground. Share experiments with teammates to get feedback and collaboratively optimize performance. + +Once ready, you can save a prompt to the Prompt Hub. Saving a prompt creates a new version. Establishing a complete commit history enables you to track changes and revert to previous versions if needed. + +## Manage prompts across environments + +When managing prompts across your dev/staging/production environments, you want to balance speed of iteration across a team with the need for stability. We reccomend using the Prompt Hub along with [commit tags](/langsmith/manage-prompts#commit-tags) to manage prompts across dev/testing/staging environments and a seperate workflow for managing prompts in production. + +### Update prompts in dev/testing/staging environments + +After testing in the Playground, you'll want to see how the prompt interacts within the context of your application. A reccomended workflow is: + +1. Set up your dev/testing/staging environments to [pull prompts from the Prompt Hub](/langsmith/manage-prompts-programmatically#pull-a-prompt) using defined commit tags (eg. `dev`, `staging`) for each environment. This creates a dynamic reference to the prompt version you want to use in a particular environment. +2. After saving a new version of your prompt, [move the commit tag](/langsmith/manage-prompts#move-a-tag) for the desired environment to the new version of the prompt. Your application will automatically pull the new version on its next request to the Prompt Hub. +3. Test the prompt across environments. + + Integrate prompt evaluations into CI/CD. [Prompt webhooks](/langsmith/manage-prompts#configure-a-webhook) can be used in order to trigger CI when a prompt has new commit, or when a commit tag is moved. Use [pytest](/langsmith/pytest) or [jest/vitest](/langsmith/vitest-jest) to run LangSmith evaluations as part of your test suite, blocking promotion if quality thresholds fail. + +### Update prompts in production + + It is not reccomended to put an API call to the Prompt Hub in the hot path of your application on every request. We reccomend either pulling the prompt into your code directly or fetching the prompt once and reusing it. + +[Prompt webhooks](/langsmith/manage-prompts#configure-a-webhook) can be used in order to keep prompts in sync with your production environment. For example a notification can be sent when a prompt gets the `prod` tag added to it, which can then be used as a trigger to pull the latest version of the prompt into your production environment. + +#### Pull prompts into your code + +A common approach is to sync prompts from the Prompt Hub into your source code, which allows them to be version-controlled and deployed alongside your application code. + +The workflow would look like: +1. A LangSmith user moves the `prod` tag to a new prompt version. +2. The webhook triggers a CI/CD pipeline (eg. GitHub Actions, GitLab CI) that pulls the version with the `prod` tag into your repository. +3. The pipeline commits the prompt change to your repo. + + +#### Store prompts in a database or cache prompts in your application + +If you want your prompts versioned independent of your application code, you can sync them to a database or cache them in your application. + +- Database: Store prompts in a database, treating them as a form of dynamic configuration. The CI/CD process would be responsible for updating the prompt in the database based on the webhook trigger. +- Cache: The application fetches the prompt once on startup and stores it in memory. On webhook receipt, invalidate the cache and pull the latest commit. From 9b4cefb64ec3503eb38efe0f6aad067d7d38c777 Mon Sep 17 00:00:00 2001 From: Tanushree Sharma Date: Fri, 19 Sep 2025 14:42:11 -0700 Subject: [PATCH 2/4] update based on feedback --- .../manage-prompts-programmatically.mdx | 2 +- .../prompt-management-best-practices.mdx | 20 +++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/langsmith/manage-prompts-programmatically.mdx b/src/langsmith/manage-prompts-programmatically.mdx index 380b4a227..4524d558c 100644 --- a/src/langsmith/manage-prompts-programmatically.mdx +++ b/src/langsmith/manage-prompts-programmatically.mdx @@ -5,7 +5,7 @@ sidebarTitle: Manage prompts programmatically You can use the LangSmith Python and TypeScript SDK to manage prompts programmatically. -We do not reccomend pulling prompts from the Prompt Hub in your production code on every request. Please see our guide on [best practices for using the prompt hub](/langsmith/prompt-management-best-practices) for more information. +We do not recommend pulling prompts from the Prompt Hub in your production code on every request. Please see our guide on [best practices for using the Prompt Hub](/langsmith/prompt-management-best-practices) for more information. ## Install packages diff --git a/src/langsmith/prompt-management-best-practices.mdx b/src/langsmith/prompt-management-best-practices.mdx index 7dac9c955..4649ca800 100644 --- a/src/langsmith/prompt-management-best-practices.mdx +++ b/src/langsmith/prompt-management-best-practices.mdx @@ -18,15 +18,15 @@ Prompt development is an iterative, experimental process. Using the [Playground] - **Prototype tools, set structured outputs** and test the interaction within the Playground. - **[Run evaluations](/langsmith/run-evaluation-from-prompt-playground)** over your prompt in the Playground. Share experiments with teammates to get feedback and collaboratively optimize performance. -Once ready, you can save a prompt to the Prompt Hub. Saving a prompt creates a new version. Establishing a complete commit history enables you to track changes and revert to previous versions if needed. +Once ready, you can save a prompt to the Prompt Hub. A new commit is created each time you save a prompt, establishing a complete commit history enables you to track changes and revert to previous versions if needed. ## Manage prompts across environments -When managing prompts across your dev/staging/production environments, you want to balance speed of iteration across a team with the need for stability. We reccomend using the Prompt Hub along with [commit tags](/langsmith/manage-prompts#commit-tags) to manage prompts across dev/testing/staging environments and a seperate workflow for managing prompts in production. +When managing prompts across your dev/staging/production environments, you want to balance speed of iteration across a team with the need for stability. We recommend using the Prompt Hub along with [commit tags](/langsmith/manage-prompts#commit-tags) to manage prompts across dev/testing/staging environments and a seperate workflow for managing prompts in production. ### Update prompts in dev/testing/staging environments -After testing in the Playground, you'll want to see how the prompt interacts within the context of your application. A reccomended workflow is: +After testing in the Playground, you'll want to see how the prompt interacts within the context of your application. A recommend workflow is: 1. Set up your dev/testing/staging environments to [pull prompts from the Prompt Hub](/langsmith/manage-prompts-programmatically#pull-a-prompt) using defined commit tags (eg. `dev`, `staging`) for each environment. This creates a dynamic reference to the prompt version you want to use in a particular environment. 2. After saving a new version of your prompt, [move the commit tag](/langsmith/manage-prompts#move-a-tag) for the desired environment to the new version of the prompt. Your application will automatically pull the new version on its next request to the Prompt Hub. @@ -36,23 +36,23 @@ After testing in the Playground, you'll want to see how the prompt interacts wit ### Update prompts in production - It is not reccomended to put an API call to the Prompt Hub in the hot path of your application on every request. We reccomend either pulling the prompt into your code directly or fetching the prompt once and reusing it. + It is not recommended to put an API call to the Prompt Hub in the hot path of your application on every request. We recommend either pulling the prompt into your code directly or fetching the prompt once and reusing it. [Prompt webhooks](/langsmith/manage-prompts#configure-a-webhook) can be used in order to keep prompts in sync with your production environment. For example a notification can be sent when a prompt gets the `prod` tag added to it, which can then be used as a trigger to pull the latest version of the prompt into your production environment. #### Pull prompts into your code -A common approach is to sync prompts from the Prompt Hub into your source code, which allows them to be version-controlled and deployed alongside your application code. +A common approach is to [sync prompts from the Prompt Hub with your source code](/langsmith/prompt-commit), so they can be **version-controlled and deployed alongside your application code**. The workflow would look like: -1. A LangSmith user moves the `prod` tag to a new prompt version. -2. The webhook triggers a CI/CD pipeline (eg. GitHub Actions, GitLab CI) that pulls the version with the `prod` tag into your repository. +1. A LangSmith user moves the `prod` tag to a new prompt commit. +2. The webhook triggers a CI/CD pipeline (eg. GitHub Actions, GitLab CI) that pulls the commit with the `prod` tag into your repository. 3. The pipeline commits the prompt change to your repo. #### Store prompts in a database or cache prompts in your application -If you want your prompts versioned independent of your application code, you can sync them to a database or cache them in your application. +If you want your prompts **versioned independenty of your application code**, you can sync them to a database or cache them in your application. This allows you to update, A/B test prompts without redeploying your application. -- Database: Store prompts in a database, treating them as a form of dynamic configuration. The CI/CD process would be responsible for updating the prompt in the database based on the webhook trigger. -- Cache: The application fetches the prompt once on startup and stores it in memory. On webhook receipt, invalidate the cache and pull the latest commit. +- Database: Store prompts in a database, treating them as a form of dynamic configuration. Your CI/CD process updates the prompt record whenever a webhook from LangSmithis triggered. +- Cache: Fetch the prompt once at application startup and keep it in memory. When the webhook fires, invalidate the cache and pull the latest commit. From a082f0f40cd6b89a33c1c0ecac301744d548dc72 Mon Sep 17 00:00:00 2001 From: Tanushree <87711021+tanushree-sharma@users.noreply.github.com> Date: Fri, 19 Sep 2025 14:44:46 -0700 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Kathryn May <44557882+katmayb@users.noreply.github.com> --- src/langsmith/manage-prompts.mdx | 10 +++++----- src/langsmith/prompt-management-best-practices.mdx | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/langsmith/manage-prompts.mdx b/src/langsmith/manage-prompts.mdx index 34a273788..80bcb6c3b 100644 --- a/src/langsmith/manage-prompts.mdx +++ b/src/langsmith/manage-prompts.mdx @@ -65,17 +65,17 @@ Some common use cases of this include: Navigate to the **Prompts** section in the left-hand sidebar or from the application homepage. In the top right corner, click on the `+ Webhook` button. -1. Add a webhook URL +1. Add a webhook URL. 2. Select the event(s) you want to trigger the webhook for: - - On each commit to a prompt - - When a new tag is created for a prompt - - When a tag is moved for a prompt + - On each commit to a prompt. + - When a new tag is created for a prompt. + - When a tag is moved for a prompt. 3. Add any required headers. 4. Test out your webhook by clicking **Send test notification**. This will send a test notification to the webhook URL you provided with a sample payload. 4. Click **Create Webhook**. -Currently, you can only configure one webhook per workspace. If you want to configure multiple per workspace or set up a different webhook for each prompt, let us know in the [LangChain Forum](https://forum.langchain.com/). +You can only configure one webhook per workspace. If you want to configure multiple per workspace or set up a different webhook for each prompt, let us know in the [LangChain Forum](https://forum.langchain.com/). ### Webhook Payload diff --git a/src/langsmith/prompt-management-best-practices.mdx b/src/langsmith/prompt-management-best-practices.mdx index 4649ca800..affa7a9e3 100644 --- a/src/langsmith/prompt-management-best-practices.mdx +++ b/src/langsmith/prompt-management-best-practices.mdx @@ -6,7 +6,7 @@ sidebarTitle: Prompt Hub best practices Prompts are a fundamental component of your LLM application's logic. A minor change can significantly change an LLM's response or tool selection, making lifecycle management important. LangSmith provides tools to manage prompts across development, testing, and production environments: - **Prompt Hub**: A central store for managing prompts. This is a especially useful source of truth for your prompts when you are collaborating with a team to iterate on them. -- **Playground**: Test and experiment with prompts and models. +- **Playground**: A sandbox for testing and experimenting with prompts and models, helping you refine their behavior before deploying. This guide shows you how to build a workflow for developing, testing, and deploying prompts with LangSmith. @@ -28,7 +28,7 @@ When managing prompts across your dev/staging/production environments, you want After testing in the Playground, you'll want to see how the prompt interacts within the context of your application. A recommend workflow is: -1. Set up your dev/testing/staging environments to [pull prompts from the Prompt Hub](/langsmith/manage-prompts-programmatically#pull-a-prompt) using defined commit tags (eg. `dev`, `staging`) for each environment. This creates a dynamic reference to the prompt version you want to use in a particular environment. +1. Set up your dev/testing/staging environments to [pull prompts from the Prompt Hub](/langsmith/manage-prompts-programmatically#pull-a-prompt) using defined commit tags (e.g., `dev`, `staging`) for each environment. This creates a dynamic reference to the prompt version you want to use in a particular environment. 2. After saving a new version of your prompt, [move the commit tag](/langsmith/manage-prompts#move-a-tag) for the desired environment to the new version of the prompt. Your application will automatically pull the new version on its next request to the Prompt Hub. 3. Test the prompt across environments. From a87752c13401425bc2dd07e72bc4bfb9c0c029ca Mon Sep 17 00:00:00 2001 From: Tanushree Sharma Date: Fri, 19 Sep 2025 14:48:37 -0700 Subject: [PATCH 4/4] feedback round 2 --- src/langsmith/prompt-management-best-practices.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/langsmith/prompt-management-best-practices.mdx b/src/langsmith/prompt-management-best-practices.mdx index affa7a9e3..ac00045da 100644 --- a/src/langsmith/prompt-management-best-practices.mdx +++ b/src/langsmith/prompt-management-best-practices.mdx @@ -36,7 +36,7 @@ After testing in the Playground, you'll want to see how the prompt interacts wit ### Update prompts in production - It is not recommended to put an API call to the Prompt Hub in the hot path of your application on every request. We recommend either pulling the prompt into your code directly or fetching the prompt once and reusing it. + It is not recommended to put an API call to the Prompt Hub in the hot path of your application on every request. Instead, we recommend either pulling the prompt into your code directly or fetching the prompt once and reusing it. [Prompt webhooks](/langsmith/manage-prompts#configure-a-webhook) can be used in order to keep prompts in sync with your production environment. For example a notification can be sent when a prompt gets the `prod` tag added to it, which can then be used as a trigger to pull the latest version of the prompt into your production environment. @@ -46,7 +46,7 @@ A common approach is to [sync prompts from the Prompt Hub with your source code] The workflow would look like: 1. A LangSmith user moves the `prod` tag to a new prompt commit. -2. The webhook triggers a CI/CD pipeline (eg. GitHub Actions, GitLab CI) that pulls the commit with the `prod` tag into your repository. +2. The webhook triggers a CI/CD pipeline (e.g., GitHub Actions, GitLab CI) that pulls the commit with the `prod` tag into your repository. 3. The pipeline commits the prompt change to your repo. @@ -54,5 +54,5 @@ The workflow would look like: If you want your prompts **versioned independenty of your application code**, you can sync them to a database or cache them in your application. This allows you to update, A/B test prompts without redeploying your application. -- Database: Store prompts in a database, treating them as a form of dynamic configuration. Your CI/CD process updates the prompt record whenever a webhook from LangSmithis triggered. -- Cache: Fetch the prompt once at application startup and keep it in memory. When the webhook fires, invalidate the cache and pull the latest commit. +- **Database**: Store prompts in a database as a form of dynamic configuration, and update them with a CI/CD process when a webhook is triggered. +- **Cache**: Cache prompts in the application in memory, and update them by invalidating the cache on webhook receipt.