Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable Azure blob storage #176

Merged
merged 8 commits into from
Nov 12, 2024
Merged

feat: enable Azure blob storage #176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
99ef026
feat: enable Azure blob storage
bvs-langchain Nov 8, 2024
7cd07a6
docs
bvs-langchain Nov 8, 2024
cdc79f0
Merge branch 'main' into brian/ls-2204-azure-support
bvs-langchain Nov 12, 2024
766315f
bump version
bvs-langchain Nov 12, 2024
a9fd29b
add required param
bvs-langchain Nov 12, 2024
e57a392
remove
bvs-langchain Nov 12, 2024
79eb763
remove sas token
bvs-langchain Nov 12, 2024
137b771
add comments
bvs-langchain Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions charts/langsmith/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ maintainers:
email: ankush@langchain.dev
description: Helm chart to deploy the langsmith application and all services it depends on.
type: application
version: 0.8.8
appVersion: "0.8.27"
version: 0.8.9
appVersion: "0.8.29"
20 changes: 12 additions & 8 deletions charts/langsmith/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# langsmith

![Version: 0.8.8](https://img.shields.io/badge/Version-0.8.8-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.8.27](https://img.shields.io/badge/AppVersion-0.8.27-informational?style=flat-square)
![Version: 0.8.9](https://img.shields.io/badge/Version-0.8.9-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.8.29](https://img.shields.io/badge/AppVersion-0.8.29-informational?style=flat-square)

Helm chart to deploy the langsmith application and all services it depends on.

Expand Down Expand Up @@ -148,23 +148,23 @@ For information on how to use this chart, up-to-date release notes, and other gu
| fullnameOverride | string | `""` | String to fully override `"langsmith.fullname"` |
| images.aceBackendImage.pullPolicy | string | `"IfNotPresent"` | |
| images.aceBackendImage.repository | string | `"docker.io/langchain/langsmith-ace-backend"` | |
| images.aceBackendImage.tag | string | `"0.8.27"` | |
| images.aceBackendImage.tag | string | `"0.8.29"` | |
| images.backendImage.pullPolicy | string | `"IfNotPresent"` | |
| images.backendImage.repository | string | `"docker.io/langchain/langsmith-backend"` | |
| images.backendImage.tag | string | `"0.8.27"` | |
| images.backendImage.tag | string | `"0.8.29"` | |
| images.clickhouseImage.pullPolicy | string | `"Always"` | |
| images.clickhouseImage.repository | string | `"docker.io/clickhouse/clickhouse-server"` | |
| images.clickhouseImage.tag | string | `"24.5"` | |
| images.frontendImage.pullPolicy | string | `"IfNotPresent"` | |
| images.frontendImage.repository | string | `"docker.io/langchain/langsmith-frontend"` | |
| images.frontendImage.tag | string | `"0.8.27"` | |
| images.frontendImage.tag | string | `"0.8.29"` | |
| images.imagePullSecrets | list | `[]` | Secrets with credentials to pull images from a private registry. Specified as name: value. |
| images.platformBackendImage.pullPolicy | string | `"IfNotPresent"` | |
| images.platformBackendImage.repository | string | `"docker.io/langchain/langsmith-go-backend"` | |
| images.platformBackendImage.tag | string | `"0.8.27"` | |
| images.platformBackendImage.tag | string | `"0.8.29"` | |
| images.playgroundImage.pullPolicy | string | `"IfNotPresent"` | |
| images.playgroundImage.repository | string | `"docker.io/langchain/langsmith-playground"` | |
| images.playgroundImage.tag | string | `"0.8.27"` | |
| images.playgroundImage.tag | string | `"0.8.29"` | |
| images.postgresImage.pullPolicy | string | `"IfNotPresent"` | |
| images.postgresImage.repository | string | `"docker.io/postgres"` | |
| images.postgresImage.tag | string | `"14.7"` | |
Expand All @@ -190,7 +190,11 @@ For information on how to use this chart, up-to-date release notes, and other gu
| config.basicAuth.initialOrgAdminEmail | string | `""` | |
| config.basicAuth.initialOrgAdminPassword | string | `""` | |
| config.basicAuth.jwtSecret | string | `""` | |
| config.blobStorage | object | `{"accessKey":"","accessKeySecret":"","apiURL":"https://s3.us-west-2.amazonaws.com","bucketName":"","chSearchEnabled":true,"enabled":false}` | Blob storage configuration Optional. Used to store inputs, outputs, and errors in Blob Storage. We currently support S3, GCS, and Minio as Blob Storage providers. |
| config.blobStorage | object | `{"accessKey":"","accessKeySecret":"","apiURL":"https://s3.us-west-2.amazonaws.com","azureStorageAccountKey":"","azureStorageAccountName":"","azureStorageConnectionString":"","azureStorageContainerName":"","azureStorageServiceUrlOverride":"","bucketName":"","chSearchEnabled":true,"enabled":false,"engine":"S3","minBlobStorageSizeKb":"20"}` | Blob storage configuration Optional. Used to store inputs, outputs, and errors in Blob Storage. We currently support S3, GCS, Minio, and Azure as Blob Storage providers. |
| config.blobStorage.azureStorageAccountName | string | `""` | Optional. Set this along with azureStorageAccountKey to use a storage account and access key. Higher precedence than azureStorageConnectionString. |
| config.blobStorage.azureStorageConnectionString | string | `""` | Optional. Use this to specify the full connection string including any authentication params. |
| config.blobStorage.azureStorageContainerName | string | `""` | Required if using Azure blob storage |
| config.blobStorage.azureStorageServiceUrlOverride | string | `""` | Optional. Use this to customize the service URL, which by default is 'https://<storage_account_name>.blob.core.windows.net/' |
| config.existingSecretName | string | `""` | |
| config.hostname | string | `""` | Base URL of the LangSmith installation. Used for redirects. |
| config.langsmithLicenseKey | string | `""` | |
Expand Down Expand Up @@ -775,6 +779,6 @@ For information on how to use this chart, up-to-date release notes, and other gu
| Ankush | <ankush@langchain.dev> | |

----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.11.3](https://github.com/norwoodj/helm-docs/releases/v1.11.3)
Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)
## Docs Generated by [helm-docs](https://github.com/norwoodj/helm-docs)
`helm-docs -t ./charts/langsmith/README.md.gotmpl`
28 changes: 28 additions & 0 deletions charts/langsmith/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,13 @@ Template containing common environment variables that are used by several servic
{{- if .Values.config.blobStorage.enabled }}
- name: FF_S3_STORAGE_ENABLED
value: {{ .Values.config.blobStorage.enabled | quote }}
- name: FF_BLOB_STORAGE_ENABLED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the old values still work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

old values are used for now, yeah. will start using the new ones after this is released

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry i mean like for people not using helm, will the old variables still be read

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah right, probably worth respecting the old values until 0.9?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea though i guess that can just be a backend thing to read both

value: {{ .Values.config.blobStorage.enabled | quote }}
- name: BLOB_STORAGE_ENGINE
value: {{ .Values.config.blobStorage.engine | quote }}
- name: MIN_BLOB_STORAGE_SIZE_KB
value: {{ .Values.config.blobStorage.minBlobStorageSizeKb | quote }}
{{- if eq .Values.config.blobStorage.engine "S3" }}
- name: S3_BUCKET_NAME
value: {{ .Values.config.blobStorage.bucketName | quote }}
- name: S3_RUN_MANIFEST_BUCKET_NAME
Expand All @@ -291,6 +298,27 @@ Template containing common environment variables that are used by several servic
key: blob_storage_access_key_secret
optional: true
{{- end }}
{{- if eq .Values.config.blobStorage.engine "Azure" }}
- name: AZURE_STORAGE_ACCOUNT_NAME
value: {{ .Values.config.blobStorage.azureStorageAccountName | quote }}
- name: AZURE_STORAGE_CONTAINER_NAME
value: {{ .Values.config.blobStorage.azureStorageContainerName | quote }}
- name: AZURE_STORAGE_SERVICE_URL_OVERRIDE
value: {{ .Values.config.blobStorage.azureStorageServiceUrlOverride | quote }}
- name: AZURE_STORAGE_ACCOUNT_KEY
valueFrom:
secretKeyRef:
name: {{ include "langsmith.secretsName" . }}
key: azure_storage_account_key
optional: true
- name: AZURE_STORAGE_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: {{ include "langsmith.secretsName" . }}
key: azure_storage_connection_string
optional: true
{{- end }}
{{- end }}
- name: FF_CH_SEARCH_ENABLED
value: {{ .Values.config.blobStorage.chSearchEnabled | quote }}
{{ include "langsmith.conditionalEnvVarsResolved" . }}
Expand Down
2 changes: 2 additions & 0 deletions charts/langsmith/templates/secrets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,6 @@ data:
initial_org_admin_password: {{ .Values.config.basicAuth.initialOrgAdminPassword | b64enc | quote }}
blob_storage_access_key: {{ .Values.config.blobStorage.accessKey | b64enc | quote }}
blob_storage_access_key_secret: {{ .Values.config.blobStorage.accessKeySecret | b64enc | quote }}
azure_storage_account_key: {{ .Values.config.blobStorage.azureStorageAccountKey | b64enc | quote }}
azure_storage_connection_string: {{ .Values.config.blobStorage.azureStorageConnectionString | b64enc | quote }}
{{- end }}
25 changes: 19 additions & 6 deletions charts/langsmith/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,23 @@ images:
aceBackendImage:
repository: "docker.io/langchain/langsmith-ace-backend"
pullPolicy: IfNotPresent
tag: "0.8.27"
tag: "0.8.29"
backendImage:
repository: "docker.io/langchain/langsmith-backend"
pullPolicy: IfNotPresent
tag: "0.8.27"
tag: "0.8.29"
frontendImage:
repository: "docker.io/langchain/langsmith-frontend"
pullPolicy: IfNotPresent
tag: "0.8.27"
tag: "0.8.29"
platformBackendImage:
repository: "docker.io/langchain/langsmith-go-backend"
pullPolicy: IfNotPresent
tag: "0.8.27"
tag: "0.8.29"
playgroundImage:
repository: "docker.io/langchain/langsmith-playground"
pullPolicy: IfNotPresent
tag: "0.8.27"
tag: "0.8.29"
postgresImage:
repository: "docker.io/postgres"
pullPolicy: IfNotPresent
Expand Down Expand Up @@ -107,18 +107,31 @@ config:
shortlived: "1209600"
# -- Blob storage configuration
# Optional. Used to store inputs, outputs, and errors in Blob Storage.
# We currently support S3, GCS, and Minio as Blob Storage providers.
# We currently support S3, GCS, Minio, and Azure as Blob Storage providers.
blobStorage:
enabled: false
engine: "S3"
# If you are using langsmith-managed-clickhouse, you may not want inputs to be stored in clickhouse for search.
# Set this as false to ensure that inputs/outputs/errors are not stored in clickhouse.
chSearchEnabled: true
# Set this to change the threshold for payloads to be stored in blob storage
minBlobStorageSizeKb: "20"
# If you are using workload identity, you may not need to store the S3 credentials in the secrets.
# Instead, you will need to add the workload identity annotation to the backend and queue service accounts.
accessKey: ""
accessKeySecret: ""
bucketName: ""
apiURL: "https://s3.us-west-2.amazonaws.com"
# The following blob storage configuration values are for Azure and require blobStorage.engine = "Azure"
# -- Optional. Set this along with azureStorageAccountKey to use a storage account and access key. Higher precedence than azureStorageConnectionString.
azureStorageAccountName: ""
azureStorageAccountKey: ""
# -- Required if using Azure blob storage
azureStorageContainerName: ""
# -- Optional. Use this to specify the full connection string including any authentication params.
azureStorageConnectionString: ""
# -- Optional. Use this to customize the service URL, which by default is 'https://<storage_account_name>.blob.core.windows.net/'
azureStorageServiceUrlOverride: ""

aceBackend:
name: "ace-backend"
Expand Down
Loading