Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fully azdevify the application #281

Merged
merged 72 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
24be759
azdevify this sample and add azd new feature in devcontainer
Nov 5, 2023
fbfd9b7
instead parameters.json file with .bicepparam
Nov 5, 2023
d32d977
processing format
Nov 7, 2023
552f254
keep consistent casing for params
Nov 7, 2023
c00f55a
move appsettings
Nov 7, 2023
b2e62b2
remove some invalid changes
Nov 9, 2023
b1a97b7
fix format issues
Nov 9, 2023
625487e
Remove depandsOn
Nov 9, 2023
65ac3c5
remove resources file
Nov 10, 2023
43b09e7
remov keys from output
Nov 10, 2023
bd29d31
remove existing reference in module
Nov 10, 2023
8587450
update ReadMe
Nov 15, 2023
afd3ca8
add rgName in listkeys
Nov 22, 2023
654762e
add use keyvault option
Nov 23, 2023
cc3e957
set default value in .bicepparam
Nov 23, 2023
9dc6a78
fix some bugs
Nov 28, 2023
9bcb2a7
update rbac
Dec 14, 2023
1a9212b
update rbac
Dec 21, 2023
eb11eab
fix some format and update infra core
Jan 3, 2024
a403d25
Update openai to v1
ChenxiJiang333 Jan 3, 2024
b88ee5f
use resourcegroup.name() instead pass in rhName
Jan 3, 2024
4bdf8c3
Merge pull request #1 from zedy-wj/openaiv1
zedy-wj Jan 3, 2024
1ecb9be
Revert "Update openai to v1"
zedy-wj Jan 3, 2024
75f230e
Merge pull request #2 from zedy-wj/revert-1-openaiv1
zedy-wj Jan 3, 2024
969aa25
add token provider
ChenxiJiang333 Jan 4, 2024
ffee3f8
Merge branch 'azdevify' into openaiv1
ChenxiJiang333 Jan 4, 2024
778e1d4
Merge pull request #3 from zedy-wj/openaiv1
zedy-wj Jan 4, 2024
5892d5d
Update requirements.txt
ChenxiJiang333 Jan 4, 2024
ce7d48d
Merge pull request #4 from zedy-wj/ChenxiJiang333-patch-1
zedy-wj Jan 4, 2024
2b683de
revert
Jan 4, 2024
397ab11
fix chat
ChenxiJiang333 Jan 4, 2024
1e21f2a
Update TextProcessingTool.py
ChenxiJiang333 Jan 4, 2024
9481a66
Merge pull request #5 from zedy-wj/newcommit
zedy-wj Jan 5, 2024
b25cce5
update KeyVault options
Jan 10, 2024
332402a
abstract auth type with define method
Jan 11, 2024
bd092df
remove invalid infra/core
Jan 11, 2024
5406c5a
update readme and storage.bicep in infra/core
Jan 12, 2024
d2145ec
Update output format
Jan 17, 2024
c83a409
default to use rbac
Jan 22, 2024
aa86081
add some comments
Jan 26, 2024
06bb879
solve conflicts
ChenxiJiang333 Jan 31, 2024
1a7303a
add readme and speech service related code in main.bicep
Feb 1, 2024
2c8abfb
fix azd deploy problem
Feb 2, 2024
5dcf10c
move code folder structure
Feb 5, 2024
779065f
fix conflict
Feb 6, 2024
cf07644
azdev final
jongio Feb 6, 2024
5186832
fix ci error and some format issue
Feb 7, 2024
7ac0727
fix some format issue
Feb 7, 2024
7ac4e5f
fix some format issues
Feb 7, 2024
b769b9a
little changes
Feb 7, 2024
5c85d84
fix conflict
Feb 7, 2024
c4e140d
fix backend issue
Feb 7, 2024
9b6f180
Fixed mismatched requirements
ross-p-smith Feb 7, 2024
4a1ef31
Fix build-frontend
ross-p-smith Feb 7, 2024
97eb155
fix function error
Feb 8, 2024
1701d36
Run black against the code base
ross-p-smith Feb 8, 2024
49ddafd
fix conflict and correct package version
Feb 9, 2024
9087ebe
try to fix ci error
Feb 9, 2024
984b7a8
fix ci errors
Feb 9, 2024
fab8895
fix azd deploy web error
Feb 9, 2024
2d92b83
fix web debug error and update main.bicep
Feb 19, 2024
4167b63
Merge remote-tracking branch 'upstream/main'
Feb 19, 2024
32c5851
resolve conflict
Feb 19, 2024
792c7f8
fix tests path
Feb 19, 2024
61f85bf
Merge remote-tracking branch 'upstream/main'
Feb 20, 2024
ce0279d
try to fix CI/build error
Feb 20, 2024
6d7bb5a
Merge branch 'main' into pr/jongio/281
ross-p-smith Feb 20, 2024
76d5b74
Merge main
ross-p-smith Feb 20, 2024
56caa4c
Fixed Requirements
ross-p-smith Feb 20, 2024
f3abef6
Black fixes
ross-p-smith Feb 20, 2024
1f6572e
Merge branch 'main' into pr/jongio/281
ross-p-smith Feb 20, 2024
215af4a
Merge branch 'main' into main
ross-p-smith Feb 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"TeamsDevApp.ms-teams-vscode-extension"
],
"settings": {
"azureFunctions.projectSubpath": "code/batch",
"azureFunctions.projectSubpath": "code/backend/batch",
"python.defaultInterpreterPath": "/usr/local/bin/python",
"python.pythonPath": "/usr/local/bin/python"
}
Expand Down
2 changes: 1 addition & 1 deletion .devcontainer/postCreate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

pip install --upgrade pip

pip install -r code/requirements.txt -r code/app/requirements.txt -r extensions/backend/requirements.txt
pip install -r code/requirements.txt -r code/backend/requirements.txt -r code/backend/batch/requirements.txt

pip install -r code/dev-requirements.txt

Expand Down
7 changes: 6 additions & 1 deletion .env.sample
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,9 @@ ORCHESTRATION_STRATEGY=openai_functions
#Speech-to-text feature
AZURE_SPEECH_SERVICE_KEY=
AZURE_SPEECH_SERVICE_REGION=
AZURE_AUTH_TYPE=rbac
# Auth type environment variables.
# When AZURE_AUTH_TYPE=rbac, please make sure variable USE_KEY_VAULT=false
# When USE_KEY_VAULT=true, please make sure to set AZURE_KEY_VAULT_ENDPOINT
AZURE_AUTH_TYPE=keys
USE_KEY_VAULT=true
AZURE_KEY_VAULT_ENDPOINT=
4 changes: 2 additions & 2 deletions .github/workflows/unittests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ jobs:
architecture: x64
- name: Install dependencies
run: |
pip install -r code/requirements.txt -r code/dev-requirements.txt -r code/app/requirements.txt
pip install -r code/requirements.txt -r code/dev-requirements.txt -r code/backend/requirements.txt
- name: Run Python tests
run: python -m pytest --rootdir=code -m "not azure"
run: cd code/; python -m pytest -m "not azure"
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore

# generated frontend files
code/app/static/
code/static/

# User-specific files
*.rsuser
Expand Down
6 changes: 3 additions & 3 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"run",
"Admin.py"
],
"cwd": "${workspaceFolder}/code/admin",
ross-p-smith marked this conversation as resolved.
Show resolved Hide resolved
"cwd": "${workspaceFolder}/code/backend",
"preLaunchTask": "pip install (code)",
},
{
Expand All @@ -31,14 +31,14 @@
"--debug",
"run"
],
"cwd": "${workspaceFolder}/code/app",
"cwd": "${workspaceFolder}/code",
"preLaunchTask": "pip install (code)"
},
{
"name": "Launch Frontend (UI)",
"type": "node",
"request": "launch",
"cwd": "${workspaceFolder}/code/app/frontend",
"cwd": "${workspaceFolder}/code/frontend",
"preLaunchTask": "npm install (code)",
"runtimeExecutable": "npm",
"runtimeArgs": [
Expand Down
2 changes: 1 addition & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"azureFunctions.deploySubpath": "code\\batch",
"azureFunctions.deploySubpath": "code\\backend\\batch",
"azureFunctions.scmDoBuildDuringDeployment": true,
"azureFunctions.pythonVenv": ".venv",
"azureFunctions.projectLanguage": "Python",
Expand Down
6 changes: 3 additions & 3 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"isBackground": true,
"dependsOn": "pip install (functions)",
"options": {
"cwd": "${workspaceFolder}/backend"
"cwd": "${workspaceFolder}/code/backend/batch"
}
},
{
Expand All @@ -26,7 +26,7 @@
},
"problemMatcher": [],
"options": {
"cwd": "${workspaceFolder}/backend"
"cwd": "${workspaceFolder}/code/backend"
}
},
{
Expand All @@ -44,7 +44,7 @@
"command": "npm install",
"problemMatcher": [],
"options": {
"cwd": "${workspaceFolder}/code/app/frontend"
"cwd": "${workspaceFolder}/code/frontend"
}
}
]
Expand Down
8 changes: 6 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@ ci: unittest build-frontend ## 🚀 Continuous Integration (called by Github Act

unittest: ## 🧪 Run the unit tests
@echo -e "\e[34m$@\e[0m" || true
@python -m pytest -m "not azure"
@cd code/ && python -m pytest -m "not azure"

build-frontend: ## 🏗️ Build the Frontend webapp
@echo -e "\e[34m$@\e[0m" || true
@cd code/app/frontend && npm install && npm run build
@cd code/frontend && npm install && npm run build

azd-login: ## 🔑 Login to Azure with azd and a SPN
@echo -e "\e[34m$@\e[0m" || true
@azd auth login --client-id ${AZURE_CLIENT_ID} --client-secret ${AZURE_CLIENT_SECRET} --tenant-id ${AZURE_TENANT_ID}
40 changes: 40 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,46 @@ az deployment group create --resource-group $RESOURCE_GROUP_NAME --template-file


![A screenshot of the chat app.](./media/web-unstructureddata.png)
### Running the sample using the Azure Developer CLI (azd)

The Azure Developer CLI (`azd`) is a developer-centric command-line interface (CLI) tool for creating Azure applications.

You need to install it before running and deploying with the Azure Developer CLI. (If you use the devcontainer, everything is already installed)

### Windows

```powershell
powershell -ex AllSigned -c "Invoke-RestMethod 'https://aka.ms/install-azd.ps1' | Invoke-Expression"
```

### Linux/MacOS

```
curl -fsSL https://aka.ms/install-azd.sh | bash
```

After logging in with the following command, you will be able to use the `azd` cli to quickly provision and deploy the application.
zedy-wj marked this conversation as resolved.
Show resolved Hide resolved

```
azd auth login
```

Then, execute the `azd init` command to initialize the environment (You do not need to run this command if you already have the code or have opened this in a Codespace or DevContainer).
```
azd init -t chat-with-your-data-solution-accelerator
```
Enter an environment name.

**Notes:** the default auth type uses keys, if you want to switch to rbac, please run `azd env set AUTH_TYPE rbac`.
zedy-wj marked this conversation as resolved.
Show resolved Hide resolved

Then, run `azd up` to provision all the resources to Azure and deploy the code to those resources.
```
azd up
```

Select your desired `subscription` and `location`. Wait a moment for the resource deployment to complete, click the website endpoint and you will see the web app page.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait a moment

A bit of a nit. But for me, it took 35 minutes for this to fully deploy. Should we re-word this to set better expectations?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jongio - Three services are currently used. In local deployment, we also noticed that packaging in python and npm takes a lot of time. In fact, adminweb and function services use the same requirements.txt file, but when using azd to deploy, because two independent services need to install packages twice, the deployment time is too long. Do you have any suggestions for optimizing deployment time?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you have a step that installs the requirements first and then uses a multi-stage docker build step that shares the locations of the packages?

The devcontainer installs all of these requirements as a PostCreate step so as long as we use a convention we can fix it for this too

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you have a step that installs the requirements first and then uses a multi-stage docker build step that shares the locations of the packages?

@ross-p-smith - If I understand correctly, you mean to install the required requirements in a container first, and then use docker to obtain them in other containers where they are needed, right?

@jongio - Do you have any ideas about the above requirements? As far as I know, when using azd deploy, the host=appservice with language=py will first find the requirements in the current path and install it. And using azd, the process from provisioning resources to deploying resources is consistent. If we want to pull other images in appservice, it seems that we can only do some operations through the appCommandLine parameter during provisioning (docker has been installed in appservice). What do you think about this? Or do you have better optimization suggestions?


You can also run the sample directly locally (See below).

### [Local deployment instructions](./docs/LOCAL_DEPLOYMENT.md)
To customize the accelerator or run it locally, first, copy the .env.sample file to your development environment's .env file, and edit it according to environment variable values table. Learn more about deploying locally [here](./docs/LOCAL_DEPLOYMENT.md).
Expand Down
33 changes: 33 additions & 0 deletions azure.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json

name: chat-with-your-data-solution-accelerator
metadata:
template: chat-with-your-data-solution-accelerator@0.0.1-beta

services:
web:
project: ./code
language: py
host: appservice
hooks:
prepackage:
windows:
zedy-wj marked this conversation as resolved.
Show resolved Hide resolved
shell: pwsh
run: cd ./frontend;npm install;npm run build;
interactive: true
continueOnError: false
posix:
shell: sh
run: cd ./frontend;npm install;npm run build;
interactive: true
continueOnError: false

adminweb:
project: ./code/backend
language: py
host: appservice

function:
project: ./code/backend/batch
language: py
host: function
Loading