2023-11-29: Fall release

We're going to start using releases more consistently. You can choose to watch just the "releases" in this repository to be notified of the latest changes. This is our first release since July, so the changelog below includes quite a few changes.

The most notable changes are:

• streaming responses in the UI and backend
• optional login and document ACLs
• rendering full PDFs instead of single pages in the UI
• support for non-Azure OpenAI
• removal of langchain-based approaches
• calculation of MD5 hashes when uploading docs (to avoid redundant uploads later)

What's Changed

• Add more tests for core/ folder by @pamelafox in #430
• Update FAQ to include common failures reasons with links by @johnrsanders in #252
• Add simple validation by @pamelafox in #445
• Using azd feature by @pamelafox in #429
• Rewrite roles.ps1 in bash by @ahnl in #351
• Fix for paths with spaces by @amithkk in #71
• Fix path to env file in README by @openjck in #153
• Add 3.8 and 3.9 to test to matrix, fix type bug by @pamelafox in #454
• Require Node 14+ by @pamelafox in #453
• Update imports to explicit named imports for prepdocs by @pamelafox in #452
• Update notebooks to chatcompletion api by @srbalakr in #463
• Tell Oryx to configure gunicorn workers dynamically by @pamelafox in #461
• Adjust regions by @pamelafox in #467
• Add more to troubleshooting section by @pamelafox in #470
• Respect ENV values for resource group and index name by @Patrick-Davis-MSFT in #469
• Startup script for App Service by @pamelafox in #464
• Ensure openai token by @gukoff in #262
• Run ruff, add to precommit by @pamelafox in #491
• Use an app factory pattern to enable app testing by @pamelafox in #495
• Port to Quart by @pamelafox in #503
• Update README.md with FAQ comparing to other repo by @pamelafox in #512
• Sample Locust loadtest by @pamelafox in #486
• Change to port 50505 by @pamelafox in #523
• Gunicorn timeout by @pamelafox in #524
• Add application insights for request and backend tracing of performance data by @tonybaloney in #515
• Update main.bicep with more allowed regions by @pamelafox in #520
• Address recent bugs by @srbalakr in #529
• Add more details to costs section by @pamelafox in #537
• Update README.md to include clear authentication/access control and steps to secure deployed app by @andyloree in #536
• Add workaround for connection reset issue by @pamelafox in #525
• Fix typo in prepdocs.py by @eltociear in #539
• Remove unneeded deployment by @pamelafox in #540
• Typo "Azure Open AI"→"Azure OpenAI" by @hyoshioka0128 in #545
• Update README.md based on Connor's flow by @pamelafox in #548
• Remove defaults for getenv by @pamelafox in #550
• Update README.md location list by @pamelafox in #551
• Update README.md FAQ re gpt-4 by @pamelafox in #553
• Bump langchain from 0.0.187 to 0.0.236 in /notebooks by @dependabot in #522
• Bump langchain from 0.0.187 to 0.0.236 in /app/backend by @dependabot in #521
• Add Azure Security for DevOps to CI Pipeline by @tonybaloney in #557
• Adding table of contents to main readme by @connorwehrum in #554
• Use Standard_LRS instead of Standard_ZRS by @debonte in #567
• Allow for subdirectory structure within data directory by @PfisterAn in #569
• Make SARIF uploads optional but automatic for the main repo by @tonybaloney in #578
• Create stale-bot.yml for tidying up backlog by @tonybaloney in #579
• Remove notebooks by @pamelafox in #588
• Update README.md with chat/ask question by @pamelafox in #584
• Add section about productionizing by @pamelafox in #577
• Update README.md deploying section by @pamelafox in #589
• Update README.md about local env by @pamelafox in #587
• Indexing: Improve vectors throughput by @srbalakr in #556
• Adds optional streaming support for chat requests by @pamelafox in #532
• Explicitly set https required by @tonybaloney in #581
• Update README.md with account reqs by @pamelafox in #609
• Fix prepdocs compatibility with openai key and add test by @pamelafox in #605
• Use environ vs getenv by @pamelafox in #611
• Update README.md with logging info by @pamelafox in #610
• Fix devcontainer name by @iemejia in #613
• Fix prompt_template override, add tests by @pamelafox in #606
• Update README.md with azd up descrip by @pamelafox in #602
• Make it easier to re-use a search service by @pamelafox in #620
• Add parameter for configuring storage sku by @pamelafox in #628
• Reword top label by @pamelafox in #627
• Only validate bicep on changes to infra folder by @tonybaloney in #631
• Run black on all Python files by @pamelafox in #632
• Adding support for non-Azure openai instances by @carlotta94c in #507
• Readme on deploying to existing RG by @pamelafox in #637
• Hotfix to prepdocs.ps1 to fix env variable by @pamelafox in #639
• Adding anchors to FAQs by @pamelafox in #638
• Comparison table by @pamelafox in #640
• Clarify which search Skus are compatible by @pamelafox in #642
• Fix prepdocs to handle empty env variables by @pamelafox in #651
• Smooth streaming, fix partial citation, and ensure visibility. by @SteveJSteiner in #659
• Adding more regions by @tonybaloney in #660
• Stabilize search query generation by @srbalakr in #652
• Hotfix: Workaround for stream bug by @pamelafox in #664
• Add support for an optional login and document level access control system. by @mattmsft in #624
• Add E2E tests using Playwright, increase accessibility of buttons by @pamelafox in #677
• Escape more Powershell possibly empty args by @pamelafox in #683
• Use pip-tools to pin transitive dependencies and dependabot for version updates by @pamelafox in #693
• Bump actions/checkout from 3 to 4 by @dependabot in #700
• Bump actions/setup-node from 2 to 3 by @dependabot in #701
• Bump ndjson-readablestream from 1.0.6 to 1.0.7 in /app/frontend by @dependabot in #699
• Bump quart from 0.19.1 to 0.19.2 in /app/backend by @dependabot in #709
• Bump dompurify and @types/dompurify in /app/frontend by @dependabot in #708
• Bump @types/react-dom from 18.2.6 to 18.2.8 in /app/frontend by @dependabot in #707
• Bump actions/setup-python from 2 to 4 by @dependabot in #702
• Bump @fluentui/react-components from 9.30.3 to 9.33.0 in /app/frontend by @dependabot in #706
• Bump react-router-dom from 6.14.1 to 6.16.0 in /app/frontend by @dependabot in #703
• Readme cost warning by @Aloncarrasco in #711
• Address documentation feedback for access control by @mattmsft in #712
• Address Issue #670 by @srbalakr in #716
• Enable CORS support by @pamelafox in #717
• Fix CORS command in readme by @pamelafox in #723
• Default log level for dev vs prod by @pamelafox in #718
• Bump langchain from 0.0.305 to 0.0.306 in /app/backend by @dependabot in #705
• Bump postcss from 8.4.25 to 8.4.31 in /app/frontend by @dependabot in #730
• Remove broken langchain-based approaches by @pamelafox in #738
• Fix embedding name so that it can be set with azd by @pamelafox in #691
• Normalize text in messages to reduce token length by @Yun-BizQA in #688
• Conform to Chat App specification for responses by @pamelafox in #679
• Update @azure/msal-browser related dependencies by @mattmsft in #745
• Clarify roles by @pamelafox in #741
• Bump prettier from 3.0.0 to 3.0.3 in /app/frontend by @dependabot in #757
• Bump quart from 0.19.2 to 0.19.3 in /app/backend by @dependabot in #755
• Bump pypdf from 3.16.2 to 3.16.3 in /scripts by @dependabot in #753
• Python version updates by @pamelafox in #761
• Grouped updates by @pamelafox in #729
• Fixed type annotations per mypy and added mypy to CICD by @pamelafox in #749
• Urllib3 update for security issue by @pamelafox in #728
• Add more Playwright tests to check all routes and buttons by @pamelafox in #743
• Bump @fluentui/react from 8.110.7 to 8.112.2 in /app/frontend by @dependabot in #759
• Bump the node-packages group in /app/frontend with 7 updates by @dependabot in #771
• Chat response by @pamelafox in #748
• Rm print statements by @pamelafox in #772
• Adding session_state to conform to ChatApp protocol by @pamelafox in #773
• Use absolute redirect uris for login and fetching tokens by @mattmsft in #784
• Daily tests by @pamelafox in #787
• Improvements to VS Code and Vite debug experience by @pamelafox in #786
• support non English languages by @ks6088ts in #780
• Add arguments required for OpenAI connection by @wdfox in #796
• Bump the node-packages group in /app/frontend with 5 updates by @dependabot in #800
• Bump @babel/traverse from 7.23.0 to 7.23.2 in /app/frontend by @dependabot in #812
• Increased maxtoken to avoid malformed JSON in chatreadretrieveread.py by @KesavarajaRK in #681
• Message builder fixes for history format and token count by @pamelafox in #778
• Add scheduler package by @mattmsft in #822
• Bump urllib3 from 2.0.6 to 2.0.7 in /app/backend by @dependabot in #820
• Fix bug due to "bot" being used instead of "assistant" in Chat.tsx by @tvhahn in #823
• Support windows by using uvicorn without extras by @pamelafox in #815
• Update Readme.md - Dev Containers by @charris-msft in #829
• Fix if for tests running conditional by @pamelafox in #833
• Run prettier on files and add to precommit/CI by @pamelafox in #834
• Add front-matter needed for Sample browser by @pamelafox in #816
• Store an MD5 hash of uploaded/indexed file and check before prepdocs by @tonybaloney in #835
• Improve follow-up questions and pipe into context by @pamelafox in #832
• Reorder tags to optimize for sample browser by @pamelafox in #853
• Fix order of few shots sent to MessageBuilder by @pamelafox in #852
• Bump werkzeug from 3.0.0 to 3.0.1 in /app/backend by @dependabot in #863
• Updated FAQ so that it reflect PR 835 by @MaciejLitwiniec in #868
• Refactor prepdocs by @mattmsft in #862
• Locust file related improvements by @pamelafox in #870
• Render entire PDFs instead of single pages by @pamelafox in #840
• Bump the github-actions group with 1 update by @dependabot in #880
• Using the option --storagekey for the prepdocs script might fail by @IRooc in #866
• Add exclude files by @pamelafox in #876
• Handle errors better especially for streaming by @pamelafox in #884
• Add more readmes/guides by @pamelafox in #889
• Bump pypdf from 3.16.3 to 3.17.0 in /scripts by @dependabot in #890
• Fix list file by @pamelafox in #897
• Including Australia East to allowed regions by @Smiddy4000 in #906
• Test and fix listfilestrategy.py by @pamelafox in #907
• Upgrade infra/core to latest from azd by @jongio in #918
• [Bug fix] Fix for mimetypes issue when running local server on Windows machines by @pamelafox in #928
• Bump the python-requirements group in /scripts with 5 updates by @dependabot in #894
• Bump the python-requirements group with 20 updates by @dependabot in #934
• [Infra] Add swedencentral to list of allowed locations for OpenAI deployment by @pamelafox in #935
• Enable uploads of large PDFs over connections with less upload speed by @pamelafox in #923
• Update customization.md with correct image filenames by @pamelafox in #926
• [Backend] Add special handling in app.py for content safety filter errors by @pamelafox in #896
• [CI] Move cron trigger to dedicated nightly workflow for better CI behavior on forks by @pamelafox in #937
• [Docs] Document how to make setting overrides permanent by @pamelafox in #955
• Rebranding from Cognitive Search to AI Search by @gmndrg in #959
• replace form recognizer with doc intelligence by @Dybe in #686
• [Docs] Rename remaining instances of Cognitive Search to AI Search by @pamelafox in #961
• Update README.md with link to samples in other languages by @achandmsft in #966
• Fix ignored capacity field for openAI deployment by @vhvb1989 in #970
• set content-type for streamed response by @kaygu in #968
• [Bug fix] Fix prepdocs logic in uploading docs with more than 1000 sections by @pamelafox in #971
• Fixing app service settings by @pamelafox in #986
• Added insights dashboard by @pamelafox in #898
• [Infra] Default AZURE_USE_APPLICATION_INSIGHTS to true, so that monitoring is always provisioned by @pamelafox in #992
• [BugFix] Fix "monitoring" related issues in main.bicep by @pamelafox in #991
• [Backend/Prepdocs] Update to latest version of search SDK by @pamelafox in #1010
• Fix: BadRequest: NetworkAcls is required for this resouce. by @amgdy in #799
• Bump cryptography from 41.0.5 to 41.0.6 in /scripts by @dependabot in #1016
• Bump aiohttp from 3.8.6 to 3.9.0 in /scripts by @dependabot in #1009
• Bump the python-requirements group with 13 updates by @dependabot in #1013

New Contributors

• @johnrsanders made their first contribution in #252
• @ahnl made their first contribution in #351
• @amithkk made their first contribution in #71
• @openjck made their first contribution in #153
• @Patrick-Davis-MSFT made their first contribution in #469
• @andyloree made their first contribution in #536
• @eltociear made their first contribution in #539
• @hyoshioka0128 made their first contribution in #545
• @connorwehrum made their first contribution in #554
• @debonte made their first contribution in #567
• @PfisterAn made their first contribution in #569
• @iemejia made their first contribution in #613
• @carlotta94c made their first contribution in #507
• @SteveJSteiner made their first contribution in #659
• @Aloncarrasco made their first contribution in #711
• @Yun-BizQA made their first contribution in #688
• @ks6088ts made their first contribution in #780
• @wdfox made their first contribution in #796
• @KesavarajaRK made their first contribution in #681
• @tvhahn made their first contribution in #823
• @charris-msft made their first contribution in #829
• @MaciejLitwiniec made their first contribution in #868
• @IRooc made their first contribution in #866
• @Smiddy4000 made their first contribution in #906
• @gmndrg made their first contribution in #959
• @Dybe made their first contribution in #686
• @kaygu made their first contribution in #968
• @amgdy made their first contribution in #799

Full Changelog: rel-vector-071823...29-11-2023