ADR for using Pull based(Integrated Vectorization) approach throughout

[komalg1]

Purpose

• Adds an ADR for the approach to take to implement Integrated Vectorization. Required by #321

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Documentation content changes
[ ] Other... Please describe:

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
code
app.py	13	13	0%	1–4, 6, 8, 11–13, 15, 17, 19–20
create_app.py	148	3	97%	199, 204, 327
code/backend
Admin.py	22	22	0%	1–6, 9, 11, 13, 15, 18–19, 21–22, 25, 32, 39, 42–44, 46, 48
code/backend/batch
AddURLEmbeddings.py	29	2	93%	37–38
BatchPushResults.py	27	0	100%
BatchStartProcessing.py	19	0	100%
GetConversationResponse.py	32	3	90%	63–65
function_app.py	15	15	0%	1–8, 11–12, 14, 17–20
code/backend/batch/utilities/common
Answer.py	24	1	95%	39
SourceDocument.py	58	4	93%	31, 35, 39, 124
code/backend/batch/utilities/document_chunking
DocumentChunkingBase.py	10	2	80%	10, 16
FixedSizeOverlap.py	19	0	100%
Layout.py	19	0	100%
Page.py	17	0	100%
Paragraph.py	9	9	0%	1–4, 7–9, 12, 15
Strategies.py	25	4	84%	24–25, 27, 29
__init__.py	7	0	100%
code/backend/batch/utilities/document_loading
DocumentLoadingBase.py	9	1	88%	13
Layout.py	12	12	0%	1–4, 7–9, 11–13, 16, 25
Read.py	12	12	0%	1–4, 7–9, 11–13, 16, 25
Strategies.py	20	8	60%	13, 15, 17, 19, 24–25, 27, 29
Web.py	19	1	94%	23
WordDocument.py	25	25	0%	1–6, 9–12, 21–24, 26–27, 29–30, 32–37, 45
__init__.py	11	0	100%
code/backend/batch/utilities/helpers
AzureBlobStorageHelper.py	71	53	25%	16, 20–22, 30, 46, 48–50, 53–54, 59, 63, 67, 70, 73–74, 79, 82, 88–89, 91, 95, 99, 103, 109, 124, 127, 139, 142, 146, 149, 151, 159–163, 186, 190–194, 196, 199–200, 204, 208, 210, 212, 216, 227
AzureFormRecognizerHelper.py	81	81	0%	1–6, 9–11, 13, 16–17, 25, 27, 35, 43–45, 52–55, 60–68, 70, 73–75, 77–78, 81, 84–86, 88–90, 93, 97–98, 105–109, 111–114, 117–131, 133, 135–137, 139–140, 143, 145–147
AzureSearchHelper.py	20	0	100%
ConfigHelper.py	63	34	46%	19–22, 30–31, 34, 39, 42, 45, 48, 54–58, 63, 68–69, 75–76, 78–80, 83–86, 88, 92–93, 99–100, 218
DocumentChunkingHelper.py	12	1	91%	21
DocumentLoadingHelper.py	12	1	91%	14
DocumentProcessorHelper.py	31	19	38%	15–17, 22, 25–35, 38–41
EnvHelper.py	120	9	92%	194, 199–200, 203–205, 215–217
LLMHelper.py	33	20	39%	11–13, 15–16, 22, 28–29, 34, 37–38, 47, 58–59, 71, 83–84, 91, 101, 109
OrchestratorHelper.py	12	4	66%	20–22, 25
code/backend/batch/utilities/loggers
ConversationLogger.py	36	28	22%	8, 11–12, 15–24, 27–30, 33–42, 46
TokenLogger.py	9	4	55%	7–8, 11, 15
code/backend/batch/utilities/orchestrator
LangChainAgent.py	72	29	59%	23–28, 30, 65–66, 71–73, 78, 98–101, 118–119, 122–125, 132–133, 138–140, 143
OpenAIFunctions.py	66	66	0%	1–3, 5–12, 14, 17–21, 56, 59, 62–64, 69–71, 76, 79, 81, 87–90, 92, 95, 102–106, 110–111, 113, 119–123, 127–129, 132, 135–136, 139, 144–146, 149–151, 156–158, 161, 164, 169
OrchestratorBase.py	32	15	53%	14–20, 31, 40–42, 49–51, 61
Strategies.py	12	7	41%	10–11, 13–15, 17, 19
__init__.py	11	1	90%	10
code/backend/batch/utilities/parser
OutputParserTool.py	39	0	100%
ParserBase.py	9	2	77%	9, 19
__init__.py	7	2	71%	7, 11
code/backend/batch/utilities/tools
AnswerProcessingBase.py	8	2	75%	8, 12
AnsweringToolBase.py	9	2	77%	9, 15
ContentSafetyChecker.py	41	25	39%	16, 18–19, 24, 30–32, 35–36, 42–43, 49–54, 57–59, 61, 65–67, 69
PostPromptTool.py	22	13	40%	11, 14–15, 17–18, 22, 29, 36–37, 45, 51–52, 60
QuestionAnswerTool.py	36	20	44%	21–24, 27–28, 33, 36, 43, 46, 50–51, 53–54, 57–59, 68, 70, 77
TextProcessingTool.py	16	9	43%	9, 12–15, 19, 21, 28, 35
code/backend/pages
01_Ingest_Data.py	120	120	0%	1–12, 18–22, 24–26, 28, 34, 41, 44, 48–49, 51, 56, 59–60, 63–72, 76–78, 81–84, 86, 89–99, 102–109, 112–114, 116, 119, 121–124, 129–134, 137, 140–141, 144, 150, 163–166, 169–170, 174, 178, 185, 199–202, 205, 210–211, 213–214, 216–218, 222, 225–226, 232–235, 242–243, 248, 250–251
02_Explore_Data.py	28	28	0%	1–8, 10, 12, 14, 20, 27, 30, 38, 41–43, 45–48, 50, 54, 58–59, 62–63
03_Delete_Data.py	51	51	0%	1–6, 8, 10, 12, 18, 25, 28, 36, 39–40, 43–47, 49, 51–55, 57–58, 60, 63–65, 67–70, 72–74, 76, 78, 81–83, 85–86, 88–90, 92–93
04_Configuration.py	69	69	0%	1–6, 8, 10, 12, 19, 26, 28, 33–38, 41–42, 45–46, 48–51, 53–54, 64–68, 71–72, 76–78, 81–82, 85–86, 89–93, 100–102, 104, 106, 114, 121–122, 129, 131, 143–144, 161–162, 166, 168–169, 185, 207–208, 210–211
TOTAL	1749	852	51%

Tests	Skipped	Failures	Errors	Time
74	0 💤	0 ❌	0 🔥	9.970s ⏱️

[komalg1]

This is the initial draft design I thought of for implementing Integrated Vectorization. I might have missed something. We can have an extended discussion and finalize the design.

[adamdougal]

Looking good! One minor comment!