KnowledgeSource not working

[mirojs]

Hello team,

After trying example of both the python and Jupiter sample code, the KnowledgeSource option to include reference files failed to work with 400 error.

Could you help to check if this is a currently available option? Thanks.

[chienyuanchang]

Hi @mirojs, thank you for sharing the issue. Could you please share more details about how you used knowledgeSources? I think the 400 error could be caused by invalid analyzer.

We have a sample notebook of using knowledgeSources in Pro mode: https://github.com/Azure-Samples/azure-ai-content-understanding-python/blob/main/notebooks/field_extraction_pro_mode.ipynb . Were you trying this sample notebook?

If you're trying to add some training data, you can refer to another notebook for using labeled training data: https://github.com/Azure-Samples/azure-ai-content-understanding-python/blob/main/notebooks/analyzer_training.ipynb

[mirojs]

@chienyuanchang Thanks for the response. I did several tests.

#1: Call api directly following the put -> post -> get pattern with input/reference files but without training data. The schema was accepted but at the end I was stopped by the 400 error no matter how

#2: I went ahead to test the sample code directly. And in order to make sure I was not missing anything, I tested both https://github.com/Azure-Samples/azure-ai-content-understanding-python/blob/main/notebooks/field_extraction_pro_mode.ipynb and https://github.com/Azure-Samples/azure-ai-content-understanding-python/blob/main/python/content_understanding_client.py, but both failed at exactly the same point as in the test #1.

#3: Eventually, I ended up by just removing the knowledgeSources property (leaving the multiple input files) and the put -> post -> get pattern worked like a charm with expected output, which proved the schema and multiple input files involvement were correct

After the above tests, I started to wonder if the knowledgeSources property is still valid? Besides, I used the json file and following exactly the same way as in the https://github.com/Azure-Samples/azure-ai-content-understanding-python/blob/main/python/content_understanding_client.py file.

Besides, the schema generation is a bit mysterious. That doesn't seem to be any rules to follow when building a customized one. Could you advise on this as well?

Appreciate for your further help. Thanks.

[chienyuanchang]

@mirojs Thank you for sharing the details. I am able to run our sample notebooks. Can you please share more details of the failed step in field_extraction_pro_mode.ipynb? There might be some Azure blob or credential setting issue if you could not run the notebook with our sample analyzer and data. I would suggest that we try to have a successful run of the provided sample schema and files first and then try to modify and use different files.

Yes, I agree that modalities and their modes/features require some special configs/settings that the users might not be able to know at first. That's why we are trying to build samples to demo and help the users understand how to use them. I've encountered similar issues while building the samples. I'm happy to help.

[mirojs]

@chienyuanchang Thanks for confirming that the feature is still working and I appreciate the help offering. I'll try to run it again and get back to you soon. In terms of my own test setup, I already successfully called the put, post & get apis that analyzed the input file from storage account. Only upon adding reference files by including the knowledgeSources property (with jsonl file list), the 400 error was raised. In terms of the schema, I started from the sample one but had to change the format somehow in order to get through. So, at the moment, I'm suspecting if the jsonl format would be the issue since that maybe the only one that is not so certain? Could you let me know where the correct jsonl format could be located? Thanks.

[chienyuanchang]

@mirojs Yes, this is the jsonl file of our example in the notebook.

{"file": "HOLDING TANK SERVICING CONTRACT.pdf", "resultFile": "HOLDING TANK SERVICING CONTRACT.pdf.result.json"}
{"file": "BUILDERS LIMITED WARRANTY.pdf", "resultFile": "BUILDERS LIMITED WARRANTY.pdf.result.json"}
{"file": "purchase_contract.pdf", "resultFile": "purchase_contract.pdf.result.json"}
{"file": "PROPERTY MANAGEMENT AGREEMENT.pdf", "resultFile": "PROPERTY MANAGEMENT AGREEMENT.pdf.result.json"}

We have to name it as sources.jsonl and save it under the same folder with reference document files and their result.json files on the blob.

[mirojs]

@chienyuanchang We tried several ways to make it work but still not successful. Is it possible that's a region issue?
On the other hand, I kind of wondering if it's meaningful to use the reference file option since it adds complexity by requiring separate OCR and putting all the files as input files can also get the results? Could you help to share if there's any benefit of using this option? Could you also share the best way of creating the schema, as seen from the documentation, its doesn't seem to be very exploit on rules? Or am I asking the right question? Thanks.

[chienyuanchang]

@mirojs it could be. Are you using australiaeast? Please try westus or `swedencentral. Here is the reference: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/language-region-support.

Pro mode is designed for advanced use cases, particularly those requiring multi-step reasoning, and complex decision-making (for instance, identifying inconsistencies, drawing inferences, and making sophisticated decisions). This also provides the option to have multiple input/reference files although we could also try to concatenate all files into one file as input, but the multi-step reasoning ability of Pro mode could offer more for complex questions. Here is the reference: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/standard-pro-modes. I will check with our Pro mode team to see if someone could join this discussion to provide more insights.

Yes, as I mentioned that the API documentation hasn't covered whole details about creating the analyzer schema. I think our sample notebooks and analyzer templates would be the best reference for now.

I feel a good way to debug now is to check the response message. If you can set the breakpoint of at this line in the content_understanding_client.py. You will be able to find more detailed error message under response.text like below.

If you could share the message, we can look deeper and find the root cause.

[juliagongms]

Hi @mirojs, thank you for contacting us about Pro mode! I want to echo @chienyuanchang's suggestion to share the full details of the error message you're getting. This will allow us to better help you with fixing the error.
Regarding your other questions:

Is it possible that's a region issue?

The error message will help us to better understand what's going on.

On the other hand, I kind of wondering if it's meaningful to use the reference file option since it adds complexity by requiring separate OCR and putting all the files as input files can also get the results? Could you help to share if there's any benefit of using this option?

Adding documents as reference files is a way of specifying documents that you don't want to be processed repeatedly with each run of the analyzer. It is a way to save on costs. However, if the combined set of documents fits within the page and file size limits (for preview.2, that is 150 pages and 100MB, as discussed in https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/service-limits#document-and-text), you may see better quality and it is a valid solution. It is definitely encouraged to give both setups a try if you're interested in comparing the options.

Could you also share the best way of creating the schema, as seen from the documentation, its doesn't seem to be very exploit on rules? Or am I asking the right question? Thanks.

This is a great question! We are also working on putting together more documentation that will be useful as references when creating your own schemas. The best practices we have shared at the moment are here: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/standard-pro-modes#pro-mode-known-limitations-and-best-practices. In general, creating fields that are specific and clear, with any elaboration or instructions/special rules or cases of concern, are encouraged. From the best practices, "Schemas should be designed with the highest level of specificity possible. For instance, instead of presenting a generalized list of inconsistencies, it's advisable to create distinct fields for each type of inconsistency, accompanied by detailed descriptions. Additionally, wherever feasible, references to specific sections of relevant documents that should be reviewed should be included."

Let us know if you have any other questions about Pro mode! Looking forward to hearing from you about the error message so we can better assist.

[mirojs]

@chienyuanchang Thanks for sharing the documents and I'm kind of going them through for a couple of time during the debugging journey.

Here's the log from the code, hopefully there's some clues in there:

#1: from home made version:

✅ Loaded main schema: InvoiceContractVerification

📤 UPLOADING INPUT FILES

📤 Uploading step2_input_1757609485_contoso_lifts_invoice.pdf to blob storage...
✅ Uploaded successfully: step2_input_1757609485_contoso_lifts_invoice.pdf

📊 Upload Summary:
Total input files uploaded: 1
📁 Results directory: /.../step2_results_1757609485

📋 Creating Analyzer with knowledgeSources: knowledge-source-analyzer-1757609485
🔧 Processing location: dataZone
📄 Input files for analysis: 1
🗂️ Reference container: pro-reference-files
🏷️ Reference prefix: ref_result_gen_1757607387_
📝 JSONL file: ref_result_gen_1757607387_sources.jsonl

🔐 Generating container SAS URL...
✅ Container SAS URL generated successfully
🔗 knowledgeSources configuration:
Container URL: https://...blob.core.windows.net/pro-reference-files?se=...
Prefix: ref_result_gen_1757607387_
File List Path: ref_result_gen_1757607387_sources.jsonl

✅ Analyzer with knowledgeSources created! HTTP 201
⏳ Waiting for analyzer to be ready...
⏳ Analyzer status: creating (attempt 1/30)
⏳ Analyzer status: creating (attempt 2/30)
⏳ Analyzer status: creating (attempt 3/30)
⏳ Analyzer status: creating (attempt 4/30)
⏳ Analyzer status: creating (attempt 5/30)
❌ Analyzer creation failed! Status: failed
Error details: {'analyzerId': 'knowledge-source-analyzer-1757609485', 'description': 'Document Analysis with Reference Knowledge Sources - 1757609485', 'createdAt': '2025-09-11T16:51:32Z', 'lastModifiedAt': '2025-09-11T16:52:34Z', 'baseAnalyzerId': 'prebuilt-documentAnalyzer', 'config': {'returnDetails': True, 'enableOcr': True, 'enableLayout': True, 'enableFormula': False, 'disableContentFiltering': False, 'tableFormat': 'html'}, 'fieldSchema': {'name': 'InvoiceContractVerification', 'description': 'Analyze invoice to confirm total consistency with signed contract', 'fields': {'PaymentTermsInconsistencies': {'type': 'array', 'method': 'generate', 'description': 'List all areas of inconsistency identified in the invoice with corresponding evidence.', 'items': {'type': 'object', 'method': 'generate', 'description': "Area of inconsistency in the invoice with the company's contracts.", 'properties': {'Evidence': {'type': 'string', 'method': 'generate', 'description': 'Evidence or reasoning for the inconsistency in the invoice.'}, 'InvoiceField': {'type': 'string', 'method': 'generate', 'description': 'Invoice field or the aspect that is inconsistent with the contract.'}}}}, 'ItemInconsistencies': {'type': 'array', 'method': 'generate', 'description': 'List all areas of inconsistency identified in the invoice in the goods or services sold (including detailed specifications for every line item).', 'items': {'type': 'object', 'method': 'generate', 'description': "Area of inconsistency in the invoice with the company's contracts.", 'properties': {'Evidence': {'type': 'string', 'method': 'generate', 'description': 'Evidence or reasoning for the inconsistency in the invoice.'}, 'InvoiceField': {'type': 'string', 'method': 'generate', 'description': 'Invoice field or the aspect that is inconsistent with the contract.'}}}}, 'BillingLogisticsInconsistencies': {'type': 'array', 'method': 'generate', 'description': 'List all areas of inconsistency identified in the invoice regarding billing logistics and administrative or legal issues.', 'items': {'type': 'object', 'method': 'generate', 'description': "Area of inconsistency in the invoice with the company's contracts.", 'properties': {'Evidence': {'type': 'string', 'method': 'generate', 'description': 'Evidence or reasoning for the inconsistency in the invoice.'}, 'InvoiceField': {'type': 'string', 'method': 'generate', 'description': 'Invoice field or the aspect that is inconsistent with the contract.'}}}}, 'PaymentScheduleInconsistencies': {'type': 'array', 'method': 'generate', 'description': 'List all areas of inconsistency identified in the invoice with corresponding evidence.', 'items': {'type': 'object', 'method': 'generate', 'description': "Area of inconsistency in the invoice with the company's contracts.", 'properties': {'Evidence': {'type': 'string', 'method': 'generate', 'description': 'Evidence or reasoning for the inconsistency in the invoice.'}, 'InvoiceField': {'type': 'string', 'method': 'generate', 'description': 'Invoice field or the aspect that is inconsistent with the contract.'}}}}, 'TaxOrDiscountInconsistencies': {'type': 'array', 'method': 'generate', 'description': 'List all areas of inconsistency identified in the invoice with corresponding evidence regarding taxes or discounts.', 'items': {'type': 'object', 'method': 'generate', 'description': "Area of inconsistency in the invoice with the company's contracts.", 'properties': {'Evidence': {'type': 'string', 'method': 'generate', 'description': 'Evidence or reasoning for the inconsistency in the invoice.'}, 'InvoiceField': {'type': 'string', 'method': 'generate', 'description': 'Invoice field or the aspect that is inconsistent with the contract.'}}}}}}, 'warnings': [], 'status': 'failed', 'processingLocation': 'dataZone', 'mode': 'pro', 'knowledgeSources': [{'kind': 'reference', 'containerUrl': 'https://....blob.core.windows.net/pro-reference-files?se=...
'fileListPath': 'ref_result_gen_1757607387_sources.jsonl'}]}

#2: from Microsoft version:

🔐 Getting access token...
✅ Access token obtained

🔐 Generating container SAS URL...
✅ Container SAS URL generated

📤 Uploading input file...
✅ Input file uploaded: https://....blob.core.windows.net/pro-input-files/microsoft_example_1757609723_invoice.pdf

📋 Creating analyzer: microsoft-example-1757609726
🔧 Using Microsoft's exact knowledgeSources pattern

🔗 knowledgeSources configuration:
kind: reference
containerUrl: https://....blob.core.windows.net/pro-reference-files?se=...
prefix: ref_result_gen_1757607387_
fileListPath: ref_result_gen_1757607387_sources.jsonl

✅ Analyzer created! HTTP 201
⏳ Waiting for analyzer to be ready...
⏳ Analyzer status: failed (attempt 1/30)
❌ Analyzer creation failed!
Error details: {
"analyzerId": "microsoft-example-1757609726",
"description": "Microsoft Official knowledgeSources Example",
"createdAt": "2025-09-11T16:55:27Z",
"lastModifiedAt": "2025-09-11T16:55:27Z",
"baseAnalyzerId": "prebuilt-documentAnalyzer",
"config": {
"returnDetails": true,
"enableOcr": true,
"enableLayout": true,
"enableFormula": false,
"disableContentFiltering": false,
"tableFormat": "html"
},
"fieldSchema": {
"name": "SimpleInvoiceAnalysis",
"description": "Simple invoice analysis with reference documents",
"fields": {
"TotalAmount": {
"type": "string",
"method": "extract",
"description": "Total amount on the invoice"
},
"ContractConsistency": {
"type": "string",
"method": "generate",
"description": "Analysis of consistency with reference contracts"
}
}
},
"warnings": [],
"status": "failed",
"processingLocation": "dataZone",
"mode": "pro",
"knowledgeSources": [
{
"kind": "reference",
"containerUrl": "https://....blob.core.windows.net/pro-reference-files?se=...,
"prefix": "ref_result_gen_1757607387_",
"fileListPath": "ref_result_gen_1757607387_sources.jsonl"
}
]
}

What's happening is that there's no much feedback from the api that could be used for debugging. Besides, upon removing the knowledgeSources property, the api works immediately.

[mirojs]

@juliagongms Thanks for commenting on the issues.

Adding documents as reference files is a way of specifying documents that you don't want to be processed repeatedly with each run of the analyzer. It is a way to save on costs. However, if the combined set of documents fits within the page and file size limits (for preview.2, that is 150 pages and 100MB, as discussed in https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/service-limits#document-and-text), you may see better quality and it is a valid solution. It is definitely encouraged to give both setups a try if you're interested in comparing the options.

To my understanding, by separating input and reference files, it avoids the effort of having to loop through all the files and hence the cost. Is that right? And by mentioning "the combined set of documents", does it mean the input file only case?

Could you also share the best way of creating the schema, as seen from the documentation, its doesn't seem to be very exploit on rules? Or am I asking the right question? Thanks.

This is a great question! We are also working on putting together more documentation that will be useful as references when creating your own schemas. The best practices we have shared at the moment are here: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/standard-pro-modes#pro-mode-known-limitations-and-best-practices. In general, creating fields that are specific and clear, with any elaboration or instructions/special rules or cases of concern, are encouraged. From the best practices, "Schemas should be designed with the highest level of specificity possible. For instance, instead of presenting a generalized list of inconsistencies, it's advisable to create distinct fields for each type of inconsistency, accompanied by detailed descriptions. Additionally, wherever feasible, references to specific sections of relevant documents that should be reviewed should be included."

It would be helpful, if at all possible, to document what each property is for or let's say basic concept explanation like items, properties, what they were built for and how flexible they could be. How many layers of sub fields can it go, what's the recommended maximum number, balance between complexity and flexibility. Sorry for being too detailed here.

Having said the above, all these could be explored at individual level, but the very rigorous nature of this api along with the mysterious 500 responses make the workload a bit heavier. Thanks for the answers, very helpful!

[chienyuanchang]

@mirojs thank you for sharing more details. The response should have error message like below if failed. This could provide more information if you could find it.

'{"id":"<request id>","status":"Failed","error":{"code":"InvalidRequest","message":"Invalid request.","innererror":{"code":"<error code>","message":"<error message>"}}}'

From the schema you're sharing, there may be some misunderstanding of prefix and fileListPath. If pro-reference-files is the container you're using, prefix would be the folder path under this container where you store the reference documents, their result.json, and the jsonl file. fileListPath would the jsonl file name like sources.jsonl. It will be like:

"prefix": "<folder name>",
"fileListPath": "sources.jsonl"

Since other part is working, maybe we can clarify this part and try first.

[mirojs]

@chienyuanchang Thanks for the update. I updated the information and it still failed without any error message received from the api. I then copy/paste and run the GitHub code using westus endpoint and got exactly the same result, failed but no error message.

The api will provide error message concerning payload data format and once this phase passed (payload accepted), there doesn't seem to be any real message that could be provided.

[chienyuanchang]

@mirojs I'm sorry that I forgot to mention a small tricky thing that please add '/' after the folder name(path). For example, if the folder path under your container is test20250915/ref_result_gen_1757607387. The variables will be

"prefix": "test20250915/ref_result_gen_1757607387/",
"fileListPath": "sources.jsonl"

If it still fails, is it possible for you to try our sample notebook for Pro mode?

[chienyuanchang]

@mirojs Hi! I would like to check if you could successfully run with the Pro mode or if you encountered any issues. Please let me know. Thank you!

[mirojs]

@chienyuanchang @juliagongms Sorry for getting back to you late because I was stuck with the api result repeatability issue. In order to get the best possible result as mentioned in the reply, I put all the files as input files, checked status following each step, tried async, yet still got fluctuating results but that looked different from hallucinations. And testing locally, it's quite repeatable, but the cloud deployment is a different story (WestUS). Any comments on this?

For the reference files processing, it still could not really work. The analyzer creation check could pass but not going further and without meaningful error message. Once the reference files were removed, however, the api chain would work again.

At this point, I believe prioritizing the repeatability issue would make more sense as there's no alternative now.

While working on the above issues, I did get further understanding of the schema structure and each property, which makes me wonder if multi-step reasoning could be further extended by employing more schema layers?

Thanks for your time!