Test sets are used to group multiple tests together. That way, when running tests, you select a test set, and each test in the test set will be executed as part of the run.
- Open the Power CAT Copilot Studio Kit application (as seen in installation instructions)
- Navigate to Test Sets.
- Create a New Copilot Test Set record.
- Provide a Name.
- Save.
| Test Type | Description |
|---|---|
| Response Match | This is the simplest type of test, and it can be immediately evaluated. Compares the copilot response with the expected response. |
| Attachments (Adaptive Cards, etc.) | Compares the copilot attachments JSON response with the expected attachments JSON. Note: this is the full array of attachments |
| Topic Match | Only available when Dataverse enrichment is configured. When the Dataverse enrichment step completes, compares the expected topic name and the triggered topic name. |
| Generative Answers | Only available if AI Builder enrichment is configured. Uses a large language model to assess if the AI-generated answer is close to a sample answer or honors validation instructions. When Azure Application Insights enrichment is configured, negative tests, such as Moderation or No Search Results can also be tested. |
After you've created a Test Set, you can add Tests to it. From the Tests subgrid, select New Copilot Test
| Column Name | Required | Description |
|---|---|---|
| Name | Yes | Name of the test. This can be an internal reference ID, e.g., TST-001. |
| Copilot Test Set | Yes | Parent test set for the test |
| Test Type | Yes | One of the above test types |
| Send startConversation Event | If enabled, the startConversation start will be sent to the copilot so that it proactively starts the conversation, and the test utterance is only sent after. | |
| Expected Position of the Response Message | Don't set a value if unsure. This option allows the to capture a specific copilot response if it sends multiple messages. For example, if the copilot first says "Hello" and then "How can I help you?", and if you want to test the second message, set 2. |
|
| Test Utterance | Yes | The message that you want to send to the copilot as part of the test |
| Expected Response | Depends | Mandatory for a Response Match. Expected response from the copilot. For a Generative Answers test, this is where you can set a sample answer or your own validation instructions for the large language model. |
| External Variables JSON | JSON record for any external or contextual value you may want to pass of the Copilot as part of the test. E.g., { "Language": "fr" } |
|
| Expected Generative Answers Outcome | Depends | Mandatory for the Generative Answers type of test. Should be either Answered or Not Answered. When Azure Application Insights enrichment is enabled, you may choose Moderated or No Search Results. |
| Expected Topic Name | Depends | Mandatory for Topic Match type of test. Name of the topic that is expected to be triggered |
| Expected Attachments JSON | Depends | Mandatory for Attachments (Adaptive Cards, etc.) type of test. Full attachments JSON array that is expected from the copilot response. |
After creating a test set, you may use Excel to bulk create or update tests.
- From your test set record
- Switch the subgrid view from Tests to Export/Import View.
- In the subgrid commands, select Export Copilot Tests.
- Make sure you set the same Copilot Test Set name (must be unique for the import to succeed).
- Fill out the required columns.
- Navigate to Tests in the left-hand navigation bar.
- Select Import from Excel
- Select your file and select Next.
- Submit for import.
- Wait for the import to complete and succeed.
Note
Learn more about Excel import and export in Power Apps model-driven apps here: How to import data
Users may duplicate a full test set by selecting the Duplicate Test Set command.
