PowerBI LLM Benchmarking Tool

This repository contains resources for benchmarking Large Language Models (LLMs) in Power BI. The benchmarking tool enables data-driven evaluation across several parameters, supporting research and thesis work on conversational AI, including custom chatbot and Genie modules.

Overview

• LLM Benchmarking: Analyze and compare the response times and effectiveness of different LLM integrations in Power BI workflows.
• Custom Chatbot & Genie: Includes datasets and evaluation logic related to proprietary chatbot and Genie implementations from thesis research.
• Power BI Reports: LLM_Testing.pbix contains prebuilt dashboards and visualizations for experiment results.

Repository Structure

archive/ — Previous versions and supporting files. datasets/ — Sample datasets used for benchmarking tests. dax/ — DAX formulas applied in report metrics. images/ — Illustrative assets for reports and documentation. LLM_Testing.pbix — The main Power BI interactive report.

Getting Started

Requirements:

• Power BI Desktop
• Basic knowledge of DAX and Power BI navigation
• Basic knowledge of LLM Prompt Engineering topics like similarity and consistency

Usage Steps:

• Download and open LLM_Testing.pbix in Power BI Desktop.

Review included datasets and DAX scripts for benchmarking logic.

Customize the report to test LLM response time, accuracy, and context handling as described in the thesis chatbot and Genie chapters.