Quick Start

This Quick-Start Guide will show you how to conduct your own audit of one of Moodle's AI-based Learning Analytics models using the plugin "Let(')s audit Learning Analytics" (LaLA).

⚠️ LaLA is still in the beta phase. It is not recommended to run the plugin on production servers, since collecting the evidence while creating a model version could temporarily lead to heavy load on the servers, impacting the experience for other users.

To use LaLA to conduct an audit, you can follow these auditing steps, which are each explained in detail later:

1. Make sure your audit environment fulfills all prerequisites.
2. Navigate to the plugin page.
3. Create a new model version of the model configuration you want to audit.
4. Download the evidence and proceed with your audit.

Prerequisites

With the Quick Start Script

You can use this bash script to quickly get a Moodle test instance with LaLA and sample data running.

You need:

• Linux or MacOS
• Docker Engine v24+
• Docker Compose v2+
• Git

Without the Quick Start Script

• Set up or have admin access to a Moodle server. You could, for example, use this Docker configuration to quickly set up a local Moodle server. Make sure that a Cron job executes adhoc tasks regularly.
• Install LaLA as described in the README.
• Have a Learning Analytics model configured. Per default, Moodle already contains a model configuration for a dropout prediction model. You can also configure additional models.
• Have some data in your Moodle database that can be used for auditing.
  💡 For anonymization and splitting of data into training and testing data, LaLA requires at least three sample entities, e.g. user enrolments in a course. LaLA does not make any assumptions about the meaningfulness and validity of the data. Therefore, which data and how much you need additionally depends on the model you want to audit and the claims you want to verify. For example, for the student dropout prediction model, you need a finished course that has several sections and a start and end date, with these dates being less than a year apart¹. Additionally, students need to be enrolled in that course, with the enrollment start and end date being set and approximately matching those of the course, and not being further than a year apart¹. Between the start and the end of a course, there needs to be student activity registered as events in the log store². The amount of events required is at least 10 times the number of students². You could obtain this kind of data by exporting a backup of a suitable finished course (including student activity) from a Moodle server in production, and importing it on your development server.
  💡 Valid test data sets for e.g. the dropout prediction model are the TWM dataset and the BKI dataset
  ⚠️ Note that data protection laws apply.

The plugin page

LaLA is an admin tool that concerns Learning Analytics. You find the link to the plugin page in the admin settings, under Analytics.

💡 LaLA installs a new role to the Moodle server: The role auf "auditor". As an admin, you can assign other users of the server to this role, to allow them to audit Learning Analytics models, while not receiving any additional privileges that teachers or admins might have. This way, LaLA enables third party audits of a Moodle Learning Analytics system.

If you have only the "auditor" role, you find the link on the front page.

💡 You can also reach the plugin page by appending the following path to your base URL: /admin/tool/lala/.

The plugin page lists all model configurations provided by the Learning Analytics system, including those configurations that have been deleted or overwritten since the installation and first access to the plugin page.

Creating a new model version

A model configuration itself is just that - the configuration. It is not a trained model and can not make any predictions. Creating a new model version of the configuration with LaLA automates all steps needed to gather the necessary evidence: Gathering data from the Moodle server, calculating ground truth ("targets") and features ("indicators"), creating a training and test split, training a model, getting predictions for the test split and collecting data related to the training and testing data (e.g. user data).

To create a new model version, hit "create new version" on the model configuration you want to audit. You can chose to create a new version with the standard settings for data collection from the model configuration or select the manual creation process where you can adjust the selected contexts or upload own model input data. Creating the version might take some time and runs in the background.

💡 When the model version creation finishes, the version creation is logged as an event in the log store used by your Moodle instance.

Download the evidence

After a model version has been created, LaLA provides the collected evidence for download. Download what you need. You can now use the evidence in your audit, for example by analyzing the evidence in a Jupyter notebook.

🛠️ In the next version of LaLA, example evidence and analysis will be provided.

---

<sup>1. Moodle (2023) Students at risk of dropping out. Available at: https://docs.moodle.org/402/en/Students_at_risk_of_dropping_out. ↩
2. Moodle (2021) Analytics FAQ. Available at: https://docs.moodle.org/402/en/Analytics_FAQ. ↩</sup>