This repository contains resources from the paper:
Toward Efficient Vibe Coding: An LLM-Based Agent for Low-Code Software Development
Nikolaos Malamas, Emmanouil Tsardoulias, Konstantinos Panayiotou and Andreas L. Symeonidis. Submitted to Journal of Computer Languages, 2025.
The goal of this package is to provide all materials, datasets, and instructions required to check and reproduce the results presented in the paper, in line with open science and reproducibility principles.
The study evaluates the DSL Agent, a conversational interface designed to assist developers in creating DSL-based software applications, and compares it against a manual web interface workflow. The evaluation was conducted during a large-scale 2-day workshop involving more than 130 participants in both days.
The main objective of the evaluation was to assess whether the DSL Agent can improve usability and productivity in DSL-based application development compared to manual DSL-based development approaches.
We measured:
- Task completion time
- Usability using the System Usability Scale (SUS)
- User feedback from questionnaires
- Interaction data logs from both days
The workshop was organized as an open call among students from the School of Electrical and Computer Engineering at Aristotle University of Thessaloniki as well as students from other departments (eg. Computer Science).
It took place over two consecutive days:
- Day 1: Manual DSL development using the Locsys web interface.
- Day 2: DSL Agent-assisted DSL development.
- Participants: 182 on Day 1, 162 on Day 2, with 138 attending both days.
- Background: Most had programming experience but no prior exposure to DSLs, low-code platforms, or the domains of the evaluated DSLs (CodinTxt and dFlow).
- Task Design: Tasks were of comparable complexity but in different domains to avoid direct transfer learning between days.
| Path / File | Description |
|---|---|
| Open call description | Detailed description of the 2-day workshop, schedule, and participant instructions. |
| Workshop presentation | Presentation of theoretical, DSL and DSL Agent-related aspects made to participants (some parts are in Greek). |
/tasks_day1.md |
Task description for Day 1 (manual Locsys interface). |
/tasks_day2.md |
Task description for Day 2 (DSL Agent). |
/tasks_extra.md |
Extra task description for following days after workshop. |
/sus_questions.md |
List of SUS questions used in the study. |
/sus_responses_anonymized.xlsx |
Anonymized SUS questionnaire responses and the conducted analysis. |
/anonymized_data.zip |
Preprocessed and anonymized Locsys database with models and prompts generated by participants in both sessions (unzip locally to run python scripts). |
/dsl_analysis.py |
Python script for computing validation rates of DSL-based development. |
/assistant_analysis.py |
Python script for computing validation rates of LLM-based interactions. |
/error_analysis.py |
Python script for computing common and trivial errors as received from the validation operation. |