Enhancing Proof Assistant Error Messages with Hints: A User Study

DOI:10.4121/79e7c4eb-81dc-492a-9ac4-69f33166de8e.v1
The DOI displayed above is for this specific version of this dataset, which is currently the latest. Newer versions may be published in the future. For a link that will always point to the latest version, please use
DOI: 10.4121/79e7c4eb-81dc-492a-9ac4-69f33166de8e

Datacite citation style

Khakimova, Maria; Cockx, Jesper; Juhošová, Sára; Reinders, Jaro (2025): Enhancing Proof Assistant Error Messages with Hints: A User Study. Version 1. 4TU.ResearchData. dataset. https://doi.org/10.4121/79e7c4eb-81dc-492a-9ac4-69f33166de8e.v1
Other citation styles (APA, Harvard, MLA, Vancouver, Chicago, IEEE) available at Datacite

Dataset

This repository contains the user study data accompanying the master thesis by Maria Khakimova titled "Enhancing Proof Assistant Error Messages with Hints: A User Study". The goal of the research was to investigate the impacts of hint-based error message enhancements in Agda on novice programmers. To do this, we enhanced three error messages with hints, and conducted a user study.


In the user study, we asked participants to resolve errors in pre-written Agda code, and rate the helpfulness of the error message. We collected the following data:

  • code compilation status (success/fail),
  • compilation timestamps, and
  • responses to the "Did you find the error message helpful?" question (on a Likert scale).


This repository contains the programming questions created for the user study, with the accompanying error messages (both original and enhanced) in programming_exercises.zip. We also provide the (anonymised) collected data in JSON format in response-data.json.


For more details, please read the provided README.

History

  • 2025-06-18 first online, published, posted

Publisher

4TU.ResearchData

Format

text/markdown, application/json, application/zip

Organizations

TU Delft, Faculty of Electrical Engineering, Mathematics and Computer Science, Department of Software Technology

DATA

Files (3)