Data and analysis underlying the research into The Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions

Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman and Willem-Paul Brinkman

Corresponding author: Siska Fitrianie, s.fitrianie@tudelft.nl
https://doi.org/10.4121/19758436

This document provides links to files underlying the data and analysis presented in the paper:
Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. The Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions. In ACM International Conference on Intelligent Virtual Agents (IVA '22), September, 2022, Faro, Portugal. ACM, New York, NY, USA. https://doi.org/10.1145/3514197.3549612

The ASA Questionnaire is an instrument for evaluating human interaction with an ASA, resulted from multi-year efforts invliving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide within the OSF work-group of Artificial Social Agent Evaluation Instrument (https://osf.io/6duf7/). It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013-2018. The paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that invlived 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models, and resulted in the selection of 90 items for inclusion of the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was select to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of agent profile.

The study is approved by the Human research Ethics Committee TU Delft date 18-12-2020 and registered at Open Science Framework https://doi.org/10.17605/OSF.IO/KZ8V4.

1. Pdf reader to read .pdf file
2. Text editor to read .csv file
3. Microsoft Excel 2003 or higher to read .xlsx file
4. R (v4.0.4) with factor analysis libraries from the package pscyh (v2.1.3) and lavaan (v0.6-8) to run R codes

Table of Content:

• Introduction
  • Previous and Current Work:

    • Technical report of the current work: ASAEvalInst-TR#08.pdf

• Method
  • Participants

    • Determining sample size:

      • Source creating simulated data: createSimulationData.R

      • Source running cfa on simulated data: cfaSimulationData.R

      • Output simulated data (n = 406): sim_406p_14agents.csv (original) and sim_406p_14agents_std.csv(standardized)

      • Output cfa result (n = 406): simCFA_406p_14agents_std.txt (based on the standardized simulated data)

    • Participants' attention check results: participants.csv

  • Material:

    • Videos and related document: OSF page - Study 5: Collecting Prototypical ASAs

  • Design and Procedure

    • Instruction for the participants: instruction.png

    • Initial questionnaire items (n = 131): 131_questionnaire_items.xlsx

  • Data Preparation and Analysis Plan
    • The Final Questionnaire Items

    • Standardization of the raw observed data (n = 532):

      • Participants' raw data (n = 532): result_all.csv (raw observed data)
        Note: see the technical report for the dataset description.

      • Raw data per agent (#agents = 14):

        • AIBO (n = 39): result_AIBO.csv

        • AMY (n = 39): result_AMY.csv

        • CHAPPIE (n = 38): result_CHAPPIE.csv

        • DEEPBLUE (n = 39): result_DEEPBLUE.csv

        • DOG (n = 39): result_DOG.csv

        • FURBY (n = 39): result_FURBY.csv

        • HAL 9000 (n = 37): result_HAL 9000.csv

        • iCAT (n = 36): result_iCAT.csv

        • MARCUS (TERMINATOR) (n = 36): result_MARCUS.csv

        • NAO (n = 36): result_NAO.csv

        • POPPY (n = 38): result_POPPY.csv

        • SARAH (n = 39): result_SARAH.csv

        • SIM SENSEI (n = 38): result_SIM SENSEI.csv

        • SIRI (n = 39): result_SIRI.csv

      • Source: dataPreparation.R
        Note: the script includes codes for retriving statistical description of the datasets

      • Output: result_all_pItem_std.csv (standardized observed data)

    • The Short Version of the Questionnaire
    • ASA Chart
• Result
  • The ASA Questionnaire Items
  • Convergent Validity Analysis

  • CFA of individual constructs (n = 532):

    • Input: result_all_pItem_std.csv (standardized observed data)

    • Source: initial_convergentAnalysis.R (Initial analysis codes) and final_convergentAnalysis.R (Final analysis codes)

    • Output: final_predictedLatent_convergentAnalysis.csv (Predicted latent scores of constructs/dimensions)
      Note: see the CFA results in the technical report Appendix Appendix F The CFA Results of Convergent Validity Analysis

  • Discriminant Validity Analysis

    • Initial EFA of predicted latent scores of constructs/dimensions (n = 532):

      • Input: final_predictedLatent_convergentAnalysis.csv (Predicted latent scores of constructs/dimensions)

      • Source: efa_initial_discriminantAnalysis.R (Initial analysis codes)

      • Output: see EFA results in the technical report Appendix G The EFA Result of Predicted Latent Scores

    • CFA of factorial models (n = 532):

      • Input: result_all_pItem_std.csv (standardized observed data)

      • Source: cfa_initial_discriminantAnalysis.R (Initial analysis codes) and cfa_final_discriminantAnalysis.R (Final analysis codes)

      • Output: final_predictedLatent_discriminantAnalysis.csv (Predicted latent scores of constructs/dimensions)
        Note: see the CFA results in the technical report Appendix H The CFA Results of Discriminant Validity Analysis

    • Final EFA of predicted latent scores of constructs/dimensions (n = 532):

      • Input: final_predictedLatent_discriminantAnalysis.csv (Predicted latent scores of constructs/dimensions)

      • Source: efa_final_discriminantAnalysis.R (Initial analysis codes)

      • Output: see the EFA result in the technical report Appendix H The CFA Results of Discriminant Validity Analysis, section CFA Results of Four EFA-based Models

    • Reliability analysis (n = 532):

      • Input: result_all_pItem_std.csv (standardized observed data)

      • Source: alpha_discriminantAnalysis.R

      • see Cronbach's alpha results in the technical report Appendix I CFA and Reliability Analysis Results of Individual

    • The ASA Questionnaire (#construct= 19, #item = 90):
      Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. Artificial Social Agent Questionnaire. https://doi.org/10.4121/19650846 4TU.ResearchData.

      Dataset 14 agents (n=532): ASAQ_ConstructValidity_p532.xlsx, see Tab: "LONG VERSION".

  • The Short Version of the Questionnaire

  • CFA of 24 individual constructs/dimensions (n = 532):

    • Input: result_all_pItem_std.csv (standardized observed data)

    • Source: cfa_individualConstructDimension.R

    • Output: see the CFA results in the technical report Appendix I CFA and Reliability Analysis Results of Individual and Appendix J Four Judges Decision on the Highest Loading Items

  • EFA of 24 representative items (n = 532):

    • Input: result_all_pItem_std.csv (standardized observed data)

    • Source: efa_shortQuestionnaire.R

    • Output: see the EFA result in the technical report Appendix K Correlation and EFA Result based on 24 Highest Loading Items

  • Absolute mean different between the long and short versions of the questionnaire (n = 532):

    • Input: result_all.csv (raw observed data)

    • Source: meanDifferent.R

    • Output: see the CFA results in the technical report Section The Short Version of the ASA Questionnaire Table 8

  • The Short version of the ASA Questionnaire (#item = 24):
    Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. Artificial Social Agent Questionnaire. https://doi.org/10.4121/19650846 4TU.ResearchData.

    • Dataset 14 agents (n=532): ASAQ_ConstructValidity_p532.xlsx, see Tab: "SHORT VERSION".

  • ASA Chart

    • The mean score of constructs or dimensions of 14 agents (n = 532):

      • Input: result_all.csv (raw observed data)

      • Source: 14agents_constructsMean.R

      • Output: constructsMean_14agents.csv

    • Generating ASA charts (#agent = 14):

      • Input: constructsMean_14agents.csv

      • Source: the ASA Chart Generator in Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. Artificial Social Agent Questionnaire. https://doi.org/10.4121/19650846 4TU.ResearchData.

      • Output: see the CFA results in the technical report Appendix L WEB-ASA of 14 ASAs used in the Study

• Discussion and Conclusion

  • The Short version of the ASA Questionnaire (#item = 24):
    Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. Artificial Social Agent Questionnaire. https://doi.org/10.4121/19650846 4TU.ResearchData.

  • Dataset 14 agents (n=532): ASAQ_ConstructValidity_p532.xlsx, for both long and short versions.