Sofia | Speech-Enabled Conversational Agents to Support the Self-Report of Mental Health and Wellbeing

---

---

Mental health self-reports are traditionally collected through in-person interviews and paper-based diary logs and health questionnaires. While these measures are considered effective and have a long history in the assessment and treatment of mental health conditions, they are increasingly being adopted into online tools, such as mobile and web apps, for their efficiency and accessibility. Although these systems have become our primary mode of interaction with the technology and can efficiently collect health self-reports in textual and visual forms, these are but one medium of interaction, which can limit users’ self-expression capacity and place a burden on users that could adversely affect their self-reporting experiences.

In contrast, adopting speech as the primary mode of interaction, conversational agents (CA) have the potential to improve self-reporting experiences by engaging users in a more natural human-to-human-like conversational interaction. In light of this potential, my PhD research investigated the feasibility of the state-of-the-art CA to support the self-report of mental health and wellbeing by assessing a state-of-the-art CA’s ability to capture the self-reports accurately and users’ perceived usability of the system in the real-world setting.

To assess the accuracy of the state-of-the-art CA in capturing users’ wellbeing self-reports, we developed a questionnaire-driven conversational agent with state-based and free-form conversational designs and evaluated the accuracy of the self-reports by contrasting 59 participants’ responses via each conversational design compared to the paper-based responses to the same questionnaire in a lab setting. In order to generate an understanding of the users’ perception of the system’s usability in the real-world setting, we developed a second version of the CA that enabled daily self-report of wellbeing by asking three open-ended questions and an additional questionnaire fortnightly. We conducted a four-week mixed-method field deployment study involving 20 people living with affective disorders who used the system daily to self-report about their wellbeing.

Based on the findings of these two empirical studies, the thesis contributes initial evidence of the CA’s feasibility for the self-report of mental health and wellbeing, suggests several considerations for the future design of CAs to support such practices, and highlights directions for future research.

---

Publications

Raju Maharjan, Kevin Doherty, Darius Adam Rohani, Per Bækgaard, and Jakob E. Bardram. 2022. What is the Difference? Investigating the Self-Report of Wellbeing via Conversational Agent and Web App. IEEE Pervasive Computing, Special Issue on Mental Health, Mood, and Emotion. DIO: https://doi.org/ 10.1109/MPRV.2022.3147374

Raju Maharjan, Kevin Doherty, Darius Adam Rohani, Per Bækgaard, and Jakob E. Bardram. What’s Up with these Conversational Health Agents? From Users’ Critiques to Implications for Design. Frontiers in Digital Health (2022).  https://doi.org/10.3389/fdgth.2022.840232

Raju Maharjan, Kevin Doherty, Darius Adam Rohani, Per Bækgaard, and Jakob E. Bardram. 2022. Experiences of a Speech-Enabled Conversational Agent for the Self-Report of Wellbeing Among People Living with Affective Disorders: An In-The-Wild Study. ACM Trans. Interact. Intell. Syst. DOI:https://doi.org/10.1145/3484508

Raju Maharjan, Darius Adam Rohani, Per Bækgaard, Jakob Bardram, and Kevin Doherty. 2021. Can we talk? Design Implications for the Questionnaire-Driven Self-Report of Health and Wellbeing via Conversational Agent. In CUI 2021 – 3rd Conference on Conversational User Interfaces (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 5, 1–11. DOI: https://doi.org/10.1145/3469595.3469600