ID 2518: Automated assessment of transcultural skills in therapeutic responses using LLMs

Bachelor’s / Master’s Thesis / Research Internship / Project

Background: Transcultural competence is a crucial factor for the success of psychotherapy. It describes the ability of therapists to work effectively with clients from diverse cultural backgrounds. Currently, this competence is often measured through self-reports, which are susceptible to biases such as social desirability. More objective methods, such as the standardized Facilitative Interpersonal Skills procedure, are promising but also time-consuming and resource-intensive.

The aim of this project is to investigate whether an LLM can reliably assess the transcultural competence of therapists based on transcripts of therapeutic responses. This approach should hopefully automate the assessment of this competence, thereby making it more accessible.

Tasks:

Depending on the scope of the project/thesis, up to three approaches should be compared:

1. Combination of Linguistic Inquiry and Word Count (LIWC) and LLM: Use of linguistic markers (e.g., the use of certain pronouns) identified by LIWC to train an LLM that uses LIWC features to determine a predicted competence score. The model is designed to learn the weighting of these linguistic markers in order to predict a competent response.
2. Direct prompting of an LLM: Direct application of an LLM that is given specific skill criteria as a prompt. The LLM is designed to evaluate the therapeutic response or competence based on these criteria.
3. Human ratings (gold standard): Assessment by human raters/experts.

The results of the two approaches (LLM with LIWC data vs. direct LLM prompting) are to be compared with human ratings (e.g., using correlation or interrater reliability).

Requirements:

• Interest in machine learning
• Very good knowledge of Python
• Good basic knowledge of statistics/evaluation (e.g., correlations)
• Interest in the interface between data science, AI, and clinical psychology

Supervisors

Dr.-Ing. Robert Richer Postdoc and Group Leader

Christina Nuhn, M.Sc. Chair of Clinical Psychology and Behavioral Health Therapy