Machine learning based improvement of the diagnostic accuracy of an online symptom checker

Master's Thesis

Machine learning based improvement of the diagnostic accuracy of an online symptom checker


Dr.-Ing. Felix Kluge, Prof. Dr. Björn Eskofier 

09/2020 – 02/2021


Inflammatory rheumatic diseases (IRD) often go along with unspecific symptoms that make the
correct diagnosis difficult [1]. The European League Against Rheumatism (EULAR) passed clear
recommendations that patients should be seen during 6 weeks after symptom onset [2] as early
treatment start is likely to improve the patient outcome significantly [3]. Various strategies have
been identified [1, 4] to implement these recommendations, however diagnostic delays are high
[5, 6]. There are different reasons for this delay, one of them is that the time until a patient is
seeking medical advice is too long, due to lack of awareness of the disease [5].
Symptom checkers (SCs) foster patient-empowerment and could be used to optimise triage decisions
in a personalised manner (the right appointment, for the right patient, at the right time). 90%
of patients with IRDs regularly use smartphones, 65% believe that using medical apps could be
beneficial to improve their health [7] and 4% consulted the internet to check their symptoms [8] previous
to a rheumatologist appointment. SCs like artificial-intelligence-driven Ada (
have been used to complete more than 15 million health assessments in 130 countries [9] and a
recent publication suggests that the diagnostic delay could significantly be reduced even for rare
diseases when using digital SCs [10]. Another SC is Rheport ( which has been
specifically designed for IRD identification. Current research (data not published) shows that sensitivity
and specificity of Rheport regarding IRDs is limited, however the majority of patients
would recommend the use of SC and their usability was well perceived.
The current Rheport algorithm is rule-based and builds on subjective expert knowledge. Machine
learning (ML), a way of mathematically modelling patterns in available training data and
making predictions on new data could help to improve the algorithm. There are a number of
general (not specifically designed for IRD) SCs that use ML techniques. These SCs include buoy
( and Isabel (, among others. SCs for IRD, however,
are mostly simple questionnaires as they can be found on Rheuma-Check
( or Arthritis ( It has recently
been shown that the use of ML can outperform conventional techniques, which has not
been shown for IRD yet [11, 12].
For a classification problem like distinguishing between patients with and without IRD, a number
of ML algorithms could be used including Naive Bayes Classifier, Decision Trees, Support Vector
Machines or Neural Networks in order to improve sensitivity and specificity of Rheport. Based
on structure, quality and quantity of data, the best algorithm for performing the required task
needs to be determined. The results will be compared to previously determined performances.
As Rheport is currently being used by several German rheumatologists, the results can directly
improve the quality of life of patients with IRD. The aim of this project is to improve Rheport’s
algorithm using ML based on self-reported multicentric real-world data.




[1] Benesova K, Lorenz HM, Lion V, Voigt A, Krause A, Sander O, et al.: Früh- und Screeningsprechstunden:
Ein notwendigerWeg zur besseren Frühversorgung in der internistischen
Rheumatologie? Zeitschrift für Rheumatologie. 2019;78(8):722-42.
[2] Combe, Bernard and Landewe, Robert and Daien, Claire I and Hua, Charlotte and Aletaha,
Daniel and Álvaro-Gracia et al.: 2016 update of the EULAR recommendations for the
management of early arthritis. Annals of the Rheumatic Diseases. 2017;76(6):948.
[3] Quinn MA, Emery P.: Window of opportunity in early rheumatoid arthritis: possibility of
altering the disease process with early intervention. Clin Exp Rheumatol. 2003;21(0392-
856X (Print)):154-7.
[4] Villeneuve E, Nam Jl Fau – Bell MJ, Bell Mj Fau – Deighton CM, Deighton Cm Fau –
Felson DT, Felson Dt Fau – Hazes JM, Hazes Jm Fau – McInnes IB, et al.: A systematic
literature review of strategies promoting early referral and reducing delays in the diagnosis
and management of inflammatory arthritis. Annals of the rheumatic diseases. 2012;72(1468-
2060 (Electronic)):13-22.
[5] Raza K, Stack R, Kumar K, Filer A, Detert J, Bastian H, et al.: Delays in assessment
of patients with rheumatoid arthritis: variations across Europe. Annals of the rheumatic
diseases. 2011;70(10):1822.
[6] Stack RJ, Nightingale P, Jinks C, Shaw K, Herron-Marx S, Horne R, et al.: Delays between
the onset of symptoms and first rheumatology consultation in patients with rheumatoid
arthritis in the UK: an observational study. BMJ Open. 2019;9(3):e024361.
[7] Knitza J, Raab C, Lambrecht A, Simon D, Hagen M, Bayat S, et al.: The Urge for Mobile
Apps in Rheumatology – a German Patient Perspective [abstract]. Arthritis Rheumatol.
2019;71 (suppl 10).
[8] Powley L, McIlroy G, Simons G, Raza K.: Are online symptoms checkers useful for patients
with inflammatory arthritis? BMC musculoskeletal disorders. 2016;17(1):362.
[9] Butcher M. Ada: Health built an AI-driven startup by moving slowly and not breaking
things Techcrunch; 2020 [Available from:
[10] Ronicke S, Hirsch MC, Turk E, Larionov K, Tientcheu D, Wagner AD.: Can a decision
support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada
DX in a retrospective study. Orphanet J Rare Dis. 2019;14(1):69.
[11] Kao H, Tang K, Chang E.: Context-Aware Symptom Checking for Disease Diagnosis Using
Hierarchical Reinforcement Learning. AAAI Conference on Artificial Intelligence. 2018.
[12] Akrout M, Farahmand A, Jarmain T, Abid L.: Improving Skin Condition Classification
with a Visual Symptom Checker Trained Using Reinforcement Learning. Medical Image
Computing and Computer Assisted Intervention. 2019.