BACKGROUND: Secondary schools are an ideal setting to identify young people experiencing mental health difficulties such as anxiety or depression. However, current methods of identification rely on cumbersome paper-based assessments, which are lengthy and time-consuming to complete and resource-intensive for schools to manage. Artemis-A is a prototype web app that uses computerized adaptive testing technology to shorten the length of the assessment and provides schools with a simple and feasible solution for mental health assessment. OBJECTIVE: The objectives of this study are to coproduce the main components of the Artemis-A app with stakeholders to enhance the user interface, to carry out usability testing and finalize the interface design and functionality, and to explore the acceptability and feasibility of using Artemis-A in schools. METHODS: This study involved 2 iterative design feedback cycles-an initial stakeholder consultation to inform the app design and user testing. Using a user-centered design approach, qualitative data were collected through focus groups and interviews with secondary school pupils, parents, school staff, and mental health professionals (N=48). All transcripts were thematically analyzed. RESULTS: Initial stakeholder consultations provided feedback on preferences for the user interface design, school administration of the assessment, and outcome reporting. The findings informed the second iteration of the app design and development. The unmoderated usability assessment indicated that young people found the app easy to use and visually appealing. However, school staff suggested that additional features should be added to the school administration panel, which would provide them with more flexibility for data visualization. The analysis identified four themes relating to the implementation of the Artemis-A in schools, including the anticipated benefits and drawbacks of the app. Actionable suggestions for designing mental health assessment apps are also provided. CONCLUSIONS: Artemis-A is a potentially useful tool for secondary schools to assess the mental health of their pupils that requires minimal staff input and training. Future research will evaluate the feasibility and effectiveness of Artemis-A in a range of UK secondary schools.

Department of Kinanthropology Charles University Prague Czech Republic

Department of Psychiatry University of Cambridge Cambridge United Kingdom

Institute of Biomedical Research of Salamanca Psychiatry Unit Department of Medicine University of Salamanca Salamanca Spain

Peterborough National Health Services Foundation Trust Cambridgeshire United Kingdom

Zobrazit více v PubMed

Erskine HE, Moffitt TE, Copeland WE, Costello EJ, Ferrari AJ, Patton G, Degenhardt L, Vos T, Whiteford HA, Scott JG. A heavy burden on young minds: the global burden of mental and substance use disorders in children and youth. Psychol Med. 2014 Dec 23;45(7):1551–63. doi: 10.1017/s0033291714002888. PubMed DOI PMC

Mental health of children and young people in England, 2017 [PAS] NHS Digital. 2018. [2021-12-11]. https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017 .

Patel V, Flisher AJ, Hetrick S, McGorry P. Mental health of young people: a global public-health challenge. Lancet. 2007 Apr;369(9569):1302–13. doi: 10.1016/S0140-6736(07)60368-7. PubMed DOI

Guidance for commissioners of child and adolescent mental health services. Joint Commisioning Panel for Mental Health. 2013. [2021-12-11]. https://www.slideshare.net/jcpmh/jcpmh-camhsguide .

Transforming children and young people’s mental health provision: a green paper. Department of Health and Social Care and Department for Education. 2017. [2021-12-11]. https://www.gov.uk/government/consultations/transforming-children-and-young-peoples-mental-health-provision-a-green-paper .

Humphrey N, Wigelsworth M. Making the case for universal school-based mental health screening. Emot Behav Difficult. 2016 Jan 08;21(1):22–42. doi: 10.1080/13632752.2015.1120051. DOI

Nemeroff R, Levitt JM, Faul L, Wonpat-Borja A, Bufferd S, Setterberg S, Jensen PS. Establishing ongoing, early identification programs for mental health problems in our schools: a feasibility study. J Am Acad Child Adolesc Psychiatry. 2008 Mar;47(3):328–38. doi: 10.1097/chi.0b013e318160c5b1. PubMed DOI

Lyon AR, Maras MA, Pate CM, Igusa T, Vander Stoep A. Modeling the impact of school-based universal depression screening on additional service capacity needs: a system dynamics approach. Adm Policy Ment Health. 2016 Mar 20;43(2):168–88. doi: 10.1007/s10488-015-0628-y. 10.1007/s10488-015-0628-y PubMed DOI PMC

Sayal K, Owen V, White K, Merrell C, Tymms P, Taylor E. Impact of early school-based screening and intervention programs for ADHD on children's outcomes and access to services: follow-up of a school-based trial at age 10 years. Arch Pediatr Adolesc Med. 2010 May 03;164(5):462–9. doi: 10.1001/archpediatrics.2010.40.164/5/462 PubMed DOI

D'Souza CM, Forman SF, Austin SB. Follow-up evaluation of a high school eating disorders screening program: knowledge, awareness and self-referral. J Adolesc Health. 2005 Mar;36(3):208–13. doi: 10.1016/j.jadohealth.2004.01.014.S1054-139X(04)00414-8 PubMed DOI

Anderson JK, Ford T, Soneson E, Coon JT, Humphrey A, Rogers M, Moore D, Jones PB, Clarke E, Howarth E. A systematic review of effectiveness and cost-effectiveness of school-based identification of children and young people at risk of, or currently experiencing mental health difficulties. Psychol Med. 2018 Sep 13;49(1):9–19. doi: 10.1017/s0033291718002490. PubMed DOI

Chafouleas SM, Kilgus SP, Wallach N. Ethical dilemmas in school-based behavioral screening. Assess Effect Interv. 2010 Aug 31;35(4):245–52. doi: 10.1177/1534508410379002. DOI

Bender TW, Fitzpatrick S, Hartmann M, Hames J, Bodell L, Selby EA, Joiner TE. Does it hurt to ask? An analysis of iatrogenic risk during suicide risk assessment. Neurol Psychiatry Brain Res. 2019 Sep;33:73–81. doi: 10.1016/j.npbr.2019.07.005. DOI

Kuiper N, Goldston D, Garraza L, Walrath C, Gould M, McKeon R. Examining the unanticipated adverse consequences of youth suicide prevention strategies: a literature review with recommendations for prevention programs. Suicide Life Threat Behav. 2019 Aug 12;49(4):952–65. doi: 10.1111/sltb.12492. PubMed DOI

Levitt JM, Saka N, Romanelli LH, Hoagwood K. Early identification of mental health problems in schools: the status of instrumentation. J Sch Psychol. 2007 Apr;45(2):163–91. doi: 10.1016/j.jsp.2006.11.005. DOI

Hallfors D, Brodish PH, Khatapoush S, Sanchez V, Cho H, Steckler A. Feasibility of screening adolescents for suicide risk in “Real-World” high school settings. Am J Public Health. 2006 Feb;96(2):282–7. doi: 10.2105/ajph.2004.057281. PubMed DOI PMC

Soneson E, Howarth E, Ford T, Humphrey A, Jones PB, Coon JT, Rogers M, Anderson JK. Feasibility of school-based identification of children and adolescents experiencing, or at-risk of developing, mental health difficulties: a systematic review. Prev Sci. 2020 Jul 15;21(5):581–603. doi: 10.1007/s11121-020-01095-6. 10.1007/s11121-020-01095-6 PubMed DOI PMC

Spratt J, Shucksmith J, Philip K, Watson C. 'Part of who we are as a school should include responsibility for well-being': links between the school environment, mental health and behaviour. Pastoral Care Edu. 2006 Sep;24(3):14–21. doi: 10.1111/j.1468-0122.2006.00374.x. DOI

Pretorius C, Chambers D, Cowan B, Coyle D. Young people seeking help online for mental health: cross-sectional survey study. JMIR Ment Health. 2019 Aug 26;6(8):e13524. doi: 10.2196/13524. v6i8e13524 PubMed DOI PMC

Rickwood D, Webb M, Kennedy V, Telford N. Who are the young people choosing web-based mental health support? Findings from the implementation of Australia's National Web-based Youth Mental Health Service, eheadspace. JMIR Ment Health. 2016 Aug 25;3(3):e40. doi: 10.2196/mental.5988. v3i3e40 PubMed DOI PMC

Sweeney GM, Donovan CL, March S, Forbes Y. Logging into therapy: adolescent perceptions of online therapies for mental health problems. Internet Interv. 2019 Mar;15:93–9. doi: 10.1016/j.invent.2016.12.001. S2214-7829(15)30018-X PubMed DOI PMC

Wiljer D, Shi J, Lo B, Sanches M, Hollenberg E, Johnson A, Abi-Jaoudé A, Chaim G, Cleverley K, Henderson J, Isaranuwatchai W, Levinson A, Robb J, Wong HW, Voineskos A. Effects of a mobile and web app (Thought Spot) on mental health help-seeking among college and university students: randomized controlled trial. J Med Internet Res. 2020 Oct 30;22(10):e20790. doi: 10.2196/20790. v22i10e20790 PubMed DOI PMC

Wong HW, Lo B, Shi J, Hollenberg E, Abi-Jaoude A, Johnson A, Chaim G, Cleverley K, Henderson J, Levinson A, Robb J, Voineskos A, Wiljer D. Postsecondary student engagement with a mental health app and online platform (Thought Spot): qualitative study of user experience. JMIR Ment Health. 2021 Apr 02;8(4):e23447. doi: 10.2196/23447. v8i4e23447 PubMed DOI PMC

Liverpool S, Mota CP, Sales CM, Čuš A, Carletto S, Hancheva C, Sousa S, Cerón SC, Moreno-Peral P, Pietrabissa G, Moltrecht B, Ulberg R, Ferreira N, Edbrooke-Childs J. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res. 2020 Jun 23;22(6):e16317. doi: 10.2196/16317. v22i6e16317 PubMed DOI PMC

Kenny R, Dooley B, Fitzgerald A. Developing mental health mobile apps: exploring adolescents' perspectives. Health Inform J. 2016 Jun;22(2):265–75. doi: 10.1177/1460458214555041. 1460458214555041 PubMed DOI

Gibbons RD, Weiss DJ, Frank E, Kupfer D. Computerized adaptive diagnosis and testing of mental health disorders. Annu Rev Clin Psychol. 2016 Mar 28;12(1):83–104. doi: 10.1146/annurev-clinpsy-021815-093634. PubMed DOI

Gibbons RD, Weiss DJ, Pilkonis PA, Frank E, Moore T, Kim JB, Kupfer DJ. Development of the CAT-ANX: a computerized adaptive test for anxiety. Am J Psychiatry. 2014 Feb;171(2):187–94. doi: 10.1176/appi.ajp.2013.13020178. 1725889 PubMed DOI PMC

Stochl J, Ford T, Perez J, Jones PB. Modernising measurement in psychiatry: item banks and computerised adaptive testing. Lancet Psychiatry. 2021 May;8(5):354–6. doi: 10.1016/s2215-0366(21)00041-9. PubMed DOI

Loe BS, Stillwell D, Gibbons C. Computerized adaptive testing provides reliable and efficient depression measurement using the CES-D Scale. J Med Internet Res. 2017 Sep 20;19(9):e302. doi: 10.2196/jmir.7453. v19i9e302 PubMed DOI PMC

Theunissen MH, de Wolff MS, Deurloo JA, Vogels AG, Reijneveld SA. Computerized adaptive testing to screen children for emotional and behavioral problems by preventive child healthcare. BMC Pediatr. 2020 Mar 12;20(1):119. doi: 10.1186/s12887-020-2018-1. 10.1186/s12887-020-2018-1 PubMed DOI PMC

Gardner W, Shear K, Kelleher KJ, Pajer KA, Mammen O, Buysse D, Frank E. Computerized adaptive measurement of depression: a simulation study. BMC Psychiatry. 2004 May 06;4(1):1–11. doi: 10.1186/1471-244x-4-13. PubMed DOI PMC

Stochl J. Simulation of artemis-a efficiency. OSF. 2020. [2021-07-12]. https://osf.io/6e9wh .

STudent Experiences during the Pandemic: STEP towards mental-health and well-being. Cambridge University. 2021. [2021-12-11]. https://www.cam.ac.uk/coronavirus/students/STEP-mental-health-and-wellbeing-study .

Turner N, Joinson C, Peters TJ, Wiles N, Lewis G. Validity of the short mood and feelings questionnaire in late adolescence. Psychol Assess. 2014 Sep;26(3):752–62. doi: 10.1037/a0036572.2014-14378-001 PubMed DOI

Reynolds CR. Concurrent validity of what I think and feel: the revised children's manifest anxiety scale. J Consult Clin Psychol. 1980 Dec;48(6):774–5. doi: 10.1037/0022-006x.48.6.774. PubMed DOI

Taylor S. Assessment of obsessions and compulsions: reliability, validity, and sensitivity to treatment effects. Clin Psychol Rev. 1995 Jan;15(4):261–96. doi: 10.1016/0272-7358(95)00015-h. DOI

Rosenberg M. Society and the Adolescent Self-Image. Princeton, New Jersey: Princeton university press; 2015.

Clarke A, Friede T, Putz R, Ashdown J, Martin S, Blake A, Adi Y, Parkinson J, Flynn P, Platt S, Stewart-Brown S. Warwick-Edinburgh Mental Well-being Scale (WEMWBS): validated for teenage school students in England and Scotland. A mixed methods assessment. BMC Public Health. 2011 Jun 21;11(1):487. doi: 10.1186/1471-2458-11-487. 1471-2458-11-487 PubMed DOI PMC

Hodgekins J, Coker S, Freeman D, Ray-Glover K, Bebbington P, Garety P, Kuipers E, Dunn G, Fowler D. Assessing levels of subthreshold psychotic symptoms in the recovery phase: the Schizotypal Symptoms Inventory (SSI) J Experiment Psychopathol. 2012 Sep 23;3(4):582–93. doi: 10.5127/jep.021211. DOI

de Beurs D, van Bruinessen I, Noordman J, Friele R, van Dulmen S. Active involvement of end users when developing web-based mental health interventions. Front Psychiatry. 2017 May 03;8:72. doi: 10.3389/fpsyt.2017.00072. doi: 10.3389/fpsyt.2017.00072. PubMed DOI PMC

Still B, Crane K. Fundamentals of User-centered Design: A Practical Approach. Boca Raton, Florida, United States: CRC Press; 2017.

Nielsen J, Landauer T. A mathematical model of the finding of usability problems. Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems; INTERCHI93: Conference on Human Factors in Computing; April 24 - 29, 1993; Amsterdam The Netherlands. 1993. pp. 206–13. DOI

Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors. 2016 Nov 23;34(4):457–68. doi: 10.1177/001872089203400407. DOI

Nielsen J. How many test users in a usability study? Nielsen Norman Group. 2012. [2021-12-11]. http://www.nngroup.com .

Artemis-A. [2021-12-11]. https://www.artemis-a.org/

Huberman AM, Miles MB. The Qualitative Researcher's Companion. Thousand Oaks, CA: SAGE Publications; 2002.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004 Feb;37(1):56–76. doi: 10.1016/j.jbi.2004.01.003. S1532046404000206 PubMed DOI

Hetrick SE, Robinson J, Burge E, Blandon R, Mobilio B, Rice SM, Simmons MB, Alvarez-Jimenez M, Goodrich S, Davey CG. Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-harm. JMIR Ment Health. 2018 Jan 23;5(1):e9. doi: 10.2196/mental.9041. v5i1e9 PubMed DOI PMC

Gardner W, Kelleher KJ, Pajer KA. Multidimensional adaptive testing for mental health problems in primary care. Med Care. 2002 Sep;40(9):812–23. doi: 10.1097/00005650-200209000-00010. PubMed DOI

De Beurs DP, de Vries AL, de Groot MH, de Keijser J, Kerkhof AJ. Applying computer adaptive testing to optimize online assessment of suicidal behavior: a simulation study. J Med Internet Res. 2014 Sep 11;16(9):e207. doi: 10.2196/jmir.3511. v16i9e207 PubMed DOI PMC

Gulliksen J, Göransson B, Boivie I, Blomkvist S, Persson J, Cajander Å. Key principles for user-centred systems design. Behav Inform Technol. 2003 Nov;22(6):397–409. doi: 10.1080/01449290310001624329. DOI

Ospina-Pinillos L, Davenport TA, Navarro-Mancilla AA, Cheng VW, Alarcón AC, Rangel AM, Rueda-Jaimes GE, Gomez-Restrepo C, Hickie IB. Involving end users in adapting a Spanish version of a web-based mental health clinic for young people in Colombia: exploratory study using participatory design methodologies. JMIR Ment Health. 2020 Feb 08;7(2):e15914. doi: 10.2196/15914. v7i2e15914 PubMed DOI PMC

Ospina-Pinillos L, Davenport T, Diaz AM, Navarro-Mancilla A, Scott EM, Hickie IB. Using participatory design methodologies to co-design and culturally adapt the Spanish version of the mental health eClinic: qualitative study. J Med Internet Res. 2019 Aug 02;21(8):e14127. doi: 10.2196/14127. v21i8e14127 PubMed DOI PMC

Callan JA, Dunbar Jacob J, Siegle GJ, Dey A, Thase ME, Dabbs AD, Kazantzis N, Rotondi A, Tamres L, Van Slyke A, Sereika S. CBT MobileWork©: User-centered development and testing of a mobile mental health application for depression. Cogn Ther Res. 2020 Sep 25;45(2):287–302. doi: 10.1007/s10608-020-10159-4. DOI

Cheng VW, Davenport TA, Johnson D, Vella K, Mitchell J, Hickie IB. An app that incorporates gamification, mini-games, and social connection to improve men's mental health and well-being (MindMax): participatory design process. JMIR Ment Health. 2018 Nov 19;5(4):e11068. doi: 10.2196/11068. v5i4e11068 PubMed DOI PMC

Dekker MR, Williams AD. The use of user-centered participatory design in serious games for anxiety and depression. Games Health J. 2017 Dec;6(6):327–33. doi: 10.1089/g4h.2017.0058. PubMed DOI

Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health. 2016 Mar 01;3(1):e7. doi: 10.2196/mental.4984. v3i1e7 PubMed DOI PMC

Nielsen J. 10 usability heuristics for user interface design. Nielsen Norman Group. 1994. [2021-03-19]. https://www.nngroup.com/articles/ten-usability-heuristics/

Holzinger A, Errath M. Mobile computer web-application design in medicine: some research based guidelines. Univ Access Inf Soc. 2007 Apr 20;6(1):31–41. doi: 10.1007/s10209-007-0074-z. DOI

Lidwell W, Holden K, Butler J. Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design. Beverly, MA: Rockport Publishers; 2010. pp. 1–272.

Mills CB, Weldon LJ. Reading text from computer screens. ACM Comput Surv. 1987 Dec;19(4):329–57. doi: 10.1145/45075.46162. DOI

Sherwin K. Usability heuristic 5: error prevention. Nielsen Norman Group. [2021-03-19]. https://www.nngroup.com/videos/usability-heuristic-error-prevention/

Husky MM, Sheridan M, McGuire L, Olfson M. Mental health screening and follow-up care in public high schools. J Am Acad Child Adolesc Psychiatry. 2011 Sep;50(9):881–91. doi: 10.1016/j.jaac.2011.05.013.S0890-8567(11)00446-1 PubMed DOI

Mitchell S, Gryczynski J, Gonzales A, Moseley A, Peterson T, O'Grady KE, Schwartz RP. Screening, brief intervention, and referral to treatment (SBIRT) for substance use in a school-based program: services and outcomes. Am J Addict. 2012 Nov;21 Suppl 1:5–13. doi: 10.1111/j.1521-0391.2012.00299.x. PubMed DOI PMC

Polihronis C, Cloutier P, Kaur J, Skinner R, Cappelli M. What's the harm in asking? A systematic review and meta-analysis on the risks of asking about suicide-related behaviors and self-harm with quality appraisal. Arch Suicide Res. 2020 Jul 25;:1–23. doi: 10.1080/13811118.2020.1793857. PubMed DOI

Wasserman E, Migdal R. Professional development- differences in teachers' attitudes between online and traditional training courses. Online Learn J. 2019 Mar 01;23(1) doi: 10.24059/olj.v23i1.1299. DOI

Aschbrenner KA, Naslund JA, Tomlinson EF, Kinney A, Pratt SI, Brunette MF. Adolescents' use of digital technologies and preferences for mobile health coaching in public mental health settings. Front Public Health. 2019 Jul 2;7:178. doi: 10.3389/fpubh.2019.00178. doi: 10.3389/fpubh.2019.00178. PubMed DOI PMC

Wozney L, McGrath PJ, Gehring ND, Bennett K, Huguet A, Hartling L, Dyson MP, Soleimani A, Newton AS. eMental healthcare technologies for anxiety and depression in childhood and adolescence: systematic review of studies reporting implementation outcomes. JMIR Ment Health. 2018 Jun 26;5(2):e48. doi: 10.2196/mental.9655. v5i2e48 PubMed DOI PMC

A qualitative study exploring the feasibility and acceptability of computerised adaptive testing to assess and monitor children and young people's mental health in primary care settings in the UK