BACKGROUND: The development of digital health solutions for current health care settings requires an understanding of the complexities of the health care system, organizational setting, and stakeholder groups and of the underlying interplay between stakeholders and the technology. The digital health solution was founded on the basis of an information and communication technology platform and point-of-care devices enabling home-based monitoring of disease progression and treatment outcome for patients with rheumatoid arthritis (RA). OBJECTIVE: The aim of this paper is to describe and discuss the applicability of an iterative evaluation process in guiding the development of a digital health solution as a technical and organizational entity in three different health care systems. METHODS: The formative evaluation comprised the methodologies of contextual understanding, participatory design, and feasibility studies and included patients, healthcare professionals, and hardware and software developers. In total, the evaluation involved 45 patients and 25 health care professionals at 3 clinical sites in Europe. RESULTS: The formative evaluation served as ongoing and relevant input to the development process of the digital health solution. Through initial field studies key stakeholder groups were identified and knowledge obtained about the different health care systems, the professional competencies involved in routine RA treatment, the clinics' working procedures, and the use of communication technologies. A theory-based stakeholder evaluation achieved a multifaceted picture of the ideas and assumptions held by stakeholder groups at the three clinical sites, which also represented the diversity of three different language zones and cultures. Experiences and suggestions from the patients and health care professionals were sought through participatory design processes and real-life testing and actively used for adjusting the visual, conceptual, and practical design of the solution. The learnings captured through these activities aided in forming the solution and in developing a common understanding of the overall vision and aim of this solution. During this process, the 3 participating sites learned from each other's feed-back with the ensuing multicultural inspiration. Moreover, these efforts also enabled the consortium to identify a 'tipping point' during a pilot study, revealing serious challenges and a need for further development of the solution. We achieved valuable learning during the evaluation activities, and the remaining challenges have been clarified more extensively than a single-site development would have discovered. The further obstacles have been defined as has the need to resolve these before designing and conducting a real-life clinical test to assess the outcome from a digital health solution for RA treatment. CONCLUSIONS: A formative evaluation process with ongoing involvement of stakeholder groups from 3 different cultures and countries have helped to inform and influence the development of a novel digital health solution, and provided constructive input and feedback enabling the consortium to control the development process.

Botnar Research Centre Nuffield Department of Orthopaedics Rheumatology and Musculoskeletal Sciences University of Oxford Oxford United Kingdom

Centre for Innovative Medical Technology Odense University Hospital Odense Denmark

Charles University 1st Faculty of Medicine Department of Rheumatology Prague Czech Republic

Institute of Rheumatology Prague Czech Republic

Research Unit of User Perspectives Department of Public Health University of Southern Denmark Odense Denmark

The Parker Institute Copenhagen University Hospital Bispebjerg and Frederiksberg Frederiksberg Denmark

Zobrazit více v PubMed

World Health Organization. 2010. Opportunities and developments in Member States http://www.who.int/goe/publications/goe_telemedicine_2010.pdf .

Dinesen B, Nonnecke B, Lindeman D, Toft E, Kidholm K, Jethwani K, Young HM, Spindler H, Oestergaard CU, Southard JA, Gutierrez M, Anderson N, Albert NM, Han JJ, Nesbitt T. Personalized Telehealth in the Future: A Global Research Agenda. J Med Internet Res. 2016 Mar 01;18(3):e53. doi: 10.2196/jmir.5257. v18i3e53 PubMed DOI PMC

The Advisory Board. 2014. Critical Disruptions: The 5 Forces Shaping Health Care's Future https://www.advisory.com/international/research/global-forum-for-health-care-innovators/research-briefs/2014/critical-disruptions-the-five-forces-shaping-health-cares-future .

KORA. 2017. Fem megatrends der udfordrer fremtidens sundhedsvæsen https://www.kora.dk/media/6844202/11325_fem-megatrends-der-udfordrer-fremtidens-sundhedsvaesen.pdf .

Mathijssen E, Vriezekolk J, Eijsbouts A, van DHF, van DBB. Support needs for medication use and the suitability of eHealth technologies to address these needs: a focus group study of older patients with rheumatoid arthritis. Patient Prefer Adherence. 2018;12:349–358. doi: 10.2147/PPA.S152759. doi: 10.2147/PPA.S152759.ppa-12-349 PubMed DOI PMC

Vos T, Flaxman AD, Naghavi M, Lozano R, Michaud C, Ezzati M, Shibuya K, Salomon JA, Abdalla S, Aboyans V, Abraham J, Ackerman I, Aggarwal R, Ahn SY, Ali MK, Alvarado M, Anderson HR, Anderson LM, Andrews KG, Atkinson C, Baddour LM, Bahalim AN, Barker-Collo S, Barrero LH, Bartels DH, Basáñez Maria-Gloria, Baxter A, Bell ML, Benjamin EJ, Bennett D, Bernabé Eduardo, Bhalla K, Bhandari B, Bikbov B, Bin AA, Birbeck G, Black JA, Blencowe H, Blore JD, Blyth F, Bolliger I, Bonaventure A, Boufous S, Bourne R, Boussinesq M, Braithwaite T, Brayne C, Bridgett L, Brooker S, Brooks P, Brugha TS, Bryan-Hancock C, Bucello C, Buchbinder R, Buckle G, Budke CM, Burch M, Burney P, Burstein R, Calabria B, Campbell B, Canter CE, Carabin H, Carapetis J, Carmona L, Cella C, Charlson F, Chen H, Cheng AT, Chou D, Chugh SS, Coffeng LE, Colan SD, Colquhoun S, Colson KE, Condon J, Connor MD, Cooper LT, Corriere M, Cortinovis M, de VKC, Couser W, Cowie BC, Criqui MH, Cross M, Dabhadkar KC, Dahiya M, Dahodwala N, Damsere-Derry J, Danaei G, Davis A, De LD, Degenhardt L, Dellavalle R, Delossantos A, Denenberg J, Derrett S, Des JDC, Dharmaratne SD, Dherani M, Diaz-Torne C, Dolk H, Dorsey ER, Driscoll T, Duber H, Ebel B, Edmond K, Elbaz A, Ali SE, Erskine H, Erwin PJ, Espindola P, Ewoigbokhan SE, Farzadfar F, Feigin V, Felson DT, Ferrari A, Ferri CP, Fèvre Eric M, Finucane MM, Flaxman S, Flood L, Foreman K, Forouzanfar MH, Fowkes FGR, Franklin R, Fransen M, Freeman MK, Gabbe BJ, Gabriel SE, Gakidou E, Ganatra HA, Garcia B, Gaspari F, Gillum RF, Gmel G, Gosselin R, Grainger R, Groeger J, Guillemin F, Gunnell D, Gupta R, Haagsma J, Hagan H, Halasa YA, Hall W, Haring D, Haro JM, Harrison JE, Havmoeller R, Hay RJ, Higashi H, Hill C, Hoen B, Hoffman H, Hotez PJ, Hoy D, Huang JJ, Ibeanusi SE, Jacobsen KH, James SL, Jarvis D, Jasrasaria R, Jayaraman S, Johns N, Jonas JB, Karthikeyan G, Kassebaum N, Kawakami N, Keren A, Khoo J, King CH, Knowlton LM, Kobusingye O, Koranteng A, Krishnamurthi R, Lalloo R, Laslett LL, Lathlean T, Leasher JL, Lee YY, Leigh J, Lim SS, Limb E, Lin JK, Lipnick M, Lipshultz SE, Liu W, Loane M, Ohno SL, Lyons R, Ma J, Mabweijano J, MacIntyre MF, Malekzadeh R, Mallinger L, Manivannan S, Marcenes W, March L, Margolis DJ, Marks GB, Marks R, Matsumori A, Matzopoulos R, Mayosi BM, McAnulty JH, McDermott MM, McGill N, McGrath J, Medina-Mora ME, Meltzer M, Mensah GA, Merriman TR, Meyer A, Miglioli V, Miller M, Miller TR, Mitchell PB, Mocumbi AO, Moffitt TE, Mokdad AA, Monasta L, Montico M, Moradi-Lakeh M, Moran A, Morawska L, Mori R, Murdoch ME, Mwaniki MK, Naidoo K, Nair MN, Naldi L, Narayan KMV, Nelson PK, Nelson RG, Nevitt MC, Newton CR, Nolte S, Norman P, Norman R, O'Donnell M, O'Hanlon S, Olives C, Omer SB, Ortblad K, Osborne R, Ozgediz D, Page A, Pahari B, Pandian JD, Rivero AP, Patten SB, Pearce N, Padilla RP, Perez-Ruiz F, Perico N, Pesudovs K, Phillips D, Phillips MR, Pierce K, Pion S, Polanczyk GV, Polinder S, Pope CA, Popova S, Porrini E, Pourmalek F, Prince M, Pullan RL, Ramaiah KD, Ranganathan D, Razavi H, Regan M, Rehm JT, Rein DB, Remuzzi G, Richardson K, Rivara FP, Roberts T, Robinson C, De LFR, Ronfani L, Room R, Rosenfeld LC, Rushton L, Sacco RL, Saha S, Sampson U, Sanchez-Riera L, Sanman E, Schwebel DC, Scott JG, Segui-Gomez M, Shahraz S, Shepard DS, Shin H, Shivakoti R, Singh D, Singh GM, Singh JA, Singleton J, Sleet DA, Sliwa K, Smith E, Smith JL, Stapelberg NJC, Steer A, Steiner T, Stolk WA, Stovner LJ, Sudfeld C, Syed S, Tamburlini G, Tavakkoli M, Taylor HR, Taylor JA, Taylor WJ, Thomas B, Thomson WM, Thurston GD, Tleyjeh IM, Tonelli M, Towbin JA, Truelsen T, Tsilimbaris MK, Ubeda C, Undurraga EA, van DWMJ, van OJ, Vavilala MS, Venketasubramanian N, Wang M, Wang W, Watt K, Weatherall DJ, Weinstock MA, Weintraub R, Weisskopf MG, Weissman MM, White RA, Whiteford H, Wiersma ST, Wilkinson JD, Williams HC, Williams SRM, Witt E, Wolfe F, Woolf AD, Wulf S, Yeh P, Zaidi AKM, Zheng Z, Zonies D, Lopez AD, Murray CJL, AlMazroa MA, Memish ZA. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012 Dec 15;380(9859):2163–96. doi: 10.1016/S0140-6736(12)61729-2.S0140-6736(12)61729-2 PubMed DOI PMC

ELECTOR. Changing the Future Care of Arthritis Patients http://www.elector.eu/

Spradley J. The ethnograpich interview. Florida, USA: Rinehart and Winston Inc; 1979. The ethnograpich interview; p. 0030444969.

Hansen MB, Vedung E. Theory-Based Stakeholder Evaluation. American Journal of Evaluation. 2010 Sep 13;31(3):295–313. doi: 10.1177/1098214010366174. DOI

Clemensen J, Larsen SB, Kyng M, Kirkevold M. Participatory design in health sciences: Using cooperative experimental methods in developing health services and computer technology. Qual Health Res. 2007 Jan;17(1):122–30. doi: 10.1177/1049732306293664.17/1/122 PubMed DOI

Simonsen J, Robertson T( Routledge International Handbook of Participatory Design. NY, USA: Routledge International; 2013. Routledge International Handbook of Participatory Design; pp. 978–0415720212.

Simonsen J, Svabo C, Strandvad S, Samsom C, Hertzum M. Situated Design Methods. In: Hansen O, editor. Situated Design Methods. -Cambridge, Massachusetts: MIT Press; 2014. p. 13.

Jaspers MWM, Steen T, van DBC, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform. 2004 Nov;73(11-12):781–95. doi: 10.1016/j.ijmedinf.2004.08.003.S1386-5056(04)00182-0 PubMed DOI

Peters DH, Adam T, Alonge O, Agyepong IA, Tran N. Implementation research: what it is and how to do it. BMJ. 2013 Nov 20;347:f6753. PubMed

Hassling L, Nordfeldt S, Eriksson H, Timpka T. Use of cultural probes for representation of chronic disease experience: exploration of an innovative method for design of supportive technologies. Technol Health Care. 2005;13(2):87–95. PubMed

Catwell L, Sheikh Aziz. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med. 2009 Aug;6(8):e1000126. doi: 10.1371/journal.pmed.1000126. PubMed DOI PMC

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical RCG. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. PubMed PMC

Eysenbach G, CONSORT- E. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. 2011;13(4):e126. doi: 10.2196/jmir.1923. v13i4e126 PubMed DOI PMC

Smith HR. MedScape. 2018. Rheumatoid Arthritis Guidelines https://emedicine.medscape.com/article/331715-guidelines .

Van VL, Wentzel J, Van GJE. Designing eHealth that Matters via a Multidisciplinary Requirements Development Approach. JMIR Res Protoc. 2013;2(1):e21. doi: 10.2196/resprot.2547. v2i1e21 PubMed DOI PMC

Castensøe-Seidenfaden Pernille, Reventlov HG, Teilmann G, Hommel E, Olsen BS, Kensing F. Designing a Self-Management App for Young People With Type 1 Diabetes: Methodological Challenges, Experiences, and Recommendations. JMIR Mhealth Uhealth. 2017 Oct 23;5(10):e124. doi: 10.2196/mhealth.8137. v5i10e124 PubMed DOI PMC

Scott KC, Karem P, Shifflett K, Vegi L, Ravi K, Brooks M. Evaluating barriers to adopting telemedicine worldwide: A systematic review. J Telemed Telecare. 2018 Jan;24(1):4–12. doi: 10.1177/1357633X16674087. PubMed DOI PMC

World Health Organization. 2012. Long-running telemedicine networks delivering humanitarian services: experience, performance and scientific output http://www.who.int/bulletin/volumes/90/5/11-099143/en/

Patton M. Developmental evaluation: applying complexity concepts to enhance innovation and use. London: The Guilford Press; 2011. Developmental evaluation: applying complexity concepts to enhance innovation and use; pp. 125–1.

Mobile Apps for Medication Management: Review and Analysis