A mHealth application for the training of caregivers of children with developmental disorders in South Africa: rationale and initial piloting

Introduction

Children with developmental disorders (DD), who live in developed countries, and their families often receive monthly early childhood therapy that can include a combination of speech therapy, physical therapy, and other types of services (1). The inability to effectively communicate with peers and family members impacts long-term development and ultimately diminishes a child’s opportunity to become an active member of society (2).

Previous studies have shown that well-designed early communication interventions for children with DD, which include autism spectrum disorder, cerebral palsy, intellectual disability, and other genetic disorders, stimulate the development of basic communication skills especially when parents and other primary caregivers learn how to provide communication opportunities during daily living activities, such as play and mealtime (3-7). Teaching parents how to expand their child’s communication development by creating engaging and stimulating learning opportunities has provided a successful theoretical approach that can be easily replicated (8). One of the keys to implementing this approach successfully is to provide strategies that decrease the adults’ directive communication while increasing their responsiveness to the child’s communication attempts (9).

South Africa currently offers limited access to education, social and health services (10). Language (12 languages are granted official status), culture, ethnicity and religion are highly diverse in South Africa. Families of children with DD in South Africa face significant barriers to receiving intervention services and many families cannot afford the resources that are required to support their children’s development. Healthcare providers in the public primary healthcare system, like speech-language pathologists (SLPs), have large caseloads which creates challenges in providing appropriate and adequate intervention services (11). Furthermore, in South Africa, children with DD often live far from tertiary hospitals where speech and language services are provided and travel to these hospitals creates an additional financial burden (12).

Mobile phone ownership and use has experienced exponential growth in South Africa over the past decade (13). However, while phone calls and data are available in many mobile plans in South Africa, the cost is still prohibitive for many South Africans, and the availability of data is scarce in comparison to what mobile providers offer in other countries. A recent report, ranked South Africa 149 out of 237 countries in terms of the cheapest mobile data (14). Smartphones (i.e., cell phones with capabilities beyond making a phone call such as browsing the Internet), can facilitate patient/provider contact, deliver health information, enable remote consultation and problem-solving, as well as provide instruction. A wide range of smartphones with varying prices are readily available to the South African public. Smartphones are capable of hosting health applications (apps) designed to be used for monitoring, diagnostics, behavioral prompts, reminders, instruction, and self-management programs. Smartphones can be successful in extending and enhancing healthcare well beyond the limitations of physical clinics (15).

The interest in using smartphones or tablets for healthcare research and operational initiatives have led to the development of a new interdisciplinary field called Mobile Health (mHealth). Researchers from different disciplines are interested in developing evidence-based mHealth interventions for an array of physical and mental health conditions (16). For example, a previous research study reported that mHealth can be a platform to provide health-related quality of life support for caregivers of pediatric patients with cancer (17). Progressive Web App (PWA) provides current and scientifically validated information on viral hepatitis and has the capability to reach large audiences and encourage engagement on viral hepatitis prevention (18). The Pync mHealth app promotes healthy diet, nutrition, physical activity, and weight loss among women at risk of breast cancer who have varying levels of health literacy and e-health literacy (19). The PixTalk app and companion website have proved to be effective in increasing the communication skills of autistic children (20). mHealth technologies offer a promising approach to overcoming some of the barriers of access to healthcare in South Africa. A previous research study implemented and evaluated a community-based hearing and vision screening program for preschool children in South Africa, supported by mHealth technology (21). In another research study in South Africa, text messages sent to women during pregnancy showed an improvement in achieving complete maternal-infant continuum of care (22).

An important consideration of mHealth interventions is evaluating stakeholders’ views on the functionality and design of the app as well as the appropriateness of the content. Incorporating stakeholders’ values and perspectives in the development process can increase the generalizability and social impact of the intervention. Involving stakeholders in the research process is particularly important when conducting applied research and is a way to address the researchers’ ethical obligation to ensure that the intervention benefits all stakeholders (2).

There are approximately four million Setswana-speakers in South Africa, making it the sixth largest first language in South Africa. Approximately two million Setswana speakers reside in the North-West province and another one million in the Gauteng province—the two provinces in which the study was conducted. The other million Setswana-speakers are dispersed throughout the country. In the following sections, our multi-institutional and multi-disciplinary team presents the mHealth app Nna’Le’wena, a Setswana phrase meaning “Me and You”. Our app provides a novel approach to early communication intervention by enhancing the standard intervention protocol with home caregiver strategies monitored and coached via mHealth. The purpose of this paper is to describe the initial piloting of the Nna’Le’wena mHealth app. In this paper, we also report the feedback about the quality of the app obtained from the caregivers enrolled in the project.

Methods

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the institutional ethics boards of the University of Pretoria (No. GW20160606HS) and of Georgia State University (No. 16562), and informed consent was obtained from all individual participants.

Participants

In this paper we focus on a subset of caregivers who participated in the project. Twelve caregivers and their children with DD between the ages of 3–6 years old who received monthly early childhood therapy at tertiary care hospitals completed at least portions of the app’s three sections and, were available for a phone interview to answer the questions on the Mobile App Rating Scale (MARS). Caregivers were recruited through speech-language therapy clinics offered by four public hospitals in and around South Africa’s capital city, Pretoria. The demographic characteristics of the caregivers are presented in Table 1. Caregivers reported that Setswana was their first language and that they were exposed to English.

Intervention

The intervention began with a 90-minute session in the caregivers’ preferred language (Setswana or English) including information about beginning communication development and interaction strategies for communicating with their children on the use of our app. The caregiver received instruction and practice opportunities on how to use the tablet and the app. A competency checklist was used to ensure that the caregiver understood the device operations and app features and could perform the required tasks independently. We employed principles of adult learning (i.e., role playing, providing opportunities for hands-on practice, and creating an open atmosphere in which caregivers were free to ask questions). Once the caregivers returned home with their children, they took the tablet with the app on it with them. The app provided a systematic framework and guidance for using the communication interaction strategies with the child over the app’s 12-week period.

App development

Focus group research was conducted ahead of the development and has been published elsewhere (23). The Nna’Le’wena app was developed over several interactive prototyping phases, a cycle which continued to improve and augment its functionality. In each phase, we included testing from SLPs in the United States and in South Africa. Caregivers were also involved in the testing phase once the app reached a stable version. We sought and encouraged the involvement of all stakeholders, specifically caregivers and SLPs at every step of the design process.

First, we designed, developed, and tested the app on a password protected website to easily collect input from researchers and SLPs in the United States and South Africa involved in the project and from caregivers who were regularly seen at the hospital clinics in South Africa. Then, we transferred the app to the tablets by using Bit Web Server (24), a web server app for Android, that includes a light web server, Hypertext Preprocessor (PHP) (25) a popular general-purpose scripting language, and My Structured Query Language (MYSQL) (https://www.mysql.com/) (26), an open-source relational database management system. We designed the app to allow caregivers to browse the content and to collect their answers to a weekly questionnaire without the need for an Internet connection. Given feedback from caregivers about the difficulties of smartphone use and based on budget limitation and availability and affordability of Apple products in South Africa, we chose the Samsung ST-350 tablet with an 8-inch screen to host the app, thereby enabling all participating caregivers to have consistent hardware for the prototype study. We provided one tablet to each of the caregivers. The web-based app also allowed us to be efficient with the development, updates, maintenance, and changes to our prototype, and allowed us to meet our specific time and budget constraints.

App content and characteristics

Development of the app content was a collaborative and iterative process between the research and design teams in the United States and South Africa. The app content, which included the text for the strategy instructions and help function, was initially revised, and edited to increase readability by one of the team who has expertise in copyediting. To ensure the cultural appropriateness of the content, the South African team reviewed the content for linguistic and cultural appropriateness and provided suggestions to increase the relevance of the activities and strategies for the South African context.

The South African team also selected and photographed culturally appropriate materials (e.g., toys, clothes, food items) to include in the app and created videos demonstrating the strategies. The South African team provided design ideas and suggestions to the graphic designer who created the logo of the app which included an iconic South African tree and stylized figures of a parent and child. The colors and fonts were selected based on feedback from the South African team following consultation with stakeholders. A prototype of the app was trialed with a caregiver of a child with an DD and this additional caregiver feedback was incorporated in the version updates.

The content for our app was based on the findings from a randomized-controlled parent-implemented language intervention study with toddlers with DD in the United States (27). The app content was presented sequentially over 12-week in a culturally appropriate manner and consisted of three sections comprised of four weekly strategies: (I) creating communication opportunities: waiting, giving small amounts of a desired item, providing choices, and disrupting the environment; (II) modeling communication: speaking slowly, simplifying your language, describing, and face-to-face interaction; and (III) responding to the child’s communication: responding, repeating and adding, filling-in-the-blank and asking open-ended questions. Each section and strategy within the app were presented in a sequenced manner over the course of four weeks. Each week focused on different strategies within the section consistently illustrated in four activities (Dressing and Bathing, Mealtime, Playing, Book reading).

Important characteristics of the app were: (I) each strategy had a help function with text written in English about the topic and a video in Setswana that provided a demonstration on how to conduct the activity; (II) caregivers could repeat strategies and activities after they initially completed the activity; (III) each page had a toll-free “Please call me number” that could be used by the caregivers if they needed additional support and could not afford to make a phone call. A researcher from the main clinic in South Africa returned all the calls; (IV) the app was written in an easy English format but included audio in both English or Setswana to assist caregivers who were not comfortable reading English or who preferred to listen to the app. Figure 1 shows four screenshots of our app.

Figure 1 Screen shots from the Nna’Le’wena app.

Measurements

In order to assess the quality of our mobile app, we used the relevant portions of the MARS, a simple, objective and reliable tool for assessing the quality of mobile apps (28). We were particularly interested in assessing the following six objective scales: engagement, quality of information, performance, ease of use, navigation, and visual appeal. We also used a subjective quality scale from the MARS that asked caregivers if they would recommend our app to other caregivers who might benefit from it. All items on the MARS were scored on a five-point Likert scale (1= no interest/irrelevant to 5= high interest/relevance).

In order to investigate the interaction of the caregivers with our app we collected: (I) log data that tracked what help functions related to what topics were consulted more frequently and, (II) weekly questionnaire data about the activity the caregivers thought worked best to engage their children and about the language in which caregivers listened to the audio instructions of the app. Log data and questionnaire data were downloaded from the tablets by the research team every time they had a face-to-face check-in with the caregiver. These check-ins were planned to coincide with the child’s monthly therapy sessions at the hospital clinic.

Statistical analysis

Data analysis was performed using IBM SPSS version 27.0 (29). The demographic characteristics of the caregivers, the log data and questionnaire data that was collected every week for 12 weeks were assessed using descriptive statistics as well as the seven items on the MARS that were rated on a five-point Likert scale.

Results

Most of the caregivers showed high levels of engagement with ten of the twelve considering the app moderately interesting or very interesting. The quality of the information provided was reported highly relevant by ten caregivers. Two of them rated the quality of the app moderately relevant. The performance of the app was rated perfect by eight caregivers. One caregiver reported that the app was mostly functional and three caregivers reported that the app worked overall but that they experienced some technical problems. Half of the caregivers considered the app intuitive and stated that they were able to use it immediately. Four caregivers reported that it was easy to learn how to use the app and two caregivers said the app was useable after some time/effort. The navigation of the app was also rated high by ten participants stating that the app was easy to use or perfectly logical. The visual appeal of the app also received a high score with ten caregivers reporting that it had a high or outstanding level of visual appeal. All the caregivers would recommend the app to other people who might benefit from it. Caregivers never selected the lowest two levels on any of the seven MARS subscales. Table 2 shows in detail the results obtained with the MARS.

Among the twelve caregivers, nine of them completed the entire 12 weeks training, one caregiver completed seven weeks, one caregiver completed four weeks and one caregiver stopped just after the first week. At the end of each week, caregivers answered questions about the language of the audio instructions used and about what activity they thought worked best to engage with their children. Caregivers reported 122 times about how the listened to the audio instructions. They listened in both English and Setswana half of the time (n=58, 47.5%), they listened in English one third of the times (n=39, 32.0%). They listened to the instructions in Setswana (n=11, 9.0%) for the least amount of time, despite reporting that Setswana was their preferred language. They did not listen to any audio instruction for approximately 15% of the times (n=14, 11.5%). Caregivers reported 117 times the activity that worked the best to engage with their children. The activity that caregivers considered as working the best was playing (n=54, 46.1%), followed by mealtime (n=38, 32.5%), book reading (n=14, 12.0%) and, dressing and bathing (n=11, 9.4%). During the 12-week app training, the caregivers accessed the sections of the app that contained the help function 140 times. The help function that was accessed most often was related to the mealtime topic (n=50, 35.7%), followed by books (n=38, 27.1%), playing (n=25, 17.9%) and dressing and bathing (n=27, 19.3%). Table 3 shows the results of the analysis of the questionnaire data and of the log file collected for the duration of the training.

Discussion

We designed, developed, and tested a web-based app for the at home instruction of caregivers of children with DD. Results from the initial piloting phase are encouraging and promising. Our findings suggest that mHealth solutions are well received by users in low- and middle-income countries even without a reliable Internet connection.

Technical observations

Effective mHealth apps, designed in research projects to work with a particular device on a particular software platform, can become obsolete by the end of the project (30). It is not a secret that the continual release of new mobile devices and development platforms can significantly impact and ultimately slow down the transfer from research prototype to finished product. Native apps (i.e., apps that run on the device in the native operating system) are preferable when the intervention requires access to various features (such as camera, calendar, alarm, sensor, location) contained within the device. Native apps work more quickly and present a more consistent look and feel. However, they require more specialized programming knowledge, take more time to be developed and must be maintained as the native operating system is updated. A web-based app (i.e., a resource that is accessed using the device web browser) is usually coded in a browser rendered language such as HTML, JavaScript or PHP, which is a more common knowledge. Web-based apps can be developed and iterated more quickly, and can be used across different operating systems. The challenges of fast evolving mHealth technology and the problems related to the development of native apps versus web-based apps were the drivers that made us decide to develop an app that is a localized web-based device that does not require an Internet connection to work (31).

We decided to provide all the caregivers who participated in this project with the same tablet. This approach helped with generalization of the results. Our project targeted a low-income population that did not have access to tablets. However, we are aware of the benefits of providing mHealth resources that can be installed directly on the devices that users own. People are more prone to use resources available on the device that they constantly use for calling, browsing, listing, and conducting their daily routine tasks.

We believe that mHealth researchers will inevitably face technical challenges that arise during the deployment process. Even the most meticulous coding can result in some unforeseen challenges in the final product release. We realized that some of the tablets were set on US eastern standard time (EST) time while they should all have been set on South Africa time zone. Also, we realized that the time setting function of the tablets were enabled and hence some caregivers changed the time on their tablet. For this reason, the time stamp of the log file could not be used to determine what times of the day the caregivers used the app most.

Intervention observations

While mHealth interventions can deliver the education content in an alternative format, they should not replace therapy sessions. For this reason, we gave the caregivers who participated in our study the opportunity to interact frequently with the app but in a paced environment without the ability to advance through the content until the instruction was completed. This approach allowed the caregivers to interact with the mHealth app in a manner that was similar to how they used other apps, rather than how they would interact with a SLP.

Caregivers reported being able to engage with their children mostly during Play and Mealtime activities. The log data also recorded these two activities as the most frequently occurring caregiver-interactions. Previous research has reported that language barriers affect the utilization of digital health tools (22). One important feature of the app was the ability to listen to audio recordings of the strategy instructions and help functions in both Setswana and English. What was surprising was that caregivers listened to English almost twice as often as Setswana. Despite being able to understand, speak and write English, we anticipated that caregivers would feel more comfortable with understanding Setswana as this was reported as their home language. For the success of mHealth interventions, it is vital to not forget the language needs and preferences of the stakeholders (i.e., end users).

Limitations and recommendations for future studies

There were limitations to our study. The Nna’Le’wena app was tested with a small number of caregivers of children with limited etiologies of DD. Only nine caregivers completed the entire 12 weeks of training. Also, we did not collect any qualitative data from the caregivers about their perceptions of the app, their confidence with using the app, or their experience with the instructional program. Such information could have helped with the further development and refining of the app. Potential bias of the participants who were involved in this phase of the app testing should also be considered.

Future research could include the development and testing of the Nna’Le’wena app other communities in different languages. Since the Nna’Le’wena app was designed as a web-based app, the possibility to deploy the instruction as a web service where caregivers access a web portal using their own devices, could be accomplished in a relative short time. The app was developed prior to the pandemic. Given changes to health care delivery since the pandemic, the app may have applicability in a range of settings and countries.

Conclusions

Our caregiver instruction app as part of early communication intervention was innovative in several ways. We are aware of no published studies on the use of mHealth interventions for teaching caregivers of children with DD among low-income populations in South Africa who have varying levels of health literacy and e-health literacy. Our app addressed the need for low-cost, effective strategies for caregiver instruction. Our app was developed using evidence-based approaches for managing communication disabilities and it is based on effective published theories. We found that multidisciplinary teams, comprised of clinical and technology experts working together throughout the entire project, are critical to the offering of effective mHealth interventions. Furthermore, the inclusion of a validated scale (i.e., MARS), permitted us to evaluate the user experience of the app which will inform future versions and updates to the app.

Acknowledgments

The authors thank Dr. Vann Morris and Ms. Vanessa Sanchez for their work on the development of the app. The authors also thank the caregivers and their children for their time and effort.

Funding: The study was supported by National Institute of Deafness and Other Communication Disorders, National Institutes of Health, Grant/Award Number: DC-015225 (to Gianluca De Leo, MaryAnn Romski, Marika King, Rose A. Sevcik, and Juan Bornman).

Footnote

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-49/dss

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-49/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-49/coif). All authors except Marco Renzi were parts of a group that received a research grant for this study from the National Institute of Deafness and Other Communication Disorders, National Institutes of Health, Grant/Award Number: DC-015225. MaryAnn Romski is also an Extraordinary Professor at the University of Pretoria, Centre for Augmentative and Alternative Communication (a nonfinancial position). The other author has no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the institutional ethics boards of the University of Pretoria (No. GW20160606HS) and of Georgia State University (No. 16562), and informed consent was obtained from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. National Center on Birth Defects and Developmental Disabilities CfDCaP. What is “Early Intervention”?: Centers for Disease Control and Prevention; 2023. Available online: https://www.cdc.gov/ncbddd/actearly/parents/states.html.
2. Sevcik RA, Romski MA. Communication interventions for individuals with severe disabilities: exploring research challenges and opportunities. Baltimore: Paul H. Brookes Publishing Co.; 2016. xxiii, 352 p.
3. Adamson LB, Romski M, Bakeman R, et al. Augmented language intervention and the emergence of symbol-infused joint engagement. J Speech Lang Hear Res 2010;53:1769-73. [Crossref] [PubMed]
4. Kaiser AP, Hancock TB. Teaching parents new skills to support their young children's development. Infants & Young Children 2003;16:9-21. [Crossref]
5. Roberts MY, Kaiser AP. Early intervention for toddlers with language delays: a randomized controlled trial. Pediatrics 2015;135:686-93. [Crossref] [PubMed]
6. Romski M, Sevcik RA, Adamson LB, et al. Parent perceptions of the language development of toddlers with developmental delays before and after participation in parent-coached language interventions. Am J Speech Lang Pathol 2011;20:111-8. [Crossref] [PubMed]
7. Sevcik RA, Romski MA, Adamson LB. Research directions in augmentative and alternative communication for preschool children. Disabil Rehabil 2004;26:1323-9. [Crossref] [PubMed]
8. Stockwell K, Alabdulqader E, Jackson D, et al. Feasibility of parent communication training with remote coaching using smartphone apps. Int J Lang Commun Disord 2019;54:265-80. [Crossref] [PubMed]
9. Brady N, Warren SF, Sterling A. Interventions Aimed at Improving Child Language by Improving Maternal Responsivity. Int Rev Res Ment Retard 2009;37:333-57. [Crossref] [PubMed]
10. Huus K, Dada S, Bornman J, et al. The awareness of primary caregivers in South Africa of the human rights of their children with intellectual disabilities. Child Care Health Dev 2016;42:863-70. [Crossref] [PubMed]
11. Kathard H, Pillay M. Promoting change through political consciousness: a South African speech-language pathology response to the World Report on Disability. Int J Speech Lang Pathol 2013;15:84-9. [Crossref] [PubMed]
12. Donohue DK, Bornman J, Granlund M. Household size is associated with unintelligible speech in children who have intellectual disabilities: A South African study. Dev Neurorehabil 2015;18:402-6. [Crossref] [PubMed]
13. Silver L, Johnson C. Internet connectivity seen as having positive impact on life in Sub-Saharan Africa. 2018. Available online: https://www.pewresearch.org/global/2018/10/09/internetconnectivity-seen-as-having-positive-impact-on-life-in-subsaharan-africa/
14. Cable.co.uk. Worldwide mobile data pricing: The cost of 1GB of mobile data in 237 countries 2023. Available online: https://www.cable.co.uk/mobiles/worldwide-data-pricing/
15. Harrison V, Proudfoot J, Wee PP, et al. Mobile mental health: review of the emerging field and proof of concept study. J Ment Health 2011;20:509-24. [Crossref] [PubMed]
16. de Korte E, Wiezer N, Bakhuys Roozeboom M, et al. Behavior Change Techniques in mHealth Apps for the Mental and Physical Health of Employees: Systematic Assessment. JMIR Mhealth Uhealth 2018;6:e167. [Crossref] [PubMed]
17. Koblick SB, Yu M, DeMoss M, et al. A pilot intervention of using a mobile health app (ONC Roadmap) to enhance health-related quality of life in family caregivers of pediatric patients with cancer. Mhealth 2023;9:5. [Crossref] [PubMed]
18. Zagarella RM, Farrelly FA. ABC Epatite Web App: raising health awareness in a mobile world. Mhealth 2021;7:58. [Crossref] [PubMed]
19. Coughlin SS, Besenyi GM, Bowen D, et al. Development of the Physical activity and Your Nutrition for Cancer (PYNC) smartphone app for preventing breast cancer in women. Mhealth 2017;3:5. [Crossref] [PubMed]
20. De Leo G, Gonzales CH, Battagiri P, et al. A smart-phone application and a companion website for the improvement of the communication skills of children with autism: clinical rationale, technical development and preliminary results. J Med Syst 2011;35:703-11. [Crossref] [PubMed]
21. Eksteen S, Launer S, Kuper H, et al. Hearing and vision screening for preschool children using mobile technology, South Africa. Bull World Health Organ 2019;97:672-80. [Crossref] [PubMed]
22. Coleman J, Black V, Thorson AE, et al. Evaluating the effect of maternal mHealth text messages on uptake of maternal and child health care services in South Africa: a multicentre cohort intervention study. Reprod Health 2020;17:160. [Crossref] [PubMed]
23. Bornman J, Romski M, King M, et al. Supporting Early Communication Skills of Children with Developmental Disorders in South Africa: Caregiver and Clinician Perspectives about Mobile Health Applications. Infants Young Child 2020;33:313-31. [Crossref] [PubMed]
24. Unpam A. Bit Web Server. 2023. Available online: https://dalfindo.id/
25. Thies C. Arntzen SB, Shane Caraveo, Andi Gutmans, Rasmus Lerdorf, Sam Ruby, Sascha Schumann, Zeev Suraski, Jim Winstead, Andrei Zmievski. Hypertext Preprocessor (PHP). 2023. Available online: https://www.php.net/
26. Oracle. My Structured Query Language (MYSQL). Oracle; 2023. Available online: https://www.mysql.com/
27. Romski M, Sevcik RA, Adamson LB, et al. Randomized comparison of augmented and nonaugmented language interventions for toddlers with developmental delays and their parents. J Speech Lang Hear Res 2010;53:350-64. [Crossref] [PubMed]
28. Stoyanov SR, Hides L, Kavanagh DJ, et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3:e27. [Crossref] [PubMed]
29. IBM Corp. IBM SPSS Statistics for Windows. 27 ed. Armonk, NY: IBM Corp; 2020.
30. Riley WT, Glasgow RE, Etheredge L, et al. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med 2013;2:10. [Crossref] [PubMed]
31. Khachouch MK, Korchi A, Lakhrissi Y, et al., editors. Framework Choice Criteria for Mobile Application Development. 2020 International Conference on Electrical, Communication, and Computer Engineering (ICECCE); 2020: IEEE.