Mobile app development in health research: pitfalls and solutions

Introduction

The use of mobile applications (apps) is pervasive across every aspect of daily life. Smartphones have been adopted faster than nearly any other technological innovation in history, such that it is now nearly universal in the United States (81% of all US adults, including 96% of adults ages 18–29) (1-3). This phenomenon has also occurred across the globe, with over 3.3 billion people using smartphones worldwide (4). Adults spend 2.5–5 hours per day on their phones, or 13–16% of their waking hours. This trend is even more pronounced among US adolescents, 95% of whom either own or have access to a smartphone, with nearly half reporting being online on a near-constant basis (5). Despite some health concerns, particularly related to excessive screen time, there is growing interest in leveraging smart phone technologies to promote health through mobile apps (6-8).

The development and use of mobile health apps is rapidly increasing, with a wide variety of functions such as self-monitoring of chronic health conditions, medication adherence reminders, and direct interactions with the health care system (9-12). Mobile health apps are increasingly being used even in situations when clinical care is provided in-person, as they can be used to help tailor patient-provider communication and support patient self-management and care engagement (9). Multiple systematic reviews, usually grouped by health condition or sub-population, have summarized the growing evidence base for the effectiveness of mobile health apps (10,11). For example, a recent meta-analysis of digital interventions that address alcohol consumption in community-dwelling populations, including many mobile health apps, found moderate-quality evidence that digital interventions decrease unhealthy alcohol consumption (13). A recent systematic review of the use of mobile health apps for substance use disorders concluded that the heterogeneity of mobile health apps made reaching a consensus about their overall effectiveness challenging (14). This review also noted that mobile health apps should fully capitalize on the technology’s capacity to tailor itself to meet the individual needs of users (14). Achieving this, however, will likely require a better understanding of how people incorporate technology into their everyday lives, as well as research into effective ways to disseminate efficacious interventions into more diverse clinical and community settings. Another review noted the great promise for mobile health apps to make an impact in low- and middle-income countries, where access to medical care is often limited but smartphone ownership is widespread (12).

A growing number of health research programs investigate app-based interventions, yet many research teams conduct their app research in isolation. There are substantial benefits to be gained by sharing knowledge across disciplines, particularly in navigating common challenges and leveraging areas of strength. In this article, we discuss opportunities and pitfalls for mobile health app research, and propose solutions to facilitate success and overcome challenges.

Opportunities

Leveraging device, operating system, and potential for scale

Mobile apps are able to leverage the strengths of host device hardware and operating systems. A review by Harari and colleagues has documented the numerous sensors and data collected by research apps (15), including accelerometer (coordinates, duration of movement), GPS scan (geolocation), clock (time), light sensor (ambient light), and microphone (audio). These data sources can be used creatively to develop interventions. For instance, a sun protection trial combined GPS data with real-time forecasts and time of day information to provide guidance regarding risk of sunburn and time until needed reapplication of sunscreen (16). A number of physical activity trials have used smartphone accelerometer data to collect physical activity data, and display such data as part of the intervention to study participants (17). Operating systems also provide rich interactive and monitoring features, including alarms, notifications, call logs, text logs, and system usage information. Data generated by these features can be highly useful; one study validated an algorithm to predict the total amount of users’ sleep based on their smartphone screen being on or off, with an average error of only 7% (18). Alarms/notifications are a main feature of many app-based interventions, providing a way to communicate updated data-informed progress towards goals, motivational messages, and re-engagement messages (17,19-21). Another important benefit of the flexibility of app systems is their capacity to interface with a wide variety of other devices such as pedometers (17) and pill bottle cap sensors (22).

The ability to positively influence health at a large scale is an intriguing advantage of successful mobile apps. Traditionally, the gold standard for impactful individual-level behavioral interventions has been evidence-based, multi-session interventions that are delivered in person. This approach provides a high level of exposure to a potentially tailored intervention, although it comes at a high cost by requiring substantial staff time and materials for each person newly engaged. Such multi-session interventions are typically sequentially planned to control the order in which a participant is exposed to an intervention, potentially enhancing the intervention’s effect, but also challenging the logistics of delivery. Mobile health apps have the potential to provide users with a high level of exposure (smartphones are ubiquitous and heavily used), while only requiring staff time that is fixed to the development and monitoring of the sequentially-designed app intervention, with low additional cost per person reached. Other technology-based intervention modalities, such as text messaging and website-based interventions, may have similar benefits of scale.

Tailoring and measurement

Apps are a natural fit for providing tailored health information, with the potential to build in automated tailoring by user groups or by disease condition. Apps are created for a broad array of conditions that require tailored materials, ranging from tele-rehabilitation for people with multiple sclerosis (23) to interventions to address childhood obesity (24). App platforms allow for each user group to receive an intervention appropriate for and customized to their experiences. This has the additional benefit of facilitating more successful inclusion of groups experiencing health disparities such as youth, sexual, and racial minorities (25,26). Through tailoring, mobile apps have the potential to engage persons in their health promotion in new and innovate ways, which are moreover less dependent on existing healthcare structures. For instance, apps can help users collect and track data on a particular health behavior and can return information tailored to that individual, such as their stage in transtheoretical model (27), thereby optimally facilitating behavioral change.

Paradata, automated process data collected as users interact with a smartphone app, is an important additional tool to gain insight on how users engage with an app (28,29). Examples of paradata include log-in/log-out times; time spent in the app overall and by each app feature; and number of clicks through each app feature. Used in combination with the primary research outcomes data, paradata provides insight into user preferences and app use patterns. It can help app researchers to understand why an app feature may or may not have met expected outcomes, and provide direction for further tailoring or updates.

Pitfalls

Technology versus research timelines

Technology product development is characterized by a period of ascent characterized by high innovation followed by a phase of maturity, and then a period of decline (30). There can be a mismatch between the timelines of NIH-funded research and technological products. NIH research grant cycles predominantly adhere to a 10-year timeline, yet technology cycles may occur more rapidly (Figure 1). To elaborate: a typical NIH-funded research cycle begins with idea generation, seed funding and baseline data collection, and first grant application (2 years), and then proceeds to a planning grant (e.g., R34/R21) and its implementation (3 years), clinical trial phase (5 years), and results dissemination (1 year), totaling a more than 10-year cycle. It is worth noting that some of these research phases may be skipped, such as if sufficient baseline data exist to avoid the seed funding period. Yet other issues may extend research timelines, such as grant resubmissions. In contrast to the linear and timeline-based process of research, there is greater variability in technological uptake, which tends to ebb and flow based on development of technological breakthroughs such as new platforms and interfaces. These types of breakthroughs may challenge the 10-year NIH cycle. For instance, if an innovative product such as the Nintendo Wii or apps such as Pokemon Go entice researchers and they initiate the NIH-funded research cycle during ascent or maturity periods, it is possible for product sales to be (I) obsolete, ended prior to research completion, (II) simultaneous, ending at the same time as the research, or (III) continuing, with sales ongoing or even growing as the research ends. In the first two scenarios, the product platform has ended before research regarding a particular intervention can be brought to scale.

Figure 1 Technology versus research cycles.

Given rapid changes in technology, it is difficult to predict when disruptive technology may break through and interrupt research plans, including those regarding app development or configuration. More subtle than disruptive innovations, but perhaps equally important, are changing norms in the design of technology interfaces. Design and user interface norms shift rapidly, with older fonts, icons, graphics, and functions quick to appear dated. Without updating, older interfaces may appear stale or nonintuitive, potentially impacting user willingness to engage with the technology. This is especially likely to impact apps grounded in gamification and graphics features, which are likely to require more frequent updating.

App development

There is an inherent information asymmetry between professional app developers’ expertise in coding and researchers possessing terminal degrees in unrelated fields. Information asymmetry is classically described as an imbalance of power that can lead to poor outcomes. Researchers with projects that involve app development either work with an app development agency, work with a free-lance app developer, or hire a full-time employee with app development experience. Due to limited experience and information, researchers are likely to have difficulty choosing the optimal approach. Moreover, any final contract may be insufficient to facilitate successful development required by the researcher (31). Once app development has begun, most researchers do not have the requisite information or experience to properly oversee the process. For example, an app developer may not be creating code of a sufficiently high quality, yet a researcher may not be able to detect this until the contract is finished and problems emerge with the app’s functioning. Moreover, researchers without app development experience may miss key areas during its design that can affect both users’ experience and interface with the app. Key areas of development include such things as coding architecture, common features (e.g., login with new technology such as fingerprint or face recognition), navigating institutional (e.g., university) and professional [e.g., Health Information Portability and Accountability Act (HIPAA)] security standards, and clarifying data export requirements. When challenges arise, any unplanned but desired app features are likely to result in substantial shifts in timelines and additions to development costs beyond the initially agreed fee.

Researchers should be particularly clear about the data export functionality of their app so that they can ensure data are being collected and exported in a way that facilitates its analysis. This area requires particular attention because it is foreign to most developers who usually focus on business-based outputs of ad clicks and sales. Even those developers accustomed to working with academics may struggle, especially as developers often code in teams, and research expertise may not be uniform across a technology team.

Technical challenges can be exacerbated by the high expenses required to create apps. Apps sufficiently nuanced to fulfill the needs of researchers, and that meet current HIPAA and other security standards, can be costly. The costs of app development range broadly, but even for a relatively low-feature health app, researchers should plan for costs over $150,000 US dollars if building from scratch (building from an existing platform can substantially change cost), a cost that may not fit into the budgets of NIH funding mechanisms typically used to fund pilot studies (32). An important component contributing to high costs are requisite security features for apps dealing with health information. Security concerns are not hypothetical; for example, in 2015, the health information of over 100 million individuals was breached (33). In other years between 2013 and 2017, more than 10 million individuals per year were impacted by health information breaches (33).

Translation and download problems

Once developed and tested in a clinical trial, apps demonstrated to be efficacious should be disseminated. Yet pathways to do so are relatively uncharted, and face two substantial problems. The translation problem is that health apps developed by researchers may never be translated into production models that can be downloaded by the public (and not just solely accessed by research subjects). In a search of NIH RePORTER conducted in late 2018, we identified 40 grants awarded to develop mobile apps relating to HIV, with 17 of these grants having completed their years of award. These grants represented a diverse portfolio of research, with target outcomes ranging from prevention (e.g., condom use) to treatment adherence, and target groups including general populations, racial/ethnic minorities, injection drugs users, men who have sex with men, and cis-gender women—indicating that any problems with performance were likely attributable to challenges spanning research domains and target populations. We performed a review of information for the 40 grants, both within RePORTER and within publications citing relevant grant numbers, to identify information such as a name or keyword that would allow us to search for resulting apps. We then searched the Google Play and iTunes App stores. We were unable to locate any apps from HIV grants in RePORTER that could be used by members of the public. We identified 2 apps that could be downloaded and used only by research participants.

The download problem is that even once made public, persons most in need of services are unlikely to download evidence-based apps unless they (I) are aware of the app, (II) believe the app provides substantial utility, and (III) believe it is better than existing apps. The app market is already inundated with non-evidence-based health apps. For instance, a review of mobile apps for HIV prevention identified 285 publicly available apps, but most (71%) were not developed by academic or public health entities, and none dealt with a key component of current prevention efforts: HIV pre-exposure prophylaxis (34). Similarly, a review of apps promulgated to support mental health found that only 10% offered support that was consistent with principles of evidence-based practice (35).

Sustainability

In addition to translation and download problems, apps that are disseminated require continual resources for updating and maintenance. Without this, apps quickly become dated and can stop functioning. This makes a post-research translation even more challenging. If not developed using a profit-seeking model, research funds are finite and tied to a specific set of proposed activities, putting most of this work beyond the scope of the vast majority of research proposals.

Solutions

Conducting research informed by theory that applies rigorous methodologies

Research grounded in behavioral theory that uses rigorous and appropriate methodologies can produce findings that are generalizable beyond the life of the technology used in the research. There are multiple examples of this in research using the now defunct personal digital assistant (PDA). For instance, a recent PDA-based study used ecological momentary assessments (EMA), a method that involves repeated sampling of subjects experiences in their natural environments in real time, and found that side-effects and self-management among cancer survivors offer opportunities for tailored care programming (36). Another study used PDA and EMA, finding evidence that supports emotional regulation being measured as a trait (37). A study using the Technology Acceptance Model found that 71% of the variation of physicians intention to use a new device was explained by domains of perceived usefulness and perceived ease of use (38). Conversely, failure to base interventions on theory or functionality that can extend beyond a single technology leads to conclusions that are outdated by their time of publication. For example, the main conclusion of a 2013 article regarding PDAs (when such devices were off-market) was that PDA use should be scaled up among nursing students. This article could have benefitted from considering the benefits of incorporating changing technology into nursing education, in general, as well as the challenges with the changing pace of such technological solutions (39).

Leverage screen capture technology to document intervention

Technology change not only produces challenges for research, but also opportunities. Journals produce electronic supplements to publications, and those dedicated to protocol publication are an optimal venue for documentation of app-based interventions. Screenshots and verbal descriptions are useful to indicate functionality of websites, and are commonly used (40). The chief limitation of this approach is that it is challenging to describe interactive platforms with static images and text. We propose an alternative: to create screen recordings and voiceovers to provide a “walk-through” for each primary function of a study app. It is likely that such videos would be a valuable resource for future researchers and developers to adapt successful interventions for new platforms. Walk-throughs can be created, at no cost other than a limited amount of staff time, by using native screen recording functionality of a mobile phone. To our knowledge, this approach has not previously been used to document app-based interventions. For proprietary intervention components, video materials could be released alongside publication of trial results to facilitate future dissemination.

Short-circuit timelines and facilitate dissemination

To improve the responsiveness of the NIH funding cycle in order to more effectively conduct research on or using technology, a number of steps could be taken. One option is for researchers to consider alternative grant mechanisms within NIH, such as the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs that are designed to support health products produced by small businesses in collaboration with research partners. The 2.5-year total research timeline for SBIR/STTR (a 6-month pilot, phase I and 2-year trial, phase II) allows it to be much more responsive to changes in technology, with a targeted goal of commercializing a product. Another option is to leverage NIH center grants and other mechanisms, such as U-level trials networks, to bring promising interventions to scale more quickly. For instance, the UNC/Emory Center for Innovative Technology (iTech), part of the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN), creates an infrastructure to share and disseminate best practices in technology-based HIV interventions (41). Leveraging collective experiences in multiple app-based interventions facilitated development of the ePrEP platform after a 1-year pilot test (40). Researchers may also consider alternative funding venues, such as foundations, that may be more flexible and responsive to the needs of app development for health research.

A separate avenue is collaboration with private enterprises that have already achieved scale in the app space. Such an approach brings substantial benefits but also limitations. In favor of this model are the likely scalability and sustainability of the work. Additionally, by being on the cutting edge of what is in the marketplace, successful app businesses have access to the newest and most innovative tools that may benefit the research. This approach, however, requires meshing the business interests of the private enterprise with the research and public health interests of academic partners. Traditional research frameworks may need to be eschewed in favor of implementation science and monitoring/evaluation frameworks. Vested business interests may limit the scope of research, and issues of technology transfer and ownership may complicate relations. To the extent possible, these should be explored prior to commitment of the partnership.

Oversight of app technical development

Inclusion of an independent and research-versed developer to the research team can directly address the problem of information asymmetry. Many universities have technology-based app development groups that can provide such oversight through collaboration or trusted independent developers. Expertise is required early and periodically throughout the project. Early on in a project, an independent expert can ensure that the initial scope of work is sufficiently detailed to provide coverage of all program needs. The added costs of including an independent developer on the research team is often well worth the upfront cost because it can minimize potential problems regarding the scope of work and oversight of project development.

Independent expert oversight should extend to code architecture and development over the life of the project. Proper code architecture is essential for the performance of the basic tasks including (I) maintenance required due to phone operating system changes over time, (II) updates to address the appearance of unexpected issues as apps are used, and (III) addition of features demanded by users or researchers. Improper architecture can lead to results that confound and frustrate researchers unfamiliar with development: a simple bug fix can make seemingly unrelated parts of the app completely nonfunctional, requiring further fixes, a process that can spiral into cycles of dysfunctions and unexpected costs. These problems can also impact study outcomes if they occur during the course of a clinical trial. Moreover, an app designed with poor or highly stylized architecture may only be accessible for work by the original developers. This creates problems if the original developer either goes out of business or decides to substantially increase their prices. Proper planning and oversight can limit these pitfalls by ensuring that, as code is developed, it can be easily worked on and maintained by a coding team independent of the developer, with an architecture favorable to research data access and future app updates.

Structural changes

Structural changes could be made to capitalize on app-based research. NIH funds a number of center grants, and a future center could focus on expertise in health app development. Such a hub could anchor key contributions, such as (I) development of an open-source coding platform to address the most common research needs or (II) providing an at-cost service center model to provide expert oversight of code architecture for NIH-funded research projects. Adding a greater resource base could facilitate development of common functionality for open-source coding platforms that already exist, such as ResearchStack for Android, ResearchKit for iOS, or frameworks that allow simultaneous development of both Android and iOS such as React Native. Additional resources could include standardized consent or programming interfaces for commonly used external services such as survey platforms. Having these openly available to researchers could provide substantial resource savings for future app development. These efforts face challenges, however, such as staying current due to rapidly changing technology and norms of development. An alternative strategy could be to develop a model to facilitate app development for research across NIH. A hub where researchers share best practices could serve as a service model to provide key oversight or other functionality requested by NIH-funded researchers, and to document and disseminate best practices of and theory-based findings from app research.

To enhance the scale of evidence-based app interventions, it may be necessary to have privately or publicly funded dissemination programs. The US Center for Disease Control’s (CDC) Evidence-Based Interventions for HIV Prevention (EBI) is a useful example of such a program. The EBI program currently houses over 120 interventions that have been demonstrated to reduce HIV transmission or improve care outcomes for those living with HIV. Dedicated funding is used to disseminate these interventions, which for non-electronic programs has included development of intervention materials such as binders, pamphlets, videos, and other printed materials as well as ongoing provision of technical assistance. None of these EBI interventions are solely or even predominantly app-driven. A similar program for evidence-based apps could provide substantial utility, with translation and maintenance costs borne by the program. Such a program could be supported by CDC or alternate funding sources such as the National Institutes of Health or private foundations. Structured and dedicated funding would allow and empower health departments and community organizations to access evidence-based app programs, and would facilitate the visibility of app interventions via their inclusion in a publicly-funded, evidence-based compendium.

Conclusions

Mobile app health research is a promising avenue for health promotion, yet its implementation comes with many new challenges. We describe some of these and discuss strategies to address them. Investing resources in app research may facilitate its development, impact, and dissemination, allowing it to fulfill its substantial promise.

Acknowledgments

Funding: This work was supported by the National Institute of Mental Health (R01MH114692), the Adolescent Medicine Trials Network for HIV/AIDS Interventions (Protocol 159) from the National Institutes of Health (U19HD089881), and the National Institute of Allergy and Infectious Diseases (R01AI143875). The work was facilitated by the Emory Center for AIDS Research (P30AI050409). The contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Lisa Hightow-Weidman) for the series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” published in mHealth. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-19-263). The series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” was commissioned by the editorial office without any funding or sponsorship. LHW served as the unpaid Guest Editor of the series. Dr. Hightow-Weidman reports grants from NICHD, during the conduct of the study. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. DeGusta M. Are smart phones spreading faster than any technology in human history. Massachusetts Institute of Technology Review, 2012.
2. Smith A. Record shares of Americans now own smartphones, have home broadband. Pew Research Center 2017;12:1-2.
3. Center PR. Mobile fact sheet. Pew Research Center: Internet Science & Tech 2017;
4. Taylor K, Silver L. Smartphone Ownership is Growing Rapidly Around the World, but Not Always Equally. 2018. Available online: https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/
5. Lenhart A, Duggan M, Perrin A, et al. Teens, social media & technology overview 2015. Pew Research Center: Internet & American Life Project, 2015.
6. Twenge JM, Martin GN, Campbell WK. Decreases in psychological well-being among American adolescents after 2012 and links to screen time during the rise of smartphone technology. Emotion 2018;18:765-80. [Crossref] [PubMed]
7. Twenge JM, Campbell WK. Associations between screen time and lower psychological well-being among children and adolescents: Evidence from a population-based study. Prev Med Rep 2018;12:271-83. [Crossref] [PubMed]
8. Peracchia S, Curcio G. Exposure to video games: effects on sleep and on post-sleep cognitive abilities. A sistematic review of experimental evidences. Sleep Sci 2018;11:302-14. [Crossref] [PubMed]
9. Tofighi B, Abrantes A, Stein MD. The Role of Technology-Based Interventions for Substance Use Disorders in Primary Care: A Review of the Literature. Med Clin North Am 2018;102:715-31. [Crossref] [PubMed]
10. Moman RN, Dvorkin J, Pollard EM, et al. A Systematic Review and Meta-analysis of Unguided Electronic and Mobile Health Technologies for Chronic Pain-Is It Time to Start Prescribing Electronic Health Applications?. Pain Med 2019;20:2238-55. [Crossref] [PubMed]
11. McCann L, McMillan KA, Pugh G. Digital Interventions to Support Adolescents and Young Adults With Cancer: Systematic Review. JMIR Cancer 2019;5:e12071 [Crossref] [PubMed]
12. Naslund JA, Aschbrenner KA, Araya R, et al. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature. Lancet Psychiatry 2017;4:486-500. [Crossref] [PubMed]
13. Kaner EF, Beyer FR, Garnett C, et al. Personalised digital interventions for reducing hazardous and harmful alcohol consumption in community-dwelling populations. Cochrane Database Syst Rev 2017;9:CD011479 [Crossref] [PubMed]
14. Nesvåg S, McKay JR. Feasibility and Effects of Digital Interventions to Support People in Recovery From Substance Use Disorders: Systematic Review. J Med Internet Res 2018;20:e255 [Crossref] [PubMed]
15. Harari GM, Lane ND, Wang R, et al. Using Smartphones to Collect Behavioral Data in Psychological Science: Opportunities, Practical Considerations, and Challenges. Perspect Psychol Sci 2016;11:838-54. [Crossref] [PubMed]
16. Buller DB, Berwick M, Lantz K, et al. Smartphone mobile application delivering personalized, real-time sun protection advice: a randomized clinical trial. JAMA Dermatol 2015;151:497-504. [Crossref] [PubMed]
17. Romeo A, Edney S, Plotnikoff R, et al. Can Smartphone Apps Increase Physical Activity? Systematic Review and Meta-Analysis. J Med Internet Res 2019;21:e12053 [Crossref] [PubMed]
18. Ciman M, Wac K. Smartphones as Sleep Duration Sensors: Validation of the iSenseSleep Algorithm. JMIR Mhealth Uhealth 2019;7:e11930 [Crossref] [PubMed]
19. LeGrand S, Muessig KE, McNulty T, et al. Epic Allies: Development of a Gaming App to Improve Antiretroviral Therapy Adherence Among Young HIV-Positive Men Who Have Sex With Men. JMIR Serious Games 2016;4:e6 [Crossref] [PubMed]
20. Henny KD, Wilkes AL, McDonald CM, et al. A rapid review of eHealth interventions addressing the continuum of HIV care (2007–2017). AIDS and Behavior 2018;22:43-63. [Crossref] [PubMed]
21. Schoeppe S, Alley S, Rebar AL, et al. Apps to improve diet, physical activity and sedentary behaviour in children and adolescents: a review of quality, features and behaviour change techniques. Int J Behav Nutr Phys Act 2017;14:83. [Crossref] [PubMed]
22. Greer JA, Jacobs JM, Pensak N, et al. Randomized Trial of a Smartphone Mobile App to Improve Symptoms and Adherence to Oral Therapy for Cancer. J Natl Compr Canc Netw 2020;18:133-41. [PubMed]
23. Thirumalai M, Rimmer JH, Johnson G, et al. TEAMS (Tele-Exercise and Multiple Sclerosis), a Tailored Telerehabilitation mHealth App: Participant-Centered Development and Usability Study. JMIR Mhealth Uhealth 2018;6:e10181 [Crossref] [PubMed]
24. Tate EB, Spruijt-Metz D, O'Reilly G, et al. mHealth approaches to child obesity prevention: successes, unique challenges, and next directions. Transl Behav Med 2013;3:406-15. [Crossref] [PubMed]
25. Villanti AC, Johnson AL, Ilakkuvan V, et al. Social Media Use and Access to Digital Technology in US Young Adults in 2016. J Med Internet Res 2017;19:e196 [Crossref] [PubMed]
26. Laing SS, Alsayid M, Ocampo C, et al. Mobile Health Technology Knowledge and Practices Among Patients of Safety-Net Health Systems in Washington State and Washington, DC. J Patient Cent Res Rev 2018;5:204-17. [Crossref] [PubMed]
27. Akinola M, Hebert LE, Hill BJ, et al. Development of a Mobile App on Contraceptive Options for Young African American and Latina Women. Health Educ Behav 2019;46:89-96. [Crossref] [PubMed]
28. Bauermeister JA, Golinkoff JM, Muessig KE, et al. Addressing engagement in technology-based behavioural HIV interventions through paradata metrics. Current Opinion in Hiv and Aids 2017;12:442-6. [Crossref] [PubMed]
29. Bonett S, Connochie D, Golinkoff JM, et al. Paradata Analysis of an eHealth HIV Testing Intervention for Young Men Who Have Sex With Men. AIDS Educ Prev 2018;30:434-47. [Crossref] [PubMed]
30. Taylor M, Taylor A. The technology life cycle: Conceptualization and managerial implications. Int J Prod Econ 2012;140:541-53. [Crossref]
31. Horvath KJ, Ecklund AM, Hunt SL, et al. Developing Internet-based health interventions: a guide for public health researchers and practitioners. J Med Internet Res 2015;17:e28 [Crossref] [PubMed]
32. How Much Does It Cost To Build an App? 2019. Available online: http://www.howmuchtobuildanapp.io/
33. Ronquillo JG, Erik Winterholler J, Cwikla K, et al. Health IT, hacking, and cybersecurity: national trends in data breaches of protected health information. JAMIA Open 2018;1:15-9. [Crossref] [PubMed]
34. Sullivan PS, Jones J, Kishore N, et al. The Roles of Technology in Primary HIV Prevention for Men Who Have Sex with Men. Curr HIV/AIDS Rep 2015;12:481-8. [Crossref] [PubMed]
35. Huguet A, Rao S, McGrath PJ, et al. A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression. PLoS One 2016;11:e0154248 [Crossref] [PubMed]
36. Paterson C. An ecological momentary assessment of self-management in prostate cancer survivors. J Cancer Surviv 2019;13:364-73. [Crossref] [PubMed]
37. MacIntyre JM, Ruscio AC, Brede E, et al. Emotion dysregulation and negative affect: Laboratory and EMA investigations in smokers. Addict Behav Rep 2018;7:65-70. [Crossref] [PubMed]
38. Basak E, Gumussoy CA, Calisir F. Examining the Factors Affecting PDA Acceptance among Physicians: An Extended Technology Acceptance Model. J Healthc Eng 2015;6:399-418. [Crossref] [PubMed]
39. Johansson PE, Petersson GI, Nilsson GC. Nursing students' experience of using a personal digital assistant (PDA) in clinical practice - an intervention study. Nurse Educ Today 2013;33:1246-51. [Crossref] [PubMed]
40. Siegler AJ, Brock JB, Hurt CB, et al. An Electronic Pre-Exposure Prophylaxis Initiation and Maintenance Home Care System for Nonurban Young Men Who Have Sex With Men: Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2019;8:e13982 [Crossref] [PubMed]
41. Hightow-Weidman LB, Muessig K, Rosenberg E, et al. University of North Carolina/Emory Center for Innovative Technology (iTech) for Addressing the HIV Epidemic Among Adolescents and Young Adults in the United States: Protocol and Rationale for Center Development. JMIR Res Protoc 2018;7:e10365 [Crossref] [PubMed]