Quality improvement and practice-based research in sleep medicine using structured clinical documentation in the electronic medical record
Sleep Science and Practice volume 4, Article number: 1 (2020)
We developed and implemented a structured clinical documentation support (SCDS) toolkit within the electronic medical record, to optimize patient care, facilitate documentation, and capture data at office visits in a sleep medicine/neurology clinic for patient care and research collaboration internally and with other centers.
To build our SCDS toolkit, physicians met frequently to develop content, define the cohort, select outcome measures, and delineate factors known to modify disease progression. We assigned tasks to the care team and mapped data elements to the progress note. Programmer analysts built and tested the SCDS toolkit, which included several score tests. Auto scored and interpreted tests included the Generalized Anxiety Disorder 7-item, Center for Epidemiological Studies Depression Scale, Epworth Sleepiness Scale, Pittsburgh Sleep Quality Index, Insomnia Severity Index, and the International Restless Legs Syndrome Study Group Rating Scale. The SCDS toolkits also provided clinical decision support (untreated anxiety or depression) and prompted enrollment of patients in a DNA biobank.
The structured clinical documentation toolkit captures hundreds of fields of discrete data at each office visit. This data can be displayed in tables or graphical form. Best practice advisories within the toolkit alert physicians when a quality improvement opportunity exists. As of May 1, 2019, we have used the toolkit to evaluate 18,105 sleep patients at initial visit. We are also collecting longitudinal data on patients who return for annual visits using the standardized toolkits. We provide a description of our development process and screenshots of our toolkits.
The electronic medical record can be structured to standardize Sleep Medicine office visits, capture data, and support multicenter quality improvement and practice-based research initiatives for sleep patients at the point of care.
Many sleep disorders have complex symptomatology and variable evolution. Carefully documenting the severity of disease, the frequency of symptoms, impact on health and quality of life, and response to treatment is essential to guiding the efficacy and safety of disease management. With the advent of the electronic medical record (EMR), it became possible to process and store data for easy evaluation and comparison. The EMR also allows immediate access to longitudinal health information, improving the quality, safety, and efficiency of patient care (Esper et al. 2010). The EMR can provide information on therapy outcomes (Ballard et al. 2015), track quality of care (Fitzsimons et al. 2013), screen for neurological disorders (Wray et al. 2014), and improve communication between specialists and primary care physicians (Zuchowski et al. 2015).
The American Academy of Neurology (AAN) has proposed quality measures for the care of adults with neurological disease (Cheng 2010; Fountain et al. 2011; Miller et al. 2013; Odenheimer et al. 2013; England et al. 2014), and guidelines for restless legs syndrome/Willis-Ekbom disease (RLS/WED) are under development (Guidelines Projects in Process, 2019). The American Academy of Sleep Medicine recently published quality measures (Trotti et al. 2015). Other guidelines for RLS/WED have been published (Allen et al. 2014; Picchietti et al. 2015; Garcia-Borreguero et al. 2016; Winkelman et al. 2016). Few studies in sleep and neurology are incorporated into EMRs (Alexander et al. 2016; Stubberud et al. 2019; Mulvey et al. 2018). A challenge to EMR documentation is that clinical data are not captured discretely, making it difficult to report performance and assess quality improvement opportunities. Structured clinical documentation support offers a solution to this problem, allowing data to be captured discretely, and making it easier to report performance.
Sleep medicine is also challenged by a lack of comparative effectiveness research. There are several approved treatments for common sleep disorders, such as RLS/WED, but it is unknown which ones are superior in efficacy and tolerability and for which patient subgroups. Traditional clinical trials enroll selected patients, use surrogate measures, follow patients for short periods, and generalize poorly to clinical practice (Tunis et al. 2003; Vickers and Scardino 2009; Patsopoulos 2011; Chalkidou et al. 2012). Data captured in the EMR could be used to identify eligible patients, assign treatments, and measure outcomes at the point of care (Staa et al. 2012). To address these unmet needs, we participated in a department-wide project addressing quality improvement and practice-based research in neurology by building a structured clinical documentation support (SCDS) and clinical decision support (CDS) toolkit within our EMR. We present a description of our process for developing a sleep toolkit to support Best Practices and provide some basic descriptive data demonstrating our discrete data collection. As an example, we present RLS/WED as one of the many sleep disorders for which our toolkit collects data. RLS/WED serves as a useful example because it is well defined, has a genetic basis, and poorly understood etiology requiring further study.
The NorthShore University HealthSystem (NorthShore) Sleep program includes fourteen physicians (six pulmonologists and eight neurologists). The toolkit is used by sleep specialists from the Department of Neurology practicing at four hospitals, and at six outpatient sites in the northern suburbs of Chicago, IL.
To build our SCDS toolkit, we had to develop the content of the note, optimize the EMR (Epic Systems Corporation) to reflect the content, and implement the toolkit in a test environment. We previously illustrated our seven-stage, stepwise progression of quality improvement and practice-based research using the EMR. Briefly, the department’s stepwise progression consists of seven steps: 1. Development and implementation of structured clinical documentation support (SCDS) EMR tools; 2. Enrollment reports of subjects encountered; 3. Data quality reports to identify missing data; 4. Descriptive reports of group characteristics such as patient reported and physcian assessment measures; 5. Quality improvement projects (baseline data); 6. Quality improvement projects (using clinical decision support tools built into the EMR to hardwire patient safety and improved outcomes); 7. Dissemination of tools and sharing of data via the NPBRN (Maraganore et al. 2016).
The sleep neurologists met every 2 weeks to standardize the progress note content to support Best Practices in treating patients with sleep disorders. We reviewed the pertinent medical literature, AAN guidelines (Guidelines, 2019), National Institute of Neurological Diseases and Stroke Common Data Elements (NINDS/NIH, 2015), the American Academy of Sleep Medicine (AASM) (Aurora et al. 2012), and the International Restless Legs Syndrome Study Group (IRLSSG) guidelines (Garcia-Borreguero et al. 2013). We included the Center for Epidemiologic Studies Depression (CES-D) scale (Radloff 1977), General Anxiety Disorder 7 item (GAD-7) scale (Spitzer et al. 2006), Pittsburgh Sleep Quality Index (PSQI) (Buysse et al. 1989), Epworth Sleepiness Scale (ESS) (Johns 1991), Insomnia Severity Index (ISI) (Morin 1993; Bastien et al. 2001), and International Restless Legs Syndrome Study Group Rating Scale (IRLSRS) (Walters et al. 2003). We designed workflows (the order and assignment of tasks to a care team (medical assistant, nurse, sleep neurologist)) according to the level of training and progress notes (the order and layout in which the content would write). Patients may also fill in their questionnaires online ahead of the office visit through our online patient portal. This process does not lengthen the office visit and ensures collection of all the necessary data. We limited the medical assistant and/or nurse assessments to 15 min each and the neurologist assessment to 60 min.
After deciding on the content, we conducted meetings with programmers from NorthShore’s EMR Optimization team every 2 weeks. They built an SCDS and CDS toolkit that included navigators (a sidebar index of processes to choose from), electronic forms (which had the ability to auto-score and auto-interpret), and summary flow sheets. We included free text fields to allow for additional information. All patients with sleep disorders were evaluated using the SCDS. The SCDS and CDS toolkit were also designed to promote patient safety (by alerting physicians of untreated anxiety or depression and directing them to a mental health orders set).
Finally, we tested the SCDS and CDS toolkit in the EMR’s development environment, made revisions, and moved the toolkit to the EMR’s production environment. The project team continued to meet every 2 weeks to make revisions based on patient encounters. Our toolkits focused on outpatient visits using the EMR’s ambulatory environment. (Screenshots of the SCDS and CDS tools appear in Additional file 1).
After toolkit implementation, we met every 2 weeks with programmers specialized in extracting, transforming, and loading data from the EMR’s data repository to sleep-specific data marts in NorthShore’s Enterprise Data Warehouse (EDW). The EDW programmers created enrollment reports for tracking patients and produced data quality reports indicating which required data was missing from office visits. These data quality reports were distributed to the care team monthly. Data not cleaned within 3 months were archived as permanently missing, and those data were no longer listed in subsequent reports. The care team learned where they were error prone from the data quality reports, and they remediated their use of the toolkits. When systematic errors occurred for many providers, the teams had the opportunity to improve their use of the toolkits or to request optimizations or a change in data requirements. The monthly reports produced only a few or no data checks per provider once the project was established.
To demonstrate what our toolkit effectively captures, as of May 1, 2019, we have evaluated 18,105 patients with our Sleep toolkit, including up to 836 fields of data per office visit (example screenshots in Additional file 1). Our toolkit generates monthly reports, such as Table 1, which shows the sleep diagnoses of patients seen at initial visit. For each of these patients, we collected detailed information related to their demographics. We also collected detailed information related to their specific symptoms, sleep medication use, and family history of sleep related disorders. An example of a detailed, descriptive report related to these patients is shown in Additional file 2. Lastly, we collected several score test measures that can be evaluated longitudinally as we follow these patients (Additional file 3).
We employed an SCDS and CDS toolkit built within our EMR for quality improvement and outcomes research. We have demonstrated the feasibility of incorporating these toolkits into clinical care and capturing high-level patient data. Additionally, we capture longitudinal data on patients by continued use of the toolkits at annual visits. Understanding how measures of patient’s well-being change over time is informative for planning appropriate care across disciplines.
As a clinical next step, we will use the data collection to aid in decisions at the point of care. Specifically, we are creating electronic pop-up boxes (best practice advisories [BPAs]) that fire at the point of care whenever a quality improvement opportunity is identified, based on data captured by the EMR. We have already implemented these to detect and manage depression or anxiety in the sleep visit. We implemented a BPA based on a patient’s CES-D and GAD-7 score. If the patient screens positive, an alert ‘pops’ up requiring the physician to place an order (medication or referral) or defer (and a comment as to the reason is required). As a quality measure, we are assessing the frequency of antidepressant or anxiolytic medication order and/or psychiatry referral order for those that screen positive. We are planning additional BPAs, specific to sleep patients. One application could be if a patient has RLS/WED and the ferritin level has not been checked in the past year, a BPA will fire and present a mouse-click option to place an order or defer (which will cascade and prompt selection of a reason for deferral). After implementing the BPA, we will track improvements in documentation of ferritin levels versus benchmark data. In another application, we could screen for impulse control disorder and augmentation. If a patient screens positive, we could alter management of dopamine agonists.
Several research applications could be undertaken using this routinely collected clinical data. We are currently conducting a DNA biobanking study, in which enrollment is prompted by a BPA that scans the EMR for eligibility criteria. Eligible patents are identified automatically at the point of care. When appropriate, they can then be consented into the study. DNA information in addition to clinical data will allow novel studies of biomarkers and clinical features as it relates to disease progression, treatment response and outcomes. Another future opportunity is to conduct pragmatic trials using the EMR. For example, we could compare a dopamine agonist versus an alpha-2 delta ligand to evaluate outcomes such as RLS/WED severity, augmentation rates, and effects on scales such as the CES-D and ESS. BPAs would trigger enrollment of eligible subjects and prompt random assignment of equivocal medications, at the point-of-care, using the data captured by our SCDS and CDS toolkit. We have already implemented this design successfully for other projects in the Department of Neurology.
The Department of Neurology at NorthShore University HealthSystem (NorthShore) includes 40 neurologists practicing at 4 hospitals and 8 outpatient sites in the north suburbs of Chicago, IL. The neurologists include generalists and subspecialists in epilepsy, neurodegenerative disorders, multiple sclerosis, neuromuscular disorders, neuro-oncology, sleep disorders, and stroke. Our toolkit creates an efficient division of labor among the medical staff that renders the process of obtaining all necessary data more clear and efficient.
Our toolkit also facilitates collaboration within our eleven neurology subspecialties that all have access to the data. The data collected can also be shared with other specialties and centers if they are willing to create a standardized toolkit. While our toolkit is currently limited to neurology subspecialities that use the same EMR, potential future applications include expanding to other subspecialties and EMR platforms.
To improve quality and conduct novel practice-based research, we are also sharing our sleep toolkit and collaborating with other departments of neurology that use the same EMR (Epic). With funding support from the Agency for Healthcare Research and Quality, the NorthShore Neurological Institute created a Neurology Practice-Based Research Network (NPBRN) using the EMR. All 15 participating sites have agreed to implement SCDS and CDS toolkits for up to 11 neurological indications (including sleep disorders), and to share de-identified data collaboratively. The NorthShore site will manage the data and provide monthly participation and data quality reports as well as quarterly descriptive cohort and quality improvement reports, which will allow us to analyze the data for demographic and clinical characteristics in diverse patient populations and to conduct multi-center research studies.
While there are many benefits to incorporating SCDS and CDS into clinical practice, there are also some limitations. The structured questions may not be asked in the same manner by different examiners. The patients’ answers may be subjective or recorded in a biased manner by the examiner. Additionally, patients’ interpretation of self-entered questionnaires may vary. There is a learning curve to using the toolkits and they require ‘standardized’ answers. However, there is always the opportunity to enter free text. In our experience, once fully implemented, the toolkits do not require additional face-to-face time and include features that can increase the efficiency of the office visit, such as the previously mentioned clearly assigned tasks to the medical staff.
Over time, we have rendered the toolkit more efficient by eliminating some of our initial questionnaires that we found redundant and unnecessary, shortening the data collection process. To address the challenge of data collection prior to the physician visit, we streamlined the process, assigning the medical staff to specific data collection tasks in addition to having the patient fill out some questionnaires online prior to coming in the office. These changes have improved our workflow, while ensuring that our data collection process is complete and efficient.
In conclusion, SCDS toolkits, as well as CDS features, may be used to standardize a sleep medicine office visit. Workflows may be optimized to deliver patient care, and progress notes transformed from unstructured text to structured documents with discrete data points, easily captured in the EMR. The toolkits can be built to enable physicians to approach common diseases according to AAN quality guidelines and support quality initiative projects. The toolkits can also aid in the enrollment of research by identifying eligible patients at the point of care. Furthermore, the data can be queried for research questions such as whether the IRLS score predicts whether a patient will be treated or whether the ferritin level predicts benefit from treatment. Overall, we anticipate that these SCDS and CDS toolkits will improve the quality of sleep medicine care and facilitate practice-based research.
Availability of data and materials
American Academy of Neurology
American Academy of Sleep Medicine
Best practice advisory
Clinical decision support
Center for Epidemiologic Studies Depression scale
Enterprise Data Warehouse
Electronic Medical Record
Epworth Sleepiness Scale
U.S. Food and Drug Administration
General Anxiety Disorder 7 item scale
International Restless Legs Syndrome Study Group Rating Scale
International Restless Legs Syndrome Study Group
Insomnia Severity Index
National Institute of Neurological Diseases and Stroke Common Data Elements
Neurology Practice-Based Research Network
Pittsburgh Sleep Quality Index
Restless Legs Syndrome/Willis Ekbom Disease
Structured clinical documentation support
Single nucleotide polymorphisms
Alexander M, Ray MA, Hébert JR, Youngstedt SD, Zhang H, Steck SE, et al. The National Veteran Sleep Disorder Study: descriptive epidemiology and secular trends, 2000-2010. Sleep. 2016;39(7):1399–410.
Allen RP, Picchietti DL, Garcia-Borreguero D, Ondo WG, Walters AS, Winkelman JW, et al. Restless legs syndrome/Willis-Ekbom disease diagnostic criteria: updated international restless legs syndrome study group (IRLSSG) consensus criteria--history, rationale, description, and significance. Sleep Med. 2014;15(8):860–73.
Aurora RN, Kristo DA, Bista SR, et al. The treatment of restless legs syndrome and periodic limb movement disorder in adults—an update for 2012: practice parameters with an evidence-based systematic review and meta-analyses. Sleep. 2012;35(8):1039–62. https://doi.org/10.5665/sleep.1988.
Ballard DW, Kim AS, Huang J, et al. Implementation of computerized physician order entry is associated with increased thrombolytic Administration for Emergency Department Patients with acute ischemic stroke. Ann Emerg Med. 2015;66(6):601–10. https://doi.org/10.1016/j.annemergmed.2015.07.018.
Bastien CH, Vallières A, Morin CM. Validation of the insomnia severity index as an outcome measure for insomnia research. Sleep Med. 2001;2(4):297–307 http://www.ncbi.nlm.nih.gov/pubmed/11438246.
Buysse DJ, Reynolds CF 3rd, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh sleep quality index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28(2):193–213.
Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials. 2012;9(4):436–46. https://doi.org/10.1177/1740774512450097.
Cheng EM. Quality improvement in neurology : AAN Parkinson disease quality measures report of the quality measurement and reporting subcommittee of. Am Acad Neurol. 2010;1(10):2021–7. https://doi.org/10.1212/WNL.0b013e3181ff96dd.
England JD, Franklin G, Gjorvad G, et al. Quality improvement in neurology: distal symmetric polyneuropathy quality measures. Neurology. 2014;82(19):1745–8. https://doi.org/10.1212/WNL.0000000000000397.
Esper GJ, Drogan O, Henderson WS, Becker A, Avitzur O, Hier DB. Health information technology and electronic health records in neurologic practice. Neurol Clin. 2010;28(2):411–27. https://doi.org/10.1016/j.ncl.2009.11.014.
Fitzsimons M, Dunleavy B, O’Byrne P, et al. Assessing the quality of epilepsy care with an electronic patient record. Seizure. 2013;22(8):604–10. https://doi.org/10.1016/j.seizure.2013.03.002.
Fountain NB, Van Ness PC, Swain-Eng R, Tonn S, Bever CTJ. Quality improvement in neurology: AAN epilepsy quality measures: report of the quality measurement and reporting Subcommittee of the American Academy of neurology. Neurology. 2011;76(1):94–9. https://doi.org/10.1212/WNL.0b013e318203e9d1.
Garcia-Borreguero D, Kohnen R, Silber MH, et al. The long-term treatment of restless legs syndrome/Willis-Ekbom disease: evidence-based guidelines and clinical consensus best practice guidance: a report from the international restless legs syndrome study group. Sleep Med. 2013;14(7):675–84. https://doi.org/10.1016/j.sleep.2013.05.016.
Garcia-Borreguero D, Silber MH, Winkelman JW, Hogl B, Bainbridge J, Buchfuhrer M, et al. Guidelines for the first-line treatment of restless legs syndrome/Willis-Ekbom disease, prevention and treatment of dopaminergic augmentation: a combined task force of the IRLSSG, EURLSSG, and the RLS-foundation. Sleep Med. 2016;21:1–11.
Guidelines. American Academy of Neurology. https://www.aan.com/Guidelines/. [Accessed 22 Mar 2019].
Guidelines Projects in Process. American Academy of Neurology. Available at: https://www.aan.com/Guidelines/home/UnderDevelopment; [Accessed 22 Mar 2019].
Johns MW. A new method for measuring daytime sleepiness: the Epworth sleepiness scale. Sleep. 1991;14(6):540–5.
Maraganore D, Frigerio R, Kazmi N, Meyers S, Sefa M, Walters S, et al. Quality improvement and practice-based research in neurology using the electronic medical record. Neurol Clin Pr. 2016;5(October):74–8. https://doi.org/10.1212/CPJ.0000000000000176.
Miller RG, Brooks BR, Swain-eEng RJ, et al. Quality improvement in neurology : amyotrophic lateral sclerosis quality measures : report of the quality measurement and reporting Subcommittee of the American Academy of neurology quality improvement in neurology : amyotrophic lateral sclerosis quality. Neurology. 2013;81:2136–40. https://doi.org/10.1212/01.wnl.0000437305.37850.f9.
Morin CM. Insomnia: psychological assessment and management. New York: Guilford Press; 1993.
Mulvey M, Veerapandiyan A, Marks D, Ming X. Electronic health record cue identifies epilepsy patients at risk for obstructive sleep apnea. Neurol Clin Pract. 2018;8(6):468–71.
NINDS/NIH. NINDS common data elements. NIH Website. 2015. https://commondataelements.ninds.nih.gov/#page=Default. [Accessed 22 Mar 2019].
Odenheimer G, Sanders AE, Swain-eEng RJ, et al. Quality improvement in neurology: dementia management quality measures. Neurology. 2013;81(17):1545–9. https://doi.org/10.1212/WNL.0b013e3182a956bf.
Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217–24.
Picchietti DL, Hensley JG, Bainbridge JL, Lee KA, Manconi M, McGregor JA, et al. Consensus clinical practice guidelines for the diagnosis and treatment of restless legs syndrome/Willis-Ekbom disease during pregnancy and lactation. Sleep Med Rev. 2015;22:64–77.
Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Measur. 1977;1(3):385–401. https://doi.org/10.1177/014662167700100306.
Spitzer RL, Kroenke K, Williams JBW, Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7. https://doi.org/10.1001/archinte.166.10.1092.
Staa TP, Goldacre B, Gulliford M, et al. Pragmatic randomized trials using routine electronic health records: putting them to the test. BMJ. 2012;344(February):e55. https://doi.org/10.1136/bmj.e55.
Stubberud AB, Moon RE, Morgan BT, Goode VM. Using the electronic medical record to improve preoperative identification of patients at risk for obstructive sleep apnea. J Perianesth Nurs. 2019;34(1):51–9.
Trotti LM, Goldstein CA, Harrod CG, et al. Quality measures for the care of adult patients with restless legs syndrome. J Clin Sleep Med. 2015;11(3):293–310. https://doi.org/10.5664/jcsm.4550.
Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624–32. https://doi.org/10.1001/jama.290.12.1624.
Vickers AJ, Scardino PT. The clinically-integrated randomized trial: proposed novel method for conducting large trials at low cost. Trials. 2009;10:14. https://doi.org/10.1186/1745-6215-10-14.
Walters AS, LeBrocq C, Dhar A, et al. Validation of the international restless legs syndrome study group rating scale for restless legs syndrome. Sleep Med. 2003;4(2):121–32.
Winkelman JW, Armstrong MJ, Allen RP, Chaudhuri KR, Ondo W, Trenkwalder C, et al. Practice guideline summary: treatment of restless legs syndrome in adults: table. Neurology. 2016;87(24):2585–93.
Wray LO, Wade M, Beehler GP, Hershey LA, Vair CL. A program to improve detection of undiagnosed dementia in primary care and its association with healthcare utilization. Am J Geriatr Psychiatry. 2014;22(11):1282–91. https://doi.org/10.1016/j.jagp.2013.04.018.
Zuchowski JL, Rose DE, Hamilton AB, et al. Challenges in referral communication between VHA primary care and specialty care. J Gen Intern Med. 2015;30(3):305–11. https://doi.org/10.1007/s11606-014-3100-x.
The authors thank the medical assistants, nurses, neurologists, EMR Optimization and Enterprise Data Warehouse programmers, administrators, and research personnel at NorthShore University HealthSystem who contributed to the quality improvement and practice-based research initiative using the EMR. The authors also thank Hannah Moulthrop, MS, for her assistance with editing and formatting the manuscript. Finally, the authors thank the neurology patients who inspire us to improve quality and to innovate our clinical practice every day.
The authors acknowledge the generous funding support of the Auxiliary of NorthShore University HealthSystem and David Zucker with respect to the initial building of electronic medical record (EMR) toolkits. The authors also acknowledge funding support from the Agency for Healthcare Research and Quality (R01HS024057).
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Screenshots of SDCS toolkit within the EMR,© 2018 EPIC Systems Corporation, used with permission
Complete descriptive report of patients at initial visit.
Score tests distributions, correlations, and principal component analysis of patients at initial visit.
About this article
Cite this article
Maraganore, D.M., Freedom, T., Simon, K.C. et al. Quality improvement and practice-based research in sleep medicine using structured clinical documentation in the electronic medical record. Sleep Science Practice 4, 1 (2020). https://doi.org/10.1186/s41606-019-0038-2
- Electronic medical record
- Sleep disorders
- Structured clinical documentation support
- Clinical decision support
- Best practices