RAPID PROCESS IMPROVEMENT WORKSHOPS: AN ECONOMIC EVALUATION AND SYSTEMATIC STUDY OF EMPLOYEE EXPERIENCE IN A HEALTHCARE SETTING by James Gregory Chan B.A. (Hons.), Simon Fraser University, 1999 M.A., The University of British Columbia, 2004 DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN HEALTH SCIENCES UNIVERSITY OF NORTHERN BRITISH COLUMBIA August 2018 © James Chan, 2018 ii Abstract Many studies purport that Lean is a useful approach to improving quality in healthcare, however, there are calls for better research designs to evaluate its effectiveness. This study used a mixed-methods approach to investigate Lean Rapid Process Improvement Workshops (RPIWs) applied to surgical services in a large health system. The main objective was to conduct an economic evaluation using the Return on Investment (ROI) method. Quantifying improvements at the RPIW event level was not feasible, so the analysis defaulted to examining the cumulative effects of RPIWs at the sector level. The quantitative results did not produce sufficient evidence to claim the outcomes justify the investments. The study went beyond ROI by statistically analyzing the effects of RPIWs on the performance measures of surgical volumes, sick time, and overtime. Temporal analysis was used to examine the performance measures within four sites considered in this study (two RPIW sites and two no-intervention control sites). As well, the intervention and control sites were compared in terms of performance differences using the same measures. Findings from these analyses were generally statistically non-significant or not conclusive. To capture the intangible effects of the RPIWs, key informant interviews explored the richness of participants’ experiences while a survey gathered data on experience and engagement from a larger number of participants. Respondents to the interviews and survey indicated that RPIWs are a desirable method for improving healthcare. The study yielded six thematic enablers to engagement, while four themes representing barriers to engagement were also identified. Ten themes also emerged as recommendations for making Lean RPIWs more engaging. This study demonstrates the difficultly of empirically calculating the ROI of RPIWs, which leads us to challenge any unfounded claims of monetary benefit from iii improvement initiatives. It also stresses the importance of understanding employees’ experience and the process of engagement, which is critical for any healthcare organization that wishes to improve its services, client care, and overall performance. The findings invite us to turn our attention away from discussions of financial ROI and toward other evidence that helps us understand how Lean can provide value in healthcare. iv TABLE OF CONTENTS Abstract ii Table of Contents iv List of Tables vii List of Figures viii Acknowledgements x Chapter One: Introduction Defining Quality in Healthcare A Brief History of Quality Improvement in Healthcare Lean in Healthcare Lean principles and tools Approaches to Lean implementation Purpose of the Study 1 1 3 6 8 12 14 Chapter Two: Literature Review Search Strategy Lean in Healthcare Literature Lean Studies in Surgical Services Studying Lean in British Columbia The Provincial Lean Network Economic evaluation Employee experience and engagement Summary 16 17 20 24 28 29 30 33 38 Chapter Three: The Research Context 40 Chapter Four: Methods Study Approach Economic Evaluation at the Event Level Research design Data requirements Costing analysis RPIW benefits Economic Evaluation at the Sector Level Research design Data collection Data analysis 52 52 54 54 58 58 61 64 64 67 68 v Employee Experience and Engagement Research design Employee Experience and Engagement at the Event Level Participants Data collection Data analysis Organizing Phase I Organizing Phase II Analysis Phase Quotations Employee Experience and Engagement at the Sector Level Participants Data collection Instrument Data analysis Chapter Five: Results Economic Evaluation at the Sector Level Return on investment Statistical analyses Statistical analyses of surgical volumes Statistical analyses of end-of-shift overtime Statistical analyses of sick time utilization Statistical analyses of patient outcome measures Direct comparisons between intervention and control groups on performance Employee Experience and Engagement at the Event Level Domain 1: Definition of engagement Domain 2: Participants’ impression of RPIWs Domain 3: Participants’ experience in the RPIWs Domain 4: Factors enabling engagement in RPIWs Domain 5: Barriers to engagement in RPIWs Domain 6: Recommendations to make RPIWs more engaging Employee Experience and Engagement at the Sector Level Response rate Demographics Engagement questions Experience questions Other experiences with Lean Results from open-ended questions 73 73 75 75 76 77 77 79 79 79 82 82 83 83 84 86 86 86 88 89 99 101 102 103 107 109 110 111 112 115 118 122 122 123 125 127 128 130 vi Chapter Six: Discussion Validity of Research Design Findings from Economic Analysis Qualitative Findings: The Soft Side of Lean Triangulation of Qualitative and Quantitative Findings The Use of Lean as a Tool Versus Philosophy Methodological Implications Limitations Future Directions 134 134 135 137 141 142 145 147 149 Chapter Seven: Conclusion 155 References 159 Appendix A: Eight Forms of Non-Value Added Waste 174 Appendix B: Metrics Framework for Lean in BC 176 Appendix C: RPIW Costing Analyses 181 Appendix D: Phillips Return on Investment Model 194 Appendix E: Lean Study Recruitment Poster 195 Appendix F: Lean Study Interview Consent Form 196 Appendix G: Lean Study Interview Guide 200 Appendix H: Lean Study On-Line Survey 203 Appendix I: Interrupted Time Series Estimates 217 Appendix J: Declarations 221 vii List of Tables Table 1. Approaches to Lean Implementation 13 Table 2. RPIWs Conducted at the HA Sites 2013–2014 57 Table 3. Example of the Costing Analysis 61 Table 4. Monetized Results of the RPIWs 87 Table 5. t-test Results Comparing Surgical Volumes by Site and Period 90 Table 6. Results from Levene’s Test for Equality of Variance 99 Table 7. t-test Results Comparing End-of-Shift Overtime by Site and Period 101 Table 8. t-test Results Comparing Sick Time Utilization by Site and Period 102 Table 9. Results of Statistical Tests for Outcome Measures 103 Table 10. Direct Comparisons Between Intervention and Control Sites on Performance Measures Table 11. 104 t-test Results Comparing Intervention and Control Sites on Performance Differences 105 Table 12. Themes for Definitions of Engagement 109 Table 13. Themes of Participants’ Impressions of RPIWs 110 Table 14. Themes of Participants’ Experiences in the RPIWs 112 Table 15. Themes of Enabling Factors for Engagement in RPIWs 113 Table 16. Themes of Barriers to Engagement in RPIWs 116 Table 17. Themes of Recommendations to Make RPIWs More Engaging 119 Table 18. Respondents by Profession 124 viii List of Figures Figure 1. Current State Value Stream Map for an Emergency Department Process 9 Figure 2. Continuous Improvement Through Lean Kaizen Events 11 Figure 3. Literature Review Flow Chart 19 Figure 4. Lean Evaluation Framework 29 Figure 5. Distinguishing Characteristics of Healthcare Evaluation 31 Figure 6. Quality Healthcare Workplace Model 36 Figure 7. Lean Implementation Journey at the HA 42 Figure 8. Logic Model for Lean in Surgical Services at the HA 44 Figure 9. High-Level Value Stream Map for Surgical Services at Site 1 47 Figure 10. High-Level Value Stream Map for Surgical Services at Site 2 48 Figure 11. Evaluation Logic of the Present Study 53 Figure 12. Example of Data Collected in the Target Progress Report 62 Figure 13. Sequential RPIW Intervention Periods 69 Figure 14. Employee Engagement and Experience Research Methods 74 Figure 15. Qualitative Data Analysis Procedure 81 Figure 16. ITS Results Pre to Intervention Period Site 1 92 Figure 17. ITS Results Pre to Intervention & Post-Intervention Periods Site 1 92 Figure 18. ITS Results Pre to Intervention Period Site 2 94 Figure 19. ITS Results Pre to Intervention & Post-Intervention Periods Site 2 94 Figure 20. ITS Results Pre to Intervention Period Site 1 vs Site 3 96 Figure 21. ITS Results Pre to Intervention & Post-Intervention Periods Site 1 vs Site 3 96 Figure 22. ITS Results Pre to Intervention Period Site 2 vs Site 4 98 ix Figure 23. ITS Results Pre to Intervention & Post-Intervention Periods Site 2 vs Site 4 98 Figure 24. Sampling Results for Employee Experience and Engagement Survey 123 Figure 25. Results from Engagement Questions 126 Figure 26. Results from Experience Questions 128 Figure 27. Results from Other Experience Questions 129 x Acknowledgements A research endeavor of this magnitude cannot be successfully undertaken without the support and assistance of many people. This study is the result of a multi-year journey that spans several jurisdictions and many networks of individuals and communities. Although I cannot describe all the linkages and synergies I have encountered over the years, I would like to express my sincere gratitude to the following people who have significantly contributed to my journey and this study. First and foremost, I am greatly indebted to my Supervisor, Dr. Jalil Safaei, who provided me with ongoing leadership and direction since the inception of this study. This study would not have been possible if not for Dr. Safaei’s knowledge, expertise, perseverance, patience, coaching, and encouragement to carry out the project to completion. Dr. Safaei has been a true mentor and inspiration to me! I am also appreciative of my Committee, Dr. Thomas Rotter and Dr. Balbinder Deo, who regularly reviewed the project and guided its progress. Dr. Rotter deserves extra special thanks for his insightful recommendations to improve the study based on his expansive expertise on this topic. I also owe thanks to Dr. Rotter for including me in his community of Lean researchers. I thank this group of esteemed colleagues, from whom I have learned a tremendous amount, and Christopher Plishka in particular for his leadership of this international group. I am honored to have had Dr. Erika Penz as the external examiner of this Dissertation. Key figures in the School of Health Sciences at the University of Northern British Columbia are thanked for their ongoing encouragement, namely Dr. Shannon Wagner and Dr. Hendry Harder. I would like to thank those who inspired me early in my journey of studying Lean, including Sean Hardiman, Don Clark, John Whelton, and Jane Bishop. Many others deserve credit for their support and encouragement throughout this study: Erin McGarvey, Renee Caillier, Bonnie Urquhart, Fraser Bell, Suzanne Johnson, Jeanette Foreman, Jennifer Mackenzie, Ellen Chesney, Dr. Stuart MacLeod, Graham Worsley, Jerry Weber, Susan Brown, Holly Buhler, Debby Kinakin, Dan Goughner, Kari Grant, Angela Trif, Linda Comazzetto, Diane Goosens, Sherri Lampman, Wendy Petillion, Dr. Graham Lowe, Heather Dawson, and of course, Steven Lewis. I would like to thank my family members, especially my beloved Heather Bowes, who provided me with the ultimate support that I needed and saw me through countless hours of devotion to this project. I also appreciate the support I received from my sister, Deanna Kimberly Chan, and my cousin Beverly Ang. While space limitations will not permit me to name the many others who have helped me in some shape or form along the way, I am truly grateful to everyone—I couldn’t have done it without you! 1 Chapter One: Introduction The use of Lean as a method for making improvements in healthcare has been on the rise since transitioning from its origins in the automotive manufacturing industry over two decades ago. Since then, many healthcare organizations have adopted Lean as a method of quality improvement to varying degrees. Correspondingly, a number of companies have arisen that offer Lean consultancy services and implementation tools to healthcare organizations that aspire to use Lean in hopes that it will be beneficial. Research in this area has also proliferated, although relatively few studies attempt to thoroughly examine the benefits of Lean in healthcare. This study attempts to go beyond anecdotal accounts of Lean’s success and address some gaps in the scientific literature on this topic. This chapter begins with defining quality in healthcare, since Lean has the potential to positively impact several aspects of quality as we will see. A brief history of quality improvement in healthcare follows, to provide the context for examining Lean as contender for improving quality. The focus will then turn to Lean as it is applied in healthcare and its main principals will be presented along with a summary of techniques that make up the Lean approach. While it can be said that many improvement methods share tools and techniques, it will become clear that Lean is unique, both in terms of its specific techniques and the mindset that it promotes. Defining Quality in Healthcare Before we can examine attempts to improve healthcare, it is necessary to establish a working definition of what “quality” in healthcare connotes. For example, the Institute of Medicine (2001) provides a succinct narrative definition of quality: “Quality is defined as the degree to which services and treatment increase the likelihood of desired outcomes and are 2 consistent with current professional knowledge” (p. 23). On the other hand, an important manual from the British National Health Services (NHS) defines quality as “…healthcare that is safe, effective, patient-centered, timely, efficient, and equitable” (Boaden, Harvey, Moxham, & Proudlove, 2008, p. 10). The issue of defining quality is important, especially when we consider that some approaches to quality improvement have historically been characterized by a series of fragmented initiatives that lack coherence (e.g., audit programs, setting national standards in healthcare, promoting self-regulation for clinicians, etc.; Whitty, 1998). Whitty argues that quality improvement approaches in healthcare should ultimately benefit patients, and that a broad service framework for quality improvement should adopt a wide-ranging approach that addresses whole services and not simply administrative or clinical issues exclusively. Whitty (1998) makes a subtle but important point. It is crucial to consider the difference between improving care provision and improving the health of consumers of services. The Institute for Healthcare Improvement, which is the leading national not-for-profit organization entirely devoted to improving health systems in the United States, warns of the distinction between improving the patient’s experience of care without improving health (Institute for Healthcare Improvement, 2009). The Institute’s approach to quality improvement, therefore, is based on what is known as the Triple Aim. This is a philosophy that seeks to optimize performance along three dimensions: 1) the health of a defined population, 2) the experience of care for individuals in that population, and 3) the cost per capita of providing care for that population. The Institute invites us to consider the complex dynamics of these three dimensions, and invites organizations to become involved in the Triple Aim by adopting these dimensions as part of core organizational strategies. 3 A Brief History of Quality Improvement in Healthcare The history of quality improvement in healthcare can be traced to the quality movement in operations management in the manufacturing industry. While a full account of the rich history of quality improvement in industry is beyond the scope of this discussion, there were a number of key figures that heavily influenced this movement (e.g., Shewhart, 1931) as well as a range of academic disciplines that made contributions (e.g., industrial engineering, operations research, organizational behaviour, human resource management, services marketing, etc.; Boaden et al., 2008). Quality improvement in healthcare has its own definitive history of evolution and key figures. One of these is Avedis Donabedian, who has left a lasting legacy for conceptualizing healthcare quality (Ayanian & Markel, 2016). In his landmark article, Donabedian (1966/2005) emphasized the need to measure quality in both technical management of illness and patient outcomes. The emphasis on physician practice is a key birthplace for the study of quality in healthcare. Quality in healthcare is rooted in the craft work of the medical profession, where quality was seen as largely dependent on the skill of individual craftspeople (e.g., physicians). This position assumes that the competence of individual practitioners is the major contributor to high quality care, and as a corollary, the assessment of quality naturally shifts to focus on clinical practice as opposed to operational processes (Boaden et al., 2008). To highlight the attention paid to clinical activities in healthcare-related quality improvement, several reforms that have taken place in the United Kingdom are illustrative. For example, during the 1980s and 1990s, national and regional initiatives were launched that required medical professionals to examine their practices through clinical audit, and 4 investigate procedures used for diagnosis, care provision, and treatment—paying particular attention to resource allocation and patient outcomes (Madhok, 2002). In the latter 1990s, these mandated clinical audits designed to promote managed competition were later replaced by the introduction of the clinical governance concept. Where clinical audits focused only on care providers, clinical governance mandated care providers, managers, and senior officials to work together toward “…continuously improving the quality of their services and safeguarding high standards of care by creating an environment in which excellence in clinical care will flourish” (Madhok, 2002, p. 78). In his 2008 NHS Next Stage Review, Lord Ara Darzi (a leading surgeon in the United Kingdom) recommended bettering healthcare services by: 1) providing better benchmarking performance data to assist clinicians with their improvement efforts, 2) introducing pay-forperformance as an incentive for improving clinical performance in hospitals, and 3) requiring publication of annual quality reports (as cited in Thorlby & Maybin, 2010, p. 10). By way of Lord Darzi’s example, it is clear that approaches to healthcare improvement have also focussed on operations and associated processes in addition to clinical practice. Perhaps the most notable figure who has shifted the focus away from clinical work and toward an industry-based, organizational change quality improvement paradigm is American physician Don Berwick. Observing that the results of clinical audits were failing to achieve meaningful change, Berwick advocated for increased process analysis, teamwork, and guideline development, in addition to health practitioners advancing their clinical skills (Boaden et al., 2008). In 1986, Berwick established the National Demonstration Project on Quality Improvement in Healthcare—the precursor to the Institute for Healthcare 5 Improvement, which is considered by many to be the most influential organization for the improvement of healthcare in the United States (Institute for Healthcare Improvement, n.d.). One of the Institute’s most effective methods of promoting large-scale improvement in healthcare is known as the “Breakthrough Series Collaborative” model (Institute for Healthcare Improvement, 2003). These initiatives are characterized by multidisciplinary teams from various healthcare departments or organizations joining forces for several months to work in a structured way to improve their services (Institute for Healthcare Improvement, 2003). Typically, in-person meetings (called Learning Sessions) are held, wherein teams receive training from clinical experts, showcase progress, and share lessons learned. In between Learning Sessions are Action Periods, where teams return to their home facilities and trial small tests of change prior to fully implementing new ways of operating. Some examples of Breakthrough Series Collaboratives undertaken in healthcare include the Cancer Collaborative, the Primary Care Collaborative, the Critical Care Program, and the Coronary Heart Disease Collaborative (Wilcock, Brown, Bateson, Carver, & Machin, 2003). Breakthrough Series Collaboratives use a plethora of proven techniques and tools for achieving improvements (Boaden et al., 2008). A few of these methods are:  Plan-Do-Study-Act (PDSA) cycles: An approach based on the scientific method used for trialing small tests of change that eventually produce significant change.  Statistical Process Control: A statistical technique for measuring processes with the goal of reducing variation.  Six Sigma: A method for reducing defects in products and services through the use of scientific method and statistical techniques. 6  Lean: A method for eliminating various forms of waste in processes while increasing efficiency. Breakthrough Series Collaboratives are an excellent way to expedite improvements across large health systems. They borrow micro-level change mechanisms from the established methods outlined above (and others) in order to bring about change on a large scale. Similarly, the Accelerated Model for Improvement (AMI) uses a variety of improvement techniques to enact positive change (Langley et al., 2009). The AMI approach uses the PlanDo-Study-Act cycle, and three clarifying questions: What are we trying to accomplish? How will we know that a change is an improvement? What change can we make that will result in improvement? The Institute for Healthcare Improvement has built upon the AMI approach by incorporating W. Edwards Deming’s theory of profound knowledge to create what is referred to as IHI-QI, which relies on diversity and an open-source approach to method and content (Scoville & Little, 2014). Rather than adopting a dogmatic approach to improvement, the IHI-QI model draws on an array of methods, including those from the scientific method, statistical process control, and industrial engineering. This discussion will now turn to an overview of a distinct approach to improvement that shares some of the history and methods of the IHI-QI model. This approach is known as Lean, and we briefly explore its application in the field of healthcare. Lean in Healthcare Put very simply, Lean is a method for improving the quality of products and services that a business delivers to its customers (Endsley, Magill, & Godfrey, 2006). Lean is not an acronym, but in fact, it is a synonymous term for the Toyota Production System created by 7 Taichi Ohno for the Toyota Motor Company in Japan (Ohno, 1988). The Toyota Production System had its origins in a visit to the Ford assembly line in United States by Toyota representatives during the 1950s, where they found inconsistencies and unnecessary steps in the production process. Later, the Toyota representatives witnessed a streamlined process when touring an American supermarket and decided to implement this way of conducting business back in Japan (de Weck, Roos, & Magee, 2011). Thus, the Toyota Production System was born and has been in development as a method to improve quality ever since. The term Lean was first coined by Jim Womack and Dan Jones in 1990, and in their 1996 seminal book Lean Thinking they define Lean as “the endless transformation of waste into value from the customer’s perspective” (Womack & Jones, 2003, p. 15). Over the last three decades, Lean manufacturing has been adopted by global industry as a method to reduce error, reduce production cost, and improve productivity with the ultimate outcome being cost savings (Motwani, 2003). Moreover, the application of Lean methodology is not limited to application in the automobile production industry. Lean thinking has expanded to a number of different sectors, including service industries such as healthcare. According to Black (2009), it was not until the 1980s that Lean began to gain momentum with manufacturers in the United States and only became popular in healthcare in the late 1990s. Evidence of Lean’s growing presence in healthcare can be seen in a 2009 survey of U.S. hospitals, where 53% reported having implemented Lean, and of those hospitals, 60% reported implementing Lean in the Emergency Department (Holden, 2011). Thus, the presence of Lean in the healthcare sector is rapidly growing, and in concert, academic literature related to this topic is also proliferating. 8 Lean principles and tools. It is important to note how Lean differs in principle from many other quality improvement approaches. Where many approaches often espouse clinical best-practices or offer change packages as a priori ideas for quality improvement, Lean necessarily involves staff in every aspect of process re-design—rather than requiring staff to react to top-down directives from senior management. In this way, cultural transformation is inherent within Lean, whereby a mindset of continuous improvement is promoted as a foundational and critical aspect. Toward this end, there are a large number of vendors that have created Lean implementation packages for purchase by companies who wish to utilize formal staff training curricula and consultancy services. Irrespective of the differing attempts to brand and market Lean systems, Womack and Jones (2003) provide universal principles that are common to all Lean strategies. They stipulate that applying Lean thinking to improve business practices involves the following five principles: 1) Specifying the value desired by the customer. 2) Identifying the stream of operations for each product or service that provides value to the customer and reducing or eliminating all forms of waste in that process. 3) Making the product or service flow continuously. By standardizing processes around best-practices, operations run more smoothly, and time is freed up for creativity and innovation. 4) Introducing “pull” between all steps where continuous flow is impossible. Lean thinking encourages a focus on the demand from the customer and re-arranges work so that production events are triggered backwards through the production chain. 9 5) Working towards perfection so that attempts to remove non-value adding activity are continuous. One of the fundamental tools in Lean is the Value Stream Map, where business processes are made visible through graphic representation (Principle 2). Figure 1 is an example of a Value Stream Map and depicts some conventional elements such as process steps, cycle times, wait times, and overall lead time for the entire process. This form of map is extremely useful for determining activities that add value to the process as opposed to activities that do not add value. Once a Value Stream Map is created in a workplace, the maps are then analyzed for specific forms of waste so that waste can be reduced or eliminated. Figure 1. Current state Value Stream Map for an emergency department process. From Lean Green Belt Healthcare Certification Participant Workbook (p. 3.2), by Leading Edge Group, 2006, Toronto, ON: Author. Copyright 2006 by Leading Edge Group. Reprinted with permission. In the Lean method, eight forms of waste are specified: 1) defects, 2) over-production, 3) waiting, 4) non-utilization of staff creativity/talent, 5) transportation, 6) excessive processing, 7) inventory, and 8) movement/motion (Womack & Jones, 2003). Definitions of each form of waste with examples in the healthcare industry are provided in Appendix A. Once these forms of waste are made visible and identified using the Value Stream Map, they are systematically removed from business processes by staff who work in the particular area. 10 Staff can select from a menu of Lean tools to reduce waste or make improvements to the system. Although a description of all the Lean tools will not be provided here, some of the fundamental Lean tools, as described by Black, Miller, and Sensel (2016), are:  5-S (Sort, Set-in-order, Shine, Standardize and Sustain): Activities focussed on organizing the workplace in order to make it more productive.  Spaghetti Diagramming: A method for exposing transportation and motion waste.  Poka Yoke: A system that uses simple, low-cost devices to reduce errors. It is used to prevent defects from being made or passed along in a process.  Takt Time: The average rate at which customers buy products or consume services, and hence the rate at which products should be manufactured or services delivered.  Kanban: A system where production is authorized from downstream operations, based on physical consumption. It is based on a pull material replenishment system, with the principle that supplies or resources are pulled through the process based on the actual customer or patient requirements.  Visual Management: A system that enables anyone to immediately assess the current status of an operation or process at a glance, regardless of their knowledge of that process.  Standard Work: A prescribed, repeatable sequence of steps or specific instructions that allow processes to be completed in a consistent and timely manner. One of the key methods used in Lean are Kaizen events (a Japanese term that translates to “continuous incremental improvement”). The term Kaizen event is used to describe how staff are convened to participate in quality improvement efforts (Womack & Jones, 2003). Kaizen events typically last up to three weeks and should involve a broad range of stakeholders to 11 assist with making improvements. Staff participating in Kaizen events work together to identify waste, remove or eliminate it, and improve processes in a collaborative effort using Lean tools. Once a Kaizen event has concluded, efforts to sustain the improvements follow, and processes are closely monitored and measured. Kaizen events are also referred to as Rapid Improvement Events (Naik et al., 2012; Wolf et al., 2013) or Rapid Process Improvement Workshops (Nelson-Peterson & Leppa, 2007) and may incorporate the Lean tools described above and many more. There are many formulations of these types of events or workshops as related to Lean. However, in this study I focus on Rapid Process Improvement Workshops as described in more detail in Chapter Three. In a systematic implementation of Lean, the process of continuous improvement should not end. Ideally, Kaizen events occur on an ongoing basis in order to produce incremental change that ultimately results in positive improvement at a larger scale. Figure 2 illustrates how a progression of Kaizen events can result in transformational change. Figure 2. Continuous improvement through Lean Kaizen events. From Leading Edge Group Lean Healthcare White Belt Workshop (Slide 92), presented by Leading Edge Group, 2009, at Northern Health, Prince George, BC. Copyright 2009 by Leading Edge Group. Reprinted with permission. 12 According to Black, Miller, and Sensel (2016), the transformational potential of Lean can only be optimized by linking all of the incremental improvements into an overall organization-wide production system. Kovacova (2012) describes a dimensional model of Kaizen events whereby improvement efforts can progress from a single point (i.e., a discrete Kaizen event), to a series of Kaizen events along a service line, to multiple Kaizen events that attempt to improve many processes within a product family or service area (such as cancer care or surgical services) known as a “plane-Kaizen” approach. This approach of purposeful and strategic improvement of multiple services can culminate in meaningful results in terms of a seamless experience for patients and the removal of functional silos that are commonplace in healthcare. Over time, Lean improvement can expand across many service areas and fully evolve to include improvement work with suppliers and supporting services, such as Human Resources and Finance (Black, Miller, & Sensel, 2016). Approaches to Lean implementation. In his book The Toyota Way, Liker (2004) describes different ways in which organizations have implemented Lean. Liker claims that many organizations emphasize rapid improvement but limit their involvement to the use of Lean tools, thereby demonstrating only a partial commitment to Lean. Far fewer organizations fully embrace Lean as a business philosophy, grow leaders who live the philosophy, and evolve a culture of continuous improvement. Other early authors in this field pointed to the fact that there are multiple interpretations of what constitutes Lean, which gives rise to many approaches to implementation (Parker, 2003; Pettersen, 2009). In their comprehensive review of Lean in healthcare, D’Andreamatteo, Ianni, Lega, and Sargiacomi (2015) go as far as to say that the definition of Lean management is actually quite ambiguous. Thus, there are many terms used 13 to describe the implementation of Lean in healthcare settings. Table 1 outlines a few approaches to Lean implementation that can be found in the literature. Table 1. Approaches to Lean Implementation Lean Implementation Approach Lean thinking Description & Literature Examples Lean thinking involves implementing the five principles of Lean as outlined by Womack & Jones (2003): specifying value, value stream mapping, create flow, implement pull, and pursue perfection. Ben-Tovim et al., 2007; Holden, 2011; Womack & Jones, 2003; Yousri, Khan, Chakrabarti, Fernandes, & Wahab, 2011 Toyota Production System Toyota Production System is a set of principles and practices developed by the Toyota Motor Company that focuses on maximizing value to the customer. Burkitt et al., 2009; Furman & Caplan, 2007; Liker, 2004 Virginia Mason Production System Virginia Mason Medical Center’s adaptation of the Toyota Production System and systematic application to healthcare. Kenney, 2011; Nelson-Peterson & Leppa, 2007; Reinertsen, 2006 Lean Six Sigma Lean Six Sigma is the combination of two approaches to performance improvement. Lean was founded on the Toyota Production System and focusses primarily on gaining efficiency through waste reduction; Six Sigma comes from Motorola and focusses on reducing variation and eliminating defects. Arthur, 2007; Matt, Woodward-Hagg, Wade, Butler, & Kokoska, 2014; Niemeijer et al., 2013 More recently, a group of researchers have developed a process for operationally defining Lean management in healthcare as part of conducting a Cochrane systematic review. Rotter et al. (2018) identified two defining characteristics of Lean management as it pertains to healthcare. To be included in their systematic review, articles must have described organizations that have integrated Lean philosophy into their mandate, policies, or guidelines (as evidenced by Lean principles and continuous improvement) and utilized at least one Lean activity (as evidenced by a Lean assessment activity such as Value Stream Mapping or a Lean improvement activity such as 5-S). 14 Having a clear definition of Lean is critical in determining an organization’s approach to implementation. In turn, understanding an organization’s approach to Lean implementation is important given the many warnings that a gradual approach to improvement is rarely effective (Grol, Bosch, Hulscher, Eccles, & Wensing, 2007; Hines, Holweg, & Rich, 2004; Proudlove, Moxham, & Boaden, 2008; Bevan, Ham, & Plsek, 2008). Organizations that use Lean tools sporadically run the risk that benefits will typically be localized, are less likely to be sustained, and may even negatively impact other parts of the system (Burgess & Radnor, 2013). Ultimately, clarity around the definition and implementation of Lean is critical to researchers, who must specify exactly what they are researching in order to expand the knowledge base in this field (Parker, 2003). Purpose of the Study The purpose of this research was to conduct a systematic study of a major Lean program in the form of Rapid Process Improvement Workshops (RPIWs) implemented in a large health authority in the province of British Columbia (BC), Canada. To date, there has not been a single study conducted in BC that examines the efficacy of Lean with sufficient rigor. More specifically, the main objective of this study was to design and execute a systematic evaluation of six RPIWs conducted for surgical services at two hospital sites in a health authority in BC during 2013/14. The first portion of the study is quantitative, which begins with economic evaluation of RPIWs. By collecting detailed data during the RPIWs and comparing cost of inputs to quantified outcomes, this study attempts a Return on Investment (ROI) analysis of RPIWs as applied in the two hospital sites. The study also aims to delineate a process for conducting an ROI analysis of Lean events and activities. The delineation of a procedure for systematically 15 studying Lean in healthcare is a contribution to the field given that other Lean studies do not clearly outline the procedures used to substantiate claims of benefit/cost savings from Lean methods. This portion of the study ends with a series of statistical tests to examine the impact of RPIWs on select performance measures in the intervention sites, and compare performances between the two intervention sites and two other hospital sites in the same health authority considered as control sites. The second portion of the study examines the impact of RPIWs on employee experience and engagement. This is an important area of study for researchers as well as stakeholders because Lean presupposes that employees are experts in their roles and fundamentally influence any attempts at improvement. Thus, understanding how employees experience RPIWs and learning about the process of engagement is critical for any healthcare organization that wishes to improve its services, client care, and overall performance. We now turn to a review of the background of the problem, particularly within the context of Lean in BC. 16 Chapter Two: Literature Review This chapter begins by describing the method used to conduct a brief literature review to identify and screen articles that form the foundation of the present study. Much of the research retrieved when this study began was anecdotal in nature, with writers primarily describing how Lean was being applied in various healthcare settings. This section will outline the challenges associated with early research and note some specific critiques of studies that lack scientific rigor and fail to investigate the efficacy of Lean in a defensible manner. Over time, research in this area became more sophisticated as investigators used more advanced study designs and produced more convincing arguments related to either the success of Lean or its inadequacies. In an era of increasing fiscal restraint and accountability, these articles have been of keen interest to many, especially in light of the fact that Lean has been promoted as a method that promises efficiency gains and cost reduction. The chapter will review some key articles that attempt to explicate the financial benefits of Lean as it has been applied to surgical services. This will reveal some lofty claims related to Lean’s ability to reduce costs in healthcare. But there are problems inherent in studies that make such claims—problems related to clearly defining Lean interventions and properly conducting cost benefit analyses. The present study attempts to build on past research by addressing some of the problems that are identified. This study also attempts to address the gap in Lean research that exists in the Canadian province of BC. The chapter will briefly touch on the Lean journey in BC over the last decade, which includes a look at an article that offers an evaluation framework as well as grey literature that provides a glimpse into attempts to empirically study Lean across the province. The efforts to study Lean in BC led me to explore literature in economic evaluation 17 and employee experience and engagement. The chapter concludes by highlighting some distinguishing characteristics of healthcare evaluation and covers some of the issues and key research articles related to the topic of employee experience and engagement with Lean. Search Strategy At the start of this study in 2011, a search for health science literature was conducted to locate studies related to Lean in healthcare. Several databases were used in the search strategy including CINAHL, Evidence Based Medicine Reviews, Pubmed, OVID Medline, PsychINFO, and the TRIP database. The keywords used in the search were: Lean, Lean Thinking, Toyota Production System, Six Sigma, PDSA, Quality Improvement, Quality Collaborative, Organizational Change, Large Scale Change, Transformational Change, Healthcare Management, and Healthcare Evaluation. The search results produced an array of literature related to the topic of Lean in healthcare published in the English language up until 2011. The yield included quantitative, qualitative, peer-reviewed, and grey literature articles, as well as governmental and nongovernmental reports. The total number of articles identified through this initial database search was not recorded. However, there was a total of 99 articles selected for examination because they provided background on Lean in healthcare or they investigated the efficacy of Lean in healthcare. The literature retrieved in this initial search provided sufficient material to lay the foundation for the research proposal. On August 1, 2014, a second database search was conducted to obtain more current literature on this topic. The databases searched were Medline, CINAHL, EBMR, PsychINFO, and TRIP. The main keywords used in this search were: Lean, Toyota Production System, Lean Management, Lean Production, Lean Methodology, Rapid Process 18 Improvement Workshop, Healthcare, Hospital, Efficacy Studies, Return on Investment, and Employee Experience. The databases were searched for literature between 2011 and 2014. An additional 64 articles were identified, and 6 of these records were excluded as being not relevant. The reference sections of each of the remaining 58 publications were screened for other relevant articles. This snowball method of locating additional articles of interest resulted in an additional 557 records being identified and screened. The screening process involved trawling the titles and abstracts of the articles to determine if the study met the following criteria: 1) self-identified as a Lean study, 2) was applied in a healthcare setting, and 3) reported end-of-project results (particularly financial gains). The screening process resulted in 349 articles being excluded. The full publications of the remaining 208 articles were retrieved and fully reviewed, and 73 of these articles were excluded for specific reasons (e.g., context was not solely healthcare; mixed improvement approaches were used such as Lean with Six Sigma; article was an editorial and/or did not report results; article was not relevant to this study; etc.). The final result was 135 records being referenced. It is recognized that the search strategy used was much less formal and precise as compared to the Preferred Reporting Items for Systematic Reviews and Meta Analysis (PRISMA) method. This is a limitation of the literature review in that it did not adhere to strict conventions, and therefore is not exhaustive. In spite of this deficiency, the literature review has been ongoing over a long period of time, and I have been systematic in arriving at the final list of materials that I have reviewed and referenced. Thus, the following section sufficiently maps the literature on the topic of Lean efficacy studies and provides a basic backdrop for this study. Key concepts and sources of evidence are noted, and some gaps in research are identified. The gaps that are 19 highlighted formed the impetus of the present study, which attempts to address some of those gaps. Figure 3 outlines the method used in conducting the literature review. Records identified through initial database search in 2011 99 records selected following screening 2014 database search conducted (parameters: 2011-2014) 64 records identified 6 records excluded as not relevant 58 records screened, references reviewed 557 additional records identified from reference lists and screened 349 records excluded as not relevant 208 full text articles retrieved and reviewed 73 full text articles excluded – did not meet inclusion criteria 135 articles referenced in this study Figure 3. Literature review flow chart. 20 Lean in Healthcare Literature In a milestone article that surveyed the existing Lean literature in healthcare, Brandao de Souza (2009) created a taxonomy that classifies Lean publications into one of six categories: manufacturing-like, managerial and support, patient flow, organizational, methodological, and speculative. It is important to point out that the author did not identify efficacy studies or evaluative studies as one of the categories in his taxonomy. Notwithstanding, the author notes that Lean studies in healthcare began to appear in the published scientific literature around the year 2000, and over 90 works were categorized using his taxonomy. This finding is consistent with other researchers, who note that Lean was introduced to healthcare in the early 2000s (Nelson-Peterson & Leppa, 2007). In contrast, the movement to improve quality in healthcare started gaining momentum more than 30 years ago, which illustrates how Lean is a relatively new improvement method being applied in the industry (Frankel, Haraden, Federico, & Lenoci-Edwards, 2017). The majority of papers published on this topic are speculative case studies—practicebased discussions that describe an application of Lean in healthcare settings (Brandao de Souza, 2009). This is important, given that speculative studies are primarily aimed at the practitioner audience and are published to assist in the application of Lean, rather than provide concrete evidence of efficacy. Moreover, case studies provide a synoptic perspective, and do not discuss Lean implementation on a large-scale basis, which is perhaps indicative of Lean’s relatively new emergence in healthcare. Other authors note that there are many how-to books available for Lean practitioners, but they do not present complete and tested methods of implementation that are applicable in different contexts (Al-Baushi et al., 2014). 21 Though there seems to exist a general agreement about the potential benefits of Lean in healthcare, Brandao de Souza (2009) concludes that it remains a challenge for academics and practitioners alike to evaluate Lean using a more critical lens. This finding is consistent with the position of other authors, who point to the paucity of research that evaluates Lean, either at the individual project level or at the systemic level (e.g., Aherne, 2007; Ovretveit, 2009; Ulhassan et al., 2013; Vest & Gamm, 2009). Vest and Gamm (2009) also examined peer-reviewed literature for evidence of effective strategies to transform U.S. healthcare organizations and concluded that studies on Lean (and Six Sigma) have fatal methodological limitations. The studies reviewed typically did not incorporate any statistical analysis, or in cases where statistical analyses were present, test assumptions were violated. Other studies failed to take serious confounding factors into consideration, such as selection bias or the failure to include comparison groups. A study by Naik et al. (2012) in a U.S. emergency department is a case in point. They studied rapid improvement events over an 18-month period and found promising results across several performance metrics. However, the study was limited by its uncontrolled design, such that the improvement in outcomes could not be causally linked to the Lean intervention. A similar study of Kaizen events conducted in an emergency department by Dickson, Singh, Cheung, Wyatt, and Nugent (2009) claimed very positive results based on some basic process improvements, but once again the study was limited by its single-group pre-test post-test design. Persoon, Zaleski, and Frerichs (2006) studied an application of Lean in their chemistry laboratory using a single-group interrupted time series design. The authors created a performance index as the outcome measure (the percent of completed processed specimens in 22 one month minus the baseline target of 80%) and graphed the data over time. The graphed data provided visual evidence of improvement caused by Lean, because as Lean principles were applied, the outcome measure changed direction. However, Vest and Gamm (2009) point out the weakness of this study from statistical and generalizability perspectives. They argue that the variation in each monthly measure was exaggerated because of the reduction by the constant baseline of 80%, and that no statistical tests were performed. Additionally, Vest and Gamm are skeptical of the generalizability of the study, since the effect size of the Lean interventions decayed and eventually disappeared over time, which undermines confidence in the belief that results could be recreated in other settings. It is illustrative to examine two studies reviewed by Vest and Gamm (2009) in greater detail. First, Zarbo and D’Angelo (2007) applied Lean principles in order to reduce defects in the surgical pathology laboratory of the Henry Ford Hospital in Detroit, Michigan. Defects were defined as “…flaws, imperfections, or deficiencies in specimen processing that required work to be delayed, stopped or returned…” (p. 1016). Over the course of a year, 77 staff members used Lean to implement more than 100 process changes, which resulted in a 55% reduction in the number of defected specimens. The study, Vest and Gamm argue, was sound in many ways (i.e., no ambiguous temporal sequence, no participant attrition, minimal threat of selection bias, and no changes in instrumentation) and appropriate statistical tests for paired pre-test post-test analysis were used. Unfortunately, the study design (single group pre-test post-test) could not eliminate the threat from history, in which case, a comparison group would have dramatically improved this study. The uncontrolled before-and-after study design is also vulnerable to the Hawthorne effect. This form of bias describes how individuals may alter their behavior as a result of 23 being observed. Thus, productivity or other improvements may be a consequence of the observation rather than the efficacy of particular interventions. Mason, Nicolay, and Darzi (2015) found this form of bias to be inherent in many of the studies included in their review of Lean applied in surgical settings. For instance, McCulloch et al. (2010) admitted that the Hawthorne effect was a serious limitation in their study of Lean at an emergency surgical unit, citing a lack of available control sites and insufficient manpower as two key barriers in overcoming this bias. Vest and Gamm (2009) have pointed to the lack of comparative analysis as a major methodological gap for research in this area for quite some time. They call for stronger study designs, suggesting that this is a feasible endeavor. The lack of comparison groups in this field of study could be addressed using interrupted time series design, or a phased-in approach, where sites that implemented interventions later could serve as controls for the sites where interventions are implemented earlier. Another issue in Lean healthcare research can be illustrated in a study of Lean applied in emergency departments by Mazzocato et al. (2012). These researchers reported significant reductions in wait-times and increases in patients completing their visit and leaving the department within four hours following Lean intervention. They also made Lean-inspired changes to employee roles and reported improvements in staff communication and coordination, problem solving, and workspace layout. The shortcomings of this study are evident both in its single case study design and in the ambiguity of the Lean interventions used. The authors did not explain the specific steps or practices they undertook to implement Lean, and only described their interventions in terms of broad Lean principles. There appears 24 to be vagueness in the description of Lean and its application in many studies, which is a noticeable gap in Lean healthcare literature. There are articles that provide sufficient descriptions of Lean interventions, for example, the study by Nelson-Peterson and Leppa (2007) that explicates the Lean methods used at the Virginia Mason Medical Center. These authors adequately explain the process of the Lean improvement events as well as the specific tools and techniques that were used. They report considerable gains in terms of re-claimed time that clinical staff could then devote to caring for patients at the bedside. However, the article does not describe the method of study used to arrive at their conclusions, which might be considered to be a major flaw of the study. It is incumbent upon researchers in this field to specify the interventions used in their studies, especially given there are many different approaches to implementing Lean in healthcare. The shortcomings of research in this area does not indicate that evaluating the effectiveness of Lean in healthcare is impossible, but rather, it appears that rigorous scientific studies are few in number (Mackenzie & Hall, 2014). Indeed, Ovretveit (2009) surmises that Lean seems to permit the clear identification of waste and the calculation of potential savings more so than other methods of quality improvement. In his 2009 review of the effectiveness of quality improvement methods, however, Ovretveit laments that most Lean articles discuss its application but notes that none were comparative or critical studies of cost effectiveness. Lean Studies in Surgical Services In recent years, there have been numerous reports of economic gains resulting from Lean interventions that can be found within the scientific literature, grey literature, and popular media—but none seem to provide sufficient details to explain how such claims are supported 25 by sound evidence (Bercaw, 2013; Black & Miller, 2008; Government of Saskatchewan, 2012; Mackenzie & Hall, 2014; Nicolay et al., 2012). In their use of Lean to improve care for esophagectomy patients, Iannettoni, Lynch, Parekh, and McLaughlin (2011) report reduced lengths of stay and a 43% decrease in cost per case after standardizing phases of care and eliminating intraoperative variability. The 43% improvement was reported as $34,678 savings per case following their Kaizen events. The costing analysis included expenses related to the intensive care unit, anesthesia, disposables, and hospital services. What is missing is the costing of the improvement interventions, which took place over the course of five years. Without tracking the investments associated with instituting the changes, the reported gains (if accurate) would likely be diminished. Similarly, Schwarz et al. (2011) conducted a Lean project that aimed to improve operating room capacity and increase throughput at their hospital. The team used value stream mapping and single-minute exchange of dies (SMED), and eventually implemented a pull system to improve surgical processes across their nine operating rooms. Statistical t-tests were used to examine throughput before and after the intervention and a significant decrease in lead time was observed (21% reduction). The researchers contend that if increased capacity could be fully optimized, a considerable cost savings could be attained (estimated at €366,000 annually). While these results sound very promising, it is important to note that the study reports a theoretical financial impact from their Lean interventions and not actual cost savings/avoidance. Furthermore, the claims of potential financial benefit are limited by a lack of proper costing analysis in this study. 26 In an 18-month prospective study of Lean in one surgeon’s operating room, Collar et al. (2012) reported significant gains in teamwork, morale, and physician resident education. Perhaps most profound was the impact of Lean on operating room performance. The authors found statistically significant reductions in turnover times and turnaround times, which translated into additional capacity for the operating room (4500 minutes for a single room used twice weekly). The authors estimate the opportunity revenue to be $330,000 per annum (opportunity revenue is defined as the maximum potential revenue gains, assuming total use of saved operating room time). With gains extrapolated to the entire suite of 35 operating rooms at their facility, they estimated a theoretical gain in excess of $25 Million. However, a proper costing analysis was not provided and, like other studies, only a theoretical financial impact was postulated rather than actual financial benefits being demonstrated. A Lean rapid process improvement workshop (RPIW) was used in another study to optimize on-time performance in two vascular operating rooms, demonstrating positive results (Warner et al., 2013). The researchers clearly described how they used value stream mapping and Pareto charts to discover factors that were contributing to late first case starts in their operating rooms. By reducing rounding time prior to surgery, implementing standardized checklists, and instituting huddles, the team was able to increase the percentage of first case starts from 39% prior to the RPIW to 71% at six-week follow up. The gains were sustained at 12-weeks and 1-year post RPIW. Warner et al. estimated the cost of operating room time to be $34 per minute, which translated to an annualized opportunity cost potential of $72,696 if more surgeries could be performed. Like the Collar et al. study outlined above, the hypothesized gains were only forecasted estimates and the costs related to the Lean interventions were not considered. 27 One study that accounted for the cost of Lean implementation was conducted by Gayed, Black, Daggy, and Munshi (2013) at a Veterans Affairs Medical Center. Over the course of nine months, these researchers employed Lean and Six Sigma techniques to make several improvements for joint replacement patients, at a cost to the system of approximately $25,000. The techniques used included process flow charting, single-piece flow, visual management, and standard work. This resulted in better discharge planning and defined care pathways, and ultimately reduced length of stay by 36%, which increased the capacity for patient care at the Veterans hospital (the need for care delivery by the non-Veterans Administration system was eventually eliminated). A return on investment (ROI) analysis was conducted and it was estimated the ROI for this project was $1 Million annually. This article was the only one found in the literature review conducted for the present study that seemed to satisfy the critiques outlined earlier (i.e., the intervention was briefly explained, statistical tests were conducted, and an economic analysis was performed). Notwithstanding, the study lacked a control group and did not monitor or report on gains beyond one year (in order to show that the benefits were truly sustained). This research was also confounded by the fact that an additional surgeon was hired between the project period and the sustainment period, which could explain the increase in surgical performance at their facility. A further challenge to researchers in this area can be found in the article by Rauh, Wadsworth, Weeks, and Weinstein (2011) who explored the concept of harvesting gains and argued that clinical quality improvement efforts will fail to deliver economic results due to the current cost structures in typical healthcare settings. Owing to rigid cost structures that are not easily manipulated to react to changes in patient volume, resource use, or patients’ health conditions, they suggested clinical quality improvements can, at best, create increased 28 capacity but will not influence cost savings to the system. Until health systems are able to reduce or eliminate unnecessary capacity (e.g., send healthcare providers home if there are fewer patients to serve) or lower utilization rates of services (i.e., provide fewer services to patients), it will be difficult to realize substantial financial gains. This argument starkly contrasts the promise of Lean, which endeavours to remove waste and increase efficiency in health systems. Clearly, the field of Lean in healthcare needs to be substantiated as a legitimate contributor to improving health systems through the use of sound research strategies and rigorous methods that will eventually enable causal claims about the ability of these methods to produce effective outcomes. Therein lies the crux of the problem. How can research demonstrate the worth of Lean in a way that will satisfy the scrutiny of reviewers that are usually best acquainted with clinical research, randomized controlled trials, and evidencebased medicine—methods and paradigms that more easily lend themselves to widely accepted mechanisms for determining efficacy (e.g., experimental design)? Moreover, how can research demonstrate the value of Lean in terms of economic benefits or cost savings? Studying Lean in British Columbia One group of researchers has designed a large-scale evaluation framework for examining Lean in healthcare (Puterman et al., 2013). While studying the Lean program at the Provincial Health Services Authority in BC, Puterman et al. produced an evaluative framework that is designed to guide a thorough and rigorous investigation of Lean in health systems. As can be seen in Figure 4, the evaluation framework of Puterman et al. identifies four components of analysis at two dimensions (i.e., short and long term time frames). 29 Figure 4. Lean evaluation framework. Adapted from “‘If You're Not Keeping Score, You’re Just Practising’: A Lean Healthcare Program Evaluation Framework,” by M. Puterman, Y. Zhang, S.K. Aydede, B. Palmer, S. MacLeod, H. Bavafa, and J. MacKenzie, 2013, Healthcare Quarterly, 16(2), p. 25. Copyright 2013 by Longwoods Publishing. Reprinted with permission. It is important to note that Puterman et al. (2013) did not evaluate the Lean program at the Provincial Health Services Authority; rather, they designed a recommended framework for evaluating Lean. The lack of any systematic studies of Lean in BC represents a knowledge gap related to the implementation and impact of Lean in the province. The Provincial Lean Network. In 2011, a Provincial Lean Network was established in an attempt to advance the measurement and reporting of Lean events across BC. The Provincial Lean Network is made up of senior representatives from each of the six Health Authorities within the BC healthcare system, and the purpose of the committee is to assist the Ministry of Health in coordinating provincial Lean activities, to help facilitate information sharing across health authorities, and to champion the use of Lean within respective regions (BC Ministry of Health, 2011). The Metrics Working Group, a sub-committee of the Provincial Lean Network, was convened to recommend appropriate measures and data elements for the evaluation of Lean in BC. As Chair of the Metrics Working Group, I was tasked with leading the development of a standardized provincial framework for the Lean system and process metrics. The working group produced the Metrics Framework (Chan et al., 2011), which is presented in Appendix B. 30 During the development of the Metrics Framework, the Metrics Working Group quickly learned of the challenges associated with designing an instrument that would support an empirical and quantitative evaluation of Lean. For example, it was noted that typical outcome metrics collected in healthcare would be difficult to consistently include in an analysis of Lean due to the proximal distance from the Lean interventions that were taking place. This realization led me to consider other important concepts in evaluating Lean. Economic evaluation. As discussed previously, publications in the field of Lean literature routinely report huge economic benefits despite their inability to be classified as bona fide economic evaluations. Notwithstanding, some Lean studies can be described as partial evaluations, which means that future research needs to employ improved study designs before we can be confident about causal claims made about the true economic value of Lean. This is not meant to suggest that such research is of no value, but rather, partial evaluations can be thought of as building blocks that contribute to our conceptualization and understanding of the economic benefits of applying Lean in healthcare. To be able to make causal claims about the economic value of Lean, we must advance beyond partial evaluation (i.e., cost description) to a higher level of sophistication. Drummond, Sculpher, Torrance, O’Brien, and Stoddart (2005) provide a succinct definition of economic evaluation as being “…the comparative analysis of alternative courses of action in terms of both their costs and consequences” (p. 9). There are two defining characteristics of this definition: 1) that economic evaluation is concerned with the inputs and the outputs of activities, and 2) a major premise of economic evaluation is that it is concerned with making choices among alternatives. To differentiate among several evaluation situations commonly 31 encountered in the healthcare evaluation literature, Drummond et al. provide a useful matrix, which is reproduced in Figure 5. Are both costs (inputs) and consequences (outputs) of the alternatives examined? No Examines only consequences No Examines only costs 1A Partial Evaluation 1B Outcome description Is there a comparison of 2 or more alternatives? Yes Cost description 3A Partial Evaluation 3B Yes Efficacy or effectiveness evaluation Cost analysis 2 Partial Evaluation Cost Outcome description 4 Full Economic Evaluation Cost-effectiveness analysis Cost-utility analysis Cost-benefit analysis Figure 5. Distinguishing characteristics of healthcare evaluation. Adapted from Methods for the Economic Evaluation of Health Care Programmes (3rd ed., p. 11), by M.F. Drummond, M.J. Sculpher, G.W. Torrance, B.J. O’Brien, and G.L. Stoddart, 2005, Oxford: Oxford University Press. Copyright 2005 by Oxford University Press. Reprinted with permission. We can see that the types of evaluations labeled in the top row of the matrix (cells 1A, 1B, and 2) do not involve the comparison of alternatives, and as such, the services or programs being evaluated can only be described. Referring back to the evaluation framework in Figure 4, the Puterman et al. (2013) approach would fit in cell 1B and be referred to as a cost description analysis because only the costs associated with a Lean program would be considered. The evaluations listed in cells 3A and 3B, however, involve the comparison of two (or more) alternatives, and can therefore be termed “efficacy/effectiveness” evaluations or “cost analysis.” But since the costs and consequences are not examined simultaneously, these are not considered to be full economic evaluations. Full economic evaluation occurs only when there are two or more alternatives considered and both costs (inputs) and consequences (outputs) of the alternatives are examined (Drummond et al., 2005). 32 There are three types of evaluations listed within cell 4: cost-effectiveness analysis, costutility analysis, and cost-benefit analysis. A full description of these techniques is beyond the scope of this paper, although a brief discussion is provided to highlight the differences among the methods. In cost-effectiveness analysis (CEA), the incremental cost and benefits of a program are compared to those of another program, with the health benefits measured in natural units that are related to the objectives of both programs (Hurley, 2010). As examples, some natural units of benefit are average blood pressure improvement, cases found, cases of disease averted, patients significantly improved, lives saved, and life-years gained. Thus, while evaluators are interested in any change in health status (effectiveness) relative to the cost of intervening, CEA does not incorporate monetary valuations of benefits into its calculations. CEA also first assumes that programs/interventions are worth pursuing and then compares two or more programs/interventions to find the most efficient way to achieve health benefits. Given that health benefits or patient outcomes are not the primary focus of the present study, CEA was not selected as the most appropriate method of economic evaluation. Cost utility analysis (CUA) is a variant of CEA (Folland, 2013). CUA examines the incremental cost of a program as compared to the incremental health improvement attributable to a program/intervention (similar to CEA), but disparate outcomes are combined into a single composite summary metric—a unit of measure called Quality Adjusted Life Years (QALYs). Health improvement is measured in QALYs gained, and the results are expressed as a cost per QALY gained (Drummond et al., 2005). One of the main strengths of CUA is that it uses QALYs to value outcomes for a wide variety of health interventions and the results from CUA studies can be compared across a broad range of health treatments. 33 Hurley (2010) provides the example of how cancer screening programs, needle exchange programs, and anti-depressant medications can all be evaluated in terms of their efficiency in producing QALYs, and how the results can then be used by policy makers to help them with investment decisions regarding these alternative interventions. As with CEA, CUA was not deemed appropriate for use in the present study because it does not value outcomes in monetary units and patient outcomes (expressed in composite summary metrics) is not the primary focus. Moreover, both CEA and CUA are often used in well-defined and specific interventions in more controlled environments, which is not the case for RPIWs. Cost-benefit analysis (CBA) is the most appropriate method of economic evaluation to employ in this study based on the evaluation scenario regarding Lean interventions. What sets CBA apart from other methods of economic analysis is that both costs and benefits/outcomes are valued in monetary terms. Because CBA converts all costs and benefits into monetary terms, the net benefit (expressed in dollars) can be compared between programs/services, and therefore, can be used to address questions related to technical and allocative efficiency (Drummond et al., 2005). The results of CBA can be expressed as a ratio of costs to benefits, or as a simple sum (possibly negative) representing the net benefit (or loss) of one program over another. While this may appear straight forward, this method can be challenging, particularly when it comes to determining the monetary value of benefits/outcomes of an intervention. Employee experience and engagement. The Province of BC has released an on-line video to educate the public on efforts to streamline health services toward greater efficiency and effectiveness. This video, available from Think Health BC (2012), showcases how Lean principles are being applied to reduce 34 wait times, eliminate waste, and foster innovation in the healthcare system. At the end of the video, the narrator emphasizes an important factor in the success of Lean: “It all comes down to people who are part of the system getting involved to change the system.” This notable quote reveals the importance of employee engagement in the effectiveness of Lean. In fact, one of the main tenants of Lean thinking is “respect for people” (Black & Miller, 2008) and a key objective is to capitalize on the skills of employees—arguably one of the most valuable resources in any organization. The Lean method invites practitioners to pay attention to human capital by listing Non-Utilized Staff Creativity as a key form of waste (also known as waste of intellect). Thus, it is implied that employee engagement is absolutely necessary if Lean is to be used to pursue high performance. While employee satisfaction has been studied in healthcare environments, far less is known about how the concept of employee engagement impacts healthcare settings (Lowe, 2012). Increasingly, though, it is being argued that high-performing organizations are those that have healthy and engaged employees, and that it is crucial to foster workplace environments where employees are valued and are provided with supports that enable them to excel (Lowe, 2010; Mintzberg, 2009). The rationale for supporting healthcare workers in this way is grounded in research that suggests managers can improve patient care experiences by improving employee satisfaction and retention (Collins, Collins, McKinnies, & Jensen, 2008; Michie & West, 2004; Rondeau & Wagar, 2006; Sikorska-Simmons, 2006; Studer, Robinson, & Cook, 2010). Research conducted in Britain claims that hospitals with alleged higher levels of staff engagement provide higher quality services and have better performance (West, Dawson, Admasachew, & Topakas, 2011). 35 Across Canada, employee engagement is recognized as being an important factor in high performing health systems. Accreditation Canada, the organization that evaluates health systems in Canada, requires health organizations to administer the Worklife Pulse Survey, which is designed to assess the current state of workplace culture and employee engagement (Nicklin & Mitchell, 2015). Accreditation Canada Surveyors use the results to suggest how an organization may improve the workplace environment and promote high quality healthcare. In the Canadian province of Ontario, the Ontario Hospital Association developed the Quality Healthcare Workplace Model (Ontario Hospital Association, 2010; see Figure 6). This model outlines the relationships between a healthy, capable, engaged workforce and productive workplaces. On the left side of the diagram, specific drivers that influence employees are listed (workplace environment, job characteristics, and organizational supports). In the middle of the diagram are the individual outcomes for employees (i.e., engagement, capability, and health and safety). If the conditions are optimal, and the workforce is engaged and healthy, it is hypothesized that positive organizational outcomes will result, such as improved quality and safety for patients, improved recruitment and retention, and increased productivity at lower costs. Although the relationships between these factors are not causal, they have been shown to be highly correlated and therefore employee engagement is worth pursuing as part of an overall strategy toward high performance (Lowe, 2012). The Ontario Hospital Association (OHA) and National Research Corporation (NRC) Picker has designed a robust instrument known as the OHA-NRC Picker Employee and Physician Experience Surveys to investigate constructs within the Quality Healthcare Workplace Model (Ontario Hospital Association, 2010). 36 Figure 6. Ontario Health Association Quality Healthcare Workplace Model. From “How Employee Engagement Matters for Hospital Performance,” by G. Lowe, 2012, Healthcare Quarterly, 15(2), p. 30. Copyright 2012 by Longwoods Publishing. Reprinted with permission. During the literature review conducted for this study, several articles that discuss employee experiences of Lean were identified. In her study of three different work groups, Parker (2003) reported negative human consequences as a result of Lean implementation. The participants reported poorer quality work designs and experienced a decline in organizational commitment after Lean was introduced. Parker’s study was conducted at an automotive manufacturing plant and not in a healthcare setting but is included here to illustrate the dismal impact of Lean that has been reported in the literature. In their systematic review of Lean in healthcare, Moraros, Lemstra, and Nwankwo (2016) also reported negative findings with regard to the employee experience of Lean. Moraros et 37 al. point our attention to a survey conducted in Saskatchewan that found an overall negative effect on employee satisfaction (Martin, 2014). They also cite a peer reviewed study which reported limited staff engagement with Lean and negative feelings toward Lean as a method of quality improvement (Ulhassan et al., 2013). Conversely, other studies of employees’ experience with Lean arrived at very different conclusions. In 2011, Schwarz et al. reported that staff motivation had improved due to increases in value-adding activities following their Lean project in surgery. Cima et al. (2011) surveyed staff before and after their Lean surgical process improvement project and the majority of respondents endorsed items related to improved communication, teamwork, and individual effort. In Canada, Deans and Wade (2011) held discussion groups during their Lean implementation at Holland Bloorview Kids Rehabilitation hospital and reported many positive quotations from engaged staff, who commented about their feelings of inclusion, their penchant for Lean techniques, and their appreciation for shared decision making. Drotz and Poksinska (2014) also found positive results after interviewing and observing staff at three different healthcare organizations. They found that Lean implementation had a very positive impact on each organization’s working environment, staff development, and overall performance. Respondents in this study reported great satisfaction with Lean, particularly because it decreased the hierarchical structures between professional groups and created a culture of cooperation and teamwork. Given the disparate findings of these previous studies, it follows that an examination of how employees experience and become engaged in Lean is an important area of investigation for the present research. 38 Summary The field of quality improvement in healthcare has been criticized for failing to provide adequate evidence of success or failure. The same charges can be levied against studies on Lean, given the results of the search for health science literature on Lean conducted for this study. The literature review revealed that most published studies on Lean are case studies, that many of these case studies espouse the benefits of Lean but are often lacking in concrete evidence of success (or failure), and that published accounts of success rely too heavily on intuition and anecdotes rather than sound scientific evidence. Many authors in this area call for more research using improved study designs and more sophisticated analyses (e.g., DelliFraine, Langabeer, & Nembhard, 2010). Although this is a limitation of the research literature related to Lean in healthcare, it also poses significant opportunity for new researchers who can contribute to the current knowledge base through advanced evaluation and knowledge translation activities. The literature review also revealed a lack of clarity in describing the specific Lean tools, techniques, or approaches used when Lean is applied in healthcare. Therefore, it is incumbent upon researchers in this field to specify the interventions used in their studies, especially given that there are many different approaches to implementing Lean in healthcare settings. It was also noted that economic evaluations in this area are few, and of the studies that attempt to demonstrate the financial benefits of Lean in healthcare, many fail to account for the cost of implementing Lean improvements, or present lofty estimates of potential gains instead of actual benefits/cost savings. Lastly, this chapter outlined some Lean research and activities that have occurred in the province of BC. A proposed evaluation framework provided the backdrop for a study that 39 examines specific aspects of Lean in healthcare, such as economic impacts and the experience of employees who work in areas where Lean has been introduced. 40 Chapter Three: The Research Context In BC, healthcare is administered and managed by six separate health authorities; five that deliver services by geographic area, and one that provides specialized care to the entire province. The BC Ministry of Health supports the health authorities in working to employ Lean and other quality improvement methods in strategic planning and operations. One of the BC health authorities was selected as the setting for this study (herein identified as “the HA”). Lean was first introduced in the HA over a decade ago. Over the years, the HA has experienced many successes using Lean, as operational leaders have been empowered to solve problems, develop people, and build partnerships that result in improved team work and better service provision. In response to the healthcare system facing increasing pressure to meet growing demand for services, the HA has increasingly turned to innovative strategies such as Lean to continue to provide high quality services using constant or declining financial and human resources. The HA began its Lean journey by using Lean tools on a sporadic basis in pockets of the organization with occasional emphasis paid to consultant-led special projects. The ad-hoc approach of promoting sporadic use of Lean tools was replaced by the advent of the Lean Promotion Office in 2011. The Lean Promotion Office was made up of five fulltime positions: one Lean Leader, and four Lean Consultants. However, there were other HA staff that had formal and informal training in Lean, with varying levels of competency. Shortly after the inception of the Lean Promotion Office, the HA contracted with the Virginia Mason Institute to provide a fulsome, consistent package of Lean training and expertise, known as the Virginia Mason Production System (Nelson-Peterson & Leppa, 2007). The HA began organizing its Lean strategy to integrate efforts toward a coherent body of quality improvement capacity. Lean activities had evolved from “point Kaizen” (i.e., ad 41 hoc projects) to concentrated work in two service areas: Hips and Knees (with focus in two hospitals), and Frail and Elderly Activation (with focus in two other hospitals). In 2013, the Lean Promotion Office updated its Strategic Plan to focus on four goals and objectives and help guide the Lean work in the organization: Goal 1: Use Lean to achieve system level process improvement. Objective: Create a formal partnership between the Senior Executive Team and the Lean Promotion Office to maximize strategic planning and achievement of organizational priorities. Goal 2: Build capacity to support a standard strategy deployment and management methodology. Objective: Create a visible line of sight amongst leadership and front-line staff by implementing the Lean management method at all levels of the organization. Goal 3: Build capacity for everyday Lean thinking across the organization. Objective: Provide direct support for adoption and spread of Lean thinking through the development of problem-solving employees at all levels of the organization for a continuous improvement culture. Goal 4: Develop a robust support system for Lean work across the organization. Objective: Develop a robust, centralized support system for implementation, adoption, spread, and sustainability of standard Lean methodology. Later in 2013, the BC Ministry of Health implemented the Pay for Performance initiative, whereby health authorities would receive a portion of their funding as a function of performance in certain domains. As a result, the Lean Promotion Office was directed by the HA Senior Executive Team to focus on surgical services, in order to reduce surgical wait times from over 52 weeks to under 52 weeks. The target date for reaching this goal was March 31, 2015. A series of six Rapid Process Improvement workshops (RPIWs) were then 42 scheduled to take place at each of two different hospitals within the HA starting in December 2013. RPIWs (also referred to as “Kaizen Events” in Chapter Two) are focused improvement projects and the primary method for improving a process using the Virginia Mason Production System (Nelson-Peterson & Leppa, 2007).1 These series of RPIWs followed the tradition of applying a plane-Kaizen approach (Black, Miller, & Sensel, 2016) along a specific service area (surgery), as opposed to the hap-hazard point-Kaizen activities that took place in the HA previously. It is noteworthy that other health systems in Canada have undertaken an even more robust approach to Lean, as is the case in Saskatchewan where a province-wide Lean reform was undertaken in 2009 (Kinsman et al., 2014; Marchildon, 2013; Provincial Auditor of Saskatchewan, 2016). Figure 7 summarizes the chronological Lean implementation journey at the HA. Figure 7. Lean implementation journey at the HA. Source: Lean Promotion Office. 1 It is important to note that ROI studies are not promoted at the Virginia Mason Medical Center. Chris Backous, senior faculty member at the Virginia Mason Institute, openly states, “…we do not focus on financial returns after every event” (Backous, 2016). The assumption is that a focus on identifying and eliminating waste will result in long term financial benefits. The staff are empowered to generate improvement ideas using Lean tools and are encouraged to make incremental improvements at the point of care delivery. The expectation of quantifying results seems to stop at the Target Progress Report. While performance targets are set and reported to be achieved, Backous does not clearly explain how small changes translate into improved financial savings or how gains are harvested and then re-invested in order to achieve increased capacity (e.g., more nurse time spent at the bedside). 43 To help clarify the associations between the content of an improvement method, how it is executed, and how it may link to patient outcomes, an overall program theory is essential to a sound evaluation approach (Parry, Carson-Stevens, Luff, McPherson, & Goldmann, 2013). A logic model was provided by the Lean Promotion Office as a way of displaying the intent behind deploying the HA’s Lean resources exclusively to surgical services (see Figure 8). The logic model supports Goal 2 of the Lean Promotion Office, that is, to support a standard strategy deployment method and create a visible line of sight amongst all levels of the organization. The focus on surgical services also linked to the overall strategic objectives of the HA, such as improving productivity, efficiency, and delivering high quality care. The logic model also illustrates a theory that attempts to explain how and why the Lean Promotion Office is supposed to work as a program. In Purposeful Program Theory, Funnell and Rogers (2011) define program theory as “…an explicit theory or model of how an intervention such as a project, a program, a strategy, an initiative, or a policy, contributes to a chain of intermediate results and finally to the intended or observed outcomes” (p. xix). Figure 8. Logic model for Lean in surgical services at the HA. Source: Lean Promotion Office. 44 45 Senior Leaders in the HA were interested in a formal study of the RPIWs in surgical services, which provided a good opportunity for focused research to occur in an applied healthcare setting. RPIWs are identified as one of the key elements of successful implementation of Lean by key experts, as well as in the journey of companies who employ Lean (Black & Miller, 2008; Graban & Swartz, 2012, 2014; Leitner, 2005; Plsek, 2014). The scope of this study, then, focusses on the 12 RPIWs conducted at two hospitals in the HA (herein identified as Site 1 and Site 2) between December 2013 and December 2014. As part of designing an effective approach to evaluating these complex interventions, I developed a thorough understanding of Lean and RPIWs through formal training (Campbell et al., 2007; Parry et al., 2013). One of the benefits of studying the RPIWs in surgical services is that the method is used systematically and very consistently in the HA. The leaders of the RPIW follow a rigorous standardized procedure for conducting each event and receive expert guidance and support from the Lean Promotion Office consultants. The concrete description of RPIW activities described below stands in stark contrast to critiques of global or vague descriptions of quality improvement interventions that appear in the literature (e.g., Hulscher, Laurant, & Grol, 2003). As part of this series of RPIWs, the Departmental leaders underwent extensive training in Lean as part of the Virginia Mason Production System’s “Lean Implementation Specialist” certification program. This program is designed to build capacity within the organization by teaching individuals how to conduct RPIWs and successfully employ Lean tools. The curriculum begins with several classes where Lean principles, tools, and applications are taught. Learners complete the rigorous, 20-module course by participating in RPIWs while 46 assuming roles with progressive levels of responsibility. Lean Promotion Office consultants oversee each RPIW to ensure that standard work is followed, and after a formal examination, learners become proficient in the RPIW method. They are then able to teach co-workers about Lean, lead additional RPIWs, and sustain the gains that are achieved. Before the specific areas of focus for the RPIWs could be determined, the Lean Promotion Office created high-level Value Stream Maps that outlined the patients’ experience as they flow through surgical services at each of the two sites. Following a thorough discussion among Departmental leaders, Senior Leadership, and the Lean Promotion Office, the domains for each RPIW were selected. Figures 9 and 10 depict the high-level Value Stream Maps for surgical services at Site 1 and Site 2. c Figure 9. High-level Value Stream Map for surgical services at Site 1. Source: Lean Promotion Office. 47 Figure 10. High-level Value Stream Map for surgical services at Site 2. Source: Lean Promotion Office. 48 49 The standard method for proceeding with an RPIW consists of eight weeks of preplanning that precedes each five-day RPIW event (Nelson-Peterson & Leppa, 2007). Individual RPIWs must have an executive sponsor, who has operational responsibility in the organizational area where the RPIW will take place. The sponsor agrees to a defined scope for the RPIW and signs an agreement before dates are selected and rooms are booked for the five-day event. The RPIW leaders then observe the current state of the target process(es), gather preliminary data, and arrange the first planning meeting with the full RPIW team. A series of three more planning meetings are held and all staff in the Department are made aware of the upcoming RPIW. This information is documented in an RPIW Project Form which names the team members and presents the general theme of the RPIW by summarizing some current problems as well as potential opportunities for improvement. Day one of the RPIW begins with introducing team members and orienting them to the planned activities for the week. A training session is delivered that explains the history and principles of Lean, describes several Lean tools, and prepares the team for employing relevant interventions that are tailored to the focus of the RPIW. Day two usually begins with Process Flow Charting, where the team maps many flows in healthcare, such as patients/families, providers, information, medication, supplies, and equipment. Process flow charting makes many forms of waste visible, and other Lean assessment tools may be used (e.g., spaghetti diagrams, root cause analysis, etc.; Rotter et al., 2018). The team proceeds by generating ideas for improvement, prioritizing them, and embarking upon systematic improvement trials using Lean tools and techniques. The improvement efforts are documented on templates called Target Progress Reports, which contain standard measures as suggestions for project teams to consider (Nelson- 50 Peterson & Leppa, 2007). Such measures include staff walking distance, parts travel distance, lead time, work(s) in progress, quality defects, and environmental/safety issues. The use of particular measures is dependent on the foci of the RPIW, as not all of the measurement domains are relevant to every situation. Data for the Target Progress Reports is collected manually by the leaders of each RPIW in collaboration with the Lean Promotion Office consultants. Baseline data that is collected at the outset of the RPIW is typically compared to post-RPIW data collected at 30, 60, and 90 day intervals to determine if any improvements were achieved and sustained. Each RPIW week ends with a final report-out of the results and identifying follow-up activities for sustaining and expanding on the team’s achievements. A substantive portion of the RPIW week is spent preparing for the final report-out, which is a formal presentation delivered to an invited audience that includes members of the unit/department where the RPIW occurred and senior leaders of the organization. The way in which the HA systematically conducts RPIWs lends itself to formal study, given the consistency and rigor that is employed. In summary, the RPIWs investigated in this study have the following core elements in every application:  Planning meetings with RPIW sponsors and Lean Promotion Office consultants.  RPIW Project Forms that outline the five-day RPIW.  Staff training in Lean.  Lean assessment activities (e.g., value stream mapping; process flow charting).  Improvement idea generation.  Systematic improvement trials using Lean tools, techniques, and methods.  Measurement of improvement using Target Progress Reports.  Report-out. 51 Additional details regarding the application of RPIWs in the HA are fully explained in an RPIW Handbook and in RPIW Leader checklists, which are forms of standard work that are made available to staff by the Lean Promotion Office. The details of how this study was conducted will be described in the next chapter. 52 Chapter Four: Methods This chapter describes the overall framework of the study, the design of components within the study, sampling techniques, data collection procedures, and data analysis procedures. The study utilized a mixed-methods design because there were several aspects of the HA Lean program that needed investigation. All components of this study were conducted in accordance with the Second Edition of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (CIHR, NSERC, & SSHRC, 2010) and with formal approval from the HA Research Ethics Board and the University of Northern British Columbia Research Ethics Board. Study Approach The approach of this study was limited to a specific sector of service delivery within the HA, as opposed to attempting to study the various impacts of Lean on the entire healthcare system. The study examined Lean as it was applied to surgical services. Furthermore, the study focused on the series of RPIWs that were held at each of two hospitals in the HA between December 2013 and December 2014. Within each RPIW, various Lean improvement techniques were utilized. The logic of the study design was built upon a modified version of the Puterman et al. (2013) evaluation framework briefly described in Chapter Two. The two main areas of focus are economic evaluation (inputs as compared to outputs/outcomes) and employee experience and engagement. Figure 11 provides a graphic illustration of the logic of this study. 53 Evaluation Plan Economic Evaluation Employee Experience and Engagement Analysis Statistical Analysis Event Level Sector Level Event Level Sector Level Event Level Sector Level Not Feasible Return on Investment Not Feasible Pre-Post Analysis In-Depth Interviews Survey t-tests Interrupted Time Series z-tests Figure 11. Evaluation logic of the present study. The two dimensions of economic evaluation and employee experience and engagement were studied simultaneously at two levels: the event level, and the service sector level. At the event level, the intent was to conduct economic evaluations of individual RPIWs and investigate employee experience and engagement through in-depth interviews. At the sector level, I conducted an economic evaluation across all RPIWs, using several statistical analyses as part of this investigation. I investigated employee experience and engagement at the sector level by surveying a broader range of staff who participated in the RPIWs. A few points are noteworthy in relation to the perspective of this study. I approached this study as an external evaluator with the guidance of the HA administration. Thus, this study is from the perspective of the HA in terms of improvement in surgical services and associated economic costs. Also, the economic evaluation was retrospective in nature, insofar as event level data was provided to me by the Lean Promotion Office after the completion of the 54 RPIWs, and sector level data was extracted from data warehouses that store routinely collected data. This chapter is divided into four sections. First, the attempt to conduct economic evaluations of individual RPIWs at the event level will be described. Second, the method used to economically evaluate all of the RPIWs combined will be explained; this perspective is referred to as the sector analysis. Third, the methods used to explore employee experience and engagement at the event level will be outlined. Fourth, the method used to explore employee experience and engagement at a broader sector level will be discussed. Economic Evaluation at the Event Level Research design. The main research question for this portion of the study was, “What are the economic benefits of RPIWs at the event level?” This question is linked to the study’s purpose of comparing cost of inputs to quantified outcomes and clearly delineating a process for conducting a cost-benefit analysis in the form of rate of ROI for individual RPIW events. The RPIWs sampled for inclusion in this study were the six RPIWs conducted in surgical services at each of two hospitals (Site 1 and Site 2) in the HA during 2013–2014. The focus of improvement for each RPIW was determined by the leadership team, using the high-level Value Stream Maps described previously. The decision to study a series of RPIWs within a single service delivery sector was purposeful so that an economic evaluation could be feasibly conducted within a reasonable time frame. The following is a brief description of the series of six RPIWs conducted at Site 1. RPIW#1 was designed to address problems related to the flow of implants in the operating room. There were challenges with ensuring that all equipment was sterilized, selected, and 55 uniquely packaged for each patient, then stored appropriately for optimal efficiency. Issues related to this process caused staff confusion, re-work, and delayed or postponed surgeries. RPIW#2 addressed problems with identifying, re-ordering, repairing, and re-assembling broken/un-usable/missing surgical instruments. Challenges associated with un-useable instruments resulted in staff searching for equipment, duplication of work, surgical delays, and frustration. RPIW#3 targeted operating room change-over time. This refers to the time period in between surgical cases when the operating room must be cleaned and prepared. A lack of role clarity, work prioritization, and poor communication often resulted in lengthy changeover times and surgical delays. RPIW#4 focused on improving the on-time performance of the first case each day. Analysis of the high-level Value Stream Map revealed poor coordination and collaboration amongst teams, and patients not being properly prepared prior to surgery. Late start times affected all subsequent surgeries, resulting in staff experiencing pressure to complete scheduled surgeries and/or the cancellation of surgeries at the end of the day. RPIW#5 examined processes upon the patient’s arrival at the hospital on the day of surgery. Difficulties related to patient registration and assessment by clinical care providers were noted, which often resulted in duplication of work and quality defects (e.g., incomplete/delayed lab results). RPIW#6 concentrated on pre-surgical screening processes, which were highly variable and frequently caused scheduling changes and other inefficiencies. There were many opportunities for improving consults with nursing, pharmacy, and anesthesia, as well as improving diagnostics and the processing of physician orders. 56 The analysis of the high-level Value Stream Map at Site 2 prompted leaders to conduct RPIWs that addressed the unique needs of their surgical department. A brief description of these follows. RPIW#1 was intended to improve the flow of information within the pre-surgical screening office. There were a large number of opportunities for improvement, including increasing patient readiness for surgery, and clarifying staff roles and responsibilities. Improvement efforts were desperately needed to address incomplete surgical booking packages, which were contributing to cancelled/postponed surgeries. RPIW#2 attempted to improve the flow of information and patients through the daysurgery department. Several processes were problematic, such as incomplete patient information, poor access to information, and short-notice scheduling changes. RPIW#3 aimed to improve the flow of emergency surgical patients in order to improve the overall operations of the surgery department and increase throughput. Issues such as poor staff communication, missing information, and long patient transport wait times were impacting elective surgery schedules. This resulted in inefficiencies across departments and poor patient experience. RPIW#4 was designed to optimize the preparation of the operating room schedule by improving a number of sub-processes that were problematic. Issues related to incomplete booking packages, management of patient wait lists, and lack of coordinated service delivery resulted in less optimal use of operating rooms and extended wait times for surgeries. RPIW#5 attempted to build on RPIW#3 and make further improvements that would result in more effective and efficient patient flow for individuals requiring emergency 57 surgery. There were many opportunities to improve surgical consults, standardize the flow of patients to various departments, and prepare patients for surgery most efficiently. RPIW#6 was intended to improve the throughput of surgical patients by addressing complications associated with wound care. This RPIW focused on enhancing Enterostomal Therapist referral and treatment processes, improving patient readiness for discharge, and reducing length of stay. Table 2 summarizes the RPIWs under study, including location, dates, and the focus of improvement efforts. Table 2. RPIWs Conducted at the HA Sites 2013–2014 It was not necessary to recruit participants for this first portion of the study, as the Target Progress Reports provided existing data from the RPIWs. Notwithstanding, the RPIW participants were given information sheets to inform them that this study utilized data from the RPIWs they were involved in. 58 Data requirements. Costing analysis. Comprehensive costing data was required in order to conduct an ROI analysis at the RPIW event level. The source of costing data was the Project Forms completed by the Lean Promotion Office for the RPIWs. The Project Forms provide an array of data such as staff time, Lean Promotion Office support, travel, and supplies. Each Project Form furnishes data that is specific to a particular RPIW, and there is some variability in the amount of resources applied to the RPIWs. The Project Forms were anonymized by the Lean Promotion Office before being provided for analysis in this study. The process for obtaining costing data and calculating costs associated with the RPIWs in this study is similar to the rigorous economic costing analysis used by Sari, Rotter, Goodridge, Harrison, and Kinsman (2017). These researchers define the cost of personnel time during participation in Lean events as indirect costs. Since the Project Forms obtained for this study listed all staff involved in the RPIWs by position title, exact wage rates for each participant were available and could be tallied across all participants. The employment benefits (Canada Pension Plan contributions, Employment Insurance premiums, vacation pay, and Extended Health premiums) were estimated as a percentage of the gross salary for all participants. Staff remuneration for time spent on the improvement event constituted the majority of costs associated with conducting the RPIWs. Costing data associated with staff remuneration was retrieved from the HA payroll dictionary. Other sources of indirect costs were not included in the costing calculations. The costs associated with meeting rooms were not included, since HA facilities were utilized and determining the exact costs of meeting rooms proved too difficult to quantify. The costs 59 related to the planning meetings among senior leaders of the RPIWs were not included, since the Lean Promotion Office did not track these costs on the Project Forms. Similarly, the staff time related to updating the Target Progress Reports at 30, 60, and 90 days post-RPIW was not included in the costing, as this information was also not collected by the Lean Promotion Office. These excluded costs represent a limitation of the study, since all costing figures are underestimated, which affects the ROI calculations. Staff time spent on implementing changes after the RPIW week were not included in the costing analysis, as improvement work is considered to be part of daily operations. This included consultations between the RPIW teams with other staff in their respective departments (i.e., non-RPIW participants), which was considered to be part of implementing improvements. The ongoing training costs for the leaders pursuing certification as Lean Implementation Specialists were also excluded, as these costs were considered to be separate from costs solely associated with RPIWs. One of the largest direct costs associated with the RPIWs (and Lean at the HA in general) was the original outlay by the organization to purchase the Virginia Mason Production System. The initial purchase cost was $271,000 and included consultancy services, training materials, tools, templates, etc. Six individuals from the Lean Promotion Office were originally trained as part of this contract package and the consultancy services continued for 24 months, during which time, coaching and certification support was provided. The inclusion of start-up costs may not be necessary in this study, however, since the Lean Promotion Office was established long before this research was initiated. Other authors suggest that start-up costs should not be applied in this sort of analysis (Howard & Pathak, 60 1999). RPIW costs borne by the HA both with and without the initial outlay will be addressed in Chapter Five. Costs associated with backfilling staff are treated as direct costs in this study because the Target Progress Reports indicated the exact costs accrued above the regular operating costs of the Department. Each RPIW was unique in terms of which staff were backfilled, with the most common position being the Registered Nurse. Casual employees were brought in at regular rates of remuneration to replace the RPIW participants. A constant rate of $100 was applied to each RPIW to roughly estimate the cost of supplies. The Lean Promotion Office consultants typically utilized flip chart paper, adhesive notes, felt pens, and rolls of large butcher paper to create Value Stream Maps and other analytic devices during the RPIW meetings. Occasionally, the Lean Promotion Office consultants had to travel to lead the RPIW meetings. The travel costs varied across the RPIWs and were made up of expenses such as car rentals (when HA fleet vehicles were not available), hotel fees, and meal per diems. While actual travel receipts were not obtained for this study, the cost estimates listed for travel are considered to be fairly accurate. Table 3 provides an example of the costing analysis for an RPIW, which includes both direct and indirect costs. Costing analyses for all the RPIWs appear in Appendix C. 61 Table 3. Example of the Costing Analysis RPIW Benefits. To do an ROI analysis at the event level, one needs to quantify and monetize all costs and benefits related to each RPIW. Target Progress Reports used to track the RPIWs would ideally provide data on the benefits realized from the RPIWs. Completed Target Progress Reports from each RPIW were obtained from the Lean Promotion Office (and anonymized prior to being submitted for analysis). In summary, the Target Progress Reports revealed many sources of waste identified by the RPIW teams, and demonstrated that several improvements were measured and documented. Most of the stated improvements were associated with efficiency or flow measures of one or several processes that were part of the RPIW. For example, Site 1RPIW#3 was focused on reducing the time to clean operating rooms between surgical cases. Data collected in the Target Progress Report for Site 1-RPIW#3 (see Figure 12) indicated a 62 reduction in lead time. Defined as the total time required to complete a specific process from beginning to end, lead time is made up of several sub-processes which culminate to produce the overall lead time. Figure 12. Example of data collected in the Target Progress Report for one RPIW in the HA. Source: Lean Promotion Office. Unfortunately, the information about improvement or benefits on the Target Progress Reports was often incomplete, inconsistent with other sources of data, or not easy to monetize. For instance, in an attempt to calculate ROI at the event level, a pilot test of quantifying improvements reported in Site 1-RPIW#3 (Operating Room Changeover) was conducted. While the Target Progress Report indicates a 26% reduction in lead time for simple cases and a 53% reduction in lead time for medium cases, data collected from the HA’s data warehouses did not reveal substantive differences in room changeover times following the Lean intervention. Any improvement in operating room changeover time suggested by manually collected data seemed to dissipate and could not be corroborated by 63 data extracted and analyzed from data warehouses. Similarly, raw data on a sample of turnaround times were analyzed and t-test results also did not produce significant findings. Further analysis of the Target Progress Reports revealed the proportion of incomplete audits required to populate the reports. For Site 1:  68% of specific process improvements did not have updated information after 30 days.  47% of specific process improvements did not have updated information after 60 days.  49% of specific process improvements did not have updated information after 90 days. The results were better for Site 2:  29% of specific process improvements did not have updated information after 30 days.  33% of specific process improvements did not have updated information after 60 days.  9% of specific process improvements did not have updated information after 90 days. Not only was the data incomplete, but many of the RPIW activities and reported results were not easily quantifiable because many of the standard measures on the Target Progress Report require considerable resources to manually collect the data. For example, to quantify the benefits of any reduction of physical steps taken, staff would need to wear pedometers and precisely report the distance travelled related to modified processes. Researchers would need to observe staff processes in great detail, calculate the gains achieved by a sample of staff, and be able to extrapolate gains across an entire department. Any quantified gains would then need to be monetized by expressing reductions in physical steps in terms of dollars saved using hourly rates of pay as reference. Given the high cost and difficulty 64 associated with attempting to quantify and monetize the Target Progress Report measures, the economic evaluation defaulted to the sector level analysis2. Economic Evaluation at the Sector Level Research design. In this study, I have tried my best to follow the recommendations based on the suggestions in the Consolidated Health Economic Evaluation Reporting Standards (CHEERS; Husereau et al., 2013). There are 25 recommendations in the CHEERS checklist which are not necessarily applicable to any single study. However, I have tried to address as many as are applicable throughout the study. The main research question for this portion of the study was, “What is the economic benefit of RPIWs at the service sector level?” This question was linked to the method of conducting an ROI analysis of RPIWs as it is applied to surgical services at the HA. The approach to conducting ROI in this study was the Phillips Return on Investment Method (Phillips & Phillips, 2007). This model is illustrated in Appendix D. Although a detailed description of each of the steps in the ROI method will not be presented here, a few points are notable. First, one of the key steps in the ROI method is isolating the effects of the program, which is often overlooked in evaluations. There are several techniques that have been used by organizations to tackle this important issue, 2 After this research study was started, a method for rigorously collecting data from Lean events was developed by the Saskatchewan Health Quality Council (province of Saskatchewan, Canada) in 2015. Called the Kaizen Tracker, this sophisticated data collection instrument can be used to more accurately estimate the costs and results of RPIWs, as it quantifies the per-minute wage cost of many professions and facilitates the calculation of benefit of various process improvements. The instrument allows the user to capture financial impacts on a variety of improvement domains, such as space, inventory, walking distance, set-up time, and cycle time reduction. The Kaizen Tracker better quantifies improvement activities and produces an estimate or desired outcome (not demonstrated benefits). While it may be possible to quantify care providers’ activities/physical steps taken, etc.—and even calculate their remuneration by the second—an evaluator would have to first assume or demonstrate that the changes are sustained and secondly have to extrapolate the savings across a number of professionals working on a unit in order to calculate cost benefit/ROI analyses. To date, data has been collected on hundreds of RPIWs across the province, but the Saskatchewan Health Quality Counsel has not completed a cost benefit analysis using this advanced data collection tool (D. Campbell, personal communication, January 8, 2018). 65 including control groups, trend line analysis, forecasting models, participant estimates, manager estimates, senior management estimates, expert’s input, and customer input (Phillips & Phillips, 2007). Another key step in the ROI method involves monetizing benefits/outcomes and making comparisons to project costs. For example, if it could be shown that Lean produced increased throughput in a health facility by reducing waste, then the cost of providing service for a single patient is reduced (given that staffing and other overhead factors remain constant). Therefore, the annual value of reduced costs per patient (i.e., cost savings) can be compared to the costs of the Lean intervention in order to calculate ROI. The general formula for ROI used in this study is as follows: ROI (%) = Benefits (Cost Savings) – RPIW Costs X 100 RPIW Costs While this formula for calculating ROI is straight forward, calculating the costs of the Lean interventions and outcomes is much more difficult. In this study, the challenge was to quantify any benefits/cost savings resulting from the RPIWs at the sector level, since quantifying benefits/cost savings at the event level was unrealistic. Three post-intervention outcomes that could be gleaned from existing data warehouses were identified and quantified in order to calculate ROI at the sector level. These three performance measures were: 1) surgical volumes3, 2) overtime utilization, and 3) sick time utilization. The first indicator (surgical volumes) is consistent with other research, which used surgical capacity as a key measure of Lean management in their study (c.f. Collar et al., 2012). Fairbanks (2007) also used surgical throughput measures in her study of using Lean and Six Sigma to improve 3 The terms “Surgical Volumes” and “Number of Surgeries” are used interchangeably for brevity in this study. 66 operating room efficiency. The rationale for selecting these measures is simple; if Lean is effective in removing waste and improving efficiency, then we can expect to see improvement in any or all of these areas. To complete the elements required for the ROI formula, the total costs of the RPIWs were tallied for both Site 1 and Site 2. An analysis of patient outcome measures was also conducted, as a means to include balancing measures and assess whether any unintended consequences may have resulted from the Lean interventions. The patient outcomes include post-surgery mortality rate, complication rate, infection rate, and surgical re-admission rate. Given that this portion of the study utilized systems data that is routinely collected in surgical services, recruitment of participants was not required. To move beyond partial evaluation, as delineated by Drummond et al. (2005), this study attempted to compare the outcomes from the RPIWs to an alternative intervention. In this case, the alternative intervention for comparison was no-intervention control groups. Thus, a quasi-experimental non-equivalent control group pre-test post-test design was utilized to conduct this research in an applied setting (Coly & Parry, 2017; Keppel & Wickens, 2004). The performance of surgical services at the two intervention hospitals (Site 1 and Site 2) were compared to similar hospitals in the HA (herein identified as Site 3 [control to Site 1] and Site 4 [control to Site 2]) to better understand the true impact of the RPIWs. These hospitals were selected as matched sites according to pre-specified organizational characteristics (e.g., Site 1 and Site 3 have similar services and numbers of beds, as do Sites 2 and 4; Coly & Parry, 2017). Using this method, the study was able to formulate reasonable comparator groups with which to assess changes in outcomes over time, and in particular, before and after the introduction of the RPIWs. 67 Data collection. To facilitate an ROI analysis at the sector level, the costing analysis conducted at the event level was summed and applied to the sector level analysis. Outcome data (i.e., number of surgeries, overtime) was extracted from the Picis OR Manager application for surgical services (Picis, 2017). Picis OR Manager is a comprehensive computerized operating room software system that manages an array of data, including waitlists, scheduling and booking information, pre-surgery screening, case records, and surgeon preference cards. The system can also capture other elements of surgical services such as case volumes, time of patient entry into operating room, surgical incision times, patient departure from operating room, etc. To calculate the ROI across all RPIWs at each site, an average cost per case was estimated based on data obtained from the MEDITECH general ledger system. MEDITECH (2018) is a leading software and service company that provides health information management services to the HA. The average cost per case for Site 1 was calculated by tallying the estimated cost per case across 26 fiscal periods (13 fiscal periods for the PreIntervention period of the study, and 13 fiscal periods for the Intervention Period)4. The sum was then divided by 26 (total fiscal periods) to arrive at $1686.00 as the estimated average cost per case for Site 1. The same calculation was conducted for Site 2, to arrive at $1034.00 as the estimated average cost per case. The costs associated with staff overtime and sick time were also obtained from the MEDITECH payroll system for inclusion in the ROI analysis. Data extraction from Picis OR Manager and the MEDITECH systems covered one year before the RPIW interventions, the year the RPIWs occurred, and one year following the RPIWs. All data associated with this portion of the study was managed according to the same high standards of ethical practice adhered to for other procedures in the study. All data was 4 The average cost per case estimates do not include payments to surgeons and anesthetists. 68 anonymized so that no names or identifying information were revealed during data collection and analysis phases. Data analysis. For the ROI analysis, an overall ROI calculation was performed to examine the cumulative effect of all RPIWs at Site 1 and Site 2. This analysis attempts to quantify any benefits/cost savings resulting from the RPIWs in terms of surgical volumes, sick time, and overtime utilization. For the purpose of this study, the monetized benefits are actually the cost savings resulting from lower surgery costs, overtime, or sick time. Any monetized cost savings were compared to the costs of conducting the RPIWs in order to determine whether the benefits/cost savings exceeded the investments at each site. The ROI analysis was performed using two distinct periods: the period before the RPIWs were introduced (i.e., the pre-intervention period), and the period when the RPIWs were conducted (i.e., the intervention period). The pre-intervention period is defined as December 10, 2012 to December 6, 2013. The intervention period is defined as December 9, 2013 to December 5, 2014. The intervention period encompasses the six RPIWs that were conducted at Site 1 and the six RPIWs that were conducted at Site 2. The ROI analysis was conducted between the Pre-intervention and Intervention period because most of the RPIWs were complete half way through the Intervention period. It was felt that sufficient time had elapsed within the Intervention year for improvements to stabilize and yield results (if, in fact, they were effective and successfully sustained). 69 To further investigate the impact of the RPIWs and the sustainability of improvements (if any), additional statistical analyses were conducted.5 To do so, a third period was considered in addition to the two periods outlined above. This was referred to as the post-intervention period and defined as December 15, 2014 to December 11, 2015. Figure 13 depicts all three of the periods under study. The series of statistical analyses were temporal in nature, that is, they examined the sites both before and after the RPIW interventions. Figure 13. Sequential RPIW intervention periods. First, to test the effect of the RPIWs on surgical volumes, t-tests were used. The average number of surgeries performed at Site 1 and Site 2 during the pre-intervention period were compared to those during the intervention and post-intervention periods at 5% significance level (p < 0.05). The maintained (null) hypothesis was: H01: The average number of surgeries performed at Site 1 and Site 2 will remain the same and will not systematically change following RPIW interventions. In addition to the t-tests for comparing the mean outcomes before and after the Lean intervention, the Interrupted Time Series (ITS) method was also used to check whether there had been any changes or interruptions in both the level and slope of the outcome time series. The ITS method uses the detailed information and disaggregated data to examine any breaks 5 Prior to analysis, all data was cleaned to ensure proper date ranges were used. For example, to compare the number of surgeries at each facility during various periods, only surgeries conducted during the day were included (evenings/ weekends/holidays/emergent cases were excluded). This process for data cleaning standardized the number of days analyzed per period. Data extracted from various databases were imported into Microsoft Excel electronic files. 70 in the structure of data. It is a very reliable method for evaluating the effects of interventions on health outcomes at the population level, especially when time series data points are large enough, which is the case in this study (Bernal, Cummins, & Gasparrini, 2016; Cruz, Bender, & Ombao, 2017; Fretheim & Tomic, 2015; Kontopantelis, Doran, Springate, Buchan, & Reeves, 2015; Penfold & Zhang, 2013). In a typical ITS design, a baseline linear model is fitted to the data to estimate the initial level (intercept) and the slope (gradient) in the outcome time series as follows: Yt = B0 + B1 t + ut Where Yt is the outcome variable, B0 is the intercept, B1 is the slope, and ut is a random error. The baseline model is then compared with an alternative model that allows for changes in the intercept as well as slope of the fitted trend line for capturing effects of intervention: Yt = (B0 +A) + (B1 +S) t + ut In this model, A represents the change in the level of the outcome series (Yt), and S denotes the change in the slope of Yt. Therefore, estimated values of A and S are used to measure the magnitudes of potential interruptions as a result of intervention, and whether such interruptions or changes are statistically significant. Secondly, Levene’s test was used to test the variability of surgeries performed at Site 1 and Site 2 during the three periods at 5% significance level (p < 0.05). A decrease in variability is considered an improvement given that a stable system has predictable cost, quality, and performance (Provost & Murray 2011; Trusko, Pexton, Harrington, & Gupta, 2007). In this case, the maintained (null) hypothesis was: H02: The variability of surgeries performed at Site 1 and Site 2 will be the same for pre-intervention and intervention (as well as post-intervention) periods. 71 Thirdly, t-tests were used to test end-of-shift overtime hours at Site 1 and Site 2 during pre-intervention, intervention, and post-intervention periods at 5% significance level (p < 0.05). End-of-shift overtime was defined as overtime pay attributed to staff within a bargaining agreement that worked beyond their scheduled shift. Overtime paid to staff on evenings, weekends, or statutory holidays were not figured into the calculation. End-of-shift overtime was used as an indicator of efficiency, and assumes that any reductions reflect a more efficient operation that is able to complete the slated surgical operations within regularly scheduled time frames. The null hypothesis here was: H03: The average number of end-of-shift overtime hours at Site 1 and Site 2 will remain the same following RPIW interventions. Fourth, t-tests were used to test sick time at Site 1 and Site 2 during pre-intervention, intervention, and post-intervention periods at 5% significance level (p < 0.05). The relevant null hypotheses for these tests were: H04: The average number of sick time hours at Site 1 and Site 2 will remain the same following RPIW interventions. Fifth, to investigate unintended consequences potentially resulting from the RPIWs, Z-tests were performed on patient outcomes. Four patient outcome measures were compared across sites: post-surgical mortality rates, complication rates, infection rates, and surgical readmission rates. The patient outcome measures were used as balancing measures in this study (Provost & Murray, 2011). The general form of the null hypothesis for these tests was: H05: The proportion of post-surgical mortality rates, complication rates, infection rates, and surgical re-admission rates at Site 1 and Site 2 will not significantly change following RPIW interventions. 72 The last series of analyses were cross-site direct comparisons between the intervention and control sites. The statistical analyses were conducted to directly compare the intervention and control sites across the three performance measures. The t-tests were used to test surgical volumes, overtime, and sick time utilization between Site 1 and Site 3 as well as between Site 2 and Site 4 across all three periods (at the 5% significance level, p < 0.05). For these tests, the null hypothesis was: H06: There will be no change in performance between intervention and control sites in terms of average differences in the number of surgeries, overtime, or sick time hours across the three periods of study. The types of quantitative analyses used in this study are consistent with recommendations found in the literature (e.g., Parry et al., 2013; Portela, Pronovost, Woodcock, Carter, & Dixon-Woods, 2015). Importantly, this study approached the economic analysis with the aim of accurately quantifying the monetary gains of interventions—to move beyond estimating theoretical cost avoidance, as many other articles do (cf. Moraros, Lemstra, & Nwankwo, 2016). The task of monetizing gains is complicated because concepts like cost savings and cost avoidance do not have widely agreed-upon definitions. Howard & Pathak (1999) note the high variability with regard to terms like cost savings, cost avoidance, and cost reduction, defining cost savings as a broad term that encompasses both cost avoidance and cost reduction. According to these authors, cost avoidance is defined as a measure to describe hypothesized future events avoided due to an intervention. Cost reduction, on the other hand, is defined as actual net dollars saved due to an intervention. Similarly, a National Association of State Procurement Officials (NASPO) research brief describes cost savings as a reduction in budgeted or projected resources (e.g., staff time, materials, and equipment) used for an 73 activity or business process as a result of a savings project (NASPO Benchmarking Workgroup, 2007). The NASPO authors define cost avoidance as “…a cost reduction opportunity [emphasis added] that results from an intentional action, negotiation, or intervention” (p. 5). The present study examined the actual outcomes of the RPIWs using both concepts of cost savings and cost reduction. Cost reduction analysis was applied to end-of-shift overtime and sick time. Cost reduction and ROI analysis were applied to surgical volumes in the following manner. If we assume a causal relationship between the RPIW interventions and increased surgical volumes—and no resources added to the service area—then the RPIWs could lower the cost per surgery, resulting in cost reduction. If the monetized benefit from lowered surgical costs exceeds the investment in the RPIWs, a positive ROI could be realized. This outcome would reflect an ROI in actual dollars resulting from the RPIWs, and not some hypothesized cost to be avoided at some future point in time. To clarify the use of these terms in this study, any cost reductions resulting from decreased end-of-shift overtime and/or sick time—and any reductions in costs per case of surgery—is reported simply as cost savings. Employee Experience and Engagement Research design. Employee experience and engagement is an important area of study if we are to learn more about whether Lean can act as a catalyst to help employees become more involved in their work and participate in process and system improvement. The general research questions for this portion of the study were, “What is the employee experience of RPIWs?” and “How did employees engage in the RPIWs?” The focus on participant experience is 74 consistent with the suggested approach to evaluating healthcare improvement offered by Parry et al. (2013). Figure 14 provides an overview of the research design used for investigating employee experience and engagement. Figure 14. Employee engagement and experience research methods. Qualitative research is an appropriate approach to investigate the employee experience and engagement dimensions of this study, as it is the preferred strategy when “how” or “why” questions are being posed, when the investigator has little control over events, and when the focus is on a contemporary phenomenon within some real-life context (Yin, 2014). Walshe (2007) concurs, stating that, “…the theoretical basis for the intervention (why and how it works) becomes more important than its empirical performance (whether it works) in any particular study” (p. 57). 75 Employee Experience and Engagement at the Event Level To examine employee experience and engagement at the RPIW event level, the qualitative tradition of phenomenology was the point of departure for this portion of the study. According to Creswell (2013): The type of problem best suited for this form of research is one where it is important to understand several individuals’ common or shared experiences of a phenomenon. It would be important to understand these common experiences in order to develop practices or policies, or to develop a deeper understanding about the features of the phenomenon. (p. 81) Thus, several in-depth, qualitative interviews were conducted with key informants that volunteered to provide information about their lived experience with RPIWs. Participants. This component of the study involved non-probability sampling, however, all participants selected for this component of the research were purposefully sampled—only HA staff who fulfilled the criterion of having experience with RPIWs at the two intervention sites were recruited for this portion of the study. At each of the RPIWs, performance metrics were displayed on a “Performance Wall,” which is typically located in a central strategic planning office for a Lean project. Participants for the interview component of this research were recruited using posters advertising the study (see Appendix E) that were located in the strategic planning office on or near the performance metrics wall. The strategic planning office was an ideal location for advertising the study since individuals involved in the RPIWs often visited this area and reviewed the performance wall. The advertisement was also widely posted around the surgical services areas (e.g., nursing stations/lunchrooms) as another way to reach staff that had experienced the RPIWs. The recruitment poster directed individuals to contact the 76 researcher by telephone if they were interested in participating in an interview for the study. There was no risk of real or perceived coercion to participate in the interviews, as staff chose to participate in the study on their own accord—no manager or other HA official promoted participation. Data collection. Individuals who indicated their interest in participation were provided a convenient appointment time to participate in an in-person interview. Interviews were set up as per the procedures outlined by Esterberg (2002), which include establishing an appropriate location and warming up. Interviews took place in a closed meeting room at a local library. Before proceeding, the participants were provided with a formal informed consent form to review and ask questions, if they had any (see Appendix F). It was made very clear that participation in the study was completely voluntary and an individual could choose to withdraw at any time during the study without penalty of any kind (i.e., a staff member's decision to decline or withdraw from participating in the study would have absolutely no impact on their employment with the HA). I used a semi-structured interview guide that was developed in consultation with the academic supervisor and other professionals. The questions revolved around two domains: the participant’s experience in the RPIWs, and their engagement in these workshops. The interview guide was pilot tested several times before it was used with study participants. The guide consisted of a framework of open-ended questions, and the participants were often asked probing questions in order to follow up on their responses and gain a better understanding of their subjective perspectives. For example, in an attempt to capture the essence of Lean as a phenomenon, the interview started with two broad, general questions 77 (Moustakas, 1994): “What have you experienced in terms of Lean?” and “What contexts or situations have typically influenced or affected your experience of Lean?” Participants were then asked to elucidate their experience in the RPIW, and explain what they thought were the enablers and barriers to the success of an RPIW. They were also asked to define engagement and were continually prompted to identify factors that may have positively influenced their engagement. Lastly, the participants were asked to describe how the RPIW was conducted and to provide recommendations for improving RPIWs. The interview guide appears in Appendix G. The qualitative interviews were audio recorded and transcribed. All recordings and transcripts were coded, electronically encrypted and password protected, and stored on a secure computer in a locked, alarmed office to maintain the confidentiality of participants and comply with ethical standards. Data analysis. The method of qualitative data analysis used in this study draws on the recommendations of Auerbach and Silverstein (2003), Creswell (2013), Miles and Huberman (2014), Richards (2005), Thomas (2006), and Yin (2014). The first step in data analysis was to enter the verbatim transcripts into NVIVO qualitative data analysis software. Computer software packages make the handling of large quantities of data more manageable by enabling the electronic coding, storage, retrieval, and analysis of data (Creswell, 2013; Hahn, 2008). Organizing Phase I. The analysis then proceeded in accordance with Richard’s (2005) approach of topic and analytic coding. It was useful to first code all of the answers to each question separately in order to view and organize the answers in isolation, an approach that is appropriate for 78 applied research settings (Lewins & Silver, 2007). This method of coding by question allowed me to retain the context of the responses throughout the analysis. I read through all of the responses to each question and coded every answer—with the single unit of analysis being either a part of a sentence or a multi-sentence chunk (Miles & Huberman, 2014). In this way, an inductive approach was used, whereby the codes used to classify the data were generated from the data as opposed to coding data using a prefabricated accounting scheme (Thomas, 2006). Large sections of each transcribed interview were re-coded again a few days later to test for the internal consistency of analysis (Miles & Huberman, 2014). The following formula was used: Reliability = Number of agreements X 100 Number of disagreements + Number of agreements Employing this formula ensured that the same codes were used to code the same blocks of data (or segments of transcribed text). In some cases, definitions of codes had to be adjusted or blocks of data re-coded. When the code-recode reliability exceeded the value of 90% as recommended by Miles and Huberman (2014), the analysis was considered reliable and the data set retained for further analysis. The transcripts were also given to a colleague experienced in qualitative data analysis who independently coded the data using the same procedure. The coded interview transcripts were then compared as an inter-rater reliability check. When discrepancies occurred, we discussed, negotiated, and made adjustments (e.g., agreed on the most appropriate code when two looked good, altered codes to become more encompassing or narrow, etc.) (Miles & Huberman, 2014). The same reliability formula was used to ensure both researchers used analogous codes to code the same blocks of data over 90% of the time. 79 Organizing Phase II. Next, the dataset was grouped by question (with their associated coded responses) into domains, in order organize the analysis by topic. There were six domains in total: participants’ definitions of engagement, their impressions of RPIWs, their experience in the RPIW, factors enabling engagement in RPIWs, barriers to engagement, and recommendations for RPIWs. Analysis phase. The data then underwent iterative phases of analytic coding (Analysis Phase in Figure 15). The codes were analyzed to identify emergent categories that summarized the codes. This produced a series of categories, which were stored as nodes in the NVIVO software. The final analysis was to group the categories into themes that were organized under each of the six domains. This analysis was organized into a matrix, following the suggestion of Miles and Huberman (2014), who argue that data displays should be a normal part of reporting conclusions in qualitative research just as they are in the quantitative tradition. The matrix is summarized and appears as a series of tables in Chapter Five. The tables display the themes and the categories that support them. Quotations. I then selected participant quotes so as to provide additional specific and concrete evidence in support of the categories and themes (Creswell, 2013). Participant quotes were chosen for presentation in the Results chapter according to the following decision rules: 1) researcher confidence, and 2) the extent to which the participant’s narrative represented exemplars of the category definitions. Although the quotes were selected from categories 80 supported by the most sources and frequently recurring codes, quotes from less recurrent codes were also considered for presentation as results. As a final means of ensuring the trustworthiness of the data and the analysis, I have attempted to supply outside observers with a chain of evidence by documenting the procedure in great detail (Yin, 2014). Figure 15 graphically depicts the data analysis procedure. In this way, I have tried to make the derivation of evidence transparent to readers so as to support the results presented in the following chapter. 81 Figure 15. Qualitative data analysis procedure. 82 Employee Experience and Engagement at the Sector Level This section addresses the last portion of the evaluation logic of this study. To expand the scope of the research beyond individual interviews with key informants, the survey method was used to study the impact of Lean on employee experience and engagement at a broader level (within the service sector). The purpose of the survey was to complement the in-depth interviews and attempt to gain a more comprehensive understanding of the intangible benefits that may be associated with the RPIWs (Fowler, 2009). A survey instrument was developed with the general research questions in mind (“What is the employee experience of RPIWs?” and “How did employees engage in the RPIWs?”). The survey method permitted an analysis across a larger number of individuals, which provided a different perspective on the impact of the RPIWs. Participants. This component of the study followed criterion-based sampling, as only HA staff who had experience with the RPIWs at Site 1 and Site 2 were recruited for the study. A Program Coordinator from the Lean Promotion Office created an electronic distribution list of eligible staff. The invitation to participate in an evaluation of Lean was sent via email directly from the Lean Promotion Office. The email contained an external link to the survey managed by the researcher. The email invitation was sent on August 4, 2015. The first reminder to complete the survey was sent on September 11, 2015. The second reminder to complete the survey before the deadline was sent on October 13, 2015. The survey was closed on November 4, 2015. In total, the survey was in field for three months. 83 An individual’s decision to decline to participate or withdraw from the survey did not result in any impact on their employment with the HA. Further, the HA administration did not have any access to the submitted surveys or responses of individual participants, thus there was no risk of real or perceived coercion associated with participation in the survey. Data collection. Individuals receiving the invitation email sent by the Lean Promotion Office who opted to proceed by clicking on the embedded external link were taken to a separate website housing the survey. The introduction page provided instructions and reviewed ethical issues. The voluntary nature of participation was outlined and the individual had to provide informed consent prior to participating in the survey. For the series of questions that followed, the participant was directed to read the statement and select from the list of response options. There were also a number of open-ended questions that provided fields wherein the participant could type in their response. I used the Fluid Surveys web survey tool to collect participant responses. Fluid Surveys is a Canadian company, and data from submitted surveys was received and stored on their secure server. Secured Sockets Layer (SSL) encryption provided extra security between the web server and participants’ browsers. I retrieved the data for analysis by exporting the dataset to Microsoft Excel. This was managed using encrypted electronic storage devices and a password protected computer located in a locked office. Instrument. I sought permission to adapt some of the survey questions from the OHA-NRC Picker Employee Experience Survey (Ontario Hospital Association, 2010). This survey is a validated and widely used instrument in the field (Lowe, 2012). I also consulted with the 84 developer of the OHA-NRC Picker survey to assist with the design of the survey used in this study (G. Lowe, personal communication, May 26, 2013). Adapted questions covered several domains of interest, with a focus on engagement and other areas of employee experience. In consultation with my supervisor and others with extensive experience in administering surveys, I added conceptual questions about respondents’ experience with Lean and the Lean training provided, and demographic questions. The survey was pilot tested, and the electronic version was field tested prior to being officially administered. The survey appears in Appendix H. Data analysis. Data collected using the Fluid Survey software and exported to Microsoft Excel, was cleaned to ensure the file was complete and in order. The dataset was coded for further analysis. Non-responses to questions were handled by leaving respondents who did not provide information out of the number of respondents in the analysis for that particular question (Fowler, 2009). The analytic plan consisted primarily of reporting descriptive statistics for each item in the survey. Two additional analyses were conducted: 1) analyzing results according to respondents’ experience with Lean, and 2) comparing responses to items according to the two sub-groups of administrators (also known as Management) and clinicians. The results of these analyses did not yield additive information so are not reported here. In the descriptive statistical analysis, results for each item considered response rates in each response option, paying particular attention to the top box scores (i.e., top two options of the Likert scale), which is standard practice for analyzing surveys of this kind (Lowe, 85 2012). Responses to options four and five on the five point Likert scale were combined to analyze the percentage of positive answers for each of the survey items. Given the sample size was limited to the number of staff who had participated in the RPIWs, it was anticipated that results should be reported using raw numbers in addition to the percentage of positive responses in an attempt to accurately represent the findings and avoid potentially misleading readers regarding the magnitude of any results. Responses to open-ended questions were analyzed using the same procedure for qualitative analysis described in the previous section. NVIVO computer software was used for coding and theming the qualitative data obtained from the survey. In the next chapter, we turn to a description of the results in this study of the impact of RPIWs in the HA. 86 Chapter Five: Results This chapter presents the study’s results in four sections. The results of the economic evaluation at the sector level are provided first, with the ROI results shown for each site where RPIWs were conducted. The study ventured beyond ROI analyses and investigated the impact of RPIWs using various statistical techniques, which are presented in the second section. The next two sections report the results from the employee experience and engagement analyses, which capture intangibles or non-quantifiable aspects of the RPIWs. The results from the thirteen key informant interviews are reviewed before the survey results are presented. Economic Evaluation at the Sector Level Return on investment. The overall research question for this portion of the study was, “What is the economic benefit of RPIWs at the service sector level?” Data was extracted from existing warehouses so that an overall ROI calculation could be conducted in order to address this question. Preintervention to intervention period data on the number of surgeries performed, sick time, and overtime utilization were quantified and monetized. The calculations and results of this analysis are shown in Table 4. As described in Chapter Four, the estimated average cost per case was $1686 for Site 1 and $1034 for Site 2. The estimated cost per case was used to calculate a dollar figure associated with the volume variability between the pre-intervention period and the intervention period for each site. The dollar figures associated with end-ofshift overtime and sick time reported in Table 4 are the actual costs coded to Site 1 and Site 2 for the specific date ranges in the pre-intervention and intervention periods. The total costs of the RPIWs at Site 1 and Site 2 were tallied for inclusion in the ROI formula. 87 Table 4. Monetized Results of the RPIWs According to the summation of the individual costing sheets, the costs of conducting the six RPIWs at Site 1 was $143,287. Entering the total cost savings for Site 1 into the ROI formula, the return on investment was: ROI (%) = $120,742 – $143,287 X 100= -15.73% $143,287 That is, for each dollar invested in implementing RPIWs at Site 1, the organization lost roughly 16 cents after the costs of the RPIWs were recovered. For Site 2, the costs of conducting six RPIWs was $146,954. After entering the cost increase (or total monetized loss) from Table 4 into the ROI formula, the return on investment was: ROI (%) = - $34,718 – $146,954 X 100= -123.63% $146,954 That is, for each dollar invested in implementing RPIWs at Site 2, the organization lost about one dollar and 24 cents. 88 As noted in Chapter Four, there was an original outlay to purchase the Virginia Mason Production System. If the initial outlay is included in the ROI analysis, the costs of implementing the RPIWs at both sites increases to $561,241 (i.e., $271,000 to introduce the Virginia Mason Production System plus $290,241 in costs associated with RPIWs at both sites). In this case, the return on investment would be: ROI (%) = $86,024 – $561,241 X 100= -84.67% $561,241 That is, for each dollar invested in implementing RPIWs at both Site 1 and Site 2, the organization lost about 85 cents. It is important to note that the annual budget for operating the Lean Promotion Office is not included in the ROI analyses. The annual budget for this department is approximately $454,000, however, this information is provided as background only. This annual budget is not included in the cost analysis of RPIWs because the costs associated with the Lean Promotion Office involvement were calculated on a per-event basis (see Appendix C). Statistical analyses. To augment the ROI results and attempt to move beyond what Drummond et al. (2005) would describe as partial evaluation, additional analyses using comparison groups as controls were conducted. Several statistical tests were performed to compare the performance of the RPIW intervention sites to the control sites where no RPIWs occurred. The results of statistical tests used to examine the surgical volumes within the four sites before-and-after the intervention (and longitudinally) will appear first. This will be followed by the results of tests that examined the variability of surgical volumes at all four sites. The results from t-tests that were used to explore the impact of RPIWs on overtime and sick time 89 utilization are presented next. The results from Z-tests used to compare patient outcomes (as balancing measures) will then be reported. Lastly, the results from direct comparisons between intervention and control sites on performance measures will be provided. Statistical analyses of surgical volumes. To examine the daily surgical volumes within the four sites for the pre-intervention period as compared to the intervention period, one-tailed t-tests were employed. The results illustrated in Table 5 show that there were no statistically significant findings; except at Site 3, which showed a significantly higher average number of surgeries performed daily in the intervention period (M = 32.00) than in the pre-intervention period (M = 30.85). The t-tests yielded the same results when surgical volumes at the four sites for the preintervention period were compared to the post-intervention period. Table 5 shows there were no significant findings; except at Site 3, which showed a significantly higher average number of surgeries performed daily in the post-intervention period (M = 32.11) than in the preintervention period (M = 30.85). Thus, we failed to reject the null hypothesis (H01) as the number of surgeries performed at Site 1 and Site 2 remained the same following RPIW interventions. In fact, the number of surgeries performed at Site 3 and Site 4 as control sites either did not change or increased, reinforcing the result that RPIWs did not improve the volume of surgeries at the two intervention sites. 90 Table 5. t-test Results Comparing Surgical Volumes by Site and Period To complement the t-test analysis and examine surgical volume variability longitudinally, the Interrupted Time Series (ITS) method was employed. The results for Site 1 are reported first, followed by the results from Site 2 as the two intervention sites. Next, the results comparing the intervention sites with their corresponding control sites (i.e., Site 1 versus Site 3, and Site 2 versus Site 4) are reported. The results are depicted here in a series of graphs, where trend lines are fitted through the actual data using the Ordinary Least Squares (OLS) estimation method. The estimations underlying each graph are reported in Appendix I. The ITS graph in Figure 16 shows the daily number of surgical cases at Site 1 over the three periods of study. Trend lines were fit to the data for the pre-intervention period, intervention period, and post-intervention period. Visually, a slight drop in level of the trend line (-3.70) appears at the beginning of the intervention period and the slope of the line seems to gradually increase (0.01 per day). Yet, statistical calculations reveal that the level change was not significant (p=0.1243) and the slope change was not significant (p=0.0728). To expand on this analysis, the pre-intervention period data was compared to two timeseries segments combined (i.e., the intervention and post-intervention periods) to account for 91 gradual changes in surgical volumes that may have occurred over an extended time period following the intervention. Figure 17 graphically illustrates the trend lines fitted to these particular time segments in the study for Site 1. The graph depicts a slight negative gradient of the trend line that spans the intervention and post-intervention periods. Statistical analysis, however, revealed that the change in level for these comparison periods was not significant (0.802; p=0.5772). The change in slope was also not significant (-0.006; p=0.3908). Number of surgeries per day 92 Number of days Number of surgeries per day Figure 16. ITS results comparing pre-intervention period to intervention period on surgical volumes at Site 1. Number of days Figure 17. ITS results comparing pre-intervention period to intervention and post-intervention periods combined on surgical volumes at Site 1. 93 The ITS analysis showed similar results for Site 2, the other site where RPIW interventions occurred. While the trend line in Figure 18 appears to reflect a slight positive gradient within the intervention period, statistical results were non-significant for changes in level (-2.25; p=0.5377) or slope (0.005; p=0.5975). To mirror the analyses conducted at Site 1, the pre-intervention period data for Site 2 was compared to the combined time series segments of intervention and post-intervention periods. Statistical analysis resulted in non-significant findings for changes in level (-1.50; p=0.4921) and slope (-0.00; p=0.9382). Figure 19 displays the results of this analysis. Number of surgeries per day 94 Number of days Number of surgeries per day Figure 18. ITS results comparing pre-intervention period to intervention period on surgical volumes at Site 2. Number of days Figure 19. ITS results comparing pre-intervention period to intervention and post-intervention periods combined on surgical volumes at Site 2. 95 To investigate surgical volumes at the intervention sites as compared to the control sites, a variant of the analyses described above were conducted. Here, the differences between daily surgical volumes performed at Site 1 and Site 3 were calculated. The difference scores were then plotted across the three periods of study, along with corresponding trend lines fit to this dataset. Logically, if surgical volumes were increasing at Site 1 over time (presumably due to the RPIW interventions), we would expect to see a positive gradient in the trend line following the introduction of the interventions. An alternative explanation for a positive slope could be that surgical volumes at Site 3 had systematically decreased over time, possibly due to some secular events. ITS analysis was used to compare the difference scores across the two sites from the preintervention period to the intervention period. Figure 20 presents, in graph form, the results of this comparison between Site 1 and Site 3. The results of statistical tests revealed no significant change in level (1.99; p=0.4371) or slope (-0.00; p=0.4223). To continue with our pattern of analysis, pre-intervention period data was compared to data from the intervention and post-intervention periods combined (on difference scores across Site 1 and Site 3). Figure 21 illustrates the findings graphically. The results of statistical tests revealed no significant change in level (0.715; p=0.6395) or slope (-0.004; p=0.5943). Number of surgeries per day 96 Number of days Number of surgeries per day Figure 20. ITS results comparing pre-intervention period to intervention period on surgical volume difference scores between Site 1 and Site 3. Number of days Figure 21. ITS results comparing pre-intervention period to intervention and post-intervention periods combined on difference scores between Site 1 and Site 3. 97 The last series of ITS analyses were conducted to compare difference scores between Site 2 and Site 4. Difference scores across the two sites from the pre-intervention period to the intervention period were graphed first (see Figure 22). There were no significant changes in level (-0.49; p=0.9142) or slope (0.00; p=0.9622). Figure 23 shows the comparison between pre-intervention period and the intervention and post-intervention periods combined. No significant changes in level (-1.43; p=0.5987) or slope (0.00; p=0.9919) were discovered following this ITS test. In summary, the ITS analyses do not provide evidence to reject the hypothesized outcome (H01). The volume of surgeries performed at Site 1 and Site 2 did not systematically change following RPIW interventions in a sustainable manner. Number of surgeries per day 98 Number of days Number of surgeries per day Figure 22. ITS results comparing pre-intervention period to intervention period on surgical volume difference scores between Site 2 and Site 4. Number of days Figure 23. ITS results comparing pre-intervention period to intervention and post-intervention periods combined on difference scores between Site 2 and Site 4. 99 To examine the variability in surgical volumes across study periods and investigate potential improvements resulting from the RPIWs, additional analyses were conducted. Levene’s test was used to examine equality of variance at the four sites. Any reduction in variance would indicate improvement (i.e., a more stable system). The results are shown in Table 6. Statistical significance was not found in any of the tests. Therefore, the null hypothesis (H02) was not rejected as the variability of surgeries performed at Site 1 and Site 2 did not systematically change in the desired direction following RPIW interventions. The variability of surgical volumes at Site 3 and Site 4 across the study periods also did not systematically change. Table 6. Results from Levene’s Test for Equality of Variance Statistical analyses of end-of-shift overtime. The next series of t-tests examined end-of-shift overtime at the four sites for the preintervention period as compared to the intervention period. Table 7 shows that there were significant findings at Site 1, where the average number of end-of-shift overtime hours were significantly lower in the intervention period (M = 14.49) than in the pre-intervention period (M = 16.65). The average number of end-of-shift overtime hours were also significantly 100 lower in the post-intervention period (M = 13.64) as comparted to the pre-intervention period (M = 16.65). At Site 2, the other intervention site, significant results were found, but this result counters support for the efficacy of the RPIWs. The average number of end-of-shift overtime hours were significantly higher in the intervention period (M = 3.52) than in the preintervention period (M = 2.56). The average number of end-of-shift overtime hours were also significantly higher in the post-intervention period (M = 3.42) as compared to the preintervention period (M = 2.56). For the no-intervention control sites, Site 3 had a significantly higher number of end-ofshift overtime hours in the intervention period (M = 5.70) than in the pre-intervention period (M = 4.01). Results also showed that Site 3 had a significantly higher number of end-of-shift overtime hours in the post-intervention period (M = 7.50) as compared to the pre-intervention period (M = 4.01). Table 7 shows that there were no significant findings at Site 4 for any of the comparison periods. So, there were mixed results for the two intervention sites as related to hypothesis H03. Site 1 yielded significantly lower end-of-shift overtime hours for the intervention and postintervention periods, whereas Site 2 showed significantly higher end-of-shift overtime for the intervention and post-intervention periods. Mixed findings were also revealed for the control sites. Site 3 had higher end-of-shift overtime (on average) for the intervention and post-intervention periods, while overtime hours were similar at Site 4 across the periods of study. 101 Table 7. t-test Results Comparing End-of-Shift Overtime by Site and Period Statistical analyses of sick time utilization. The t-test was used to examine employee sick time at the four sites. For both intervention sites, sick time utilization was found to be significantly lower during the intervention period compared to the pre-intervention period (see Table 8). On average, Site 1 had significantly less daily sick time utilization in the intervention period (M = 37.22) than in the preintervention period (M = 39.96). Site 2 also showed significantly less daily sick time utilization in the intervention period (M = 2.56) than in the pre-intervention period (M = 5.75). However, sick time comparison between pre-intervention and post-intervention periods showed no significant findings; for both intervention sites, the average amount of daily sick time utilization was similar across these periods. A different pattern emerged for the control sites. At Site 3, sick time hours were similar across all periods of study (no significant findings). Site 4, however, had significantly more daily sick time utilization in the intervention period (M = 14.43) than in the pre-intervention period (M = 5.21), and also significantly more when the pre-intervention period (M = 5.21) was compared to the post-intervention period (M = 9.65). 102 These results show partial support for rejecting the null hypothesis H04. The average amount of sick time utilization at Site 1 and Site 2 was significantly lower in the period immediately following the RPIW interventions, but similar when pre-intervention and postintervention periods were compared. For the control sites, Site 3 had similar sick time utilization (on average) for all periods of analysis as hypothesized. Site 4 showed higher sick time utilization across the periods of study. Table 8. t-test Results Comparing Sick Time Utilization by Site and Period Statistical analyses of patient outcome measures. Table 9 displays the results of the analysis of patient outcome measures. These analyses are included in the study as balancing measures, to examine any unintended consequences that may have resulted from the RPIWs. The Z-test was used to compare the post-surgical mortality rates, complication rates, infection rates, and surgical re-admission rates across the periods for all sites. In every case, no significant findings resulted. Thus, the null hypothesis (H05) was not rejected as the balancing measures were not significantly lower following RPIW interventions. Likewise, these patient outcome measures were similar at both of the no-intervention control sites. 103 Table 9. Results of Statistical Tests for Outcome Measures Direct comparisons between intervention and control groups on performance. Table 10 shows the results of direct comparisons between intervention and control sites on the performance measures of average number of surgeries, average overtime hours, and average sick time hours across the three periods of study. The top section of the table lists the average number of surgeries by period for each of the sites. This allows comparisons on average surgical volumes by period between Site 1 and Site 3 and between Site 2 and Site 4. In the middle section of the table, one can make comparisons between Site 1 and Site 3 on average end-of-shift overtime hours by period, and the same comparisons can be made 104 between Site 2 and Site 4. Average sick time hours appear in the bottom section of the table for comparisons between intervention sites and their matched control site. Table 10. Direct Comparisons Between Intervention and Control Sites on Performance Measures If the RPIWs had an impact on these variables, we would expect to see statistically significant changes in the average differences of these measures between the intervention sites and control sites from pre-intervention to intervention period (and from the preintervention to post-intervention period). Several t-tests were calculated to statistically test the performance differences between intervention and control sites on surgical volumes, endof-shift overtime, and sick time utilization across study periods. Table 11 displays the results of these analyses. 105 Table 11. t-test Results Comparing Intervention and Control Sites on Performance Differences In terms of the average differences in the number of surgeries, Site 1 and 3 showed no significant change in the first analysis. When the average differences were compared between the pre-intervention and post-intervention periods, the result was significant. The average difference in the pre-intervention period was (M = 9.911) as compared to the postintervention period (M = 8.056). This reduction in the average differences for the number of surgeries is explained by an increase in the average number of surgeries at Site 3 (see Table 10). This finding is contrary to expected results. If the RPIWs were impacting average differences on surgical volumes in a desirable way, we would expect to see a greater average number of surgeries at Site 1. When the average differences in the number of surgeries were compared between Sites 2 and 4, both period comparisons were not statistically significant. In other words, there was no change in the number of surgeries at Site 2 as a result of the intervention. With regard to average differences of end-of-shift overtime hours, a different pattern emerged (see Table 11). For Site 1 vs 3, the average difference was significantly lower in the 106 intervention period (M = 8.784) than in the pre-intervention period (M = 12.641). This means that there was a reduction in the number of end-of-shift overtime hours at Site 1 and an increase at Site 3. This result provides some indication of improvement from the RPIWs. As for the comparison between the pre-intervention and post-intervention periods, the average difference was even lower in the post-intervention period (M = 6.133 compared to the M = 12.641), which is to say that there is more evidence of improvement from the RPIWs in the post-intervention period. In comparing Site 2 with Site 4, the average differences in end-of-shift overtime were significantly higher in the intervention period (M = 1.885) than in the pre-intervention period (M = 0.869). This finding is also contrary to expected results. If the RPIWs were impacting average differences in overtime hours in a desirable way, we would expect to see a reduction in end-of-shift overtime at Site 2. In fact, we observe that the number of overtime hours has increased at Site 2 in the intervention period (see Table 10). The same result holds in the comparison between pre and post intervention periods, although the magnitude of change is smaller. For average differences in the sick time hours between Sites 1 and 3, the mean was significantly lower in the intervention period (M = 15.280) than in the pre-intervention period (M = 20.063). This could be taken as evidence for the effectiveness of RPIWs in reducing the average sick time hours at Site 1. However, the average differences did not change when the pre-intervention period was compared to the post-intervention period. Therefore, no evidence of sustained improvement was observed in the post-intervention period. Lastly, the change in average differences in sick time were statistically significant in both period comparisons for Site 2 vs 4. The average difference was significantly lower in the 107 intervention period (M = -11.869) than in the pre-intervention period (M = 0.538). This reduction is evidence of improvement at Site 2, and more so, a reflection of greater sick time utilization at Site 4 (see Table 10). The average difference was also significantly lower in the post-intervention period (M = -4.508) compared to the pre-intervention period (M = 0.538). Again, this reduction is evidence of improvement at Site 2, and somewhat a result of more sick time at Site 4 in the post-intervention period. On the whole, there is some evidence of improvement due to RPIWs (e.g., reduction in sick time) but the results are not conclusive throughout the comparison of indicators across all periods and between intervention and control sites. Therefore, H06 cannot be clearly and consistently rejected or supported. We now turn to the results of the analysis of employee experience and engagement to explore the effects of the RPIWs outside of the ROI and statistical analyses. Employee Experience and Engagement at the Event Level Collecting qualitative data through semi-structured interviews yielded several reliable themes related to employee experience and engagement. Respondents provided insights into their overall experience in the RPIWs as well as their general impressions of Lean. They were also asked to define what engagement means to them, which helped to uncover several factors that enable employee engagement in RPIWs. Information provided by respondents also produced a list of barriers to engagement in RPIWs. Finally, the interviews produced numerous recommendations to improve RPIWs and make them more engaging. These findings help us understand how and why employees got engaged in the RPIWs and provide insight into the strengths and weaknesses of the method. 108 In total, 13 individuals responded to the recruitment posters that advertised the study and agreed to an in-depth interview. These staff members represented a wide range of service providers in healthcare, including:  3 Senior Administrators  2 Physicians (Surgeons)  2 Program Support Administrators  1 Nursing Manager  1 Patient Care Coordinator (Nursing Supervisor)  2 Registered Nurses  1 Licenced Practical Nurse  1 Housekeeping Staff As described in Chapter Four, the results from the in-depth interviews were organized into six domains: participants’ definition of engagement, their impressions of RPIWs, their experience in the RPIW, factors enabling engagement in RPIWs, barriers to engagement, and recommendations for RPIWs. Presentation of the results begins with the domain title and list of emergent themes. This is followed by a table summarizing the constitution of themes: the dominant categories that make up the themes (not all categories are shown), and the number of codes and sources that support the categories. Miles and Huberman (2014) recommend counting in data analysis as this method allows one to quickly view the content of a large amount of data while protecting against bias and maintaining analytical honesty. Finally, some illustrative quotes from respondents are provided to show individuals’ lived experiences and bring the results to life. 109 Domain 1: Definition of engagement. Four themes emerged from respondents’ definitions of engagement: Listening and Learning, Motivation, Responsibility to Improve the System, and Supportive Environment. These themes were foundational to interpreting responses to other questions in the interview, as they aid in understanding what participants mean when they refer to engagement. Table 12 lists the themes in this domain, summarizes the categories that make up the themes, and shows the number of supporting codes and sources. Table 12. Themes for Definitions of Engagement Foundational to engagement was being motivated at work and wanting to do a good job. o Category: Motivated at Work Participant: “To me, it has to do with both attitude at work as well as motivation to make improvement.” Participant: “It’s actually wanting to do good work.” o Category: I Want To Do a Good Job Participant: “Most of my engagement just comes from doing a good job, being efficient, and being good in my primary role.” Participant: “I guess I care. Because I wouldn’t be doing this if I didn’t care.” 110 It was also noted that being in a supportive environment is very important for engagement. o Category: Being in a Supportive Collegial Environment Participant: “You can’t stay engaged though by yourself. Engagement is not a one man show. Engagement is a bunch of people.” Participant: “I just try to enjoy the people that I am working with and appreciate everybody as different and individual but we are all here to provide patient care, so how do we work together as a team that changes daily to have a safe environment for patients? And try to enjoy that as opposed to redirect people to be a certain way. People have strengths that I don’t and vice versa. So complimenting each other and building as a team as opposed to saying – no, you disagree with me and I will shut you out....” Domain 2: Participants’ impression of RPIWs. The qualitative interviews revealed three themes that encapsulate participant’s overall impressions of RPIWs: Collegial Approach to Improvement, Systematic Method for Improvement, and Takes Time to Understand (see Table 13). Overall, participants give the impression of liking RPIWs because it creates an environment of improvement and provides the tools and understanding necessary for staff to bring about positive change in the workplace. Table 13. Themes of Participants’ Impressions of RPIWs Quotes that support two dominant categories help us see how participants appreciate the unique aspects of RPIWs as an improvement method, although it is evident that it takes time to understand and learn. 111 o Category: A Framework for Thinking That Emphasizes Waste Removal Participant: “And the top thing for this, for me, is always about – when you do process improvement, everything else doesn’t teach you this, but Lean does. And that’s about driving the waste out of the system. You can do process improvement, process mapping, but it kind of stops at a certain point. It doesn’t take it to that next step that Lean does in the fact that it’s really always thinking about ‘how can I recapture capacity because I am driving waste out.’ That’s the way my brain thinks. If I getting rid of this, then I can recapture something. I can park it here, you know what I mean. It’s something in the bank. That’s the cool thing about Lean.” o Category: From Fogginess to Clarity Participant: “Well, to start with, I would say that, and I remember the early days when I started in observation and looking at some of the education materials online, I was a little ‘Wow, I’m not sure how all of this hangs together’ and what really I am going to be able to use in the toolkit and the methodology. And then I did the entry level, familiarize yourself with the ABCs of Lean and that still was not clear because it was still more like an entry that I had had with [another healthcare system], so I was still kind of perplexed and unclear of ‘Well I could see how this could work but I got another repertoire of tools that have worked and steaded me well, so not sure what the difference is yet.’ And it wasn’t until I read in detail the Virginia Mason book, because it resonated with me as a health care organization and actually how they had recreated the assembly line into health care and how that really worked and held true. What really resonated with me in my experience to go ‘I get it’ was many of the experiences in the book where they gave examples of how they had regained efficiency by driving waste out of the system. There were beautiful examples of an ambulatory care setting and a cancer care setting, and the book was an easy read, and it’s just a lovely tool. I really believe that’s a great tool to indoctrinate as an introduction to health care and Lean. My experience to start was a little foggy, I was in the fog, I’ll say, as I’m crossing the water. I started to get some clarity and went ‘Bang, ok, I get how this might help.’” Domain 3: Participants’ experience in the RPIWs. Two themes emerged from the exploration of participants’ experiences in the RPIWs, one positive (Organized and Efficient) and one that appears to be more negative (Unclear Expectations). Table 14 lists the themes and their dominant categories as well as the number of codes and sources. 112 Table 14. Themes of Participants’ Experiences in the RPIWs Elaborating on the theme of experiencing the RPIW as being organized and efficient, the following quote hones in on the specificity and conciseness of the RPIW objectives as being beneficial. o Category: Objectives Specific and Concise Participant: “They were extremely specific and I think that’s why I enjoyed the process because you do whittle it down to something you can do something about in the week that you have the RPIW. So you’re not overwhelmed by the size of the problem because they get it down to such a specific thing that you are trying to focus on and so, in that way, you have high success because you are not trying to solve all the world’s problems, you’re just taking this one little facet of it.” Another participant provided details regarding the theme of unclear expectations. o Category: Lack of Involvement of Staff Participant: “I don’t think we even did interviews with staff. I don’t think we actually talked to anyone on the floor about what their hopes were, what their problems were, what their value stream is. It was kind of like – here it is. I think that was a bit disengaging and it made it really hard to actually do the measures. It made it hard to help guide the sponsors into where this RPIW should be focused. That one, I don’t think was as successful just because of that.... The deliverables were a bit loose. When it comes to the target sheet, it was very loose and I don’t think we had a lot of hard outcomes from it.” Domain 4: Factors enabling engagement in RPIWs. Six themes were identified as enablers to engagement in RPIWs: Culture of Continuous Improvement, Dedicated Time Permitted, Gradual Engagement, Interest in Improvement, Like Lean Method, and Social Aspects. Table 15 lists the six themes, their dominant categories, and the associated codes and number of sources. 113 Table 15. Themes of Enabling Factors for Engagement in RPIWs The theme Culture of Continuous Improvement was supported by the dominant categories of having all staff involved and management support for Lean culture. Participants were very candid with their comments that buttress the categories. o Category: Involvement by All Levels of Staff Participant: “So often you hear frontline staff say – I’ve told them that a million times but no-one ever listens. If they feel they are going to be listened to and valued, I think that is really important and that can certainly be an enticement to take part.” Participant: “Educating and actually participating in an RPIW. I think that when there are so few people that know what Lean is and how it’s functioning in an organization, it lacks the momentum that it could have sweeping through an organization and down to the grassroots. So the more of those people you can involve in it. My impression of how it was done in the operating room, we had the directors and the high level people were all trained in it, and then the managers were trained in it, and they did a bit of training. And then it got down a little bit down to my level, but it didn’t go all the way down to the level that it needs to in order to facilitate the actual change. Because if you don’t convince those people that this is a good thing, then all bets are off. Because they will just practice the way they’ve always done it, unless you can convince them otherwise, unless they can see through trial and error that it has a purpose and it has a value.” 114 o Category: Management Support for Lean Culture Participant: “I think for employees to really get engaged, it’s really about the management structure. We can hold sessions and we can have an RPIW week and everybody has bought into it and people are participating. But it can’t stop just with events. It shouldn’t be a Lean event. It should be a Lean culture. The events maybe are highlights and help us focus where our areas of improvement need to be but you really need to develop that cultural piece. It should be shown by example. The management team – it should be part of their normal language, to look for ways to talk about process improvements, to talk about PDSA [Plan-Do-Study-Act] cycles, and engagement of staff.” The theme Gradual Engagement revealed how participants can progress from learners to promoters that help spread Lean methods and culture. o Category: From Learning About Lean to Promoting Participant: “When I first heard of it, I knew nothing about it. I went from an education of myself to what Lean was about, to participating in what Lean was about, to promoting others to participate in Lean and for others to learn what they can about it so that we can use it in our workplace.” The theme Interest in Improvement emerged as a key factor of engagement, and was supported by two dominant categories. o Category: Wanted to Learn More About Improvement Participant: “Wanting to understand it would actually be applied to our workplace. Not just going through the steps and then reading and answering the questions, but really thinking – how could this be implemented and integrated, how do I understand it so that I can share it with others?” o Category: Better Understanding of Lean Participant: “It’s interesting because the Lean Promotion Office support person who was in the thing at the last RPIW said this was best RPIW I ever attended with the most engaged. The group was engaged right through. We broke at Friday noon to give the final report and they continued to work on things until 4:00 in the afternoon. They were so engaged, they wanted to keep going. We wondered what was different about that. I think what is was, was they really grasped the concepts. I think sometimes with the Lean RPIW, and it may be just because everybody is being certified, so the first time you do your presentation in Lean, you’re kind of reading off of slides. The detriment for the participants is that they are not really understanding the concepts. They don’t understand how important the concepts are.” 115 Participant: “I think the more they understood about Lean itself, the merits of it, the more likely they are to get involved.” It was obvious many participants simply have a penchant for Lean, as evidenced by the emergent theme Liked Lean Method. Two dominant categories and supporting comments underscore the affinity people have for Lean and how much they appreciate the methodology. o Category: Liked How Lean Provides a Method for Improvement Participant: “I think, same with Lean. Lean provides people with some methodology that they feel that there is a process, that they have some control over contributing to the process. Or at least they have an understanding of how change occurs through PDSA [Plan-Do-Study-Act] cycles, through process flow mapping, all of the tools that we use in Lean. I think it helps people to understand how they could contribute to that system.” o Category: Provides Common Method and Language for Improvement Work Participant: “It’s got me engaged because I see it was a good methodology to improve, whether it’s efficient or quality or whatever.” Participant: “Lean has affected my engagement in that I can see potential for improvement, the opportunity to be involved with a group that really broke down the steps and identified inefficiencies. Knowing that applied to room turnover, there are inefficiencies that we overlook in a lot of the aspects of our work. If efficiency is an end goal, then Lean is a factor every day because you are always looking to improve.” Domain 5: Barriers to engagement in RPIWs. In contrast to the enablers of engagement, the analysis yielded four reliable themes that represent barriers to engagement in RPIWs. These themes are: Apathy, Financial Barrier, Mis-Fit of Lean to Healthcare, and Resistance. Table 16 lists themes, dominant categories, and associated codes and number of sources. 116 Table 16. Themes of Barriers to Engagement in RPIWs A lack of initiative or interest in improvement was the dominant category that supports the theme of Apathy. Quotes from three participants capture the notion of Apathy concisely and elude to the consequences of Apathy in the RPIWs. o Category: Lack of Initiative or Interest in Contributing to Improvement Participant: “I shared that one with the employee that was not engaged and I truly don’t think, even towards the end of the RPIW, she was really engaged. She didn’t want change to occur and I think it was more, and I hate to say this, I had to say this has to go through, we have to try some things. And I think that because of that, some of the sustainability was not there post. So if you don’t have employee engagement, you can do as many RPIWs as you want, but you are not going to have your sustainability.” Participant: “Non-engagement of people that are on the RPIW. I recall one that we did where we had an employee, she was a dominant employee within that department, and could well have jeopardized that RPIW because she was not willing to trial things to change. I had to have a strong conversation with her because it really was impacting how the RPIW was flowing. There wasn’t any willingness to try anything.” Participant: “People would do it but I didn’t see a whole lot of excitement from the whole thing.” Other participants pointed to a mis-fit between Lean and healthcare, explaining it in terms of the high variance in healthcare and the non-linearity of operations in the industry. o Category: Variance or Non-Linearity of Healthcare Process Participant: “When you think that the average nursing unit has probably 50 staff. At any given time, there is probably 12 that are working. So you’ve got 50 staff, you’ve got 100 physicians that come on to that unit, at [Site 1] for example. And then you’ve 117 got all the allied health, all the people that are involved depending on the process you are looking at. So you might have a team of 200 or 300 people who all do their work differently, who are faced with so many different variables because we are not a widget line. Things don’t happen in sequential order. Even when you are drawing a process map and taking them through that exercise, it’s so difficult for them to think linearly because nothing happens in a line. Nothing happens – this is step one, this is step two, this is step three. It’s not like I’m building a car. At first, we build the frame, and then put the wheels on and then add the chassis. This is human interaction; all the pieces actually go in circles down the process, not in a line. Every person deals with their little circles differently. That is a big barrier.” Participant: “I don’t know what they did in Toyota. Say they are manufacturing brake pads. That is very consistent and is sustained all the time. Even if you have the same surgery, say vascular creation of an AV fistula in which a patient is usually less acute, there are different patient needs every time. So you don’t go from step A, B, and C. You still, step A, bring the patient into the room, but you might have to transfer them differently. You might have to speak or interact with them differently. They might have different mobility issues. So even though the concept is the same from step to step, the subtleties are different and they are not controlled, like in a factory.” Resistance was a robust theme that emerged as a barrier to engagement. Participants voiced concerns related to added workload, but perhaps more troubling was resistance in the form of active sabotage. o Category: Added Duties Resulting for Staff Participant: “The RPIW process is probably seen by many groups as extra work. When you are busy working already, to add extra work like this, can be off-putting to some people to carry one more burden.” Participant: “I’m strictly speaking from a nursing point of view at this particular point, but most times when you say to a nurse – we are going to make some changes around here – automatically the first thing they are thinking is – you are going to give me more duties. That’s the first thing that is going to pop into their head. Or you are going to take away something that is going to make me have to work harder to make up for. Obviously not a very good mindset. But that, in healthcare, is one of the things you are going to have to face, or expect. You are going to expect that. Then you say – well no, we are just looking at every single section and see where we can improve it. But to see their point, some of the things that we ended up coming up with were increasing their duties. We trialed having the nurses in the room cleaning the gas machine, which was…at other places I’ve worked that was a nursing job, but here traditionally hadn’t been. We ended up kind of making their point, even though it didn’t survive through the length of the whole Lean project.” 118 o Category: Sabotage and Resistance Participant: “I had one destructor where I had to actually to bring the sponsors back and say – I think we’ve got an issue. Process owners, I talked to them. We’ve got an issue in one of the team cohort, they’re actually doing some sabotaging under the ground, you need to know this. I’m not telling you that I’m going to take them off the team but I’m telling you you’re going to have some issues post because he doesn’t want things to change. He likes the current reality.” Participant: “I remember in [another location] we did one in [a different Department] and there was actually a member of the team, a worker bee so to speak, and they were going to have nothing to do with this. Even though they had to be in the room, they would go out and they would undo everything we would do. So that was difficult. So then it’s the learning conversation – no, this is an initiative that is being supported by your sponsors and you need to be part of this. So the staff can become a barrier so you just have to mentor them.” Participant: “It became part of a job description issue. The nursing staff was asked to clean a piece of equipment and some of the nurses went to the union and said this is not our role, and we are taking somebody else’s job. It became very political.” Domain 6: Recommendations to make RPIWs more engaging. In total, ten themes emerged from the analysis that serve as recommendations to make RPIWs more engaging: Clarify Expectations, Improve Measurement, Improve Training, Increase Time to do Improvement Work, Increase Sustainability of Improvements, Involve More Staff, Make More Substantive Improvements, More Support, Shorten the RPIW, and Spread Improvements. Table 17 lists the ten themes, their dominant categories, and associated codes and number of sources. For brevity, only a select number of quotes are presented to provide depth to the dominant categories and themes. 119 Table 17. Themes of Recommendations to Make RPIWs More Engaging One participant highlighted the need to clarify expectations in order to make RPIWs more engaging. Specifically, good communication is essential. o Category: Good Communication Participant: “I know for one of the RPIWs that we did, we thought we had good communication. This is feedback from staff and working with the staff of the department after post RPIWs. We thought we’d done a really good job of communicating of the upcoming RPIW, what we were doing, and then throughout. In actual fact, the staff felt that we didn’t communicate well enough. We put posters up, we emailed, we mentioned it in staff meetings but the staff still felt that we hadn’t communicated.” 120 Another participant recommended that measurement be improved in order for RPIWs to become more engaging. o Category: Good Measurement Participant: “And you have to have good data. So your measurement, your premeasurement has to be really, really good so that you’ve got a starting point from there as to your improvement. And that’s really critical in the role and both team leader and workshop leader have to roll up their sleeves and make sure that you get those measurements, because they’re not easy. That’s laborious work.” Many recommendations were made to improve the training in order to improve engagement. Participants found parts of the training difficult to grasp, unrelated to the context of their responsibilities, or poorly delivered. o Category: Lack of Understanding of Lean Concepts or Tools Participant: “And also being a bit distracted, not really understanding some of the activities. Unfortunately, that’s part of the challenge with learning; you need to get out of your comfort zone to stretch your mind a bit. Sorting the cards, I think that was one activity, I didn’t really understand what we were doing. Or the paper airplanes; what is the point of spending time on doing this that I think has no relevance to room turnovers. It’s just showing how you can streamline processes, and cut out excess waste. There was value in it, but in the moment, it’s frustrating.” o Category: Improve Training Participant: “The first day, which is half teaching, I think it used to be way too long. There was a lot of slide reading and it really depended on who the Lean person was or the person who was being certified, whether they were a good presenter or whether they were someone who wasn’t comfortable doing it. That had a huge impact on how that was delivered. But I think they have really cut down the content and really hit some more of the key messages. There probably is a bit of room still to make some adjustments to that, but we’ve seen a lot of improvement, I have anyway, from it taking 8 hours and now being comfortable to be doing in 4 hours. I think process flow mapping – facilitating something so scattered is not everyone’s forte. And it also really depends on the group that you have in the room as well. But just some more training around – how can we facilitate or how can we run that process, because it really does feed the rest of the week.” 121 If RPIWs are to become more engaging, participants stressed that the improvements need to be sustained. o Category: Lack of Meaningful, Sustained Improvement Participant: “If I participated in a Lean RPIW as a student of Lean and I thought – we wasted on that last one because we didn’t get any sustainability and we didn’t get any true improvements, meaningful improvements. If I’m thinking that, sure the heck the staff are thinking that, that participated. And they are going back – well, it was a great week, I didn’t have to give patient care, I got a little bit of a break, straight day shift. So what did you gain out of it? Well, we worked on a few things. So where are the improvements? Well, you know. There was no outcome. So, I think that if we’re not really thoughtful about how we are using the RPIWs, the perspective of the staff is – they want to see what is the real change that came out of that.” A further recommendation was the theme of involving more staff. One participant noted over-representation of managerial staff involved in the RPIW, while many other comments expressed the need to generally involve everyone that works on a unit so that Lean can be most engaging and successful. o Category: Lack of Involvement of Staff Participant: “I think that perhaps a broader range of people involved in the workshop could only benefit the outcome. I saw a lot of management involved in this workshop, but I didn’t see a lot of frontline workers. In fact, I didn’t see any frontline workers. I didn’t see anesthetists. I didn’t see doctors, surgeons. I didn’t see surgical nurses involved in the workshop other than they were there and they were instructed.” Participant: “I think we need to have more frontline ownership, and I don’t mean managers and directors. I mean nurses who are doing shift work or booking clerks who are in the work every day. If you can share and spread some of that responsibility for follow up; ultimately it falls to the manager to do. But just really having more hands-on engagement by the people who are doing that work, I think will really help to sustain it. If everything is coming from the director, or from the manager or the PCC [Patient Care Coordinator], I don’t think you get that same kind of buy-in.” 122 Some participants criticized improvements as superficial, perhaps because the scope of the RPIW was too broad. o Scope of RPIW Too Broad Participant: “The ones that really flounder are ones that have picked too broad a topic to be able to actually fit in a rapid process improvement workshop. When you are looking at the workshop, you’ve really got 3 days to really nail down – what are we going to focus on, what is the improvement idea we are going to focus on, how might we put in some improvements and PDSA [Plan-Do-Study-Act] cycles? It’s such a rapid process that I think, just in terms of the ones I’ve been involved in, the ones that didn’t do as well were trying to look at too broad a process.” Following the overview of learning based on the in-depth interviews, the next section examines employee experience of the RPIWs with more breadth. Employee Experience and Engagement at the Sector Level A summary of the results from the custom survey are presented here, including information about the response rate, demographics, and participants’ experience with RPIWs. Responses to the questions incorporated from OHA-NRC Picker survey specifically related to engagement are reviewed first. Next, we examine the responses to the questions from the OHA-NRC Picker survey related to how employees experience their work. A third section looks at the results from the set of conceptual questions, before themes arising from qualitative questions are presented along with selected quotes. Response rate. In total, there were124 individuals who were invited to take part in the survey based on the eligibility criterion of having participated in the RPIWs. There was no response (did not access the survey) from 76 individuals. Seven individuals opened the survey but did not proceed further—they either closed the survey prior to entering any information (n=6) or 123 opted not to consent to the survey (n=1). An additional six individuals provided consent but did not respond to any of the survey questions. The remaining 35 individuals were considered to have participated in the survey—they opened the survey, provided their consent, and responded to at least one of the survey questions. This represents a response rate of 28%. Figure 24 outlines the sampling procedure and total sample attained for analysis. Figure 24. Sampling results for employee experience and engagement survey. The results presented below are formulated from the 35 surveys that were either partially or fully complete. Given the low response rate, it was only possible to report descriptive statistics (e.g., analysis of top-box scores). It was not possible to analyze the results using more sophisticated statistical techniques. Demographics. Of the 29 respondents that indicated their professional position, 15 were in the management category, and 14 reported being a clinical provider. The vast majority reported working in an acute care setting (n=26). There were no physicians represented among respondents. This may be partly explained by the fact that not many physicians participated in the RPIWs. Also, only HA email addresses were used in the correspondence sent by the 124 Lean Promotion Office, and many physicians do not regularly use the HA email system. Table 18 shows the breakdown of respondents by specific professional position. Table 18. Respondents by Profession Category Management Response Option n % 7 24% 3 10% Other – Admin 5 17% Licensed Practical Nurse 2 7% Patient Transport/Porter 1 3% Pharmacist 1 3% Physiotherapist 2 7% 6 21% 2 7% 29 100% Administrator Manager a Clinical Providers Registered Nurse b Other – Clinical Total Responses Note. aOther – Admin includes staff development educator, infomatics, human resources, clerical, and clerk. bOther – Clinical includes sterilization and porter. The majority of respondents had only participated in one RPIW (n=22). Two respondents had participated in two RPIWs, while ten respondents had participated in more than three RPIWs. In general, this finding mirrors another indicator of experience, Lean training. About half the respondents reported having completed three brief on-line training sessions (n=15) and a similar number reported receiving an orientation to Lean on the first day of an RPIW (n=14). This is not a completely accurate measure of experience because respondents could indicate involvement in several training opportunities. However, given only eight respondents claimed they had completed Lean Implementation Specialist Certification training, it is implied that the majority of respondents have limited training in Lean. Respondents also reported the length of time that they had been involved in Lean. Most respondents can be categorized as less experienced, with 12 individuals reporting less than 125 one year of involvement in Lean and 13 individuals reporting between one and two years of involvement. Seven respondents reported having two to three years of experience, and only three reported more years of involvement. Engagement questions. This section reports the results from items characterized as engagement questions on the survey. Six questions (Q9-Q14) were adapted from the OHA-NRC Picker survey, and according to Gibbons and Schutt (2010), these particular items measure what researchers have identified as being central to the concept of employee engagement. The analysis of these items consisted of combining responses from the top two response options in the Likert scale (agree and strongly agree), and calculating the percentage of positive responses for each question. Lowe (2012) encourages organizations to interpret employee engagement survey results in this manner, so that items receiving positive ratings of 60% or higher are considered strengths, while those with positive ratings of 40% or lower are considered to be areas for improvement. Figure 25 shows the results for this portion of the survey. The percentage of positive responses in questions Q9 to Q12 range from 60% to 78%, all above the thresholds recommended by Lowe (2012). The majority of respondents indicated that Lean has influenced them to look forward to Lean activities, has inspired better job performance, has similar values to their own, and has made them proud to tell others about the Lean work on their unit/Department. These results provide persuasive evidence that employee engagement is influenced by participation in Lean activities. 126 However, questions Q13 and Q14 have percentage of positive responses of 43% and 40% respectively, indicating opportunities for Lean to increase job satisfaction and improve perceptions of the workplace. Figure 25. Results from engagement questions. There were two additional questions (Q15 and Q16) in the survey that attempted to assess engagement. They asked respondents to rate their engagement both before and after their involvement in Lean activities (using a five-point Likert scale ranging from Strongly Disengaged to Strongly Engaged). Both questions yielded percentage of positive responses of 87%, even though the post-Lean involvement question had four more endorsements of strongly engaged. This result is considered to be unremarkable, since the small sample size did not permit statistical analysis to examine significance. 127 A further analysis was conducted to examine potential differences between administrative versus clinical respondents. Although one might speculate that administrative staff may be more inclined to be engaged in improvement methods, there were no marked differences across groups or over time. Experience questions. The next series of questions (Q18 to Q26) were also adapted from the OHA-NRC Picker survey, and measure how employees experience their work (Lowe, 2012). Figure 26 shows the results for these nine employee experience questions. The percentage of positive responses ranged from 46% to 72%. Only two questions, Q23 and Q25, showed results that perform above the 60% threshold and can be classified as strengths (Lowe, 2012). This implies that individuals from diverse backgrounds generally felt welcome during the Lean work, and that people were consulted about changes that affect them. The other questions in this section produced results that fell below the 60% threshold, indicating room for improvement in the areas of staff having opportunities to make suggestions, use their skills, support one another, work together, and treat each other with respect. The performance of items Q20 and Q26 suggest a need for improvement in terms of managers acting on feedback and workplace organization. 128 Figure 26. Results from experience questions. It should be noted that the engagement and experience questions adapted from the OHANRC Picker survey may no longer retain their psychometric qualities of reliability and validity because they were altered from their original form. Therefore, the positive results from the engagement questions and the relatively neutral results from the experience questions should be interpreted with caution. Other experiences with Lean. The next set of survey questions (Q27 to Q36) were designed to examine conceptual areas that one might expect to be impacted by RPIWs. The results appear in Figure 27. The 129 percentage of positive responses for these items ranged from 28% to 63%. In fact, only one item exceeded the threshold to be considered a strength. The percentage of positive response rate of 63% for question Q27 suggests that respondents thought the Lean work improved the quality of care or services delivered in their area. The remaining questions performed below the threshold and are therefore considered as improvement opportunities. These items are related to safety, access, reliability, work flow, workload, time management, teamwork, job expectations, and skill development. Figure 27. Results from other experience questions. 130 Results from open-ended questions. There were several open-ended questions on the survey that produced qualitative results. A key question (Q17) asked participants to explain how Lean influenced their engagement at work (if it had any influence). Data was analyzed and organized into the following ten emergent themes:  Lean improved my confidence with improvement.  Lean engenders feelings of accomplishment.  Lean helped me feel more valued.  Lean helped me to engage others.  Lean improved teamwork.  Lean improves empathy for others.  Lean increased understanding of work processes.  Lean increases responsibility.  Lean promoted new improvements.  Lean did not influence engagement. Lean did not have any influence on engagement for one individual. The respondent explains why in the following quote. Participant: “My engagement has remained as it was prior to my Lean experience. I feel the workshop was a time to recognize changes that could assist in the day to day operation of the clinic. I feel that the human experience cannot be measured in a unit of time to do a job. I feel the Lean workshop explained how the clinic could do the job in the time needed but in fact missed viewing the real time needed to deal with individuals with health challenges in the hospital setting. I have negative feelings still about the workshop because after implementing some changes that worked I felt the process of getting there was over-analyzed and the real changes were minimal compared to the week-long multiple party involvement used to get there. I felt that Lean was telling our clinic that in fact you can do the work with the staff you have, 131 (and here is the proof on paper) yet the true work is still not accomplished without additional staff.” For the most part, respondents were positive in reporting how Lean has influenced their engagement at work. The following selected quotes illustrate how Lean has increased understanding of work processes, helped people feel valued, and improved teamwork. Participant: “Lean has had a huge impact as to how I look at issues in the hospital related to access and flow as well as quality.” Participant: “Participation allowed me to meet and interact with individuals within the hospital who affect my ability to perform my job and who are affected by my job performance. Participation enabled me to understand the perspectives of these individuals and how our jobs impacted each other's work.” Participant: “I think that doing the Lean projects makes you feel like you mean something to [the HA]. As a regular employee, you don’t have very many opportunities for promotion, this really gives people the feeling of being a part of a ‘management’ style team.” Participant: “Lean has encouraged me to speak to co-workers and interdisciplinary staff around discharges to make a more stream-lined process for the patient and staff.” Another open-ended question (Q5) invited suggestions for improving the Lean training that participants received. The following are the five themes constructed from the analysis of responses to this question:  Better prepare the home team prior to the RPIW  Improve sustainability of the changes  Make training more interactive  Reinforce the training with follow up sessions  Use Canadian examples in training 132 The following quotes provide more depth related to the theme of improving sustainability. Participant: “We have also had some challenges with follow-up of the action items post RPIW. I would like to see a different process created so that there is more supportive oversight in place to work through the action items. We have made some errors in the past where the staff were not experienced enough or had the skills required to stay on top of the action items.” Participant: “However, we did not receive any follow up which I think is important. There are some key issues that should be addressed and many of our wonderful ideas have failed as some of the stakeholders refuse to follow the suggestions.” Participant: “During RPIWs, the emphasis on getting ‘something’ done within the week led to focusing on the quickest (instead of the most impactful) ideas – without any regard for a strategic plan for follow up.” Participant: “Although it was a positive experience to work on the gemba, with front line staff...participants were sometimes not well supported to accomplish follow up items (e.g., dedicated time and resources).” Another open-ended question (Q8) invited suggestions for improving the services of the Lean Promotion Office, producing the following ten themes:  Consider time of year when planning RPIWs  Expand the Lean Promotion Office as Lean expands in the HA  Guide the changes to speed up improvements  Improve communications with the home team  Improve coordination of all initiatives in the HA  Improve sustainability  Involve the Lean Promotion Office in new buildings and renovations  Link improvement ideas across RPIWs  Recognize that some goals are unattainable  Share the Lean strategic plan with certified implementation specialists 133 Finally, an open-ended question (Q37) invited participants to provide general comments about Lean. A total of nine themes emerged from the analysis:  Inadequate staff resources for success  Lack of sustainability  Lean project was not effective  Make Lean training during RPIW more interactive  Need more follow up and leadership support  Resistance to project changes  Suggest more Lean staff for service areas  Use plain language  Staff enjoy and support Lean The following quotes briefly capture the essence of what many of the respondents conveyed in terms of how they enjoy and support Lean in the organization. Participant: “I love Lean…it should be implemented in every aspect of healthcare as there are too many white elephants in the system. I would love to participate in more of the sessions.” Participant: “I think that Lean is a great opportunity for any staff member to feel like they are part of something bigger than their regular assignments.” Participant: “Lean has revolutionized the way I look at my work at the hospital.” In the next chapter, the results will be discussed in relation to the study’s research questions and the broader implications of the study. 134 Chapter Six: Discussion This chapter discusses the findings in terms of four sections of the evaluation framework used in this study: 1) the event level quantitative economic analysis, 2) the sector level quantitative economic analysis, 3) the qualitative interviews of individuals’ experiences, and 4) the analysis of staff experiences at a broader level via an internal survey. The chapter will review how each of the major research questions were addressed at the various levels. The discussion will highlight how this research adds to the current literature in this area and how the findings relate to other research in the field. It also provides some recommendations for Lean practitioners and for future research. Validity of Research Design One of the main findings of this research is actually how difficult it is to conduct a study of Lean interventions that will withstand scientific scrutiny. To address calls in the literature for more rigorous methods (e.g., Anderson, Røvik, & Ingebrigtsen, 2014; DelliFraine et al., 2010; Leggat, Bartram, Stanton, Bamber, & Sohal, 2015; McIntosh, Sheppy, & Cohen, 2014; Nicolay et al., 2014; Poksinska, 2010; Shojania & Grimshaw, 2005; Vest & Gramm, 2009), a great deal of effort was spent trying to apply the principles and procedures of sound research design to this area of study. It quickly became apparent that the resource requirements to pursue such a study are considerable, if not formidable for most organizations. A large amount of data related to Lean interventions needs to be manually collected at the process level, or extracted from data warehouses that store routinely collected data. Moreover, a researcher or team of researchers needs to be involved to manage the research process— something many organizations will not be able to dedicate resources toward. Ideally, individual staff members would collect their own data and regularly display it openly to 135 communicate performance and use data for improvement. But even if staff at the point of care are able to complete these tasks, the fact remains that resources (i.e., staff time) are required to collect data, process it, and post/report on it. Arguably, all of this work can only occur in an environment where continuous improvement is a cultural norm—that is, organizations that are considered mature when it comes to quality improvement. The difficulty in executing a study of this nature immediately arose when the two research questions related to economic evaluation were considered: 1) “What are the economic benefits of RPIWs at the event level?” 2) “What is the economic benefit of RPIWs at the service sector level?” Findings from Economic Analysis As mentioned in previous chapters, answering the first question in the economic analysis proved not to be feasible. The Target Progress Reports at the event level that could potentially be used as sources of data were incomplete. The method of manually collecting data for populating the Target Progress Reports at 30, 60, and 90 days was partial and discontinuous. In some cases, the data reported seemed questionable. Due to the challenges associated with manual data collection and monetization of improvements at the event level, attention turned to the second research question and analysis at the sector level. The shift to examining the RPIWs at the sector level is consistent with previous research, which found that efforts to improve individual components of the surgical process are unlikely to have substantive or sustainable impact (Cima et al., 2011). After examining the cumulative effects of the six separate RPIWs at each of the two sites, this study did not find sufficient quantitative evidence to claim that the outcomes justify the investments. 136 To further test the validity of the ROI analysis, statistical analyses were employed. Referred to in improvement literature as enumerative analysis (Massoud et al., 2016), the study employed traditional statistical methods (e.g., t-tests) using data from pre-intervention, intervention, and post-intervention periods. In the majority of cases, the t-tests did not show significant improvement gains. In addition to the enumerative analysis, the study undertook an analytic approach (Provost, 2011). Here, Interrupted Time Series (ITS) was used to examine surgical volume data longitudinally in an attempt to better understand the efficacy of the interventions. The ITS graphs on surgical volumes at Site 1 and Site 2 did not show any consistent improvement following the interventions, therefore, we conclude that there was no overall impact of the RPIWs on surgeries at these sites. The ITS graphs revealed a high variability of surgical volumes for these sites, which we can characterize as “not within statistical control”6 (indicating un-predictability of performance both before and after the RPIW interventions). The ITS graphs clearly show that gains achieved from the RPIWs, if any, were not sustained and therefore could not be consistently quantified for inclusion in an ROI analysis. Moreover, performance at the no-intervention control sites reflected their comparators in that both Site 3 and Site 4 demonstrated high variability (barring dips in the summer months, which may be explained as seasonal variance). One last but important point should be made as it relates to ITS analysis and this study. ITS analysis requires multiple data points over time (both before and after the intervention), which necessitates a considerable time-lapse between the time the intervention was 6 Statistical Process Control Charts were also computed in this study, but are not shown in favor of presenting ITS results. The control charts supported that the surgical volumes were not within statistical control. 137 administered until the time the results are available. This should alert us to resource costs associated with tracking data over extended periods (Coly & Parry, 2017). The design of this study attempted to meet the requirements of full economic evaluation as posited by Drummond et al. (2005) and consider costs, consequences, and alternatives in the analysis. Ultimately, a less than ideal approach was undertaken because it was not possible to meet all of the design elements required for full economic evaluation. Not only were there limitations in terms of the costing analysis, but it was not possible to quantify all of the potential benefits or cost savings. Also, the alternatives used in the study were nointervention control sites as opposed to alternative improvement methods or interventions. So, in spite of efforts to address all of the elements of full economic evaluation outlined by Drummond et al., readers are cautioned to interpret this study as a form of partial evaluation. The findings from this study are consistent with a Provincial Auditor of Saskatchewan report (2016) that examined the use of Lean in government ministries and concluded “…the measures do not provide information on whether the use of Lean is delivering results in demonstrating a return on investment in Lean…” (p. 171). However, in the same publication, a recurring message from interviews with ministries, agencies, and other sector agencies indicated that focusing on reports of cost savings, productivity gains, and cost avoidance did not capture other improvements respondents felt they had achieved through the use of Lean. Qualitative Findings: The Soft Side of Lean To expand on the findings from the economic and quantitative portion of this study, we needed to examine the so called “soft data.” Goddard, Mannion, and Smith (1999) consider soft data to be as important as hard data, arguing that a blend of hard and soft approaches is commonly used during the stages of gathering, processing, interpreting, and disseminating 138 data in both positivist and qualitative traditions. In the case of the present study, the qualitative findings can be used to complement the quantitative findings, not to replace them. Furthermore, the inclusion of qualitative methods in this study serves to avoid the gap of researching RPIWs without investigating the direct impact on employees—a situation that often occurs in Lean research and is well described by Holden (2011). Any study that ignores the effects of Lean on employees runs the risk of not learning about negative impacts. Failure to learn of such negative impacts precludes any corrective action, which could result in apathy, resistance, and the evaporation of any positive cultural transformation. Badurdeen & Gregory (2012) have pointed to the need to focus on the soft side of Lean, which they suggest is more difficult to understand than quantitative information. They argue that Lean researchers and managers should focus on respect for people, such as empowering staff to become competent and trusted problem solvers, because it is those very staff who can contribute to the cultural transformation that is critical to an organization’s success. Massoud et al. (2016) take this one step further by calling for a new epistemological paradigm for learning about healthcare improvement, one that may help researchers understand the context of improvement work and determine what might be generalizable. Not only do these researchers ask, “Did the improvements work?”, but they also challenge us to explore why did the improvements work, how we can attribute the results to improvement efforts, and how we can replicate positive changes. In the case of this study, to help understand whether the RPIWs actually worked, we can draw on results from the two research questions related to employee experience and engagement with RPIWs: 1) “What is the employee experience of RPIWs?”, and 2) “How did employees engage in the RPIWs?” 139 One of the key emergent themes from the qualitative interviews suggest that the RPIWs did work—if we interpret the theme of “Organized and Efficient” coupled with the dominant category of “Objectives Specific and Concise” as positive outcomes. It may be argued that these findings point to the effectiveness of RPIWs as an improvement method, although this focus is somewhat different than determining if the improvement activity produced measureable results in a specific area. This speculation must be considered against other emergent themes like “Unclear Expectations” and the dominant category “Lack of Involvement of Staff”, which seem to contradict the efficacy of RPIWs. Other emergent themes from the qualitative interviews portray employee experience with RPIWs in the affirmative (i.e., respondents revealed their positive impressions in that RPIWs are desirable, systematic, and collegial methods for improvement—while recognizing that it takes time to understand the method). The implication here is that RPIWs can be effective, if adequate time is taken to educate staff and include as many people as possible in consulting, problem solving, and shared decision-making processes. With regard to engagement, the thematic results can help inform us about how or why the RPIWs worked. Results from the qualitative interviews identified six themes as enablers to engagement. The enabling themes point to having a culture of continuous improvement as being a major factor in engagement. It was noted that engagement is gradual, and that engaged individuals are interested in improvement in the first place—the implication being that such individuals should be sought out when starting improvement projects. It was also discovered that many staff simply like Lean, because it is a systematic process and provides a common language and approach to improvement. This finding is consistent with a study by Mazzocato, Savage, Brommels, Aronsson, and Thor (2010) who found that Lean brought a 140 structured approach to problem solving and improvement to healthcare, which is an environment where operations are often not explicitly designed. Four themes also emerged from the interviews as barriers to engagement, which helps us to understand how or why the RPIWs perhaps did not work. The four barriers identified (i.e., apathy, financial barriers, mis-fit of Lean to healthcare, and resistance) are similar to some barriers to Lean implementation reported in previous research. In 2011, Brandao de Souza and Pidd conducted informal interviews at three different healthcare organizations in England and discovered that staff perceived Lean to be a management approach that is appropriate for manufacturing, but not for healthcare. Resistance was also noted in their study, although characterized as resistance to change as opposed to active resistance to RPIWs implied in the present study. It is important to understand the barriers to Lean engagement so that attempts can be made to anticipate and actively counteract them. To overcome mis-perceptions and resistance, Brandao de Souza and Pidd called for debunking myths about Lean’s applicability to healthcare processes and demonstrating successes in manufacturing-like areas within a hospital in order to gain the confidence of skeptics. This study also yielded information for the providers of RPIWs in terms of useful recommendations. Information gleaned from the qualitative interviews revealed ten themes that recommend improvements to the Lean Promotion Office services, while the survey produced five themes that recommend improvements to Lean training. The narrative information gained from the interviews, however, produced learning with far more richness and depth. The favoring of in-depth interviews over the survey method is consistent with Tsianakas et al. (2012), who promote using key informant interviews to fully explore the complexity of experience and produce more valuable improvement recommendations. 141 Hopefully, the Lean Promotion Office in the HA would find all of the recommendations useful in further developing their program and projects. It is still unknown which recommendations on how to make Lean more engaging are most important. Likewise, it is not clear which enablers or barriers to engagement are most important. Perhaps future research could explore these topics further and possibly establish an order of priority importance for these factors. Triangulation of Qualitative and Quantitative Findings Learning comes from not only what has worked and why, but also what has not worked and why not. One finding stands out when it comes to what did not work. When qualitative and quantitative results are triangulated, a lack of sustainability is clearly problematic. What is not as obvious is why the improvements were not or could not be sustained. The qualitative analysis did reveal some thematic barriers to engagement; for example, mis-fit with healthcare, and resistance. This assumes that engagement could be a shortcoming when it comes to explaining poor sustainability. However, when we consider the key informant recommendations for improving RPIWs, a lack of engagement may not be the only factor that influences sustainability. We can gain further insight into what did not work when we consider some thematic areas for suggested improvement, such as Better Clarification of Expectations, Improving Training, Involving More Staff, Improving Measurement, and making More Substantive Improvements. There is an element of irony if we apply Lean principles to Lean interventions themselves. If Lean interventions are deemed unsuccessful or improvements are not sustained, then by definition, the result is the production of waste rather than waste removal. 142 Radnor, Holweg, and Waring (2012) offer a partial explanation for the lack of quantitative evidence of improvement with Lean. They argue that the implementation of Lean might be considered to be on the fringes of service transformation with results that lead to impressive efficiency gains in the short term, but in most cases, they stall or fail to materialize into more widespread and sustained improvements. Their research suggests: Within these conditions, the implementation of Lean is likely to hit some low-lying glass ceiling, whereby small service improvements are made, and often remade, without the underlying lessons being learnt or more system-wide improvements evident. In this sense, those undertaking Lean tasks appear almost trapped in a continually repeating cycle of improvement, with work returning to the status quo in between. (p. 369) Badurdeen & Gregory (2012) seem to agree by claiming that less than 2% of companies that try to adopt Lean actually succeed. The Use of Lean as a Tool Versus Philosophy In healthcare, the implementation of Lean has been very rigorous in some organizations. The approach taken by John Black and Associates and the Virginia Mason Medical Center are examples of Lean being implemented with zeal (Black, Miller, & Sensel, 2016; Plsek, 2014). In some circles, there is a debate about whether Lean should be applied as a tool in the cadre of healthcare improvement techniques, versus being viewed as a philosophy (Ackerman, Hemphill, & Cowan, 2011; Balle & Regnier, 2007; Malmbrandt & Ahlstrom, 2011). Some authors insist that Lean must be implemented with the vigor of a philosophy and argue further that five to six of the technical tools of Lean must be ongoing simultaneously (Bhasin & Burcher, 2010). Bhasin and Burcher also point to a lack of direction, planning, and project sequencing as contributing factors when companies fail in their Lean implementation efforts. 143 Scoville and Little (2014) of the Institute for Healthcare Improvement adopt a more neutral stance, positing that Lean shares a common development history alongside other improvement methods and that various approaches can be complementary. Scoville and Little go on to say that creating a true Lean organization requires a transformational and stead-fast, long-term commitment that begins with senior leadership and transcends throughout an organization. Other healthcare leaders argue that Lean should be seen as one tool among many in the toolbox for quality improvement (e.g., Pinney, Page, Jevsevar, & Bozic, 2016). Boaden et al. (2008) actually warn against the exclusive promotion of a single mindset or approach to improving healthcare. Derek Feeley, Chief Executive Officer of the Institute for Healthcare Improvement, referred to an over-reliance on Lean as restrictive, and prefers Lean to be viewed as one of many tools that can be used for improving healthcare (D. Feeley, personal communication, October 9, 2014). With this debate in mind, it should be noted that the Lean Promotion Office in the HA is very small as compared to leading healthcare organizations that have adopted Lean in earnest (e.g., Virginia Mason Medical Center, Park Nicollet Health Services, ThedaCare). The entire Lean Promotion Office consisted of only five employees when the RPIWs were investigated for this study. This relatively light application of resources makes it difficult for teams to follow through with the sustainability of improvements and continue a series of RPIWs or other improvement work using a “plane-Kaizen” or service area approach. This fact may also account for the diminished gains, lack of sustainability, and/or in-ability for work units to gain traction and advance in their quality improvement journeys. The importance of an organization’s commitment to Lean is underscored when we consider the Rotter et al. (2017) review of scientific literature as part of the development of a 144 protocol for a Cochrane systematic review. These authors limited their study of Lean to organizations that demonstrated a philosophical commitment through the use of Lean principles, tools, and a dedication to continuous improvement. In the case of the present study, it can be argued that by dabbling in Lean through sporadic application of RPIWs, the HA has not committed to Lean, which may partly explain the lack of sustained results. The RPIW audits were not complete at 30, 60, and 90 day follow up—revealing the difficulties associated with follow-through and sustainability. Part of an approach to gaining more traction and sustainability is the training curricula that is part of Lean implementations. At Virginia Mason, for example, staff are seconded from their regular positions and cycled through the Lean Promotion Office where they receive extensive applied training over an extended period before returning to other areas of the workforce (E. Noel, personal communication, August 10, 2015). The Lean Promotion Office at the HA adopts a similar approach, whereby selected leaders undergo Lean Implementation Specialist training in order to build a critical mass of individuals who are then equipped to sustain and spread improvement work. Since this study was conducted, the HA also implemented Lean Daily Visual Management as an operating system at several of its hospitals, which helps considerably with sustainment by strengthening accountability and attending to improvement work in an ongoing and systematic manner (Barnas, 2014; Berlanga & Husby, 2017; Graban & Swartz, 2014; Taher, Landry, & Toussaint, 2016; Toussaint, 2015; Wellman, Hagan, Jeffries, & Bailey, 2017; White, 2016). A final word about Lean as a philosophy is warranted in terms of considering the influence of broader culture on the success of Lean. It may be that cultures who value collectivism over individualism may be at an advantage when it comes to implementing 145 Lean. In his 2016 article, Schonberger highlights the keiretsu system (groupings of many mutually supportive businesses) in his discussion of Japanese production management and its rise to dominance in the field of operations management during the 1980s. Schonberger also describes Theory Z, which attempts to explain a distinctive type of Japanese management characterized by lifetime employment, long-term staff development, group work, shared decision making, and worker commitment to the company. These cultural phenomena persevere in Southeast Asian countries, who have stayed the course with management initiatives as compared to many Western companies, who treat industrial improvement approaches as here-today, gone-tomorrow fads (Schonberger, 2006). Methodological Implications This study contributes to the field by delineating a process to conduct a detailed analysis of RPIWs and illustrates the pre-requisites for improvement efforts to be empirically examined. The basic requirements for conducting defensible research in this area include a description of the program theory that underlies the improvement effort and an explicit evaluation plan that outlines the logic of the study. Other key building blocks include clarifying the nature of the intervention (in this case, providing a robust description of RPIWs), explaining data collection (whether observational methods or system-level indicators are used), and having available and relevant data. The ROI formula, costing data, and method for monetizing actual benefit/cost savings should also be explicated. For all components of the study, whether the focus was on economic evaluation or employee experience, the research methods were made clear—thereby illuminating the process of investigation to promote thorough understanding and effective knowledge exchange. 146 This study also raises standards in the field through a call for more detailed and transparent studies of Lean. One implication is that we should not simply accept statements about the financial benefits of Lean without considerable scrutiny. Unsubstantiated claims are often made about the financial benefits of Lean, yet such claims go unchallenged. For example, Graban & Swartz (2014) claim that $250,000 was saved annually by making small Lean changes in an Emergency Department. They also claim $180 Million in construction cost savings at Seattle Children’s Hospital allegedly resulting from Lean. More dramatic claims are listed in the book by Lean healthcare gurus Black, Miller, and Sensel (2016). Not only was it estimated that over $42 Million was saved through 440 RPIWs conducted in Saskatchewan within three years, these authors suggested that the cost savings would have been double if all regions in the province had reported data. Further, they cited a study by Creative Healthcare that boasted a seven-to-one return on investment resulting from Lean activities across ten organizations. Upon my closer inspection though, Creative Healthcare could not produce a report or any kind of documentation of their ROI study (I. Lazarus, personal communication, November 13, 2017), which completely undermines the validity of statements by Black, Miller, and Sensel. The present study teaches us to challenge any claims of monetary benefit from improvement initiatives that do not provide clear procedures for conducting financial analyses. The method designed by the Institute for Healthcare Improvement is a case in point (Martin, Neumann, Mountford, Bisognano, & Nolan, 2009). It comes replete with a custom Microsoft Excel spreadsheet, called the QI Savings Tracker Worksheet. However, the method falls short of facilitating economic analysis insofar as no instructions on how to describe units of measure or quantify productivity gains are provided, no explanations about 147 how to calculate cost savings from improvement projects are offered, and no discussions of how to estimate the re-investments from savings achieved are presented. Simply entering year-over-year figures from departmental expense reports is tantamount to providing a black box for estimating economic benefit and by no means provides convincing evidence of gains resulting from improvement initiatives. Limitations There are a few limitations that should be noted with regard to this study. First, as acknowledged in Chapter Four, the exclusion of various activities in the costing analysis poses a limitation in terms of underestimating the total cost of RPIW interventions (which has implications for the ROI analysis). If all of the costs associated with the RPIWs were collected, quantified, and included in the ROI formula, more negative ROI results would have been reported. Still, this study included sufficient costing data to calculate ROI, which is a challenge in healthcare research where the cost of implementing quality improvement interventions is rarely reported (Kilpatrick, et al., 2005). Secondly, the survey used in the study was based on validated questions adapted from other surveys of employee engagement, but the alteration of the questions means the survey questions may no longer retain their psychometric qualities of reliability and validity. The survey also had a small sampling frame and limited response rate. Yet, the survey and the in-depth interviews do capture the experiences of individuals who participated in the RPIWs at the HA. Since the present study was conducted, a validated tool has been developed (Kaltenbrunner, Bengtsoon, Mathiassen, & Engstrom, 2017). It is recommended that future research use this tool to explore Lean in this context as the instrument was developed 148 specifically for use in healthcare. The 16-item questionnaire measures staff perceptions of Lean adoption, using 14 principles within four domains (philosophy, processes, people and partners, and problem solving). Thirdly, there were other improvement activities underway at the time of this study. Two sites were participating in a large-scale initiative known as the National Surgical Quality Improvement Program (NSQIP). There are many sub-components of this initiative and all interventions are aimed at improving clinical outcomes for patients. For instance, NSQIP involves improvement activities in the areas of urinary tract infection reduction, enhanced recovery after colorectal surgery, post-operative pneumonia prevention, and surgical site infection reduction. Given the clinical nature of these interventions, NSQIP was not considered to be a competing initiative to the RPIWs, nor a confounding factor in this study. Although a recent report (BC Patient Safety & Quality Council, 2017) mentions how NSQIP may have positively impacted operative Length of Stay (which theoretically can increase surgical throughput capacity), the report does not include a control group comparison, and is too broad in its provincial scope to accurately estimate the impact of NSQIP at the two HA sites in the present study. Fourth, although an attempt was made to match the intervention sites with nonintervention control sites, this matching could not completely prevent any systematic bias, as there could be characteristics unique to each site, such as other improvement activity that may have impacted individual performance (Coly & Parry, 2017). In other words, the quasiexperimental design could not negate the impact of other events that occurred at the same time as the intervention (which could also have influenced performance at intervention sites or at control sites). Furthermore, this study could not adjust for patient-level characteristics in 149 the data that was analyzed. For example, although the quantitative surgical data was cleaned to remove surgeries conducted in the evenings, on weekends, or on Statutory holidays, it was not feasible to classify the surgeries by type or other patient-level characteristics. Lastly, while this study focused on the six RPIWs at each of two sites during 2013–2014, additional Lean improvement work continued after 2014 (including RPIWs that were tangentially linked to surgical services). However, ongoing improvement work was not thought to affect the integrity of this study, given that any additional improvement work should have resulted in improved outputs/outcomes on the performance indicators, but substantive/significant improvement was not observed. Thus, even by stacking the deck with more Lean improvement work, the results did not reflect outputs/outcomes that were empirically observable. Future Directions In their article “Negativity Toward Negative Results”, Matosin, Frank, Engel, Lum, & Newell (2014) remind us that there is much to learn from neutral or negative findings. To illustrate this point, they argue that “science is, by its nature, a collaborative discipline, and one of the principle reasons why we should report negative results is so our colleagues do not waste their time and resources repeating our findings” (Matosin et al., 2014, p. 172). One of the conclusions of the present study, then, is to caution others from pursuing ROI analysis of RPIWs as it may not be a worthwhile endeavor. Perhaps others have already arrived at this conclusion, which may partially explain the dearth of substantive studies on this topic. In their study on the design and evaluation of complex interventions in healthcare, Campbell et al. (2007) state that “…it may become clear that an intervention is unlikely to be cost effective 150 in the current environment and does not warrant the cost of a large randomized trial” (p. 459), which is a position similar to steering others away from conducting ROI studies. So instead of requiring rigorous economic evaluation of RPIWs, perhaps a moderate approach is acceptable in this field whereby organizations are expected to pursue the measurement of RPIW elements, but forego a fulsome and scientific economic analysis. A moderate approach might seem reasonable if we consider the amount of effort it takes organizations to become competent at simply measuring results at a basic level as opposed to achieving advanced proficiency (K. Mate, personal communication, December 13, 2017). Rather than invest resources to produce sound ROI studies, it may be sufficient to encourage organizations to reach a level of proficiency so that 30, 60, and 90 day post-RPIW measures can be effectively and reliably completed in order to better ensure the sustainability of change efforts. Not only would such an approach help to circumvent waste produced by unsuccessful improvement projects, but it could represent the building blocks of large scale change. In fact, this is the history of the Lean journey at Boeing, the world’s largest aerospace company and leading manufacturer of commercial jetliners (www.boeing.com/company). Boeing’s journey involved the evolution of many of its workshops (e.g., Just-in-Time Workouts, Quality Circles, etc.) into what are now called RPIWs (Leitner, 2005). Boeing did not deem it necessary to prove value for money for each RPIW. The success of their transformation occurred through the sheer volume of RPIWs, and according to Leitner, the evidence for the worth of Lean and RPIWs were revealed over time. It may be that it takes considerable time (perhaps decades) before gains from individual improvement efforts can be realized at scale and the measurable benefits can be detected via high level indicators. 151 The recommendation here, then, is to turn our attention away from the typical empty ROI discussions related to Lean. Too often, authors and/or presenters at healthcare conferences make brief reference to ROI without any thorough discussion—and quickly move on to other topics (Asch, Pauly, & Muller, 2016). Dr. Don Berwick, President Emeritus and Senior Fellow at the Institute for Healthcare Improvement, concurs: “It seems as though presenters mention the term ROI as a device to capture attention but the details are often missing” (D. Berwick, personal communication, December 12, 2017). In contrast, this study invites us to turn our attention toward Lean’s ability to add value in healthcare, perhaps by focussing on Lean’s ability to engender a culture of continuous improvement (Mann, 2015; Liker & Hoseus, 2008; Liker & Meier, 2006). It appears that advocates of Lean interventions may have to rely on having faith in the effectiveness of Lean and the “soft” evidence that exists to justify the continued promotion of Lean in healthcare. One thing is clear. If economic evaluations of Lean are to be properly achieved, a financial team must be assigned prior to any interventions so data can be better gathered, tracked, and analyzed throughout the investigation. The need for a partnership between improvement professionals and finance experts is a prerequisite to demonstrating value and ROI in healthcare. This suggestion echoes a key recommendation made in a recent white paper on optimizing the business case for safer healthcare (Institute for Healthcare Improvement, 2017). Given the staggering complexity of healthcare, it may be more important to refine the methods and stay the course in terms of implementing improvements rather than invest in resources to justify costs. Paradoxically, there may be a much higher cost to the system if we were to discontinue improvement efforts and do nothing to address any dimension of quality. 152 Perhaps future research could compare the opportunity cost of Lean with how alternative improvement approaches (including making no effort) impacts clinical, operational, or experiential outcomes. Some researchers are addressing this area and have attempted to prove the business case for quality improvement, although arguably, the details remain incomplete (Swensen, Dilling, McCarty, Bolton, & Harper, 2013). In the final analysis, proving value for money is much more difficult in the complex environment that is healthcare as compared to manufacturing, where Lean emerged. In a ground-breaking study by Spear and Bowen (1999), the authors highlight the fundamental difference whereby processes in healthcare are far more complex than the linear production line that is typical in manufacturing industries. This means manufacturers can more easily conceptualize operations and design improvements to production systems. In factories where production is relatively linear, management can also more easily isolate particular segments of production, measure and quantify processes and outcomes, and more easily utilize advanced techniques such as statistical process control to make improvements. While quantification is used in healthcare, it is not as easy to apply advanced Lean concepts such as pull systems or sophisticated statistical concepts and techniques (cf. Ibrahim, Mansour, & Abed, 2011). Moreover, the product is quite different between manufacturing and healthcare industries. Where manufacturers deal with inanimate objects, healthcare works with the additional complexity of human beings, all of whom are unique. People get ill in a multitude of different ways; they also heal at different rates. Owing to the uniqueness of human beings and the complexity of the healthcare system, it is much more difficult to apply Lean in healthcare to begin with, let alone quantify its benefits at a granular level. 153 Notwithstanding, it is important to note that there are standard operating procedures in healthcare, although they are usually referred to as best/better practices or clinical practice guidelines. This implies that Lean can be a useful method to help standardize operations within healthcare and improve the efficiency of the system, although this aspiration needs to be qualified due to the fact that healthcare practitioners often rely on the craft and/or art of clinical practice more so than on strict adherence to formal instructions, codes, and policies. Radnor, Holweg, and Waring (2012) concur, acknowledging that efforts to promote more evidenced-based and standardized clinical practice are shown to be inconsistent with the variability and ambiguity of clinical practice. Regardless, our efforts to quantify improvements may be encouraged by Hauser, Kutschera, and Romac (2017) who suggest a balance between standardizing basic processes while recognizing that other processes will remain highly variable, random, unpredictable, and elusive of measurement. Perhaps the nature of healthcare will always pose a challenge to researchers who wish to study Lean, which by its very nature is based on relatively linear thinking and characterized by order and causality. Future research will continue to grapple with measuring Lean within the context of healthcare as a socio-technical system, with all of its inherent complexities and constant adaptations (Hollnagel, 2012; Plsek & Greenlaugh, 2001; Rask & Johansson, 2008). In conclusion, there are many reasons why it may not make sense to put in the time and effort needed to prove the ROI of Lean. First, there is high variability when it comes to how Lean is introduced and implemented in healthcare. As Poksinska discovered in 2010, there is no single correct way of implementing Lean in healthcare, and it is more often the case that Lean is adapted in a system as opposed to being simply adopted (Poksinska, 2010). This means that any gains that can be attributable to Lean will be applicable only to the particular 154 setting where the intervention is studied. In other words, any ROI from Lean found in one setting will not necessarily be generalizable to another setting, which renders findings from specific institutions less valuable to decision makers at other institutions. Radnor, Howleg, and Waring (2012) also point to powerful professional groups and regulatory systems within healthcare as forces that make Lean harder to apply as compared to other industries. They argue that healthcare is a highly political and complex organizational setting—with deeply embedded or institutionalized ways of functioning—and as such, they found Lean to be applied in a disjointed fashion rather than being systematically adopted as it is at Toyota. They speculate that Lean will likely not realize the same magnitude of positive effect as seen at Toyota, because healthcare staff often experience Lean as a series of one-off improvement efforts instead of a cultural transformation that is characterized by improved problem solving and continuous improvement. There is also the challenge of encouraging physician engagement and participation in improvement. The Transactional Cost Model of health economics stipulates that we can make improvements in the management of the hospital (as one side of the drivers of efficiency of a hospital), but will be left with mediocre results if the activities/orders of physicians (the other major driver of healthcare) are not addressed (Hurley, 2010). Therefore, healthcare will need to work on many fronts and use many approaches in order to realize true benefits—particularly those that can be quantified. Indeed, in the words of the renowned heath policy and research consultant Steven Lewis, “Lean is mostly aimed at technical efficiency. The notion that one thing can solve the problems of healthcare is absurd” (S. Lewis, personal communication, June 16, 2017). 155 Chapter Seven: Conclusion This study used a mixed-methods design to examine Rapid Process Improvement Workshops from an economic perspective and to gain insight into the experience of staff members. It is the first study of its kind to clearly define RPIWs as one of the Lean methods applied in a specific healthcare setting (surgical services) and provide a transparent calculation of return on investment. In this way, the study adhered to calls in the literature for more rigorous methods to be used in quality improvement research. However, meeting all of the requirements for full economic evaluation proved too difficult, so in the end, the study must be interpreted as a form of partial evaluation. This highlights just how difficult it is to conduct a study of Lean interventions. It may be that the resource requirements to prove the value of Lean projects is excessive for most organizations, or perhaps only feasible in places that exhibit a mature culture of quality improvement. Economic analysis at the RPIW event level turned out to be not feasible since the manually collected data from each separate RPIW was inconsistent and non-monetizable. The cumulative effects of the six separate RPIWs at each of the intervention sites were analyzed. However, the findings did not provide adequate quantitative evidence to claim that the outcomes justify the investments. The statistical analyses for the effectiveness of RPIWs were also inconclusive, which may partially explain why many hospitals report impressive results at the project level but cannot reflect the aggregate gains in quantifiable terms that are demonstrable and believable. Thus, this study teaches us to challenge any unfounded claims of monetary benefit from improvement initiatives because improvement efforts simply do not automatically translate into cost savings. 156 Improvement initiatives provide the opportunity for cost savings, but improvements must be sustained and the gains must be quantified, which is perhaps the greatest challenge. This study showed the cumulative effect of several RPIWs and how that effect can be conceptually harvested in terms of increased surgical volumes. Unfortunately, the longitudinal statistical analysis did not show consistent improvement gains, which leads us to encourage organizations to reach a level of proficiency of measurement so sustainability of change efforts can be achieved and eventually culminate in positive, large scale change. Operating systems, such as Lean Daily Visual Management, are needed to complement RPIWs so gains can be measured on an ongoing basis and improvements sustained. This is how cultural transformation occurred at successful companies such as Boeing, where proving value for money for each RPIW was abandoned in favor of conducting a variety of improvement events over many years. It may be that healthcare will have to rely on the belief, rather than on continuous empirical proof, that Lean and other quality improvement methods can systematically and dramatically improve the industry. Although the primary focus of this study was on economic analysis, the findings invite us to turn our attention away from discussions of financial return on investment and toward other evidence that helps us understand how Lean can provide value in healthcare. RPIWs are complex interventions that are heavily influenced by the process of implementation and the contexts where they are introduced. The method of implementing RPIWs was elucidated in this study in order to better understand what improvement teams were doing and what they were trying to accomplish. The RPIW method used in this study is arguably the most comprehensive and professional product of its kind in the industry. The fact that this study did not report favorable ROIs should not be taken as unequivocal evidence that RPIWs are 157 ineffective. On the contrary, the employee experience analysis showed that staff are generally favorable to RPIWs, which is encouraging as it indicates a degree of belief in the Lean approach to improvement. The engagement analysis helps us understand what helped or hindered the intervention by yielding six thematic enablers to engagement and four themes that represent barriers. It is undetermined which enablers or barriers are most important; perhaps future research could delve deeper into this topic to prioritize an order of importance for these factors. The survey results also indicated some problems with the Lean training, which helps us to understand more about the implementation of the RPIWs. If the Lean training was not successful in teaching individuals the improvement method, this may partly explain the inconsistency of results since staff may have been unable to utilize learning to sustain improvements. In this sense, the RPIWs were still in the testing phase in this HA at the time of this study. The experience and engagement results are encouraging in that the RPIW model was well received and staff showed interest in applying RPIWs in their workplace (as evinced by the participation in the RPIWs). In fact, RPIWs have continued to take place regularly at the HA since this study was conducted, and the Lean Promotion Office has maintained its place as a well-respected facilitator of improvement within the system. This is an accomplishment and a testament to longevity in the current zeitgeist, where the profile of Lean has diminished in BC. Lean does not have the same profile as in years past, and many Lean teams have been reduced in size, integrated with quality departments, or eliminated within the health authorities. This shift in BC has occurred against a backdrop of Lean maintaining prominence in many hospitals in the United States. This study may inspire a resurgence of 158 Lean in healthcare by pointing out the pitfalls of proving value for money, a concept that does not only apply to Lean but is relevant to any improvement method or intervention. Perhaps the findings here may steer interest away from discussions about return on investment and instead emphasize the need to focus on sound process measurement, increased sustainability, and the relentless pursuit of improvement in spite of imperfect evidence of value for money. 159 References Ackerman, J.D., Hemphill, R., & Cowan, D. (2011). Lean is a tool in the toolbox, not the silver bullet. Annals of Emergency Medicine, 58(4), 398-399. https://doi.org/10.1016/j.annemergmed.2011.04.036 Aherne, J. (2007). Think Lean. Nursing Management,13(10), 13-15. https://doi.org/10.7748/nm.13.10.13.s9 Al-Baushi, S., Sohal, A.S., Singh, P.J., Al Hajri, A., Al Farsi, Y.M., & Al Abri, R. (2014). Readiness factors for lean implementation in healthcare settings – a literature review. Journal of Health Organization and Management, 28(2), 135-153. https://doi.org/10.1108/JHOM-04-2013-0083 Anderson, H., Røvik, K.A., & Ingebrigtsen, T. (2014). Lean thinking in hospitals: Is there a cure for the absence of evidence? A systematic review of reviews. BMJ Open, 4:e003873. https://doi.org/10.1136/bmjopen-2013- 003873 Arthur, J. (2007). Lean six sigma demystified. New York: McGraw-Hill. Asch, D.A., Pauly, M.V., & Muller, R.W. (2016). Asymmetric thinking about return on investment. New England Journal of Medicine, 374(7), 606-608. https://doi.org/10.1056/NEJMp1512297 Auerbach, C.F., & Silverstein, L.B. (2003). Qualitative data: An introduction to coding and analysis. New York: New York University Press. Ayanian, J.Z., & Markel, H. (2016). Donabedian’s lasting framework for health care quality. New England Journal of Medicine, 375(3), 205-207. https://doi.org/10.1056/NEJMp1605101 Backous, C. (2016, June 15). Measuring success when the lean improvement work is complicated [Blog post]. Retrieved from https://www.virginiamasoninstitute.org/ 2016/06/measuring-success-lean-improvement-work-complicated/ Badurdeen, F., & Gregory, B. (2012). The softer side of lean: Analyzing corporate culture can point the way to necessary changes. Human Resource Management International Digest, 20(6), 50-53. https://doi.org/10.1108/hrmid.2012.04420faa.003 Ballé, M., & Régnier, A. (2007). Lean as a learning system in a hospital ward. Leadership in Health Services, 20(1), 33-41. https://doi.org/10.1108/17511870710721471 Barnas, K. (2014). Beyond heroes: A lean management system for healthcare. Appleton, WI: ThedaCare Center for Healthcare Value. BC Ministry of Health. (2011). Lean in British Columbia’s Health Sector, Annual Report 2010-11. Victoria, BC: Author. 160 BC Patient Safety & Quality Council. (2017). Improved outcomes = improved access: Through a standardized approach to measurement as part of a strategy to improve care for surgical patients in BC. Vancouver, BC: Author. Retrieved from https://bcpsqc.ca/resource/nsqip-report-improved-outcomes-improved-access/ Beasley C. (2009). The Triple Aim: Optimizing health, care, and cost. Healthcare Executive, 24(1), 64-65. Ben-Tovim, D.I., Bassham, J.E., Bolch, D., Martin, M.A., Dougherty, M., & Szwarcbord, M. (2007). Lean thinking across a hospital: Redesigning care at the Flinders Medical Centre. Australian Health Review, 31(1), 10-15. https://doi.org/10.1071/AH070010 Bercaw, R.G. (2013). Lean leadership for healthcare: Approaches to lean transformation. Boca Raton, FL: CRC Press. Berlanga, G.A., & Husby, B.C. (2017). Lean daily management for healthcare field book. Boca Raton, FL: CRC Press. Bernal, J.L., Cummins, S., & Gasparrini, A. (2016). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348-355. https://doi.org/10.1093/ije/dyw098 Bevan, H., Ham, C., & Plsek, P.E. (2008). The next leg of the journey: How do we make high quality care for all a reality? Coventry, UK: NHS Institute for Innovation and Improvement. Retrieved from https://www.researchgate.net/publication/238689896_ The_next_leg_of_the_journey_How_do_we_make_High_Quality_Care_for_All_a_reality Bhasin, S., & Burcher, P. (2010). Lean viewed as a philosophy. Journal of Manufacturing Technology Management, 17(1), 56-72. https://doi.org/10.1108/17410380610639506 Black, J. (2009). Transforming the patient care environment with lean six sigma and realistic evaluation. Journal for Healthcare Quality, 31(3), 29-35. https://doi.org/10.1111/j.19451474.2009.00028.x Black, J., & Miller, D. (2008). The Toyota way to healthcare excellence: Increase efficiency and improve quality with lean. Chicago, IL: Health Administration Press. Black, J., Miller, D., & Sensel, J. (2016). The Toyota way to healthcare excellence: Increase efficiency and improve quality with Lean (2nd ed.). Chicago, IL: Health Administration Press. Boaden, R., Harvey, G., Moxham, C., & Proudlove, N.C. (2008). Quality improvement: Theory and practice in healthcare. Coventry, UK: NHS Institute for Innovation and Improvement/Manchester Business School. Brandao de Souza, L. (2009). Trends and approaches in lean healthcare. Leadership in Health Services, 22(2), 121-139. https://doi.org/10.1108/17511870910953788 161 Brandao de Souza, L., & Pidd, M. (2011). Exploring the barriers to lean health care implementation. Public Money and Management, 31(1), 59-66. https://doi.org/10.1080/09540962.2011.545548 Burgess, N., & Radnor, Z. (2013). Evaluating Lean in healthcare. International Journal of Health Care Quality Assurance, 26(3), 220-235. https://doi.org/10.1108/09526861311311418 Burkitt, K.H., Mor, M.K., Jain, R., Kruszewski, M.S., McCray, E.E., Moreland, M.E.,…& Fine, M.J. (2009). Toyota production system quality improvement initiative improves perioperative antibiotic therapy. The American Journal of Managed Care, 15(9), 633642. Campbell, N.C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F.,…Kinmonth, A.L. (2007). Designing and evaluating complex interventions to improve healthcare. BMJ, 334(7591), 455-459. https://doi.org/10.1136/bmj.39108.379965.BE Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council of Canada. (2010). TriCouncil policy statement: Ethical conduct for research involving humans, December 2010. Retrieved from http://www.pre.ethics.gc.ca/pdf/eng/tcps2/TCPS_2_FINAL_Web.pdf Chan, J., Bryan, F., McGarvey, E., Yang, K., Arthur, J., & Worsley, G. (2011). Metrics framework. In Lean Evaluation in the Health Authorities. Victoria, BC: Ministry of Health. Cima, R.R., Brown, M.J., Hebl, J.R., Moore, R., Rogers, J.C., Kollengode, A.,…Deschamps, C. (2011). Use of lean and six sigma methodology to improve operating room efficiency in a high-volume tertiary-care academic medical center. Journal of the American College of Surgeons, 213(1), 83-92. https://doi.org/10.1016/j.jamcollsurg.2011.02.009 Collar, R.M., Shuman, A.G., Feiner, S., McGonegal, A.K., Heidel, N., Duck, M.,…Bradford, C.R. (2012). Lean management in academic surgery. Journal of the American College of Surgeons, 214(6), 928-936. https://doi.org/10.1016/j.jamcollsurg.2012.03.002 Collins, K.S., Collins, S. K., McKinnies, R., & Jensen, S. (2008). Employee satisfaction and employee retention: Catalysts to patient satisfaction. Health Care Manager, 27(3), 245251. https://doi.org/10.1097/01.HCM.0000318755.24332.4b Coly, A., & Parry, G. (2017). Evaluating complex health interventions: A guide to rigorous research designs. Washington, DC: Academy Health. Retrieved from: http://www.ihi.org/resources/Pages/Publications/Evaluating-Complex-HealthInterventions-Rigorous-Research-Designs.aspx Creswell, J.W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Thousand Oaks, CA: Sage. 162 Cruz, M., Bender, M., & Ombao, H. (2017). A robust interrupted time series model for analyzing complex healthcare intervention data. Statistics in Medicine, 36(29), 46604676. https://doi.org/10.1002/sim.7443 D’Andreamatteo, A., Ianni, L., Lega, F., & Sargiacomo, M. (2015). Lean in healthcare: A comprehensive review. Health Policy, 119(9), 1197-1209. https://doi.org/10.1016/j.healthpol.2015.02.002 Deans, R., & Wade, S. (2011). Finding a balance between “value added” and feeling valued: Revising models of care. The human factor of implementing a quality improvement initiative using Lean methodology within the healthcare sector. Healthcare Quarterly, 14(Special issue 3), 58-61. https://doi.org/10.12927/hcq.0000.22579 DelliFraine, J.L., Langabeer, J.R., & Nembhard, I.M. (2010). Assessing the evidence of Six Sigma and Lean in the healthcare industry. Quality Management in Health Care, 19(3), 211-225. https://doi.org/10.1097/QMH.0b013e3181eb140e Dickson, E.W., Singh, S., Cheung, D.S., Wyatt, C.C., & Nugent, A.S. (2009). Application of lean manufacturing techniques in the emergency department. The Journal of Emergency Medicine, 37(2), 177-182. https://doi.org/10.1016/j.jemermed.2007.11.108 Donabedian, A. (2005). Evaluating the quality of medical care. The Milbank Quarterly, 83(4), 691-729. (Reprinted from The Milbank Memorial Fund Quarterly, 44(3, Pt 2), 1966, 166-203.) https://doi.org/10.1111/j.1468-0009.2005.00397.x Drotz, E., & Poksinska, B. (2014). Lean in healthcare from employees’ perspectives. Journal of Health Organization and Management, 28(2), 177-195. https://doi.org/10.1108/JHOM-03-2013-0066 Drummond, M.F., Sculpher, M.J., Torrance, G.W., O’Brien, B.J., & Stoddart, G.L. (2005). Methods for the economic evaluation of health care programmes (3rd ed.). Oxford: Oxford University Press. Endsley, S., Magill, M.K., & Godfrey, M.M. (2006, April). Creating a lean practice. Family Practice Management, 13(4), 34-38. Esterberg, K.G. (2002). Qualitative methods in social research. Boston, MA: McGraw-Hill. Fairbanks, C.B. (2007). Using Six Sigma and Lean methodologies to improve OR throughput. AORN Journal, 86(1), 73-82. https://doi.org/10.1016/j.aorn.2007.06.011 Folland, S., Goodman, A.C., & Stano, M. (2013). The economics of health and healthcare (7th ed.). New Jersey: Pearson. Fowler, F.J. (2009). Survey research methods (4th ed.). Thousand Oaks, CA: Sage. 163 Frankel, A., Haraden, C., Federico, F., & Lenoci-Edwards, J. (2017). A framework for safe, reliable, and effective care. White Paper. Cambridge, MA: Institute for Healthcare Improvement. Fretheim, A., & Tomic, O. (2015). Statistical process control and interrupted time series: A golden opportunity for impact evaluation in quality improvement. BMJ Quality and Safety, 24, 748-751. https://doi.org/10.1136/bmjqs-2014-003756 Funnell, S.C., & Rogers, P.J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco, CA: John Wiley & Sons. Furman, C., & Caplan, R. (2007). Applying the Toyota Production System: Using a patient safety alert system to reduce error. The Joint Commission Journal on Quality and Patient Safety, 33(7), 376-386. https://doi.org/10.1016/S1553-7250(07)33043-2 Gayed, B., Black, S., Daggy, J., & Munshi, I.A. (2013). Redesigning a joint replacement program using Lean Six Sigma in a Veterans Affairs hospital. JAMA Surgery, 148(11), 1050-1056. https://doi.org/10.1001/jamasurg.2013.3598 Gibbons, J., & Schutt, R. (2010). A global barometer for measuring employee engagement (Research working group report no. 1460-09-RR). New York: The Conference Board. Goddard, M., Mannion, R., & Smith, P.C. (1999). Assessing the performance of NHS hospital trusts: The role of ‘hard’ and ‘soft’ information. Health Policy, 48(2), 119-134. https://doi.org/10.1016/S0168-8510(99)00035-4 Government of Saskatchewan. (2012, August 30). Province accelerates Lean journey to improve health care [Press release]. Retrieved from https://www.saskatchewan.ca/ government/news-and-media/2012/august/30/province-accelerates-lean-journey-toimprove-health-care Graban, M., & Swartz, J.E., (2012). Healthcare Kaizen: Engaging front-line staff in sustainable continuous improvements. Boca Raton, FL: CRC Press. Graban, M., & Swartz, J.E., (2014). The executive guide to healthcare Kaizen: Leadership for a continuously learning and improving organization. Boca Raton, FL: CRC Press. Grol, R.P., Bosch, M.C., Hulscher, M.E., Eccles, M.P., & Wensing, M. (2007). Planning and studying improvement in patient care: The use of theoretical perspectives. The Milbank Quarterly, 85(1), 93-138. https://doi.org/10.1111/j.1468-0009.2007.00478.x Hahn, C. (2008). Doing qualitative research using your computer: A practical guide. Thousand Oaks, CA: Sage. Hauser, R., Kutschera, H.J., and Romac, B. (2017). Lean complexity through tailored business streams. In K. Richter & J. Walther (Eds.), Supply chain integration challenges in commercial aerospace (pp. 209-219). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-46155-7_14 164 Hines, P., Howleg, M., & Rich, N. (2004). Learning to evolve: A review of contemporary lean thinking. International Journal of Operations and Production Management, 24(10), 994-1011. https://doi.org/10.1108/01443570410558049 Holden, R.J. (2011). Lean thinking in emergency departments: A critical review. Annals of Emergency Medicine, 57(3), 265-278. https://doi.org/10.1016/j.annemergmed.2010.08.001 Hollnagel, E. (2012). FRAM: The functional resonance analysis method: Modelling complex socio-technical systems. Burlington, VT: Ashgate Publishing Company. Howard, K.B., & Pathak, D.S. (1999). Determining the difference among cost savings, cost avoidance, and cost reduction. Pharmacy Practice Management Quarterly, 19(3), 1-7. Hulscher, M.E.J., Laurant, M.G.H., & Grol, R.P.T. (2003). Process evaluation on quality improvement interventions. Quality and Safety in Healthcare, 12(1), 40-46. https://doi.org/10.1136/qhc.12.1.40 Hurley, J.E. (2010). Health economics. Whitby, ON: McGraw-Hill Ryerson. Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D.,…Loder, E. (2013). Consolidated Health Economic Evaluation Reporting Standards (CHEERS) – explanation and elaboration: A report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value in Health, 16(2), 231250. https://doi: 10.1016/j.jval.2013.02.002 Iannettoni, M.D., Lynch, W.R., Parekh, K.R., & McLaughlin, K.A. (2011). Kaizen method for esophagectomy patients: Improved quality control, outcomes, and decreased costs. The Society of Thoracic Surgeons, 91(4), 1011-1018. https://doi.org/10.1016/j.athoracsur.2011.01.001 Ibrahim, M.S., Mansour, M.A.R., & Abed, A.M. (2011). Improve six-sigma management by forecasting production quantity using image verification quality tool. International Journal of Advances in Engineering and Technology, 1(4), 332-342. Institute for Healthcare Improvement. (n.d.). Institute for Healthcare Improvement: About us. Retrieved from http://www.ihi.org/about/pages/default.aspx Institute for Healthcare Improvement. (2003). The Breakthrough Series: IHI’s collaborative model for achieving breakthrough improvement. IHI Innovation Series white paper. Boston, MA: Author. Institute for Healthcare Improvement. (2017). Optimizing a business case for safe health care: An integrated approach to safety and finance. Cambridge, MA: Institute for Healthcare Improvement / National Patient Safety Foundation. Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press. 165 Kaltenbrunner, M, Bengtsoon, L., Mathiassen, S.E., & Engström, M. (2017). A questionnaire measuring staff perceptions on Lean adoption in healthcare: Development and psychometric testing.” BMC Health Services Research, 17(235), 1-11. https://doi.org/10.1186/s12913-017-2163-x Kenny, C. (2011). Transforming health care: Virginia Mason Medical Center’s pursuit of the perfect patient experience. New York: Productivity Press. Keppel, G., & Wickens, T.D. (2004). Design and analysis: A researcher’s handbook (4th ed.). New Jersey: Pearson Prentice Hall. Kilpatrick, K.E., Lohr, K.N., Leatherman, S., Pink, G., Buckel, J.M., Legarde, C., & Whitener, L. (2005). The insufficiency of evidence to establish the business case for quality. International Journal for Quality in Health Care, 17(4), 347-355. https://doi.org/10.1093/intqhc/mzi034 Kinsman, L., Rotter, T., Stevenson, K., Bath, B., Goodridge, D., Harrison, L.,…Westhrop, G. (2014). “The largest Lean transformation in the world”: The implementation and evaluation of lean in Saskatchewan healthcare. Healthcare Quarterly, 17(2), 29-32. Kontopantelis, E., Doran, T., Springate, D.A., Buchan, I., & Reeves, D. (2015). Regression based quasi-experimental approach when randomization is not an option: Interrupted time series analysis. BMJ, 2015:350:h2750. https://doi.org/10.1136/bmj.h2750 �� � �� , L. (2012). The development of models and methods Kaizen. Technical University of ���e���� e, Slovakia. Retrieved from https://www.sjf.tuke.sk/ transferinovacii/pages/archiv/transfer/22-2012/pdf/193-197.pdf Langley, G.J., Moen, R.D., Nolan, K.M., Nolan, T.W., Norman, C.L., & Provost, L.P. (2009). The improvement guide: A practical approach to enhancing organizational performance (2nd ed.). San Francisco, CA: Jossey-Bass. Leading Edge Group. (2006). Green belt Lean certification for healthcare participant workbook. Toronto, ON: Author. Leading Edge Group. (2009, July). Leading Edge Group Lean healthcare white belt workshop. Presented at Northern Health, Prince George, BC. Leggat, S.G., Bartram, T., Stanton, P., Bamber, G.J., & Sohal, A.S. (2015). Have process redesign methods such as Lean been successful in changing care delivery in hospitals? A systematic review. Public Money and Management, 35(2), 161-168. https://doi.org/10.1080/09540962.2015.1007714 Leitner, P.A. (2005, May). The Lean journey at the Boeing company. Presented at the ASQ World Conference on Quality and Improvement Proceedings, Seattle, Washington. Retrieved from http://www.johnblackandassociates.com/uploads/2/0/7/8/20782048/thelean-journey-at-boeing.pdf 166 Lewins, A., & Silver, C. (2007). Using software in qualitative research: A step-by-step guide. London: Sage. Liker, J.K. (2004). The Toyota way: 14 management principles from the world’s greatest manufacturer. New York: McGraw-Hill. Liker, J.K., & Hoseus, M. (2008). Toyota culture: The heart and soul of the Toyota way. New York: McGraw-Hill. Liker, J.K., & Meier, D. (2006). The Toyota way fieldbook: A practical guide for implementing Toyota’s 4Ps. New York: McGraw-Hill. Lowe, G. (2010). Creating healthy organizations: How vibrant workplaces inspire employees to achieve sustainable success. Toronto, ON: Rotman/UTP Publishing. Lowe, G. (2012). How employee engagement matters for hospital performance. Healthcare Quarterly, 15(2), 29-39. Mackenzie, J., & Hall, W. (2014). “Lean” in Canadian health care: Doing less while achieving more. Ottawa, ON: The Conference Board of Canada. Madhok, R. (2002). Crossing the quality chasm: Lessons from health care quality improvement efforts in England. Baylor University Medical Center Proceedings, 15(1), 77-85. Malmbrandt, M., & Åhlström, P. (2011). An instrument for assessing lean service adoption. International Journal of Operations and Production Management, 33(9), 1131-1165. https://doi.org/10.1108/IJOPM-05-2011-0175 Mann, D. (2015). Creating a Lean culture: Tools to sustain Lean conversions (3rd ed.). Boca Raton, FL: CRC Press. Marchildon, G. (2013). Implementing Lean health reforms in Saskatchewan. Health Reform Observer, 1(1): Article 1. https://doi.org/10.13162/hro-ors.01.01.01 Martin, A. (2014, September 11). Health workers dislike ‘Lean,’ survey finds. The Saskatoon Star Phoenix, pp. 11. Martin, L.A., Neumann, C.W., Mountford, J., Bisognano, M., &Nolan. T.W. (2009). Increasing efficiency and enhancing value in healthcare: Ways to achieve savings in operating costs per year. Cambridge, MA: Institute for Healthcare Improvement. Mason, S.E., Nicolay, C.R., & Darzi, A. (2015). The use of Lean and Six Sigma methodologies in surgery: A systematic review. The Surgeon, Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland, 13(2), 91-100. https://doi.org/10.1016/j.surge.2014.08.002 167 Massoud, M.R., Barry, D., Murphy, A., Albrecht, Y., Sax, S., & Parchman, M. (2016). How do we learn about improving health care: A call for a new epistemological paradigm. International Journal of Quality in Health Care, 28(3), 420-424. https://doi.org/10.1093/intqhc/mzw039 Matosin, N., Frank, E., Engel, M., Lum, J.S., & Newell, K.A. (2014). Negativity toward negative results: A discussion of the disconnect between scientific worth and scientific culture. Disease Models and Mechanisms, 7, 171-173. https://doi.org/10.1242/dmm015123 Matt, B.H., Woodward-Hagg, H.K., Wade, C.L., Butler, P.D., & Kokoska, M.S. (2014). Lean six sigma applied to ultrasound guided needle biopsy in the head and neck. Otolaryngology–Head and Neck Surgery, 15(1), 65-72. https://doi.org/10.1177/0194599814528659 Mazzocato, P., Holden, R.J., Brommels, M., Aronsson, H., Backman, U., Elg, M., & Thor, J. (2012). How does lean work in emergency care? A case study of a lean-inspired intervention at the Astrid-Lindgren Children’s Hospital, Stockholm, Sweden. BMC Health Services Research, 12(28), 1-13. https://doi.org/10.1186/1472-6963-12-28 Mazzocoto, P., Savage, C., Brommels, M., Aronsson, H., & Thor, J. (2010). Lean thinking in healthcare: A realist review of the literature. Quality and Safety in Health Care, 19(5), 376-382. https://doi.org/10.1136/qshc.2009.037986 McCulloch, P., Kreckler, S., New, S., Sheena, Y., Handa, A., & Cachpole, K. (2010). Effect of a “Lean” intervention to improve safety processes and outcomes on a surgical emergency unit. BMJ, 2010:341:c5469. https://doi.org/10.1136/bmj.c5469 McIntosh, B., Sheppy, B., & Cohen, I. (2014). Illusion or delusion: Lean management in the health sector. International Journal of Health Care Quality Assurance, 27(6), 482-492. https://doi.org/10.1108/IJHCQA-03-2013-0028 MEDITECH. (2018). EHR Solutions – Hospitals & Health Systems. Retrieved from https://ehr.meditech.com/ehr-solutions/financial-staff Michie, S., & West, M.A. (2004). Managing people and performance: An evidence based framework applied to health service organizations. International Journal of Management Reviews, 5/6, 91-111. https://doi.org/10.1111/j.1460-8545.2004.00098.x Miles, M., & Huberman, M. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage. Mintzberg, H. (2009, July-August). Rebuilding companies as communities. Harvard Business Review, 87, 140-143. Moraros, J., Lemstra, M., & Nwankwo, C. (2016). Lean interventions in healthcare: Do they actually work? A systematic literature review. International Journal for Quality in Health Care, 29(2), 150-165. https://doi.org/10.1093/intqhc/mzv123 168 Motwani, J. (2003). A business process change framework for examining Lean manufacturing: A case study. Industrial Management and Data Systems, 103(5), 339346. https://doi.org/10.1108/02635570310477398 Moustakas, C. (1994). Phenomenological research methods. Thousand Oaks, CA: Sage. Naik, T., Duroseau, Y., Zehtabchi, S., Rinnert, S., Payne, R., McKenzie, M., & Legome, E. (2012). A structured approach to transforming a large public hospital emergency department via lean methodologies. Journal for Healthcare Quality, 34(2), 86-97. https://doi.org/10.1111/j.1945-1474.2011.00181.x NASPO Benchmarking Workgroup. (2007). Benchmarking cost savings and cost avoidance. Lexington, KY: National Association of State Procurement Officials. Nelson-Peterson, D.L., & Leppa, C.J. (2007). Creating an environment for caring using lean principles of the Virginia Mason Production System. The Journal of Nursing Administration, 37(6), 287-294. Nicklin, W., & Mitchell, J. (2015). The Canadian health accreditation report: Building a stronger health system through leadership. Presentation of Accreditation Canada. Retrieved from chlnet.ca/wp-content/uploads/Accreditation_Canada-May-2015.pdf Nicolay, C.R., Purkayastha, S., Greenhalgh, A., Benn, J., Chaturvedi, S., Phillips, N., & Darzi, A. (2012). Systematic review of the application of quality improvement methodologies from the manufacturing industry to surgical healthcare. British Journal of Surgery, 99(3), 324-335. https://doi.org/10.1002/bjs.7803 Niemeijer, G.C., Flikweert, E., Trip, A., Does, R.J., Ahaus, K.T., Boot, A.F., & Wendt, K.W. (2013). The usefulness of Lean Six Sigma to the development of a clinical pathway for hip fractures. Journal of Evaluation in Clinical Practice, 19(5), 909-914. https://doi.org/10.1111/j.1365-2753.2012.01875.x Ohno, T. (1988). Toyota Production System: Beyond large-scale production. Portland, OR: Productivity Press. Ontario Hospital Association. (2010). OHA-NRC Picker employee and physician experience surveys backgrounder. Ottawa, ON: Author. Ovretveit, J. (2009). Does improving quality save money? A review of evidence of which improvements to quality reduce costs to health service providers. London: The Health Foundation. Parker, S.K. (2003). Longitudinal effects of lean production on employee outcomes and the mediating role of work characteristics. The Journal of Applied Psychology, 88(4), 620-634. Parry, G.J., Carson-Stevens, A., Luff, D.F., McPherson, M.E., & Goldmann, D.A. (2013). Recommendations for evaluation of health care improvement initiatives. Academic Pediatrics, 13(6 Suppl), S23-30. https://doi.org/10.1016/j.acap.2013.04.007 169 Penfold, R.B., & Zhang, F. (2013). Use of interrupted time series analysis in evaluating health care quality improvements. Academic Pediatrics, 13(6 Suppl), S38-44. https://doi.org/10.1016/j.acap.2013.08.002 Persoon, T.J., Zaleski, S., & Frerichs, J. (2006). Improving preanalytic processes using the principles of lean production (Toyota Production System). American Journal of Clinical Pathology, 125(1), 16-25. Pettersen, J. (2009). Defining lean production: Some conceptual and practical issues. The TQM Journal, 21(2), 127-142. https://doi.org/10.1108/17542730910938137 Phillips, J.J., Buzachero, V., Phillips, P.P., & Phillips, Z. L. (2013). Measuring ROI in healthcare: Tools and techniques to measure the impact and ROI in healthcare projects, programs, and initiatives. New York: McGraw-Hill. Phillips, J.J., & Phillips, P.P. (2007). Show me the money: How to determine ROI in people, projects, and programs. San Francisco, CA: Berrett-Koehler. Picis. (2017). Picis OR Manager Product Sheet [Computer software]. Retrieved from http://chriscarroll.com/wp-content/uploads/2015/12/Picis-OR-Manager-Datasheet.pdf Pinney, S.J., Page, A.E., Jevsevar, D.S., & Bozic, K.J. (2016). Current concept review: Quality and process improvement in orthopedics. Orthopedic Research and Reviews, 8, 1-11. https://doi.org/10.2147/ORR.S92216 Plsek, P. (2014). Accelerating health care transformation with lean and innovation: The Virginia Mason experience. Boca Raton, FL: CRC Press. Plsek, P.E, & Greenlaugh, T. (2001). Complexity science: The challenge of complexity in health care. BMJ, 323, 625-628. https://doi.org/10.1136/bmj.323.7313.625 Poksinska, B. (2010). The current state of Lean implementation in healthcare: Literature review. Quality Management in Health Care, 19(4), 319-329. https://doi.org/10.1097/QMH.0b013e3181fa07bb Portela, M.C., Pronovost, P.J., Woodcock, T., Carter, P., & Dixon-Woods, M. (2015). How to study improvement interventions: A brief overview of possible study types. Postgraduate Medical Journal, 91, 343-354. https://doi.org/10.1136/postgradmedj-2014003620rep Proudlove, N., Moxham, C., & Boaden, R. (2008). Lessons for Lean in healthcare from using six sigma in the NHS. Public Money and Management, 28(1), 27-34. https://doi.org/10.1111/j.1467-9302.2008.00615.x Provincial Auditor of Saskatchewan. (2016). Report of the Provincial Auditor to the Legislative Assembly of Saskatchewan (2016 Report – Volume 2). Regina, SK: Author. Retrieved from https://auditor.sk.ca/pub/publications/public_reports/2016/Volume_2/ 2016_V2_Full_Report.pdf 170 Provost, L.P. (2011). Analytical studies: A framework for quality improvement design and analysis. BMJ Quality and Safety, 20(Suppl 1), i92-i96. https://doi.org/10.1136/bmjqs.2011.051557 Provost, L.P., & Murray, S.K. (2011). The health care data guide: Learning from data for improvement. San Francisco, CA: Jossey-Bass. Puterman, M., Zhang, Y., Aydede, S.K., Palmer, B., MacLeod, S., Bavafa, H., & Mackenzie, J. (2013). “If you're not keeping score, you're just practising”: A lean healthcare program evaluation framework. Healthcare Quarterly, 16(2), 23-30. https://doi.org/10.12927/hcq.2013.23372 Radnor, Z.J., Holweg, M. & Waring, J. (2012). Lean in healthcare: The unfilled promise? Social Science and Medicine, 74(3), 364-371. https://doi.org/10.1016/j.socscimed.2011.02.011 Rask, K., & Johansson, J. (2008). Similarities and differences between lean production, tayloristic, and socio-technical systems revealed in the methodology characteristics map. Paper presented at the 18th International Conference on Flexible Automation and Intelligent Manufacturing, S���, Sweden. Retrieved from https://www.divaportal.org/smash/get/diva2:1005068/FULLTEXT01.pdf Rauh, S.S., Wadsworth, E.B., Weeks, W.B., & Weinstein, J.N. (2011). The savings illusion: Why clinical quality improvement fails to deliver bottom-line results. New England Journal of Medicine, 365:e48. https://doi.org/10.1056/NEJMp1111662 Reinertsen, J.L. (2006). Interview with Gary Kaplan. Quality and Safety in Healthcare, 15(3), 156-158. https://doi.org/10.1136/qshc.2006.018283 Richards, L., (2005). Handing qualitative data: A practical guide. Thousand Oaks, CA: Sage. Rondeau, K.V., & Wagar, T.H. (2006). Nurse and resident satisfaction in magnet long-term care organizations: Do high involvement approaches matter? Journal of Nursing Management, 14(3), 244-250. https://doi.org/10.1111/j.1365-2934.2006.00594.x Rotter, T., Plishka, C., Adegboyega, L., Fiander, M., Harrison, L., Flynn, R., Chan, J., & Kinsman, L. (2017). Lean management in health care: Effects on patient outcomes, professional practice, and healthcare systems (Protocol). Cochrane Database of Systematic Reviews, 11, Art. No. CD012831. https://doi.org/10.1002/14651858.CD012851 Rotter, T., Plishka, C.T., Adegboyega, L., Harrison, E.L., Sari, N., Goodridge, D., Flynn, R., Chan, J.G., Fiander, M., Poksinska, B., Willoughby, K., & Kinsman, L. (2018). What is lean management in health care? Development of an operational definition for a Cochrane Systematic Review. Evaluation and the Health Professions, 1-25. https://doi.org/10.1177/0163278718756992 171 Sari, N., Rotter, T., Goodridge, D., Harrison, L., & Kinsman, L. (2017). An economic analysis of a system wide Lean approach: Cost estimations for the implementation of Lean in the Saskatchewan healthcare system for 2012-2014. BMC Health Services Research, 17, 523. https://doi.org/10.1186/s12913-017-2477-8 Schonberger, R.J. (2006). Japanese production management: An evolution—with mixed success. Journal of Operations Management, 25, 403-419. Schwarz, P., Pannes, K.D., Nathan, M., Reimer, H.J., Kleespies, A., Kuhn, N.,…& Zugel, P. (2011). Lean process for optimizing OR capacity utilization: Prospective analysis before and after implementation of value stream mapping (VSM). Langenbecks Archives of Surgery, 396(7), 1047-1053. https://doi.org/10.1007/s00423-011-0833-4 Scoville, R., & Little, K. (2014). Comparing Lean and quality improvement. IHI White Paper. Cambridge, MA: Institute for Healthcare Improvement. Shewhart, W.A. (1931). Economic control of quality of manufactured product. New York: D. Van Nostrand Company. Shojania, K.G., & Grimshaw. J.M. (2005). Evidence-based quality improvement: The state of the science. Health Affairs, 24(1), 138-150. https://doi.org/10.1377/hlthaff.24.1.138 Sikorska-Simmons, E. (2006). Linking resident satisfaction to staff perceptions of the work environment in assisted living: A multilevel analysis. The Gerontologist, 46(5), 590-598. https://doi.org/10.1093/geront/46.5.590 Spear, S., & Bowen, H.K. (1999, September-October). Decoding the DNA of the Toyota Production System. Harvard Business Review, 77(5), 96-106. Studer, Q., Robinson, B.C., & Cook, K. (2010). The HCAHPS Handbook: Hardwire your hospital for pay-for-performance success. Gulf Breeze, FL: Fire Starter Publishing. Swensen, S.J., Dilling, J.A., McCarty, P.M., Bolton, J.W., & Harper, C.M. Jr. (2013). The business case for health-care quality improvement. Journal of Patient Safety, 9(1), 44-52. https://doi.org/10.1097/PTS.0b013e3182753e33 Taher, D., Landry, S., & Toussaint, J. (2016). Breadth vs. depth: How to start deploying the daily management system for your lean transformation. Journal of Hospital Administration, 5(6), 90-96. https://doi.org/10.5430/jha.v5n6p90 Think Health BC. (2012). Lean: Improving efficiency [Video]. Available from http://unionofone.squarespace.com/think-health-bc/ Thomas, D.R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27(2), 237-246. https://doi.org/10.1177/1098214005283748 Thorlby, R., & Maybin, J. (Eds.). (2010). A high-performing NHS? A review of progress 1997-2010. London: The Kings Fund. 172 Touissaint, J. (2015). Management on the mend: The healthcare executive guide to system transformation. Appleton, WI: ThedaCare Center for Healthcare Value. Trusko, B.E., Pexton, C., Harrington, H.J., & Gupta, P. (2007). Improving healthcare quality and cost with six sigma. New York: FT Press. Tsianakas, V., Maben, J., Wiseman, T., Robert, G., Richardson, A., Madden, P.,…Davies, E.A. (2012). Using patients’ experiences to identify priorities for quality improvement in breast cancer care: Patient narratives, surveys, or both? BMC Health Services Research, 12:271. https://doi.org/10.1186/1472-6963-12-271 Ulhassan, W., Sandahl, C., Westerlund, H., Henriksson, P., Bennermo, von Thiele Schwarz, M., & Thor, J. (2013). Antecedents and characteristics of lean thinking implementation in a Swedish hospital: A case study. Quality Management in Health Care, 22(1), 48-61. https://doi.org/10.1097/QMH.0b013e31827dec5a Vest, J.R., & Gamm, L.D. (2009). A critical review of the research literature on Six Sigma, Lean, and Studergroup’s Hardwiring Excellence in the United States: The need to demonstrate and communicate the effectiveness of transformation strategies in healthcare. Implementation Science, 4:35. https://doi.org/10.1186/1748-5908-4-35 Walshe, K. (2007). Understanding what works—and why—in quality improvement: The need for theory-driven evaluation. International Journal for Quality in Health Care, 19(2), 57-59. https://doi.org/10.1093/intqhc/mzm004 Warner, C.J., Walsh, D.B., Horvath, A.J., Walsh, T.R., Herrick, D.P., Prentiss, S.J., & Powell, R.J. (2013). Lean principles optimize on-time vascular surgery operating room starts and decrease resident work hours. Journal of Vascular Surgery, 58(5), 1417-1422. https://doi.org/10.1016/j.jvs.2013.05.007 de Weck, O.L., Roos, D., & Magee, C. (2011). Engineering systems: Meeting human needs in a complex technological world. Boston, MA: MIT Press. Wellman, J., Hagan, P., Jeffries, H., & Bailey, C. (2017). Leading the lean healthcare journey: Driving culture change to increase value. Boca Raton, FL: CRC Press. West, M., Dawson, J., Admasachew, L., & Topakas, A. (2011). NHS staff management and health service quality: Results from the NHS staff survey and related data. Aston Business School. London: Department of Health and Social Care. Retrieved from http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG uidance/DH_129643. White, B. (2016). Lean daily management for healthcare: A strategic guide to implementing lean for hospital leaders. Boca Raton, FL: CRC Press. Whitty, P. (1998). The National Health Service in England considers on the Government’s plans to improve quality of health care. Quality in Health Care, 7(4), 227-231. http://dx.doi.org/10.1136/qshc.7.4.227 173 Wilcock, P.M., Brown, G.C., Bateson, J., Carver, J., & Machin, S. (2003). Using patient stories to inspire quality improvement within the NHS Modernization Agency collaborative programmes. Journal of Clinical Nursing, 12(3), 422-430. https://doi.org/10.1046/j.1365-2702.2003.00780.x Wolf, L., Costantinou, E., Limbaugh, C., Rensing, K., Gabbart, P., & Matt, P. (2013). Fall prevention for inpatient oncology using lean and rapid improvement event techniques. Health Environments Research & Design Journal, 7(1), 85-101. https://doi.org/ 10.1177/193758671300700108 Womack, J.P., & Jones, D.T. (2003). Lean thinking: Banish waste and create wealth in your corporation. New York: Simon & Schuster. (Original work published 1996) Yin, R.K. (2014). Case study research: Design and methods (5th ed.). Thousand Oaks, CA: Sage. Yousri, T.A., Khan, D., Chakrabarti, R., Fernandes, K., & Wahab, K. (2011). Lean thinking: Can it improve the outcome of fracture neck of femur patients in a district general hospital? Injury, International Journal of Care for the Injured, 42(11), 1234-1237. https://doi.org/10.1016/j.injury.2010.11.024 Zarbo, R.J., & D’Angelo, R. (2007). The Henry Ford production system: Effective reduction of process defects and waste in surgical pathology. American Journal of Clinical Pathology, 128(6), 1015-1022. https://doi.org/10.1309/RGF6JD1NAP2DU88Q 174 Appendix A Eight Forms of Non-Value Added Waste Waste Definition Examples Defects Work that contains errors or • Filing errors made in documents lacks something of value • Dealing with service complaints • Making mistakes due to incorrect information or miscommunication • Filling out inpatient admission cards incorrectly • Handwriting orders in an illegible manner • Improper equipment set-up Over Production Producing more than the customer requires (redundant work) • Making photocopies of forms that are never used • Providing copies of reports to people who have not asked for them and will not actually read them • Processing piles of documents that sit in queue at the next workstation • Carbon-copying emails unnecessarily Waiting Idle time created when people/information/ equipment/supplies not ready or on hand • Patient waiting to see their physician • Early morning admissions waiting for surgeries that won’t be performed until later in the day • Patients waiting for support services such as internal transport • Staff waiting for equipment Non-Utilized Staff Creativity Not using people’s mental, creative, and physical skills and abilities • Not taking staff’s input into resolutions • Not involving staff in improvement/change processes early on 175 Waste Definition Examples Transportation Movement of product or materials between operations • Moving individual files from one location to another • Moving supplies into and out of storage rooms • Moving equipment for surgeries into and out of operating rooms • Moving equipment for procedures in and out of procedure rooms • Transferring charts from other buildings on-site Inventory More materials on hand than are required • Stockpiling office and clinical supplies that won’t be used for weeks or months • Storing excess supplies whose “useby” dates expire before they are used • Maintaining expensive implants that can be ordered on a just-in-time basis Movement Movement of people that does not add value • Physicians and staff looking for items that should be clearly labelled • Physicians walking to another location to check an online note during a patient exam • Clinicians going from one building on campus to another to attend a meeting Excessive Processing Activities that do not add value from patient perspective • Performing tests that aren’t needed • Redundant capture of info upon admission • Recording and logging of the same data multiple times • Writing information by hand when direct input to a computer could eliminate the step • Producing a paper copy when a computer file is sufficient Adapted from The Toyota Way to Healthcare Excellence: Increase Efficiency and Improve Quality with Lean (2nd ed., pp. 33-50), by J. Black, D. Miller, and J. Sensel, 2016, Chicago, IL: Health Administration Press. 176 Appendix B Metrics Framework for Lean in BC Dimension Description Process Measure Themes (System Voice: Changes to the Organization) Outcome Measure Themes (Patient or Staff voice) Client Centered Putting clients and families first:  Number of new processes instituted that are  Patient/family satisfaction Services Care and services are provided in a client centered o e.g., as a result of a specific client-centred manner that is respectful and o e.g., number of cases having interdisciplinary process improvement initiative responsive to the needs, preferences, teams for patient care; number of patients who  Health outcomes related to client centered culture and values of individuals, are partners with health care providers in their improvements families and communities; care plan; conversations that take place in  Staff satisfaction working in “client centered” acceptability; patient experience / confidential environment; noise level; patient services satisfaction. walking distance. Patient and Staff Keeping people safe: Unsafe acts in  Compliance with safety practices (hand hygiene, Safety the care system’s delivery of safety checklists, medication reconciliation) services and organizational  Number/rate of adverse events structures are prevented and o e.g., number of patients who experience no mitigated. adverse events; number of patients for whom defect-free process occurs; number of patient falls; errors in medication administration; unintended cut during surgery; other specific medical errors; rate of safety-related defects in certain processes (nature of defect to be defined).  Number/rate of ‘never events’  Number/rate of near-miss events  Compliance with safe staff practices (e.g., for lifting, wearing protective gear) Effectiveness  Number of staff injuries 177 Whether care provided was  Number of protocols/guidelines/SOPs or other consistent with established recommended care processes instituted protocols, standards, and guidelines o e.g., Clinical Care Management guidelines; (process measures). standard care plans for specific major conditions (such as stroke, sepsis, kidney disease)  Standard work implemented; compliance with standard work.  Number of cases where care is consistent with protocols/guidelines or other recommended care processes  Number/rate of adverse event outcomes (e.g., surgical site infection or other health care associated infection, pressure sores, adverse medication reaction)  Mortality rate o e.g., Hospital Standardized Mortality Rate, mortality rate in low-mortality CMGs  Number/rate of healthcare associated infections (C. difficile, MRSA)  Readmission rate o e.g., emergency readmission rate within 30 days for same Major Clinical Category Dimension Description Process Measure Themes (System Voice: Changes to the Organization) Outcome Measure Themes (Patient or Staff voice) o e.g., Clinical Care Management guidelines; standard care plans for specific major conditions (such as stroke, sepsis, kidney disease) Whether the desired outcomes were achieved for patients/clients (outcome measures). Accessibility Providing timely and equitable services: The most appropriate services are easily obtained (timeliness, geographic access, building access).  Percent of cases that achieved intended health outcomes, which would be defined for each event (e.g., measures of pain, symptoms, function and quality of life)  Readmission rates o e.g., emergency readmission rate within 30 days for same Major Clinical Category  Wait time for services in the hospital  Percent patients/clients whose condition o e.g., average/median wait time for diagnostic deteriorated during wait for care imaging, specialist consultation, biopsy, home  Patient satisfaction with wait time care services, activation unit services o e.g., when informed of queue position  Time between decision to admit in ED and admittance to an inpatient bed o e.g., percent of ED patients admitted within 2/6/10 hours of decision to admit  Hospital occupancy rate  Compliance with wait time recommendations or benchmarks o e.g., Percent of non-emergency surgeries completed within the benchmark wait time frame (according to individual patient priority)  Number of patients waiting, by procedure or service  Patient volumes o e.g., number of patients booked per day, per slate; number of procedures per day/week/month  Wait time from referral by GP to first appointment with community care services 178 Process Measure Themes (System Voice: Changes to the Organization) Outcome Measure Themes (Patient or Staff voice) Continuity of Service Experiencing coordinated and  Average/median wait time between services seamless services: Care and services (hospital to community) are coordinated and consistent  Percent of patients with discharge plan across the continuum of care.  Percent of cases where discharge from hospital includes notification of primary care physician within 24 hours  Number of defects in process related to coordination of care o e.g., in the flow of information; in shift handovers  Patient/family satisfaction with continuity of care  Percent of cases that achieved intended health outcomes (e.g., measures of pain, symptoms, function and quality of life)  Readmission rates o e.g., emergency readmission rate within 30 days for same Major Clinical Category Efficiency Making the best use of resources:  Turnaround/cycle time Resources are allocated to achieve o e.g., average turnaround time for lab work for optimal outputs with minimal ED patients; OR turnaround time inputs, reducing waste, re-work and  Patient throughput effort. o e.g., average number of scans per MRI/CT scanner; average/median surgeries per OR; number ED patients served within 4/6 hours (by CTAS level)  Hospital occupancy rate (as a result of a process and not policy change)  Amount of resources used/required/saved o e.g., number of pieces of equipment needed per process/procedure; space required for a service (e.g., waiting room); dollars, equipment, medications, gowns/gloves/masks; number of footsteps; number of steps in a process; nursing and care aide hours worked per inpatient day; nursing overtime hours as a percent of productive nursing hours; total nursing overtime hours  Resource storage o e.g., inventory size; amount of space for storage Dimension Description 179 Dimension Description Process Measure Themes (System Voice: Changes to the Organization) Outcome Measure Themes (Patient or Staff voice)  Errors in document processing o e.g., number of billing errors; number of errors in forms; number of incorrect referrals  Amount of time spent in direct patient care o e.g., nursing and care aide hours worked per inpatient day  Amount of time spent in non-patient care o e.g., setting up; searching for supplies, equipment, or patients; briefing time at shift change  Patient length of stay  Patient satisfaction with length of stay o e.g., percent medical cases whose length of stay is less than or equal to Expected Length of Stay Work Life Supporting wellness in the work environment: Optimal individual, client and organization health and outcomes are provided in the work environment.  Scope of practice o e.g., percent staff reporting practicing to full (appropriate) scope; percent staff reporting their roles & responsibilities are clearly defined (where they weren’t previously)  Workload o e.g., average/median number of patients per care provider; total patient count per time period  Autonomy  Overtime o e.g., nursing overtime as a percent of productive nursing hours  Staff satisfaction with scope of practice, workload, autonomy, specific process change  Staff injury rate o e.g., rate of newly accepted LTD claims; staff injury rate per 100 person years of employment  Sick Time o e.g., sick leave as a percent of productive hours  Staff turnover rate  Vacancies in ‘difficult to fill’ positions  Rate of newly accepted LTD claims Adapted from “Metrics Framework” in Lean Evaluation in the Health Authorities (pp. 20-25), by J. Chan, F. Bryan, E. McGarvey, K. Yang, J. Arthur, and G. Worsley, 2011, Victoria, BC: Ministry of Health. 180 181 Appendix C RPIW Costing Analyses 182 183 184 185 186 187 188 189 190 191 192 193 194 Appendix D Phillips Return on Investment Model From Measuring ROI in Healthcare (p. 62), by J.J. Phillips, V. Buzachero, P.P. Phillips, and Z.L. Phillips, 2013, New York: McGraw-Hill. Copyright 2013 by Authors. Reprinted with permission. . 195 Appendix E Lean Study Recruitment Poster 196 Appendix F Lean Study Interview Consent Form Informed Consent to Participate in a Research Study As a staff member at [the HA], you are being invited to participate in a research study that will investigate the Lean quality improvement program at [the HA]. This information sheet will ensure that you understand the purpose of this study, what you are being asked to do, and your rights as a participant. If you have any questions about the study or what is requested of you, please feel free to ask any questions you may have before participating. Title of Study: Investigating Lean at [the HA]: A systematic study of employee experience and economic evaluation of quality improvement. Thesis Supervisor: Dr. Jalil Safaei, PhD. Affiliation: Associate Professor, University of Northern British Columbia: Department of Economics. Telephone: [(XXX) XXX-XXXX]. Principal Investigator: James Chan, BA (Hons), MA, PhD student. Affiliation: University of Northern British Columbia: School of Health Sciences. Telephone: [(XXX) XXX-XXXX]. The project is being conducted by James Chan who will produce a graduate dissertation in order to fulfill the requirements for the degree of Doctor of Philosophy in the School of Health Sciences at the University of Northern British Columbia (UNBC). James Chan will be working under the supervision of Dr. Jalil Safaei, Associate Professor at UNBC. Purpose: The general purpose of this study is to assess the multiple components of the Lean quality improvement program. The first portion of the study will examine how employees of a health system experience Lean, and what impact Lean might have on engagement. Understanding how employees experience Lean and learning about the process of engagement is critical for any healthcare organization that wishes to improve its services, client care, and overall performance using Lean. What is Required: If you have been involved in a Rapid Process Improvement Workshop (RPIW), you are invited to participate in this first portion of the study, which involves obtaining your input and suggestions about the Lean Program during an individual interview with the Principal Researcher. The interview will take about 30-60 minutes to complete. 197 Monetary Compensation: If you choose to participate in the study, you will not receive any monetary compensation for providing your input. You may be permitted to participate in an individual interview during your regular working hours, in which case, you will be receiving remuneration according to your regular rate of pay. No additional monetary compensation will be provided. Audiotaping and Transcribing of Interview: You are being asked to provide permission for the interview to be audiotaped and transcribed so that an accurate record of your description of your experiences can be created. The audiotapes and transcripts are for research purposes only and will not be shared with any other persons or institutions. Only James Chan will have access to the audiotapes and transcripts of the interview. Confidentiality: Your responses to the interview will be coded, so that numbers will be used on the interview records instead of any identifying information in order to protect your identity. Your name will only appear on the consent forms, which will be kept separate from the interview records in a locked filing cabinet, located in a secure and alarmed office that is accessible only to James Chan. The interview records will be kept in a locked filing cabinet, located in a secure and alarmed office that is accessible only to James Chan. No personal names or identifying information will be entered into electronic files—and any data that is entered into a computer will be encrypted, password protected, and stored in a secure alarmed office. Given that the information you provide will be aggregated and distributed widely, complete confidentiality cannot be maintained, but your anonymity will be maintained (see Dissemination of Information below). No names or identifying information will be included in the dissemination of the research results. Right to Decline or Withdraw: Participation or non-participation in this study will have no effect on your employment with [the HA]. Your participation is completely voluntary and you may choose to withdraw at any time during the study without penalty of any kind. If you decide to withdraw at any point before the study is complete, the information that you have provided thus far will be destroyed. Your decision to decline or withdraw from participating in this study will have absolutely no impact on your employment with [the HA]. Disposal of Data: All data will be kept in a secure location (with interview records kept separate from personal information) for seven years after the completion of the study. The data will then be destroyed by shredding the surveys, interview records, and consent forms and by deleting all tape recordings and computerized data files (including electronic storage devices). Potential Risks: There are minor risks anticipated by you participating in this research. Specifically, social risk (e.g., potential loss of privacy) and psychological/emotional (e.g., potentially feeling uncomfortable). If you experience any distress as a result of participating in this study, professional counseling is available at no cost to you through [the HA’s] Employee and Family Assistance Program (Telephone: [X-XXX-XXX-XXXX]). 198 Dissemination of Results: Once the study is complete, a Dissertation will be produced. Any information that you offer in terms of your responses in the interviews or on the surveys will be summarized in the Dissertation, and therefore, your responses will be kept confidential and will not be personally identifiable. At the end of the study, a summary of the Dissertation will be made available to all participants on the HA website. The Dissertation report will also be submitted to the UNBC Library. As well, article(s) from the Dissertation will be presented at professional conferences, and submitted to peer-reviewed journals for publication. No names or identifying information will be included in the dissemination of the research results. Debriefing: At the end of the study, participants can request a copy of the final Dissertation by contacting James Chan at [(XXX) XXX-XXXX] or via email at [email address provided]. Other important Information and Contacts: You will be given a copy of the signed informed consent form for your own files. If you have any comments or would like further information about this study, please contact James Chan by telephone at [(XXX) XXXXXXX] or via email at [email address provided]. You may also contact Jalil Safaei, Associate Professor at the University of Northern British Columbia, by telephone at [(XXX) XXX-XXXX] or via email at [email address provided]. If you would like to verify the ethical review of this study, or raise any concerns or complaints that you may have, please contact the Office of Research at the University of Northern British Columbia at (250) 960-6735 or via email at reb@unbc.ca. If you would like to verify the ethical review of this study by [the HA], or if you have any concerns about your rights as a research participant, you may contact the Chair of the [HA] Research Ethics Board at [(XXX)-XXX-XXXX] or via email at [email address provided]. Potential Benefits of Participating in the Study: We do not anticipate that you will directly benefit from participating in this study. However, the information gained from this survey is invaluable to [the HA] for decision making regarding the future directions of the Lean Program, as well as for quality improvement efforts in general. Importance of This Research: The information derived from this research will be very useful to the Lean Program and HA Administrators. Findings derived from this study will assist HA Administrators in understanding the effectiveness and cost-benefits of the program, and will also provide evidence to inform decisions regarding the strategic directions of the program. 199 Informed Consent to Participate in a Research Study I have read the above information concerning the study entitled, “Investigating Lean at [the HA]: A systematic study of employee experience and economic evaluation of quality improvement.” I understand that I am being asked to participate in a research study and I have received and read an information sheet that describes the study. I understand the conditions of my participation, including the requirement to participate in an interview and that my responses to any questions will be kept confidential (only the two researchers named in the information sheet will have access to the information I provide). I also understand that there are minor risks to participating in this research (e.g., potential social risk of loss of privacy). If I experience any distress as a result of participating in this study, professional counseling is available to me at no cost. I have had adequate opportunity to consider the information in the document, and to discuss or ask questions pertaining to the study. I understand that my participation in this study is entirely voluntary and that I may refuse to participate or withdraw from the study at any time without explanation or penalty of any kind. Participation or non-participation in this study will have no effect on my employment with [the HA]. This study was explained to me by (Print Name): ____________________________ Date: ____________________ I have received a copy of this consent form and the information sheet, and my signature indicates that I agree to participate in the study. Printed Name of Research Participant: _______________________ Signature of Research Participant: ___________________________ Date: ____________________ Printed Name of Witness: ___________________________ Signature of Witness: _______________________________ Date: ____________________ 200 Appendix G Lean Study Interview Guide Introduction: Thank you for agreeing to meet and participate in this interview. The intent of this interview is to seek your perspectives on how you experience Lean, and explore what impact Lean might have on engagement. Understanding how employees experience Lean and learning about the process of engagement is critical for any healthcare organization that wishes to improve its services, client care, and overall performance using Lean. At this point I would like to remind you that any identifiable information that you may share during the interview will be kept strictly confidential. Furthermore, your name will not be used, mentioned, or linked to any responses within any ensuing reports summarizing the results of our study. I would also like to ask your permission to record this interview? Moustakas (1994) General Questions: I would like to start by asking you a couple of very general questions:  What have you experienced in terms of Lean?  What contexts/situations have typically influenced/affected your experience of Lean? Background Question and RPIW Questions: 1. What training do you have in Lean? Probing Questions:  Can you explain more about your training in Lean?  Where did you obtain your Lean training?  Do you have any suggestions for improving Lean training? 2. Have you participated in an [HA] Lean Rapid Process Improvement Workshop? (This is a screening question to ensure participant has a basic level of experience with Lean.) Probing Questions:  Were you informed about the Lean Rapid Process Improvement Workshop prior to your involvement? (Question explores the introduction of the Lean RPIW.)  How were you informed about the RPIW? 201 3. Please describe your role in the Rapid Process Improvement Workshop. (Question explores participant’s experience in the Lean RPIW.) 4. Were the objectives of the Rapid Process Improvement Workshop clear? (Question explores the direction of the RPIW.) Probing Questions:  Can you please describe the objectives of the RPIW?  Please explain how the RPIW addressed specific objectives. 5. Please provide your opinion of how the Rapid Process Improvement workshop was conducted. (Question explores participant’s opinion of how the RPIW was conducted.) 6. What were the enablers to the success of the Rapid Process Improvement Workshop? (Question explores participant’s experience of what contributes to success of an RPIW.) Probing Questions:  Can you please explain further? 7. What were the barriers to the success of the Rapid Process Improvement Workshop? (Question explores participant’s experience of what impedes success of an RPIW.) Probing Questions:  Can you please explain further? 8. How were the process changes sustained following the Rapid Process Improvement Workshop? (Question explores participant’s experience of what contributes to the sustainability of Lean improvement efforts. The phrase “process changes sustained” will be defined as improvements that were targeted and specifically enacted during the RPIW – such changes will be identified in the RPIW Target Progress Report and other documents. They will be measured using the Target Progress Report at 30, 60, and 90 days post-intervention.) 9. What do you think could be improved regarding the Rapid Process Improvement Workshop? (Question explores participant’s experience of the RPIW and their opinion of what might contribute to improvement.) 202 Employee Engagement Questions: 10. How would you define “employee engagement?” 11. Were you engaged in the Rapid Process Improvement Workshop? (Question explores participant’s engagement in the RPIW. Engagement will be defined as regular attendance and active participation in the RPIW.) Probing Questions:  Why or why not?  How were you engaged in the RPIW? 12. How has your engagement with Lean evolved? (Question explores participant’s engagement with Lean.) 13. What factors help with Employee Engagement as it pertains to Lean? (Question explores participant’s opinion of how employee engagement may be improved using Lean.) 14. If you believe that the [HA] Lean Program has positively impacted the extent to which you are engaged in your work, please explain HOW the Lean Program has influenced your engagement. 15. How do you stay engaged at work? (Question explores how staff maintain engagement at work in general.) 16. How do you keep staff engaged? (Question is for Leaders and explores how they attempt to maintain staff engagement in general.) 17. How do you think Employee Engagement in Lean can be improved? (Question explores participant’s opinions of how engagement can be improved with regard to Lean.) 18. Any negative situations related to Lean that you’d like to share regarding Employee Engagement? (Question explores participant’s experience of how Lean might interfere with employee engagement.) 203 Appendix H Lean Study On-Line Survey Informed Consent to Participate in a Research Study TITLE OF STUDY: Investigating Lean at [the HA]: A systematic study of employee experience and economic evaluation of quality improvement. INVITATION and STUDY PURPOSE: As a staff member at [the HA], you are invited to participate in a survey that will investigate the Lean Program at [the HA]. You have received this survey invitation because you may have participated in a Lean initiative at [the HA]. This informed consent form will ensure that you understand the purpose of the study, what you are being asked to do, and your rights as a participant. We will use the information you provide to identify areas for improvement for the Lean Program at [the HA]. Please note: This survey is being conducted for a doctoral research project and the preliminary informed consent section is therefore quite detailed to meet requirements. Once you proceed to the survey it is brief and should take approximately 15 minutes to complete. STUDY TEAM: Principal Investigator: James Chan, BA (Hons), MA, PhD Student, School of Health Sciences, University of Northern British Columbia, [(XXX) XXX-XXXX]. Supervisor: Dr. Jalil Safaei, Associate Professor, Department of Economics, University of Northern British Columbia, [(XXX) XXX-XXXX]. This research is part of James Chan’s doctoral studies, which are being completed through the School of Health Sciences at the University of Northern British Columbia (UNBC). Anonymous and aggregated data collected through this research will be included in published journal articles, conference presentations and as part of his doctoral Dissertation. STUDY PROCEDURES: If you decide to participate in this study, you will be asked questions about your experience with Lean at [the HA]. We anticipate the survey will take approximately 15 minutes to complete. STUDY RESULTS: Results from the survey will be incorporated into James Chan’s PhD Dissertation and a summary of it will be made available to all participants on the [HA] website. Anonymous and aggregated data collected through this research may also be included in published journal articles, conference presentations and as part of James Chan’s doctoral Dissertation. No names or identifying information will be included in the dissemination of the research results. If you would like to receive a copy of the Dissertation, please contact James Chan at [(XXX) XXX-XXXX] or at [email address provided] 204 POTENTIAL RISKS OF THE STUDY: There are minor risks anticipated by you participating in this research. Specifically, social risk (e.g., potential loss of privacy) and psychological/emotional (e.g., potentially feeling uncomfortable). However, if you experience any distress as a result of participating in this study, professional counseling is available at no cost to you through [the HA’s] Employee and Family Assistance Program (Telephone: [X-XXX-XXX-XXXX]). POTENTIAL BENEFITS OF THE STUDY: We do not anticipate that you will directly benefit from participating in this study. However, the information gained from this survey is invaluable for decision making regarding the future directions of Lean at [the HA], as well as for quality improvement efforts in general. VOLUNTARY PARTICIPATION: Participation or non-participation in this study will have no effect on your employment with [the HA]. Your participation is completely voluntary and you may choose to withdraw at any time during the survey without penalty of any kind. If you decide to withdraw at any point before the survey is complete, the information that you have provided thus far will be deleted. Your decision to decline or withdraw from participating in this survey will have absolutely no impact on your employment with [the HA]. CONFIDENTIALITY: When you press the Submit button at the end of the electronic survey, your responses go directly to the Principle Investigator. [The HA] Administrators will not see the responses of individual participants. Given that the information you provide will be aggregated and distributed widely complete confidentiality cannot be maintained, but your anonymity will be maintained in the dissemination of results. This online survey is hosted by FluidSurveys, an American-owned company (SurveyMonkey). While FluidSurveys currently stores survey data in Canada and has put measures in place to adhere to Canadian privacy legislation, the information you provide though this survey may now be subject to U.S. laws, including the U.S. Patriot Act which allows authorities access to the records of internet service providers. If you choose to participate in this survey, you understand that your responses to the survey questions may be accessed outside of Canada. The privacy policy for FluidSurveys can be found at the following link: http://fluidsurveys.com/about/privacy. The data will be managed by the Principle Investigator and Secured Sockets Layer (SSL) encryption will be used to provide extra security between the web server and your browser. Study data will be stored on a password protected hard drive and in a locked filing cabinet in a locked research office for 7 years, after which it will be destroyed. PAYMENT: You will not receive any monetary compensation for participating in this survey. You are permitted to complete the survey during your regular working hours, in which case, you will be receiving remuneration according to your regular rate of pay. No additional monetary compensation will be provided. 205 CONTACT FOR INFORMATION ABOUT THE STUDY: If you have any comments or would like further information about this survey, please contact James Chan by telephone at [(XXX) XXX-XXXX] or via email at [email address provided]. You may also contact Dr. Jalil Safaei, Associate Professor at the University of Northern British Columbia, by telephone at [(XXX) XXX-XXXX] or via email at [email address provided]. If you would like to verify the ethical review of this survey, or raise any concerns that you may have, please contact the Office of Research at the University of Northern British Columbia at (250) 960-6735 or via email at reb@unbc.ca CONTACT FOR COMPLAINTS: If you have any concerns about your rights as a research subject and/or your experiences while participating in this study, you may contact the UNBC Office of Research at (250) 960-6735 or email at reb@unbc.ca 206 CONSENT STATEMENT: Before you begin the survey, please read the following consent statement and choose one of the options below. I have read the information that came with my invitation to this survey. I understand that my participation is voluntary and anonymous. I understand any risks and benefits. Do you consent to participate in this study? Yes No Messaging to participants that click “No”: You did not provide consent to participate in this survey. If you wish to provide consent and continue, please use your browser's 'Back' button and select 'Yes' to the consent question. Thank you for your time. INSTRUCTIONS: As an organization, [the HA] has used Lean tools and methods for approximately 10 years. [The HA] began its Lean journey by using Lean tools on a sporadic basis in pockets of the organization with occasional emphasis paid to consultant-led special projects. The individual project approach was replaced by the advent of the Lean Promotion Office. The Lean Promotion Office is currently made up of six positions: one Lean Leader, four Lean Consultants, and a Lean Program Coordinator. However, there are other HA staff that have had formal and informal training in Lean, and competencies range from basic to more advanced levels. Since the inception of the Lean Promotion Office, [the HA] has been gradually organizing its Lean strategy to integrate efforts toward a coherent body of quality improvement capacity. One of the key breakthroughs was contracting with the Virginia Mason Institute in order to bring a fulsome, consistent package of Lean training and expertise into the organization (known as the Virginia Mason Production System). By 2011, [the HA] had developed a comprehensive Lean Strategic Plan and activities had evolved from individual projects to more concentrated work along specific service areas. For the purpose of this survey, we would like respondents to think of the most recent Lean work in [the HA] and the actions of the Lean Promotion Office (as opposed to sporadic Lean work that has taken place in the past). If you respond to the questions with the work of the Lean Promotion Office in mind, it will help us interpret the survey results in terms of the latest formal Lean Program at [the HA], rather than the intermittent Lean work that has taken place during the early years of Lean being introduced to the organization. 207 Please use the following buttons (located at the bottom of each page) to navigate through the survey:  For each item in the survey, read the statement, and click on the response option that you prefer (e.g., “agree”, “disagree”, etc.).  SCROLL DOWN and Click "Next" to go to the next page once you have completed the question(s) or if you do not wish to answer the question(s).  Click "Back" to go back to the previous page. IMPORTANT: You must complete your responses to the survey in one session – there is no auto-save function in the system so you cannot complete part of the survey and then return to it later. The survey should only take about 15 minutes to complete. If you need assistance completing this survey, or if you have any technical difficulties, please email your questions to [email address provided], or call [(XXX) XXX-XXXX]. Please click “Next” to continue with the survey. The following questions inquire about your engagement in Rapid Process Improvement Workshops (RPIWs) and other Lean work. In brief, Rapid Process Improvement Workshops are designed to identify waste and remove or eliminate it, with the goal of improving processes and increasing value to customers in a collaborative effort using Lean tools. 1. How many Lean Rapid Process Improvement Workshops have you participated in? Please check the box that describes your experience most accurately. Zero One Two Three Four Five More than five 2. Aside from Rapid Process Improvement Workshops, what other Lean activities have you participated in? Please describe and provide examples: 208 3. How long have you been involved in Lean activities? Less than 1 year 1 year to 2 years 2 years to 3 years 3 years to 4 years 4 years to 5 years Over 5 years The following questions inquire about your experience with Lean Education at [the HA]. 4. Which Training Program did you participate in to learn about Lean at [the HA]? Please check all that apply. General Electric Lean Training Intro to Lean on iLearn at [the HA] Leading Edge Group Lean Training Intro to Lean-Learning Hub at the BC Ministry of Health Lean Implementation Specialist Certification Program (delivered by the Lean Promotion Office at [the HA]) Lean orientation at Day 1 of a Rapid Process Improvement Workshop (delivered by the Lean Promotion Office at [the HA]) Other, please specify: (including Lean training taken outside of [the HA]) ______________________ Not applicable – I have not taken any Lean training. 5. If you participated in the Lean Implementation Specialist Certification Program delivered by the Lean Promotion Office, what suggestions do you have to improve the training? 209 The next few questions inquire about the team of internal Lean Consultants at the Lean Promotion Office at [the HA]. 6. The internal Lean consultants provided sufficient support to help successfully conduct Lean work. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 7. I feel that the internal Lean consultants help to support improvement of [the HA] as an organization. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 8. What suggestions do you have to improve the services provided by the internal Lean consultants? This section of the survey inquires about your Employee Engagement experience. 9. I look forward to participating in the Lean activities facilitated by the Lean Promotion Office (for example, Rapid Process Improvement Workshops). Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 210 10. Participating in Lean activities supported by the Lean Promotion Office has inspired better job performance in me. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 11. The principles of the Lean Promotion Office and program are similar to my values. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 12. I am proud to tell others about the Lean work done in my unit/Department that was supported by the Lean Promotion Office. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 13. I am more satisfied with my job as a result of my participation in Lean work facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 211 14. My department or unit at [the HA] is a better place to work as a result of the Lean improvements supported by the Lean Promotion Office. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 15. Before my involvement in Lean activities facilitated by the Lean Promotion Office at [the HA], I would rate my engagement as an employee as: Please check the box that describes your experience most accurately. Strongly disengaged Disengaged Neither engaged nor disengaged Engaged Strongly engaged 16. After my involvement in Lean activities facilitated by the Lean Promotion Office at [the HA], I would rate my engagement as an employee as: Please check the box that describes your experience most accurately. Strongly disengaged Disengaged Neither engaged nor disengaged Engaged Strongly engaged 17. If you believe that your involvement in Lean activities facilitated by the Lean Promotion Office at [the HA] has positively impacted the extent to which you are engaged in your work, please explain HOW your involvement in Lean has influenced your engagement. 212 The following two questions are related to Capabilities and Initiative at [the HA]. 18. I have more opportunity to make suggestions to improve the work of my unit/team/department because of our involvement in Lean work facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 19. I have more opportunity to use my skills because of the Lean work facilitated by the Lean Promotion Office. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree For this section, we would like your input regarding Leadership Responsiveness and Support. 20. Managers act on feedback that staff provide as a result of Lean activities facilitated by the Lean Promotion Office. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 213 21. We support one another during Lean Rapid Process Improvement Workshops facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 22. The Lean Rapid Process Improvement Workshops facilitated by the Lean Promotion Office have helped us to work together and help each other out. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree The following two questions are about Respect at [the HA]. 23. People from diverse backgrounds feel welcome during work that is facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 24. We treat each other with respect during our Lean work that is facilitated by the Lean Promotion Office. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 214 This section inquires about your experience with Decision Input and Resources. 25. I was consulted about changes that affect my unit/team during Lean work that was facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 26. My work unit is better organized to meet the needs of clients/service users following our participation in Lean work facilitated by the Lean Promotion Office at [the HA]. Please check the box that describes your experience most accurately. Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree This section of the survey inquires about your experience with Lean work in general at [the HA]. For each of the following statements, please select the box that describes your experience most accurately. Strongly disagree 27. The Lean work completed in my area has improved the quality of the care and/or services we deliver. 28. The Lean work completed in my area has improved the safety of the care and/or services we deliver. 29. The Lean work completed in my area has improved access (e.g., by decreasing wait times) for care and services in my area. Disagree Neither agree nor disagree Agree Strongly agree 215 Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 30. The Lean work completed in my area has made client services/patient care more reliable (i.e., consistent). 31. The Lean work completed in my area has made the work flow better. 32. The changes made to my work area through the Lean Program at [the HA] have made my job more manageable (e.g., easier workload). 33. The Lean work done in my area has allowed me to make better use of my time and skills. 34. Through participation in Lean activities at [the HA], teamwork with my colleagues has improved. 35. The Lean work at [the HA] has provided clearer expectations about my job. 36. I have had an opportunity to develop new skills as a result of the Lean Program at [the HA]. 37. Lastly, please provide any other comments you may have about Lean. Demographic section: Please provide some information about yourself so we can compare the experiences of different staff groups. 38. Which area in [the HA] do you work? Acute Care Home / Community Care Residential Care Please specify... ______________________ 216 39. Please check the box that best describes your role/position at [the HA] most accurately. Administrator Manager Physician RN LPN Laboratory Technologist/Technician Occupational Therapist Physiotherapist Psychologist/Counsellor Pharmacist Patient Transport/Porter Registered Dietician/Nutritionist Respiratory Therapist Social Worker Speech Language/Audiology Therapist Other, please specify... ______________________ 40. Please describe the service/department you are employed in at [the HA]. Thank you very much for participating! Please click 'Submit' to complete your survey. 217 Appendix I Interrupted Time Series Estimates K1: Interrupted Time Series Estimates: Site 1 Pre-Intervention Period to Intervention Period K2: Interrupted Time Series Estimates: Site 1 Pre-Intervention Period to Intervention Period combined with Post-Intervention Period 218 K3: Interrupted Time Series Estimates: Site 2 Pre-Intervention Period to Intervention Period K4: Interrupted Time Series Estimates: Site 2 Pre-Intervention Period to Intervention Period combined with Post-Intervention Period 219 K5: Interrupted Time Series Estimates: Pre-Intervention Period to Intervention Period on surgical volume difference scores between Site 1 and Site 3 K6: Interrupted Time Series Estimates: Pre-Intervention Period to Intervention Period and Post-Intervention Period combined on difference scores between Site 1 and Site 3 220 K7: Interrupted Time Series Estimates: Pre-Intervention Period to Intervention Period on surgical volume difference scores between Site 2 and Site 4 K8: Interrupted Time Series Estimates: Pre-Intervention Period to Intervention Period and Post-Intervention Period combined on difference scores between Site 2 and Site 4 221 Appendix J Declarations Declaration of Conflicting Interests: The author declares no potential conflicts of interest with respect to the research, authorship, and/or publication of this Dissertation. Funding: This research is an un-funded study.