INFORMATION TO USERS This manuscript has been reproduced from the microfilm master. UMI films the text directly from the original or copy submitted. Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer. The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps. Photographs included in the original manuscript have been reproduced xerographically in this copy. Higher quality 6” x 9” black and white photographic prints are available for any photographs or illustrations appearing in this copy for an additional charge. Contact UMI directly to order. ProQuest Information and Learning 300 North Zeeb Road, Ann Art)or, Ml 48106-1346 USA 800-521-0600 UMI* THE EFFECTS OF A PRECISION TEACHING / DIRECT INSTRUCTION READING PROGRAM ON THE READING ACHIEVEMENT OF ELEMENTARY SCHOOL STUDENTS by Tamara Rothenbusch B.Ed., Simon Fraser University, 1987 THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF EDUCATION in CURRICULUM AND INSTRUCTION © Tamara Rothenbusch, 1999 THE UNIVERSITY OF NORTHERN BRITISH COLUMBIA Fall 1999 All rights reserved. This work may not be reproduced in whole or in part, by photocopy or other means, without the permission of the author. 1 ^ 1 National Library of Canada Bibliothèque nationale du Canada Acquisitions and Bibliographic Services Acquisitions et services bibliographiques 395 WeHinglon Street Ottawa ON K1A0N4 Canada 395, rue WeWngton Ottawa ON K1A0N4 Canada YaurlUm VatmréUnne» O urm NoBrntmnne» The author has granted a non­ exclusive hcence allowing the National Libraiy of Canada to reproduce, loan, distribute or sell copies of this thesis in microform, paper or electronic formats. L’auteur a accordé une licence non exclusive permettant à la Bibliothèque nationale du Canada de reproduire, prêter, distribuer ou vendre des copies de cette thèse sous la forme de microfiche/film, de reproduction sur papier ou sur format électronique. The author retains ownership of the copyri^t in this thesis. Nei&er the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author’s permission. L’auteur conserve la propriété du droit d’auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son autorisation. 0-612-62498-6 CanadS APPROVAL Name: Tamara M. Rothenbusch Degree: Master of Education Thesis Title: THE EFFECTS OF A PRECISION TEACHING / DIRECT INSTRUCTION READING PROGRAM ON THE READING ACHIEVEMENT OF ELEMENTARY SCHOOL STUDENTS Examining Committee: Chair: Dr. Martha MacLeod A s s o c i^ P ro fe ssw ^ k ^ a ir of I S ^ in g Supervisor Dr. Peter MacMillan Assistant Professor Education Dr. Bryan Hartman Professor Education Dr. Anita Hubley Assistant Professor Psycholo E x te n m ^ x a ^ m e r Dr. Fred Egglestone Principal: College Heights Secondary School School District No. 57 (Prince George) Date Approved: ' / u ABSTRACT This research study evaluates a reading program which was implemented in a pilot school in School District No. 56 in 1994. The reading program combines the teaching strategies o f Precision Teaching, fluency-building, and Direct Instruction. The Canadian Tests o f Basic Skills Reading and Vocabulary subtest scores o f students in the treatment school were compared to those of students in four combined control schools in the pre­ treatment and the post-treatment years. In addition to an analysis involving all students, the scores o f Aboriginal and non-Aboriginal students were analyzed separately to determine whether the two groups responded in a similar manner to the reading program. Results o f this study indicate a consistent trend in that the students in the treatment school, including Aboriginal and non-Aboriginal groups, demonstrated significant and important gains in their Reading subtest scores. No differences, however, were demonstrated in their Vocabulary subtest scores. Students in the control group, including Aboriginal and non-Aboriginal students, consistently demonstrated decreases in both their Reading and Vocabulary subtest scores. Results o f this study provide evidence that, although significantly behind the control group before the reading program's implementation, after implementation, the treatment school students exhibited comparable reading achievement to those of the control schools. ui TABLE OF CONTENTS Abstract 11 Table o f Contents iii List o f Tables V List o f Figures vi Acknowledgement vii CHAPTER ONE: INTRODUCTION Background Definitions Direct Instruction Precision Teaching and Fluency Momingside Model o f Generative Instruction Problem Statement Hypotheses 1 1 3 3 4 4 7 8 CHAPTER TWO: LITERATURE REVIEW SRA's Reading Mastery Precision Teaching and Fluency History, Philosophy, and Methodology General Research Reading Research Direct Instruction History, Philosophy, and Methodology Project Follow Through Research Since Project Follow Through Project Follow Through Follow-up Studies Special-Education Students Meta-Analyses Momingside Academy Criticisms o f the Reading Program Summary 10 10 12 12 16 21 25 26 28 31 32 35 38 40 42 46 CHAPTER THREE: METHOD Subjects Measures Procedures 47 48 51 53 IV Reading Program in the Treatment School Reading Approaches in the Control Schools The Provincial Learning Assessment Program Collection and Treatment o f Data S3 55 58 60 63 CHAPTER FOUR: RESULTS Preliminary Analysis Main Analysis Descriptive Comparison o f the Treatment and Control Groups All Students - Reading All Students - Vocabulary Aboriginal Students - Reading Non-Aboriginal Students - Reading Aboriginal Students - Vocabulary Non-Aboriginal Students - Vocabulary Statistical Comparison o f the Treatment and Control Groups Effect Size Summary 65 65 67 67 67 69 70 71 73 74 76 78 79 CHAPTER FIVE: DISCUSSION Summary o f the Study Interpretation o f the Results Comparison o f the Research Limitations o f the Study Implications for Practice Implications for Future Research 80 80 82 85 88 90 92 REFERENCES 96 APPENDICES Appendix A: Opinions o f Teachers in the Treatment School Appendix B: Elementary Teacher Questionnaire Appendix C: School Board Approval Appendix D: Letter o f Intent to Principals Appendix E: Letter to Aboriginal Band Councils / Research Study Permission Form 104 113 117 119 121 Ethics LIST OF TABLES Table 1 Numbers o f Students Involved in the Study SO Table 2 Provincial Learning Assessment Program Results 60 Table 3 Analysis o f Variance Summary for Comparison o f the Control Schools 66 Table 4 CTBS Reading Subtest Data 68 Table 5 CTBS Vocabulary Subtest Data 69 Table 6 CTBS Reading Subtest Difference Scores o f Aboriginal (AB) and Non-Aboriginal (Non-AB) Students 71 Table 7 CTBS Vocabulary Subtest DifferenceScores o f Aboriginal (AB) and Non-Aboriginal (Non-AB) Students 74 VI LIST OF FIGURES Figure 1 CTBS Reading Subtest National Percentile Rank Means o f the Treatment and Control Groups Before and After Treatment 68 Figure 2 CTBS Vocabulary Subtest National Percentile Rank Means o f the Treatment and Control Groups Before and After Treatment 70 Figure 3 CTBS Reading Subtest National Percentile Rank Means o f Aboriginal Students in the Treatment and Control Groups Before and After Treatment 71 Figure 4 CTBS Reading Subtest National Percentile Rank Means o f Non-Aboriginal Students in the Treatment and Control Groups Before and After Treatment 72 Figure 5 CTBS Reading Subtest Mean Difference Scores o f Aboriginal (AB) and Non-Aboriginal (Non-AB) Students in the Treatment and Control Groups 73 Figure 6 CTBS Vocabulary Subtest National Percentile Rank Means o f Aboriginal Students in the Treatment and Control Groups Before and After Treatment 74 Figure 7 CTBS Vocabulary Subtest National Percentile Rank Means o f Non-Aboriginal Students in the Treatment and Control Groups Before and After Treatment 75 Figure 8 CTBS Vocabulary Subtest Mean Difference Scores of Aboriginal (AB) and Non-Aboriginal (Non-AB) Students in the Treatment and Control Groups 76 vu ACKNOWLEDGEMENT I sincerely thank the personnel o f School District No. 91 (Nechako Lakes) and the five Aboriginal Band Councils involved in my research. Without their assistance and cooperation, this study would not have been possible. My heartfelt appreciation goes to my supervisor. Dr. Peter MacMillan, for his patience, support, and expertise. I also thank Dr. Bryan Hartman and Dr. Anita Hubley, members o f my supervising committee, for their advice and encouragement. Finally, I am indebted to my family and fiiends for their continuous thoughts and prayers. INTRODUCTION Background In 1994, School District No. 56 decided to pilot an alternative reading program in response to a growing concern regarding the reading ability o f its students.^ This concern was voiced by administrators, teachers, and parents who believed, as a result o f observing students and from analyzing yearly reading test scores, that students' reading abilities had declined in recent years. Blame for this problem was placed on various factors, and it is not within the scope o f this paper to discuss them or to attempt a definitive explanation. Whatever the cause, it was believed that some o f the reading programs and/or instructional methods which had been used in the past were not entirely successful in ensuring the development o f acceptable reading abilities in students. Because several learning assistance teachers in the district had been using the SRA (Science Research Associates) Reading Mastery and Decoding programs developed by Engelmann (Osborn, 1988) with a fair amount o f success, and because there existed an appeal for such a traditional, "back to basics ' reading approach, the School Board decided that two schools would be chosen for pilot projects to implement this program schoolwide. In addition to the Reading Mastery program and its emphasis on Direct Instruction, the schools would also adopt the Momingside Model of Generative Instruction as ^ In September, 1996, School District No. 55 (Bums Lake) and School District No. 56 (Nechako) amalgamated to form School District No 91 (Nechako Lakes). Prior to this. School District No. 56 served the communities o f Fraser Lake, Fort Fraser, Fort St. James, Tachie, and Vanderhoof. Because the reading program was implemented in 1994 and prior to amalgamation, there are several references in this paper to School District No. 56. developed by Dr. Kent Johnson, founder and director o f Momingside Academy, a private learning center in Seattle, Washington (Johnson & Layng, 1994). As well as emphasizing Direct Instruction, the Momingside Model also practices the instructional strategies o f Precision Teaching and fluency-building. Therefore, in the 1994-1995 school year, two schools were chosen as the district's pilot sites to implement the program. These two schools seemed to be logical choices as pilot sites, because students' reading abilities were a concem at both schools and the staffs were willing and committed to teaching an alternative reading program. Both schools' administrators were highly supportive o f the innovation and allowed for much inservice and training o f staff members. Teachers, principals, and support staff o f both schools received intense training in Direct Instruction and the Momingside Model methods during the summers o f 1994, 1995, and 1996. As well, during these 3 school years, Johnson and several instructors from Momingside Academy traveled regularly from Seattle to the district to provide additional inservice, assessment, and feedback regarding the program's implementation. Although these two pilot sites were involved in implementing the treatment reading program, the Canadian Tests o f Basic Skills (King-Shaw, 1989) scores o f only one school are included in the present study's analysis. It was decided that the second pilot school not be involved in the study's analysis for two reasons. Firstly, this school contains students in Kindergarten through Grade 3, and because only students in Grade 3 and up are tested on the Canadian Tests o f Basic Skills, small and inconsistent sample sizes existed. Secondly, Grade 3 students were excluded from the Canadian Tests o f Basic Skills testing prior to 1993, and so pre-treatment data from the 1992-1993 school year were unavailable. The treatment school is situated in a rural-residential community in central British Columbia with a population o f approximately 450. Ten staff members teach at this school and approximately 110 students from Kindergarten to Grade 7 are in attendance. O f the school's total population, approximately 25-30% are Aboriginal.^ The majority o f the students in this school are from families that are involved in the industries o f forestry or agriculture. Definitions The major components o f the treatment reading program - Direct Instruction, Precision Teaching, and fluency - will be described briefly here. As well, the Momingside Model o f Generative Instruction's philosophy and methodology will be discussed. Additional detailed definitions and descriptions are provided in the forthcoming chapter o f this paper. Direct Instruction Direct Instruction is a teaching approach which emphasizes a structured, skillsbased, and academically-focussed learning environment. It involves the use o f scripted lesson presentations, teacher signals for students to answer aloud and in unison, and immediate feedback and correcting procedures. Direct Instruction emphasizes a mastery approach, where skills are taught until fluent. ^Aboriginal ancestry includes Status Indians, Non-Status Indians, Inuit, and Metis students (Ministry o f Education, 1999). Aboriginal identification is made on a volunteer basis. Precision Teaching and Flueruy Precision Teaching involves charting and analyzing student performance o f isolated skills. Both students and teachers regularly monitor and analyze the data in order to determine student achievement and program effectiveness. Fluency can be defined as the combination o f speed and accuracy in performing skills. Fluency is true mastery in that it involves automatic performance o f behavior. Precision Teaching and fluency are directly related in that fluency-building is achieved through practicing isolated behaviors, and charting and monitoring this practice is a convenient method o f on-going evaluation. M omingside M odel o f Generative Instruction Momingside Academy, which incorporates both Direct Instruction and Precision Teaching strategies, is described by Johnson, its founder and director, as "a learning laboratory for designing programs and procedures with a focus on essential skills for school success and a 13-year research base" (Johnson & Layng, 1994, p. 174). It first opened in 1980 for students unsuccessful in previous educational settings due to learning and/or behavior problems. Students generally spend 1 to 3 years at Momingside Academy before rejoining the public school system. The Momingside Model o f Generative Instruction is based on the underlying principle that all students can be successful if provided with proper placement, proven instructional materials, and the opportunity to practice essential skills to fluency. Reading, writing, math, critical thinking, and study skills are emphasized and taught using databased methods, where students' performance at these skills is charted and analyzed on a daily basis so that instmction can then be adjusted accordingly. The four .major components o f the Momingside program are precision placement testing, the use o f Direct Instruction, fluency-building o f fundamental skills so that performance becomes automatic, and the integration o f component and composite skills (once fluent, lower level skills are combined into more complex and meaningflil activities). The aim o f generative instruction is that students learn how to become independent and self-motivated learners as a result o f being given the tools to measure and improve their own skills. Independent learners are able to retain, use, and apply their fluent skills in new situations and with everyday problems (Johnson & Layng, 1992, 1994; Johnson, 1997). According to Johnson (1991), "Application practice after fluency-building is key. By building skills to rates that make them useful and automatic, and then applied in real-world contexts, student retention and application is dramatically increased" (pp. 65-66). Momingside Academy refers to its components o f Precision Teaching, fluencybuilding, and Direct Instruction as technologies, meaning that they can be taught to others and replicated in other settings. In 1995, the Momingside Learning Systems, a training and consulting company, was founded and has since helped to implement the Momingside Model of Generative Instruction in several schools throughout the United States in addition to School District No. 91. Another important component o f Momingside Academy is its emphasis on current research. As a result, it is continuously conducting relevant classroom research to evaluate its methods and to further develop strategies and materials conducive to student success. An important issue to note regards the degree to which the treatment school has replicated Momingside Academy's model. The major differences between Momingside Academy and Fort Fraser Elementary School are class size and curriculum. As a private school, Momingside Academy has smaller class sizes and employs several teacher-aids and assistants. Their entire instructional day is spent on academics and those skills deemed necessary for school success. In contrast, reading classes in the treatment school can consist o f up to 25 students, and the instructional day must incorporate all Ministrymandated curricular subjects, some o f which are non-academic (e.g. Art, Physical Education, Music). As a result, certain difBculties have been faced by the teachers at this school in implementing the Momingside model, most o f which concem the management o f charting and analyzing individual student progress in large classrooms, and in providing students with the instruction and practice necessary to achieve fluency while still ensuring that other mandated curricular objectives are met. As a result o f these difBculties and for practicality's sake, the reading program at the treatment school is not an exact replication o f the Momingside model. During the first year o f implementation, it was agreed that the model be followed as closely as possible in order to experience its effects and to better understand the underlying philosophy as a result. Since then, however, teachers have been able to use their professional experience and knowledge to adjust and fine-tune the Momingside model to better suit their and their students' needs. In fact, since 1994 and the implementation o f the reading program, the treatment school has incorporated Momingside instructional models in writing, math, study skills, problem-solving and critical thinking skills as well. Teachers have also been using Precision Teaching, fluency-building, and Direct Instruction strategies in the teaching o f content classes such as French, Science, and Social Studies. A more detailed description o f the opinions o f teachers involved in teaching the reading program in the treatment school is provided in Appendix A. Problem Statement Now that the reading program is in its 6th year in Fort Fraser Elementary School, it is necessary to determine its effectiveness. Results o f this research could have major implications regarding the future choice o f reading programs throughout the school district. Archival data from the Canadian Tests o f Basic Skills exist in varying degrees o f completeness from the late 1980s to the present, and it is therefore possible to utilize these data to conduct an evaluation o f the treatment reading program. The intention o f the present study, therefore, is to determine whether or not the reading program has been successful in increasing students' reading abilities as measured by the Reading and Vocabulary subtests o f the Canadian Tests o f Basic Skills (CTBS). Related to this general problem statement are two specific research questions: 1. Will there be an equal difference between the treatment school's (Precision Teaching/Direct Instruction reading program) and the control group schools' (combination o f other reading programs) mean CTBS Reading subtest scores in the pre-treatment years and in the post-treatment years? 2. Will there be an equal difference between the treatment group's (Precision Teaching/Direct Instruction reading program) and the control group schools' (combination o f other reading programs) mean CTBS Vocabulary subtest scores in the pre-treatment years and in the post-treatment years? It is also important to determine whether differences between the treatment and control groups, if any, are equal before and after implementation o f the treatment reading program due to the possibility o f district-wide trends. For example, it is possible that both the treatment group and the control group exhibit equal increases or decreases in student 8 scores from the pre-treatment to the post-treatment years for a reason other than implementation o f the treatment reading program. However, if only the treatment school's scores increase from the pre- to the post-treatment years, one could attribute the difference to some factor unique to this school, one such example being implementation o f the reading program. The two research questions will also be investigated for Aboriginal students and non-Aboriginal students as two separate groups in order to determine whether the treatment reading program is effective for both. However, means and standard deviations of Aboriginal students will not be reported in the present study due to confidentiality agreements between the researcher and the five band councils involved. Hypotheses The two statistical hypotheses of the study relate to the two research questions aforementioned. The hypotheses will be tested for all students as well as for Aboriginal and non-Aboriginal students as separate groups. 1 - H o : Ppce(rXc-t) ~ PpoiKrXc-t) H i ; |.lpre(rXc4) ^ Ppo«(rXc-t) » where |i p r e ( r x c - t ) is the difference between the control group's (c) and the treatment group's (t) mean CTBS Reading subtest scores (r) in the pre-treatment years, and Ppo««rxc-t) is the difference between the control group's (c) and the treatment group's (t) mean CTBS Reading subtest scores (r) in the post-treatment years 2 . H o . ltpre(vXc-t) ~ Ppo#(vXe-t) H i : Ppre(vXc-t) ^ PpoiXvXc-t). where (Xp„(vKc-t) is the difiference between the control group's (c) and the treatment group's (t) mean CTBS Vocabulary subtest scores (v) in the pre-treatment years, and m»rt(vKc-t) is the difiference between the control group's (c) and the treatment group's (t) mean CTBS Vocabulary subtest scores (v) in the post-treatment years. 10 LITERATURE REVIEW This literature review includes definitions and comprehensive descriptions o f the treatment reading program's major components: SRA's Reading Mastery program. Precision Teaching and fluency, and Direct Instruction. Recent and relevant research concerning Precision Teaching, fluency, and Direct flistruction will be summarized and critically analyzed. As well, criticisms o f the reading program's underlying philosophy will be discussed. SRA's Reading Mastery The Science Research Associates' Reading Mastery program is a basal reading series designed to teach reading comprehension and decoding skills. The program was developed by Engelmann in the 1960s after extensive research and field testing (Engelmann, Hanner, & Johnson, 1989; Osborn, 1988). Reading Mastery is a traditional, "back-to-basics " program which values efihciency and productivity. It is scientific in nature, behavioristic, and emphasizes objectivity and measurability. The philosophy underlying Reading Mastery is that students who possess the prerequisite skills will leam more and in less time than if taught with other reading programs. As well, the authors o f Reading Mastery claim that all students can leam given adequate teaching presentations and practice. The program's goals, as stated in the Reading Mastery Teacher's Guides (Engelmann & Hanner, 1988), include developing those composite skills believed to influence and determine future reading progress: "the ability to decode accurately, read at a reasonable rate, understand what is read, leam important relationships and facts through reading, retain information that is important, and find reading an enjoyable activity" (p. 1). 11 Reading Mastery involves the applied science orientation o f component / composite analysis, or the task analysis o f the skills needed to attain certain objectives. The program breaks down each composite skill into small, specific component tasks or subskills which are then taught to mastery using the Direct Instruction method. Lessons are based on cumulative skill development, where students are provided with practice throughout the remainder o f the program in applying the skills taught. Once students have received enough practice and immediate feedback so that the task becomes automatic and mastery is achieved, it is believed that they are then able to retain the skill, apply it in more complex situations, and are prepared to leam related skills. The program's authors claim that reteaching o f these skills becomes unnecessary and that every student is able to experience a high rate o f success. The Reading Mastery program consists o f six levels which correspond to Grades 1 through 6. Lessons are paced in a quick and lively manner so that more material is covered in less time and so that no time is wasted. Fast pacing o f lessons, according to the program developers, also results in greater student achievement, increased on-task student behavior, and decreased class management problems. Teachers must follow a script and expect students to respond in a specified manner as well. The scripts are designed so that lessons can be presented precisely, quickly, and consistently. Continuous assessment is a major component o f Reading Mastery. Through strategies such as unison responding and daily checks o f oral reading and written assignments, teachers are regularly made aware o f each student's progress and are then expected to plan accordingly. In this way, the program essentially follows a "teach-assessreteach" model. 12 Since its development, Engelmann and his colleagues have continued to extensively field-test, revise, and design programs in order to maximize their effectiveness with students in classrooms (Grossen, 1996). According to Bateman (1991), "The documented success o f Engelmann and his colleagues' Direct Instruction reading programs with thousands o f hard-to-teach and high-risk children is unsurpassed in the annals of reading history" (p. 11). Grossen agrees: "The large research base o f Reading Mastery distinguishes it fi’om all other currently available commercially developed reading programs" (p. 1). Precision Teaching and Fluency In this section, a comprehensive description o f the history, philosophy, and methodology o f Precision Teaching and fluency will be provided. As well, research regarding the effects o f Precision Teaching and fluency-building strategies on the development o f general learning and, more specifically, reading skills, will be summarized and discussed. History, Philosophy, and Methodology Precision Teaching finds its roots in the behaviourist theory o f the 1950s and 1960s. Teaching machines and programmed instruction as developed by Skinner (1968), the "father o f behaviorism", fulfilled the scientific, objective, and measurable orientation of this era. The economic values of productivity and efGciency influenced education in that learning to read was considered a scientific process in which each skill was broken down into smaller tasks which were taught specifically and in isolation with consistent practice. Programmed instruction meant that twice as much could be taught in the same amount of time because students were kept busy and active (Skinner). As well, with Skinner's 13 teaching machines came the change from response rate as the only source o f evaluation and measure o f behavior to percentage correct or accuracy measures (Binder, 1993). Lindsley, a student o f Skinner's during the 1950s, began to question the assessment measure o f percentage correct in his work with psychotic behavior in adults and children. He stated. Percentage ignores speed and fluency. Sole attention to percentage correct often produces highly accurate, painfully slow learners who have very low tolerance for error-filled, courageous learning. Students often t)ecome fearful o f making errors, which in t u r n can stifle creativity and exploration. (1990, p. 10) Lindsley's (1992) findings led him to the field o f education and teacher training and, in the 1960s, to the development o f Precision Teaching, which he defined specifically as "basing educational decisions on changes in continuous self-monitored performance frequencies displayed on standard celeration charts" (p. 51). Precision Teaching involves the breaking down o f composite tasks into isolated skills and regularly measuring and analyzing student performance o f these skills so that decisions can be made regarding the educational program's effectiveness (Beck & Clement, 1991; Snyder, 1992; West, Young, & Spooner, 1990). Student behaviors are charted and graphed so that regular and frequent monitoring is possible by both teachers and the students themselves. Changes in behavior are made apparent and predictions can be made concerning future behavior. As well, sound decisions can be made regarding the effectiveness o f instruction and curricular materials. It is believed that performance improves when appropriate and immediate feedback is available (Haughton, 1977) and that charting can both accelerate and help to predict academic performance (Lindsley, 1990). As well, Farr (1987) stated that any feedback which allows the student to 14 understand and then be able to correct errors is necessary for learning and retention. According to Mercer and Mercer (1993), factors which promote student learning include established goals and expectations, monitoring progress and providing feedback, and students' self-regulation o f their learning. Many teachers today base student progress on informal observations and on their own interpretations o f student performance. Although professional teacher judgment may be sound and valid, it could also prove to be inadequate and erroneous. When students record and chart their behavior, however, learning is observed at a glance, and performance can be compared to previous levels as well as to formerly established goals and aims. In addition, students take an active role by assuming more responsibility for their own learning. Precision Teaching and fluencybuilding, therefore, appear to fulfill at least one o f the three principles o f learning as stated by EC's Ministry o f Education (1994), that "Learning requires the active participation o f the student" (p. 1). With the implementation o f Precision Teaching came the notion o f fluency. In charting and analyzing student performance, it was discovered that to achieve success at any task or composite skill, both accuracy and speed in performing the prerequisites or component skills were necessary (Binder, 1993). Fluency, therefore, can be defined as this combination o f accuracy and speed. A more precise definition o f fluency is given by Binder (1988): Fluency is a combination o f accuracy plus speed which ensures that students will be able to perform easily in the presence o f distraction, will be able to retain newly-leamed skills and knowledge, and will be able to apply what they've learned to acquire new skills or to real-life situations. Fluency is "second nature" knowledge, near-automatic performance, the ability to perform without hesitation. In short, fluency is true mastery, (p. 12) 15 Other terms have been used which are related to the concept of fluency, such as mastery, overlearning, and automaticity. Bloom (1986) developed the theory o f automaticity in which he considered automatic behavior as being regular and efiBcient because it could be executed without conscious attention and also because one could be simultaneously thinking about something else. An automatic skill could be performed accurately and rapidly, with a minimum o f wasted effort. Bloom believed that practice and the "overlearning" o f skills resulted in automaticity. He used the training schedules o f athletes and musicians as examples to emphasize the importance o f practicing individual subskills daily in order to achieve automaticity and outstanding performance in the more complex skill. Practice, therefore, is the connection between the concepts o f Precision Teaching and fluency. Practice is necessary to ensure fluency, and Precision Teaching is a practical and efficient means of monitoring and assessing practice. Snyder (1992) stated, "Building fluency is the purpose o f the Precision Teaching method... making it much easier for students to quickly grasp more and more complex tasks, often without the need for instruction in those tasks" (p. 31). Teachers' observations o f their classrooms and individual students have generated various theories and opinions regarding Precision Teaching and fluency. For example. Binder (1988) claimed that increased speed o f response is related to improved retention, resistance to distraction, and the generalization o f learning to more complex tasks. In the same article, he stated that students' performance and rates o f learning can be improved by daily, timed practice sessions. Farr (1987) agreed that practice enhances fluency. He claimed that overlearning increases the degree o f learning which, in his opinion, is the 16 single most important determinant o f long-term retention, and that the greater the degree o f learning, the slower the rate o f memory loss. Farr stated, "Skills which have become more highly organized, cohesive, and automated through practice show negligible decay over long periods o f nonuse" (p. S-8). Haughton (1977) claimed that individuals who are fluent at a task can tolerate longer durations and more repetitions due to their increased comfort level. He stated, "Indications o f doing something better are high speed and accuracy o f the desired performance" (p. 33). Mercer and Mercer (1993) agreed that achieving automaticity with a skill not only improves retention but also increases students' ability to solve higher-level problems. More recent observations o f Johnson and Layng (1996) further support Binder's (1988) and Mercer and Mercer's (1993) claims that fluency is related to the generalization o f learning to more complex tasks. According to Johnson and Layng, not only are their students able to apply and extend fluent skills in new situations, but they are also able to independently combine these skills in creative and unique ways. For example, students fluent in component skills were observed writing advanced sentence constructions in English and constructing formulas and answers to story problems in Math without any teacher instruction. Johnson and Layng have concluded that creative thinking and problem-solving skills may be natural long-term benefits o f fluency-building. General Research According to Binder (1996), research regarding the use o f Precision Teaching and fluency-building methods is limited and support in the literature is scarce. He believes this is due to three major reasons; first, most teachers involved with Precision Teaching are practitioners whose main objective is to improve student learning and not to pursue 17 publication; second. Precision Teaching discoveries are occurring and progressing so rapidly that teachers are unwilling to report data which could become out-of-date by the time o f publication; and, third, those who had published findings experienced discouragement at the educational community's lack o f interest in their work. As Binder commented. Thus, Precision Teaching and its discoveries have remained more an oral than a written tradition in the field o f behavior analysis, based on the personal exchange o f charted data fi*om many thousands o f single-subject classroom interventions and on charts presented at professional conferences, (p. 4) The call for more research and data to prove Precision Teaching methods has also been made by Berquam (1985), who further claims that "if all the available charts could be compiled, the evidence that Precision Teaching works would be overwhelming" (p. 323). Much o f the available research on Precision Teaching and fluency is observational and non-empirical in nature, and most of the studies to be discussed in this section involve teachers' observations o f their classrooms or individual students after incorporating Precision Teaching and fluency-building strategies into their teaching repertoires. As a result, valid and reliable statistical data are scarce. As well, without employing a true experimental or quasi-experimental research design that incorporates control groups and pre- and post-test measures, one cannot determine the actual independent variable responsible for the change in students' behavior and achievement levels. Whether the cause is the Precision Teaching or fluency-building method, some other teaching strategy, or simply teachers' enthusiasm at using a new technique remains uncertain. As well, for some o f the studies mentioned, the number o f subjects is not available and some researchers do not indicate whether the students are average in ability or 18 whether they possess learning difficulties. This lack o f information causes the studies' validity and replicability in the classroom to be questionable. Other cautions regarding some o f the research discussed are that very small samples are used and, for some o f the studies, intervention periods are short-temL This leads one to speculate whether effects are long-term or due to the novelty o f the program. An additional consideration concerns the quality o f teaching prior to implementation o f the Precision Teaching methods, as their effectiveness may have more to do with teachers' previous ineffective strategies than the Precision Teaching itself. In spite o f these criticisms and cautions, it remains worthwhile and informative to discuss several o f the existing publications concerning Precision Teaching and fluency. Perhaps the best known research in Precision Teaching involves the Great Falls Precision Teaching Project conducted in Montana in the 1970s with approximately 450 elementary-aged students (Beck & Clement, 1991). Unfortunately, the statistical significance o f this study's results is unknown, as only nonstatistical comparisons o f national percentile ranks on the Iowa Test o f Basic Skills (ITBS) are provided. Teachers in several schools throughout the school district provided daily opportunities for students to practice basic skills (spelling, math facts, reading), chart their daily performance, and be a part o f instructional decision making. Teachers who implemented these methods in their classrooms found that "students showed marked improvements in classroom assignments, overall concentration and work habits, and displayed obvious enhancements in self­ esteem" (p. 8). After daily fluency practice over a 3 year period, students in Sacajawea School, the project's principle training site, demonstrated fi-om 20 to 40 percentile points improvement in their basic skills performance on the ITBS. In a second longitudinal study 19 included in the project, fourth grade students from Sacajawea School who had been taught since the first grade with Precision Teaching methods were compared with other fourth grade students in the school district who had not been taught with Precision Teaching methods. The Precision Teaching students out-performed the other students by over 20 percentile points in reading and 40 percentile points in math on the ITBS. A final study in the project compared two groups o f Grade 3 students from several schools within the district, one which was taught using Precision Teaching methods. Even though the control group's scores were higher prior to the intervention, the Precision Teaching group came out ahead by 24 percentile points in math and 32 percentile points in reading on the ITBS after the intervention. Research involving individual student case studies and the analyzing o f data from Precision Teaching classrooms has suggested that students who are fluent at a skill are able to work for longer durations and demonstrate increased attentions spans (Binder, Haughton, & Van Eyk, 1990). For example, individual students with severe mental retardation and behavior disorders who practiced fine motor skills such as grasping and releasing objects, putting puzzle pieces in place, and writing numbers to fluency demonstrated increased endurance and attention span not only with the skills practiced but also with subsequent, more advanced work. The authors claim that students diagnosed as ADD (Attention Deficit Disorder) are often just lacking fluency in prerequisite skills such as grasping and releasing objects and reading and writing numbers. Research by West et al. (1990) also indicates that developing fluency in various skills allows students to leam more quickly and results in increased academic gain. Fourth grade students at Valley Dale School in California performed daily practice to build 20 fluency in reading, math, and spelling skills. The average improvement in national percentile rank per student after testing with the California Test o f Basic Skills at the end o f the school year was more than 17 percentile points. Another study involving 34 Grade 1 children, 11 o f whom were special-education students, examined the building o f math facts fluency with daily timed trials. After 10 weeks, students assigned to the experimental condition not only demonstrated an increase in the percentage o f math problems answered correctly, but also increased in their levels o f on-task behavior (e.g. remaining focused during seatwork; being attentive during lessons) as compared to students not involved in the fluency-building practice (Miller, Hall, & Reward, 1995). The previously discussed research seems to indicate that rather than teach to the point o f accuracy alone, one must advance to the next step, fluency-building, in order to ensure not only accuracy o f performance, but speed o f performance as well. At Momingside Academy, for example, the focus is on "retention, endurance, and application, in addition to the accuracy o f performance" (Snyder, 1992, p. 31). This is achieved through "classroom instruction followed by timed practice, charting o f progress, feedback, positive reinforcement, and deciding what to leam and practice next" (Snyder, p. 29). Once fluency is reached, teachers should provide opportunities for students to apply these skills in various contexts and in new situations. For example, a study by Mercer and Mercer (1993) indicated a positive relationship between fluency of handwriting (the number of words written) and other measures o f writing skills, including generating story ideas. 21 It would also seem necessary for teachers to ensure that students receive sufScient practice in the prerequisite skills before attempting more complex tasks. Many instructional materials today, however, do not provide an adequate number o f examples or practice items to ensure that students reach the stage o f fluency. Moving too quickly from component skill to composite skill without achieving fluency could lead to frustration and failure. Skinner (1982; 1984) believed that the solution was to teach component skills first, allow for sufficient practice in performing these skills, and to provide immediate and frequent feedback and reinforcement. By doing this, he felt, students would leam "twice as much in the same amount o f time and with the same amount o f efifort" (1984, p. 947). Reading Research Various theories and studies specific to the teaching o f reading and relating to Precision Teaching and fluency-building are also available for discussion. Stahl, DufifyHester, and Stahl (1998) refer to a body of research dating back to the 1920s which claims that certain principles o f good phonics instruction are necessary when teaching reading. These principles include an early and explicit emphasis on letters and their sounds, the decoding of words, practice in reading words, and automatic word recognition. The recognition o f letters and words should be to the point o f fluency, so that recognition is possible without conscious efifort. The authors conclude that the ideal reading program combines direct and systematic teacher instruction with the use o f children's story-books and novels. Carver (1990) claimed that the most important outcome o f Bloom's (1986) automaticity theory is that it inspired additional research into rate and practice in the area o f reading. Repeatedly practicing the recognition o f words is believed to be the best way 22 to achieve automaticity for those words, and practice in decoding words results in increased reading rate and, therefore, less attention is required to decode and more attention is available to understand the sentence meaning (Carver; Dickinson, Wolf, & Stotsky, 1993). Samuels, Schermer, and Reinking (1992) agreed that through practice, automaticity is achieved and tasks become easier and require less attention. They extended this theory to reading and claimed that the decoding o f words must become automatic and fluent before attention can be given to the comprehension o f the text. Students, therefore, cannot focus on reading for meaning (what most would consider to be the real goal o f reading) until they are able to read and decode fluently. Howell and Lorson-Howell (1990), as well, claimed that fluent, automatic readers utilize various phonic, semantic, and syntax skills without being aware o f them, which therefore firees the mind to attend to text meaning. According to Bloom (1986), "While we are in the process o f reading connected discourse, we may also be making judgments about the ideas, enjoying the story, getting new ideas and insights, or being involved in other conscious processes" (p. 73). If decoding is automatic, the focus of attention is at the semantic level, and the reader is able to incorporate their personal experiences and associations into the reading situation (LaBerge & Samuels, 1985). Slocum, Street, and Gilberts (1995) believed that the pleasure gained from reading about topics o f interest and comprehending what is read is a naturally-occurring reinforcement. The ability to read fluently, therefore, allows the reader to cover more text and to experience additional reinforcement. Thus begins the logical cycle o f choosing to read more frequently for pleasure and, as a result, becoming yet a better and more fluent 23 reader due to the increased practice. Six studies specifically relating to Precision Teaching and its efifects on reading skills will be discussed in this literature review. O f the six studies, three involve very small samples, one does not state the number o f subjects involved, and four o f the studies have short intervention periods (between 5 and 8 weeks). As well, o f the six studies, only two consist o f an actual experiment using control groups, although one o f these does not employ any statistical measures to back its claims. These are limitations which make a cause-and-effect relationship between the Precision Teaching methods used and increased reading achievement impossible to infer. Perhaps the students in these studies would have made similar gains with alternative teaching strategies or even with no change fi'om the previous methods used. A conservative interpretation o f the following studies' results is therefore necessary. On the other hand, their limitations do not appear sufiBcient to totally negate their findings, and to rule out Precision Teaching and fluency-building as possible influential factors could also be viewed as erroneous. In a study by Kinder (1994), four Kindergarten students with learning difficulties underwent a 6-week program which involved daily timings and charting o f emergent literacy skills (identifying initial consonant sounds, sight vocabulary, and printing the alphabet). N ot only did all students demonstrate marked improvement in each o f the skill areas targeted, but also they enjoyed being timed and charting their progress. A similar thesis conducted by Nitti (1990) involving five low-achieving students in an 8-week Precision Teaching program concluded that the continuous monitoring and fluencybuilding components o f Precision Teaching greatly facilitated student learning by resulting in improved reading comprehension. As well. Idol and Rutledge (1993) concluded that 24 students with reading difSculties averaged 2 months curricular progress per month o f Precision Teaching instruction and fluency practice with letter and word sounds, although the measures used to determine curricular progress were not mentioned in their article. The authors concluded that being fluent in reading sounds enabled students to decode unfamiliar words more effectively. Another study conducted by Kessissoglou and Farrell (1995) showed that students who were taught to read words using Precision Teaching techniques demonstrated substantially more progress in reading comprehension than two other groups o f students taught with other teaching methods. The authors admitted, however, that the sample in this study was small (12 students). As well, although gains were measured by the Neale Analysis o f Reading Ability (NARA), no statistics were mentioned to determine whether the difference was significant. A follow-up study conducted 5 weeks after the original experiment indicated that students in the Precision Teaching group had maintained their previous levels o f performance. As well, the students demonstrated positive attitudes regarding the program. A larger study involving 10 classrooms over a 2 year period determined that after implementing Precision Teaching and fluency-building strategies, students made 40% greater gains in math and 25% greater gains in reading fi-om previous years' achievement test scores (Mordecai, 1977). Teachers also reported that students were excited and highly motivated as a result o f being involved in their own learning. Lastly, Tan and Nicholson (1997) conducted an experiment which assigned 42 below-average readers to three reading groups, two o f which employed the use o f flashcards to the point o f overlearning and fluency to teach word recognition. The other 25 reading group involved no word recognition training. Although the study consisted o f only five training sessions, the results indicated that students who received fluencybuilding practice demonstrated significantly greater performance on measures o f comprehension and passage recall than did students in the control condition. Allington and Cunningham (1996) stated that fluency with reading and spelling words is essential to the teaching o f reading and writing. Accuracy in recognizing words, therefore, is not sufficient in order to read successfiilly. Rather, teachers must ensure that students move beyond accurate reading and towards automatic reading. Because reading is a complex task consisting o f many smaller skills, it follows that these single skills should be taught prior to the more difficult and composite task o f reading. Two o f the key components o f both SRA's Reading Mastery program and the Momingside Model o f Generative Instruction are the teaching o f each skill to automaticity and providing opportunities to apply the skills in related and more complex situations. Direct Instruction In this section, a comprehensive description o f the history, philosophy, and methodology of Direct Instruction will be provided. The Project Follow Through research study, which compared 13 instructional approaches including Direct Instruction, will be discussed, as well as several o f its follow-up studies. As well, research regarding the effects o f Direct Instruction on the learning o f special-education students will be summarized and analyzed. Lastly, two meta-analyses of Direct Instruction research as well as findings o f the Momingside Academy will be discussed. 26 History, Philosophy, and Methodology In 1976, after «(tensive research conducted with teachers and in classrooms, Rosenshine created a definition o f effective teaching as being characterized by certain essential components, which together he labelled, "direct instruction". According to Rosenshine (1979), direct instruction involved clear teacher direction using academically focused activities and sequenced, structured materials. Specific components included clear goals, monitored student performance, questions presented at a low cognitive level so that correct answers were abundant, immediate feedback, sufficient instructional time, and extensive coverage o f content. Rosenshine stated, "In direct instruction the teacher controls instructional goals, chooses materials appropriate for the students' ability, and paces the instructional episode. Interaction is characterized as structured, but not authoritarian. Learning takes place in a convivial academic atmosphere" (p. 38). From his research, Rosenshine concluded that these instructional criteria were correlated with students' increased academic achievement. Direct Instruction (with upper-case initial letters), as distinguished from Rosenshine's direct instruction (with lower-case initial letters), refers to the specific instructional approach developed by Engelmann and his colleagues. In addition to incorporating all o f Rosenshine's essential instructional components, Engelmann's Direct Instruction model constitutes a specifically-developed curriculum for teaching a variety o f skills and subject areas (Engelmann & Gamine, 1982). This method, like Rosenshine's, emphasizes a stmctured, highly-academic learning environment where students are actively involved and are experiencing a high rate o f success. 27 Both thé Reading Mastery program and the Momingside Model o f Generative Instruction employ Direct Instmction. Direct Instruction involves teaching essential skills as effectively and efiBciently as possible with the proactive goal o f preventing academic failure (Gamine, Silbert, & Kameenui, 1997). The Direct Instmction method assumes that all students can leam, that academically disadvantaged students must be taught at a faster rate in order to catch up with their more advanced peers, and that the learning o f basic skills and their subsequent application in more complex and higher-developmental tasks must be emphasized. Stemming from these assumptions are the two main principles o f the model; "Teach more in less time, and control the details o f what happens" (Engelmann, Becker, Gamine, & Gersten, 1988, p. 303). Direct Instmction emphasizes both a mastery approach, in which skills are leamed to a level o f fluency, as well as cumulative review o f those skills leamed (Gamine et al., 1997). It is "a system o f teaching that attempts to control all the variables that make a difference in the performance o f children" (Adams & Engelmann, 1996, p. ix). The Direct Instmction method consists o f five components: a physical layout where students are positioned for close monitoring and correcting o f mistakes by the teacher; a fast and lively pacing of instmction wherein students are highly involved so that they remain motivated, fewer management problems occur, and more material is covered; the use o f teacher signals for students to answer aloud and in unison so that immediate feedback is possible and so that all students, not only the higher performers, are frequently and actively engaged; the consistent use o f simple praise which results from specific student action; and the use o f specific and immediate correcting procedures. The Direct Instmction method involves the teaching o f detailed tasks and activities using a scripted 28 presentation. In this way, the use o f pretested examples and sequences allows the instructional environment to be controlled and eliminates any ambiguity o f teacher communication. Skills are broken down into small tasks which are taught specifically and in isolation with consistent practice throughout lessons. Review is constant and errors are immediately corrected. Continuous monitoring and assessment o f student progress enables teachers to determine whether mastery has been reached or whether remediation is necessary. Once the skill has become automatic and mastery has been achieved, students are provided with opportunities to apply the skill in related and more complex situations (Gamine et al., 1997). Project Follow Through O f all the research conducted on Direct Instruction, the largest and perhaps bestknown study involves Project Follow Through. In 1967, as a response to U.S. President Lyndon Johnson's "War Against Poverty" initiative, a national educational experiment entitled "Project Follow Through" was launched to determine the most effective educational model to teach reading to poor and disadvantaged students. This project became the most expensive and extensive research study in the history o f education (Grossen, 1996). At a cost o f over $1 billion (U.S.), 13 instructional approaches were implemented by 4000 teachers to teach reading to 84,000 children in 170 different communities across the United States (Weikart, 1985). The experimental models were based on educational philosophies which ranged fi'om Skinner’s behaviourism to those promoting child development, language experience, cognitively-oriented curriculum, and discovery learning as developed by such theorists as Dewey, Freud, and Piaget (Becker, 1978). One o f the approaches chosen for implementation in Project Follow Through was 29 the Direct Instruction model developed by Engelmann and which had been used successfully in the early 1960s with disadvantaged preschool children. His theory maintained that disadvant%ed children needed to be taught more in less time in order to catch up to their more advantaged peers (Engelmann, 1992). Each o f the etperimental teaching models was implemented in various locations throughout the United States. Students' performance levels were determined when they first began the program and at the end o f each school year until they completed Grade 3. These data were collected by the Stanford Research Institute and analyzed by Abt Associates, an independent agency awarded the contract for the study's data analysis (Watkins, 1996). Eleven outcome subtests were chosen for the evaluation in order to measure student performance across three broad areas: basic academic skills, problem­ solving and higher-order thinking skills, and student self-concept. For each outcome subtest and at each Project Follow Through site, a nonparticipating control group was matched to the experimental group and evaluated to compare results. The study reached its conclusion in 1973 with controversial and, still today, disputed results. According to the initial findings o f Abt Associates, Engelmann's Direct Instruction model came out ahead o f all the other instructional approaches in academic, cognitive, and affective measures (Stebbins, 1977). Specifically, students taught with the Direct Instruction model progressed fi'om well below the 25th national percentile in reading, math, and spelling on the Metropolitan Achievement Test (MAT) to the 50th national percentile or above, which was considered the national norm OBecker, 1978; Becker & Gamine, 1980). 30 An interesting finding concerned Direct Instruction students' superior performance in self-esteem and affective measures. According to Becker (1978), although the Direct Instruction model did not intentionally set out to improve students' self-concepts, this outcome became a natural by-product o f Direct Instruction as an effective teaching method. In fact, several o f the study's other models which were described as humanistic and whose goals included the positive development o f student self-esteem resulted in lower affective scores than their comparison groups and compared to their pretest scores at the study's inception. Other ironies included the findings that students taught with cognitively-oriented methods were relatively weak in higher-order thinking skills, and that those models emphasizing discovery and self-directed learning scored the lowest in academic and affective measures. As well, data fi'om Project Follow Through supported the findings that IQ gains among the Direct Instruction groups were maintained through the third grade and that it was more beneficial to begin Direct Instruction at the Kindergarten level than at later grades (Becker). Due to the controversial nature o f the study, the data were reanalyzed at least three more times, each o f which came to the similar conclusion that the Direct Instruction model resulted in the highest achievement scores across all measures examined (Watkins, 1996). Surprisingly, however, at the study's conclusion, the American Department o f Education seemingly ignored the results, made no recognition o f Direct Instruction's success, and even went on to support and recommend some o f the less successful educational models (Engelmann & Gamine, 1982). Much o f the controversy, some believe, stemmed fi’om negative attitudes regarding the behavioural philosophy underlying the Direct Instruction approach. According to Pennypacker (1994), "The results were not 31 only unexpected, they were contrary to some cherished tenets o f invented wisdom embraced by the academic educational establishment” (p. 16). Engelmann (1992) himself stated a similar opinion: After all these years, Fm still not sure I understand why it was so important for the establishment to discredit Direct Instruction. It's true that we do not do things the way t h ^ do it in traditional classrooms. B ut what we do works and what they do doesn't. If society is concerned with kids, it would seem reasonable to find what works and to use it, regardless o f what our prejudices might be. I f we don't rely on hard data, our prejudices don't have much to support them. Apparently, the key decision-makers had a greater investment in romantic notions about children than in the gritty detail o f actual practice o r the fact that some things work well. (p. 6) Regardless o f Project Follow Through's outcome, it did become the precursor to additional research on Direct Instruction. Unlike the research on Precision Teaching and fluency-building previously discussed, there have been a large number o f studies conducted on the effectiveness o f Direct Instruction with a variety o f student types and in a variety o f locations. According to Grossen, an American education researcher and editor o f the Association for Direct Instruction Journal, "Every educator in the country should know that in the history o f education, no educational model has ever been documented to achieve such positive results with such consistency across so many variable sites as Direct Instruction " (1997, pp. 6-7). Research Since Project Follow Through Most o f the research on Direct Instruction was conducted in the 1970s and 1980s, and many studies are longitudinal, follow-up evaluations o f Project Follow Through. Of the 10 studies and two meta-analyses which will be discussed in this section, most occurred in the 1980s and 1990s. The intention was to locate the most recent and relevant research available. The studies discussed here, therefore, only represent a small fi*action of 32 all the Direct Instruction research conducted since Project Follow Through was initiated in the late 1960s. Almost all o f the 10 studies to be discussed compare groups or classrooms o f students without random assignment o f subjects to treatment or control conditions. There is almost equal representation o f urban and rural locations, average and learning-disabled students, and primary and intermediate aged students. Almost one half o f the studies evaluate reading ability only, while the rest look at a combination o f skill areas such as reading, language, spelling, and math. Unfortunately, not all o f the studies include statistical measures or results to back their claims. It is necessary to keep this limitation in mind when considering their findings. Another issue o f caution worth mentioning involves the control groups used in the Direct Instruction research to be discussed. Although most o f the studies do employ equivalent comparison groups which are typical o f regular classroom practice, others involve the comparison o f strategies and approaches which appear uncommon or not entirely conducive to teaching the skills being measured. The internal validity o f such studies, therefore, may be somewhat questionable. Ideal research conditions, o f course, would employ control groups most similar to conditions naturally existing in typical classrooms. Project Follow Through Follow-up Studies. The following four studies are follow-up evaluations o f Project Follow Through. In one study, Gersten, Becker, Heiry, and White (1984) conducted a secondary analysis of Project Follow Through data from 1969 to 1977 to compare the yearly academic gains of 1500 low-IQ and normal/high-IQ students taught with Direct Instruction. The results revealed that students with low- and 33 normal- to high-IQ demonstrated similar and substantial gains each year in math and reading on the Wide Range Achievement Test (WRAT) and the Metropolitan Achievement Test (MAT). They concluded that if taught with Direct Instruction, students with low IQs are capable o f achieving the same growth as their normal-IQ peers and can therefore be mainstreamed in regular classrooms as opposed to being assigned to specialeducation conditions. This study also indicates that Direct Instruction intervention at the primary level can have long-term benefits for below-average students, preventing them fi'om falling farther behind their average-ability peers throughout their subsequent school years. Two similar longitudinal follow-up studies looked at the achievement levels o f Grades 5 and 6 students on the Wide Range Achievement Test (WRAT) and the Metropolitan Achievement Test ^ iA T ) who had been taught using the Direct Instruction model during Project Follow Through (Becker & Gersten, 1982; Meyer, Gersten, & Gutkin, 1983). Both used Cohen's d index to calculate efifect sizes, which involves dividing the difference between the experimental group's and comparison group's mean scores by the groups' pooled standard deviation (Cohen, 1992). According to Cohen, when testing the difference between independent means, an effect size o f 0.20 is defined as small, an effect size o f 0.50 is defined as medium, and an effect size o f 0.80 is defined as large. The study by Becker and Gersten (1982) involved over 1000 Grades 5 and 6 students who were tested on the reading subtest o f the WRAT. Effect sizes between Project Follow Through and non-Project Follow Through students ranged fi'om 0.38 to 0.56. Chi-square values also indicated that Project Follow Through students achieved significantly higher scores in math problem-solving on the MAT. Meyer et al. (1983) used 34 scores from the MAT to determine that 91 Follow Through students maintained scores at or above grade level in Grades 5 and 6, and that these students also scored higher than the remainder o f students in the district. Meyer et al.'s analysis resulted in efifect sizes o f 0.58 in Math and 0.38 in Reading of Follow Through students at the end of third grade. The results o f these two studies indicate that the low-income Direct Instruction students maintained their advantage over the middle-class comparison groups in reading and math problem-solving throughout their intermediate years. A final longitudinal Project Follow Through study looked at the achievement levels of low-income, disadvantaged students once they reached high-school and beyond (Gersten & Keating, 1987). Files were retrieved fi'om the high-schools to which the Follow Through students and comparison group students had dispersed, and comparisons between the two groups were based on math and reading achievement tests, graduation rates, and college applications and acceptance. The study indicated that, at the Grade 9 level, 172 Project Follow Through students demonstrated higher achievement levels in math and reading than 279 comparison group students. As well, results showed that the Project Follow Through students experienced fewer attendance problems, repeated fewer grades, dropped out less often, and had higher high-school graduation and college acceptance rates. For example, 60% o f the Project Follow Through students graduated fi'om high-school, as compared to 38% o f non-Project Follow Through students. As well, 34% o f Project Follow Through students were accepted to college, whereas only 17% o f non-Project Follow Through students were accepted. Direct Instruction, therefore, appears to result in long-term positive efifects that extend to the high-school level and beyond. It is also important to note that despite coming fi'om disadvantaged, low socio- 35 economic areas, these students were able to achieve substantial, long-term progress as compared to their middle-class peers. Special-education students. Direct Instruction and the Reading Mastery program are often described as being best-suited for special education students. Six studies will be discussed which involve non-average learners. The authors o f these studies refer to their subjects as "learning-disabled" (Stephens, 1993), "low-achievers" ^ re n t, DiObilda, & Gavin, 1986), "disadvantaged" and "lacking language skills" (Dowdell, 1996), or as possessing "mild disabilities" (Marston, Deno, Kim, Diment, & Rogers, 1995) or "reading difficulties" (Francis, 1991; Somerville & Leach, 1988). The studies' sample sizes range from 40 to 176 elementary-aged students. Four o f the studies involve 1 year intervention periods, while the remaining two studies' interventions are 10 weeks and 12 weeks in duration. The results o f Somerville and Leach's (1988) study demonstrated significantly greater reading achievement gains for the experimental Direct Instruction group as opposed to a comparison group taught with psychomotor and self-esteem enhancement approaches. These alternative approaches are considered to be beneficial by educators who support the theory that psychomotor delay and/or low self-esteem may be the underlying causes o f difficulty in learning basic skills. The results o f Brent et al.'s (1986) study demonstrated significantly greater reading achievement gains for the experimental Direct Instruction group as opposed to a comparison group taught with a regular classroom reading program. Brent et al.'s study also examined the effects o f experienced and non-experienced Direct Instruction teachers on students' reading ability. They determined that students o f teachers with two years o f 36 Direct Instruction experience demonstrated significantly greater reading performance than those taught by teachers possessing only one year's Direct Instruction experience. Perhaps this finding relates to other studies' findings that teachers trained in Direct Instruction ofien begin with skeptical and even negative attitudes toward the program. After they have taught the program for an extended period o f time and after having experienced its effects, teachers' negative attitudes generally change considerably for the better. Two such studies are discussed in greater detail later in ^ p e n d ix A. In consideration o f Brent et al.'s study, however, it does seem reasonable to suggest that teachers' attitudes toward an instructional program have the potential to influence students' achievement in that program. In the study by Marston et al. (1995), six teaching strategies were compared. Results indicated that two o f these teaching strategies. Direct Instruction being one, produced significantly greater achievement gains in students' reading ability. The alternative reading approaches employed in this study were o f a wholistic, cognitivelyoriented nature, such as peer-tutoring and reciprocal teaching strategies. The experimental group receiving Direct Instruction and the control group receiving a regular classroom reading program in Dowdell's (1996) study received very similar post-test scores on the Iowa Tests o f Basic Skills (ITBS). However, the subjects were not randomly assigned, and the pre-test determined that differences between the two groups were apparent fi'om the start. The Direct Instruction group began instruction with lower scores than the control group and after one year, experienced a statistically significant gain o f 1.06 years, as compared to the control group's 0.45 year's gain. Although a gain o f 1.06 years is the average expected growth for students in a school 37 year, it can probably be assumed that students would have experienced less progress, as was the case with the control group, if they had not received Direct Instruction. This study indicates that students with low abilities are capable o f demonstrating average achievement and, therefore, defeating the cycle o f falling farther behind their peers with each school year. The study by Francis (1991), which compared 261 students in Grades 3 to 6 taught with either Reading Mastery and Direct Instruction or with a regular classroom reading program, indicated that the Direct Instruction students achieved higher Vocabulary and Reading Comprehension scores on the Gates-MacGinitie Reading Tests. However, statistically significant gains were only found with Grade 6 students in Reading Comprehension and Grade 3 students in Vocabulary. Francis also notes that teachers expressed extremely positive attitudes regarding the teaching o f Reading Mastery and Direct Instruction and the observed progress o f their students. The final study by Stephens (1993) employs a one-group pre-test-post-test design to determine the efifects o f implementing a Direct Instruction reading program on students' reading achievement. The results determined that 73% o f the students involved in the Direct Instruction implementation gained 1 or more grade levels in reading as measured by the Kaufinan Test o f Educational Achievement (K-TEA). Pre-tests administered prior to implementation o f the program indicated that these students were 2 to 6 grade levels behind in reading achievement. Due to these positive results, Stephens recommended that schools incorporate a Direct Instruction reading program in the primary grades which employs a structured format, increased time on task, and which teaches to mastery. This, she believed, would prevent increasing failures by students in the higher grades and, as a 38 result, may also prevent these students from being labelled as handicapped or learningdisabled. Additional positive side-efifects that Stephens mentioned as a result o f her study, and which have also been experienced in the treatment school involved in the present study, include increased collaboration and improved morale among teachers. As these six studies indicate. Direct Instruction appears to be an effective teaching strategy to assist students who possess some degree o f learning disability or reading difficulty. It is also interesting to note that although two o f the studies discussed consist o f short intervention durations o f 10 weeks ^ ia rsto n et al., 1995) and 12 weeks (Somerville & Leach, 1988), both significantly favoured Direct Instruction as the more effective teaching strategy. Their findings, therefore, demonstrate that even short-term use o f Direct Instruction can produce substantial gains in special-education students' reading ability. Meta-Analyses. In addition to the studies mentioned above, two relevant meta­ analyses o f Direct Instruction research have been conducted in the last decade which are worthy o f discussion in this literature review. White's (1988) meta-analysis involves 25 studies of special-education students, whereas Adams and Engelmann's (1996) research examines 34 studies involving both regular and special-education students. Both meta­ analyses involve studies' results gathered through searches o f databases and recent publications. Both used Cohen's d index to calculate the effect sizes for each study, which involved dividing the difference between the experimental group's and comparison groups' mean scores by the groups' pooled standard deviation (Cohen, 1992). The overall effect size was then determined by averaging all o f the comparisons' effect sizes. According to the authors o f both meta-analyses, an effect size o f 0.25 is considered to be educationally 39 significant. This refers to interventions which produce a change in students' performance by 0.25 o f a standard deviation. Adams and Engelmann claim that effect sizes o f 0.50, whereby students' performance has changed 0.50 o f a standard deviation, are uncommon in educational research. White's (1988) meta-analysis includes only those studies in which subjects were assigned to experimental and comparison groups before an intervention occurred. The results o f his research indicated that the Direct Instruction groups demonstrated an overall average effect size o f 0.84. As well. White's meta-analysis indicated equally high effect sizes for both lower-level thinking skills such as word-attack and higher-level thinking skills such as reading comprehension. Adams and Engelmann's (1996) meta-analysis includes only those studies which used comparison groups, had similar pre-test scores between groups, included necessary statistical information, and incorporated Direct Instruction curriculum developed by Engelmaim and his associates. From the 34 studies which met these criteria, 173 individual comparisons were analyzed As a result, an overall average effect size o f 0.87 was calculated for these comparisons. In addition, Adams and Engelmaim conducted a simple polling o f means to determine the percentage of studies favouring Direct Instruction. Their results indicated that 87% o f the studies showed Direct Instruction to be more effective than alternative teaching strategies. Due to the simplicity o f this polling procedure, Adams and Engelmann also polled the statistically significant differences, using an alpha level o f .001, which again resulted in the majority (64%) o f studies favouring Direct Instruction. 40 Adams and Engelmann (1996) also examined several variables as part o f their meta-analysis. This examination showed similar, educationally significant effect sizes for regular and special-education students, elementary and secondary school-aged students, and interventions lasting up to a year and those lasting more than a year. As well, effect sizes for nine different academic subject areas were all 0.3 S or greater, with 0.69 being the effect size for reading. For the sake o f comparison, Adams and Engelmann cited a metaanalysis on the Whole Language reading approach conducted by Stahl and Miller (1989) which yielded an overall effect size o f 0.09. Overall, Adams and Engelmaim's (1996) meta-analysis demonstrates extremely positive results in favour o f Direct Instruction. As well, their findings appear valid considering the statistical measures employed and considering the method o f determining studies worthy o f inclusion. O f particular relevance to the present study are the large effect sizes for Direct Instruction in teaching reading and with average students. Some educators consider Direct Instruction and Reading Mastery as best suited for specialeducation students. This research appears to indicate otherwise. In summary, a large body o f research on Direct Instruction in teaching various skills and with learners o f various ages, abilities, and socio-economic statuses exists. Although not all o f the studies discussed meet ideal research conditions, this does not appear sufGcient to negate the evidence which favours Direct Instruction as an effective teaching strategy. Momingside Academy Limited research has been conducted which examines the effect o f both Precision Teaching and Direct Instruction combined in an instructional program. Perhaps the 41 longest-running study which does incorporate both strategies involves Momingside Academy, the private learning center in Seattle, Washington, whose teaching methods have been adopted by the present study's treatment school. Momingside Academy offers a money-back guarantee that their students will gain, academically, a minimum o f two years for every one year in their school. In the 11 years &om 1981 to 1992, students at Momingside Academy experienced a gain average o f 2.5 grade levels per year in Reading Language Arts, and Math as measured by the California Achievement Test and the Metropolitan Achievement Test (Johnson & Layng, 1992). When this school's curricular program was implemented with adults at Malcolm X College in Chicago, similar and even higher gains in student test scores were the result (Binder, 1993; Johnson, 1997). Johnson, Momingside Academy's founder and director, attributes the success o f these schools, for the most part, to their emphasis on fluency-based instruction (Johnson & Layng 1994). Momingside Academy has also analyzed data from the standardized Gates MacGinitie Reading Tests to determine the reading program's effectiveness in the two schools involved as pilot sites in School District No. 56. The results indicate that in the 2nd year o f implementation, the two schools experienced gains o f 1.5 to 3.5 years in reading and writing (Johnson, 1997). Although the present study examines the same general research topic and involves one o f the schools, its analysis differs from Momingside Academy's in that it utilizes data from the Canadian Tests o f Basic Skills over several years and involves control schools for comparison. 42 Criticisms o f the Reading Program There are, as is to be expected and as is the case with any teaching method, several criticisms o f the philosophy underlying Precision Teaching, fluency-building, and Direct Instruction. These instructional components are all based on the behavioural theory o f education. The major principles associated with the behavioural approach are that behaviour is learned, is influenced by the context in which it occurs, and must be observable. Teaching and learning, therefore, involve changing student behaviour (Blackman, 1984). In order to determine the extent to which student behaviour has changed, data-based evaluation and the gathering o f empirical evidence is necessary. According to Binder (1995), "Without direct, standard measurement o f outcomes, it is not possible to objectively evaluate or compare interventions" (p. 106). Those who take exception to Skinner's behaviourist theory because o f its technical and seemingly nonhumanistic orientation will most certainly find fault with data-based instruction, fluencybuilding exercises, and teacher-directed instruction. They might say that a scientific, analytic, and empirical approach to education neglects the emotional and social aspects o f teaching and learning, and that students cannot be considered "robots" or "guinea pigs" whose every behaviour teachers set out to control and manipulate. There seems to exist some animosity and rejection regarding rote learning o f basic skills as well. Some believe that it stifles creativity, initiative, and imagination in students and that drill and practice strategies are boring and seemingly without purpose from the students' perspectives. The trend in education today appears to have a combined humanistic and cognitive-developmental emphasis wherein students' self-image and social development are accentuated in addition to their problem-solving and critical thinking 43 skills. As Skinner has commented: ...the solution conflicts with deeply entrenched views o f human behavior. [Some] tend to feel threatened by any kind o f scientific analysis o f human behavior, particularly if it leads to a "technology " that can be used to intervene in people's lives. A technology o f teaching is especially threatening. (1984, p. 948) Binder and Watkins (1990) claim that behavioural methods in education are rejected because they are out o f fashion. They boldly state that educators "don't want systems that disturb the normal distribution o f grades in the student population" (p. 74). Solity"s (1991) opinion regarding behavioural methods' lack o f representation in the school system concerns accountability: Becoming aware o f a child's failure to progress rarely becomes an opportunity for critical self-reflection on the part o f the teaching profession. An approach to teaching which adopts this as its starting point is, therefore, less likely to find favour than one which rests more readily with the alternative construction, namely that a failure to learn can be attributed to specific learning personal characteristics o f the child, or the child's home environment, (p. 163) Jones and Slate (1996) conducted a study involving 504 teachers who were asked to agree or disagree with statements relating to behavioural methods in education. Results showed that o f the many behavioural issues represented by these statements, teachers were generally supportive o f all except for two. Statements relating to the use o f objective and scientific measures to evaluate student learning were commonly rejected, as were statements concerning teacher accountability and responsibility for student learning. Instead of holding teacher instruction and instructional materials responsible, teachers tended to blame other factors for students' learning difficulties, such as society, parents, and television. The study's authors concluded that "Rejection o f scientific standards is likely to result in decisions being based on faulty criteria such as opinion or emotion" (p. 44 38). The behavioural perspective, on the contrary, places hill responsibility for student learning on teachers and instructional materials. The underlying premise is that every child can achieve if given adequate instruction (Skinner, 1984). Engelmarm (1992) claims that "The system is sick because the vast majority o f people in it—from educational researchers to teachers—lack technical understanding o f the single aspect o f the school that justifies their existence—instruction" (p. 13). Mothus (1997), fi’om research which will be discussed later in this paper, suggests that teaching practices, and not the student, are responsible for student learning, and that educational practice needs to be driven by proven classroom research as opposed to current trends or fads. Stahl et al. (1998) state their concern that current classroom reading programs are based ".. .on false allegations popularized by the media and accepted by some legislators and administrators describing the limited success o f past reading programs" (p. 351). Gamine (1992) states that education needs to be more like the medical profession, which is not characterized by fads but which has a scientific perspective and access to carefully-evaluated tools. Another opinion is that o f Axelrod (1992) who believes that effective teaching strategies and materials have been developed and are available, and it is now the responsibility o f behavioural educators to make these practices known. He places partial blame for the general rejection o f these practices on many universities' lack o f behavioural courses and faculty trained in these approaches. Another criticism concerns the emphasis on teaching subskills in isolation and out of context as opposed to a more "top-down", meaning-oriented, wholistic approach. This view is held by AUington and Walmsley who, in 1995, stated that "skills hierarchies 45 dissected literacy learning into so many parts that in both testing and teaching we lost sight o f the forest by focusing on individual trees" (p. 5). Goodman (1986) also claimed that "When schools break language into bits and pieces, sense becomes nonsense, and it's always hard for Idds to make sense out o f nonsense" (p. 8). Despite this controversy, it seems as though the goals o f both a wholistic approach and a "bottom-up" approach, at least in the case o f reading, are similar. The ultimate goal o f a subskills approach to reading is that students become fluent readers, able to comprehend story meaning and able to bring their own experiences and associations to the reading process. Bloom (1986) stated, "Reading to automaticity cannot be developed if children read only for classroom purposes. The habit o f reading (ital. orig.) for several years is necessary for automaticity to develop" (p. 76). Demonstration o f a love o f literature and language, reading for purpose and understanding, and the ability to apply thinking skills are the long-term goals o f the Reading Mastery program (Engelmann, Osborn, Osborn, & Zore^ 1988), goals which appear not unlike those o f any wholistic approach to reading instruction. Gamine et al. (1997) attribute failure in today's schools to a number o f factors, including insufficient teaching o f preskills, the teaching o f too many skills at a superficial level and without adequate review and practice, not enough time spent in actual reading, and the lack o f reading instruction for unmotivated students due to the belief that learning to read is a natural and intrinsically-motivated process. The answer, according to these authors, lies in improving teaching practices and methodology, which they believe can be achieved by implementing effective, efficient, research-proven methods such as Precision Teaching and Direct Instruction. 46 Summary The debate over which educational theory or reading approach is superior will never be solved. Many teachers would agree that an open and flexible view towards combining the best in all approaches while maintaining the overriding goal o f attempting to meet students' needs is where the answer lies. The purpose o f the present study^ therefore, is not to claim one right teaching style, but rather to provide a small though necessary piece o f evidence in an area where such research is limited. Not only will the study hopefliUy be able to provide some insight into the benefits o f Precision Teaching, fluency-building, and Direct Instruction in the classroom, but it will also be able to determine, at least to some extent, the effectiveness o f combining these components into an instructional reading program, where Direct histruction is used to teach foundational skills and concepts, and Precision Teaching is used to assist students in achieving fluency and as an assessment and decision-making tool. 47 METHOD The present study involves an evaluation o f the reading program in the treatment school. This is achieved through a comparison o f the treatment school's and four control schools' Canadian Tests o f Basic Skills Reading and Vocabulary subtest scores in the pre­ treatment and post-treatment years. Because different groups o f students were in attendance and tested in the pre-treatment and the post-treatment years, the study's analysis involves a between-groups comparison. The present study is described as quasi-experimental. Random assignment o f subjects to treatment or control conditions was not possible, because students were in predetermined, intact classrooms. Also, the treatment reading program was purposefully, and not randomly, implemented in the treatment school prior to the researcher's interest in the study. Control schools were included as part o f the present study's analysis to investigate trends in students' Canadian Tests o f Basic Skills scores over the past several years. For example, if both the treatment school's scores and the control schools' scores increase from the pre-treatment to the post-treatment years, one could attribute the increase to a factor other than the reading program, such as district-wide programs, or a difference in overall student population over time due to socio-economic or community changes. However, if only the treatment school's scores demonstrate an increase from the pre- to the post-treatment years, one could attribute the difference to some factor unique to this school, one such possibility being implementation o f the reading program. As well, a difference, if any, in the treatment school's scores between pre- and post-treatment could be underestimated without a comparison to the control schools, because the control 48 schools may exhibit an overall increase or decrease in scores between pre-treatment and post-treatment not exhibited by the treatment school. For example, if the treatment school exhibits no difference between pre- and post-treatment scores, one might assume that the program has had no effect. However, if the control schools all exhibit substantial declines between pre- and post-treatment scores, then the no difference exhibited by the treatment school indicates that the reading program has had some effect in comparison to the control schools' results. This chapter includes a description o f the subjects, measures, procedures, and ethics involved in the present study. The reading program in the treatment school, the reading approaches in the control schools, and the Canadian Tests o f Basic Skills will be described in detail. Reference will also be made to the Provincial Learning Assessment Program as an additional measure o f comparison. Subjects Before amalgamation in 1996, School District No. 56 covered approximately 160,000 square miles with a population o f 22,000 (Vanderhoof District Chamber o f Commerce, 1996). The school district consisted o f 11 elementary and elementarysecondary schools, two o f which were the pilot sites for the treatment reading program. O f the nine remaining schools, four were chosen as controls for the present study. O f the five not chosen, three schools were not included as controls due to incomplete Canadian Tests o f Basic Skills data fi'om the years involved in the study's analysis. Another school is situated in Tachie, an Aboriginal reserve north o f Fort St. James, and was not included in the study due to its unique implementation o f curriculum and instructional methods. The final school not included in the study is quite unlike the treatment school in that it 49 typically has no Aboriginal students in attendance. The one treatment school involved in the present study is located in the community o f Fort Fraser, and the four control schools are located in the communities o f Vanderhoof Fraser Lake, and Fort St. James. These communities can be classified as rural-residential, where forestry and agriculture are the largest industries. The treatment school consists o f Kindergarten to Grade 7. One o f the control schools consists o f Grades 4 to 12 (although data from students only in Grades 4 to 7 were used), and the remaining three schools consist o f Kindergarten to Grade 7. Approximately 230 student scores fi’o m the treatment school and 2300 student scores fiom the control schools were used in the study. Data fiom students who were involved in the treatment reading program for a minimum o f one year were included for analysis. School records were used to determine which students met this criterion. Because the Canadian Tests o f Basic Skills is designed to test individuals at a Grade 3 level and above, students in the study ranged fiom Grade 3 to Grade 7 and in age fiom 8 to 13 years. The mean age o f the students in the treatment school was 10.8 years. The mean age o f the students in the control schools was 11.0 years. Table 1 provides descriptive information regarding the grades o f students involved in the study. 50 Table I Numbers o f Students Involved in the Study School Pre-treatment Treatment Post-treatment Pre-treatment Control 1 Post-treatment Pre-treatment Control 2 Post-treatment Pre-treatment Control 3 Post-treatment Pre-treatment Control 4 Post-treatment Pre-treatment Combined Controls Grade 3 Year Post-treatment 1991-92 1992-93 1995-96 1996-97 1991-92 1992-93 1995-96 1996-97 1991-92 1992-93 1995-96 1996-97 1991-92 1992-93 1995-96 1996-97 1991-92 1992-93 1995-96 1996-97 1991-92 1992-93 1995-96 1996-97 — 20 17 10 — 35 28 40 — 41 22 17 — -------—• — 53 39 40 — 130 89 97 Grade 4 Grade 5 Grade 6 16 14 16 9 17 15 ■■■ — 11 16 48 9 13 Grade 7 Total 9 •57 34 74 66 98 69 170 158 107 168 153 35 164 33 150 124 198 263 232 197 567 533 705 514 — 21 18 50 — — ■■■ — 27 20 26 32 30 18 42 49 17 37 29 31 33 31 47 25 39 40 35 34 19 27 30 — — — 32 ■ ■■ — 32 34 45 48 56 38 161 80 145 110 38 31 47 58 39 49 116 87 157 97 36 33 40 38 53 50 52 37 114 155 163 122 — 54 — 40 21 53 54 46 33 176 81 151 88 Aboriginal ancestry is an ethnic variable which the Ministry o f Education requires all schools to collect. Students in School District No. 91 are designated as Aboriginal if they identify themselves, or if a family member identifies them, as having Aboriginal ancestry. Approximately 24% of the students involved in the study fi’om the treatment school were o f Aboriginal ancestry, and approximately 17% o f the students involved in the study firom the four control schools were o f Aboriginal ancestry. Numbers o f male and female student participants were fairly equal across all the schools involved. In the treatment school, 54% o f the students involved in the study were 51 male and 46% were female. In the control schools, 51% o f the students involved in the study were male and 49% were female. The number o f special-education students involved in the study was not available This is due to the schools' many and diverse definitions o f "special-education" students, which range fi’om those possessing mild learning difBculties to those with severe learning disabilities and physical handicaps. In addition, students with special needs are not always easily identified in the school system. The Canadian Tests o f Basic Skills' scores o f students designated by district testing as EMH (Educably Mentally Handicapped), however, were not included in the study. Measures The Canadian Tests o f Basic Skills (CTBS) Elementary Multilevel Battery consists o f six different test levels designed for students fi’om age 8 to age 14 (King-Shaw, 1989). The purpose o f the CTBS is to determine where students are in the development o f basic skills necessary for success in society. This knowledge is useful for evaluating instructional programs and materials as well as for making instructional decisions regarding individual students. The tests are nongraded and are made up o f a continuous scale fi'om low level Grade 3 to high level Grade 9. Each test level consists o f the following subtests; Reading, Vocabulary, Language (Spelling, Capitalization, Punctuation, Usage, Expression), Work-Study (Visual Materials, Reference Materials), and Mathematics (Concepts, Problem Solving, Computation). The Reading and Vocabulary subtests were determined as measures o f the variable o f interest in the present study, that o f reading achievement. Therefore, only these two subtests' scores were analyzed. 52 The Reading subtests o f the CTBS were developed to represent as many sources o f students' everyday reading as possible. Emphasis is placed on understanding and drawing inferences from the passages. With each increasing test level, there is an increase in the number and complexity o f items requiring higher-order thinking skills. The Vocabulary subtests o f the CTBS were developed by employing two classification systems. The first system, the content-area skills classification, is a computer-based system which generates words taken from a variety o f required and recommended school textbooks. The second system, the linguistic/structural classification, ensures that words depicting different parts o f speech, such as nouns, verbs, and connectives, are given equal representation on the subtests. The CTBS test editions prior to 1998 were standardized in 1987 using approximately 3200 students per grade level and samples representative o f all Canadian provinces and school sizes. Test norms are provided in grade equivalents, percentile ranks, and stanines. The CTBS claims to be valid in that it is based on over SO years o f continuous research. Minority and gender concerns were addressed by careful test item construction and review. Intemal-consistency reliability refers to the correlation among items on a single subtest, that is, whether subtest items measure the same trait or ability. The intemalconsistency reliability (Kuder-Richardson formula 20) o f the Reading subtest ranges from .90 to .94. The intemal-consistency reliability o f the Vocabulary subtest ranges from .83 to .88. These scores were judged by the researcher as adequate indicators of intemalconsistency reliability for the two subtests. 53 Intercorrelations among subtest scores refers to the similarity with which students score on both tests. The intercorrelations among the Reading and Vocabulary subtest scores range from .70 to .78 (King-Shaw, 1990). A separate and independent study by Wright (1976), which involved approximately 2700 students in Grades 4, 6, and 8, demonstrated similar intercorrelations between the CTBS Reading and Vocabulary subtests (r = .80 to .82). These intercorrelations were judged by the researcher as being sufficiently high to suggest that although the Reading and Vocabulary subtests are separate scales, they are also related. It must be assumed that all teachers who administered the CTBS over the years included in the present study's analysis followed the outlined procedures in a nearlyidentical manner. As well, the variability o f student characteristics such as boredom, fatigue, motivation, and illness during the tests' administration must be considered as relatively equal across the various groups and settings Procedures Random assignment o f subjects to treatment or control conditions in the present study was not possible. Instead, intact classrooms and groups o f students were utilized. The independent variable in the study is the reading program. It consists o f two levels; the reading program in the one treatment school, and the reading approaches in the four control schools. The dependent variables are the Canadian Tests o f Basic Skills Reading and Vocabulary subtest scores. Reading Program in the Treatment School Since 1994, the reading program in the treatment school has consisted o f the SRA (Science Research Associates) Reading Mastery and Decoding basal reading programs 54 combined with the Precision Teaching instructional strategy as adopted from the Momingside Model o f Generative Instruction. Direct Instruction is emphasized throughout all aspects o f the program. Most o f the teachers at this school have received training in Direct Instruction and the Momingside model. Reading in all the groups is taught for approximately 90 minutes per day for 5 days a week. The reading groups range in size from 5 to 25 students, with the average reading group size being approximately 15 students. Students are arranged in groups according to their ability levels and not their ages or grades, although a maximum age difference o f 3 years is generally allowed due to the emotional and physical differences between students who range in age by this number. Student placement into specific reading groups is determined by various test scores from both the fall and spring, although previous program levels completed and teacher input are just as influential in making this decision. Students who find certain program levels too difBcult or not challenging enough are moved into more appropriate levels. Reading levels include Reading Mastery I, n , m , IV, V, and VI, as well as Reading Mastery Fast Cycle I and II (for use with Grades 1 and 2 students o f above-average ability levels in reading), and Decoding B l, B2, and C (for use with students who are more than 2 years below grade level and/or who experience difGculty in decoding sounds and words). Students who have completed Reading Mastery VI are taught a literature-based program which emphasizes novel studies, higher-order thinking skills, and more advanced levels o f reading comprehension and written skills. A similar but lower-level literaturebased program is also available for younger students who read at or above grade-level. 55 Prior to 1994, reading programs used by teachers in the treatment school ranged from teacher-directed, structured, phonics-based, basal reader programs to less structured, whole language, literature-based programs As well, reading programs which utilized various combinations o f these approaches were common. Reading Approaches in the Control Schools The Ministry o f Education's Language Arts English Primary-Graduation Curriculum Guide (1990) identified its central aim as: "to enable each student to experience literature and to use language with satisfaction and confidence, striving for fluency, precision, clarity, and independence” (p. 16). This was to be achieved through the program's goals, which were to provide students with opportunities for using language in communication, culture, thinking, and learning. These opportunities could exist in a wide variety o f forms and experiences which embodied the four communication strands of speaking, listening, reading, and writing. The teacher was viewed as the "instructional decision maker" (p. 11) and was granted professional autonomy in choosing a language arts program that met the mandated curriculum's program goals. In 1996, the Ministry o f Education, Skills and Training published an updated version o f its Language Arts English Curriculum Guide entitled English Language Arts K to 7 Integrated Resource Package This new version incorporates many components from the 1990 version and continues to allow teachers the freedom to use a language arts program o f their choice which fulfills the Ministry-mandated requirements. The goal for the new English Language Arts K to 7 curriculum is to "provide opportunities for students to learn to use and appreciate language through a variety o f communication forms in a variety o f contexts" (p. 3). This is to be achieved by providing a language arts program 56 that encourages students to ...communicate efifectively in written, spoken, and visual forms; develop positive attitudes toward language learning; make connections to other areas o f study and to life outside the classroom; think critically, creatively, and reflectively; appreciate their own culture and the culture o f others'; and use technology, (p. 2) Because teachers possess the professional freedom to choose the instructional materials and methods to meet the Ministry's curriculum requirements, there exists a wide variety o f language arts programs in use throughout the school district. An anonymous questionnaire was given to elementary teachers in the four control schools and one treatment school to determine what types o f reading programs and teaching strategies had been used over the past few years (see ^ p e n d ix B). Approximately 65 questionnaires were given out and 16 completed questionnaires were returned. The 16 completed questionnaires represented all 5 schools involved in the study and were therefore judged to be indicative o f the types o f reading programs and teaching strategies found there. Because no personal gain or incentive existed for teachers to return the questionnaires, a 25% return rate was not viewed as unexpected or inadequate. According to the completed questionnaires, teachers have used basal reading series, literature-based reading series, phonics-based programs, novel studies, and teachermade reading programs fairly equally since 1990. There did not appear to be a general shift or trend in the use o f reading programs in the control schools from pre-1990 to 1997. However, many teachers responded that their use of reading programs had changed over the past few years, although their descriptions o f the changes were quite varied and did not indicate a trend towards the use o f any particular reading program. 57 The completed questionnaires also indicated that teachers have taught reading using small groups, the whole class, same-ability grouping, mixed-ability grouping, a traditional style, and a wholistic style Airly equally since 1990. There did not appear to be a general shift or trend in the use o f strategies to teach reading in the control schools from pre-1990 to 1997. As was the case with reading programs, most teachers stated that their use of strategies in teaching reading had changed over the past few years, although the changes were varied and did not indicate a general trend towards the use o f any specific teaching strategy. Most reading classes in the control schools are heterogeneous regarding students' reading abilities. M ost classrooms consist o f 20 to 30 students, and most fulfill the Ministry's recommended time allotment for teaching Language Arts, which is approximately 90 minutes per day, 5 days a week. An additional issue warranting discussion regards the teaching o f components of the treatment reading program in the four control schools. For example, many learning assistance teachers throughout the school district have used and are currently using Reading Mastery in their special-education classrooms, although without the Precision Teaching or fluency-building components. As o f the 1999-2000 school year, the four control schools use various components o f the treatment reading program only in the occasional classroom or by the occasional teacher and, in most cases, in an adapted format. Considering the quasi-experimental nature o f the study, however, and given the fact that classrooms and schools, in general, are dynamic, changing, and unpredictable situations, it can probably be assumed that the location and analysis o f pure control groups with absolutely no exposure to any o f the treatment reading program's components would 58 be an impossibility. As well, it can accurately be stated that the treatment school is the only one o f the five schools involved in this study to implement a reading program schoolwide that follows the Momingside model and which incorporates the combined components o f Reading Mastery, Direct Instruction, and Precision Teaching. The Provincial Learning Assessment Program Because students were not randomly assigned to the treatment and control conditions o f the present study, it may be useful to consider the results o f a different available assessment measure, the Provincial Learning Assessment Program (PLAP). The results o f the PLAP can provide some comparisons with other levels o f the school system, and it may therefore be useful to compare the treatment school's outcomes o f available PLAP evaluations with the school district's average outcomes. This comparison may provide evidence regarding similarities or differences between the treatment school and other schools in the district, including the four control schools. However, it must be stressed that the PLAP is only one assessment measure o f only a limited group of students. The PLAP consists o f annual assessments mandated by the Ministry o f Education for students in Grades 4, 7, and 10. A different subject area is evaluated each year, although not all annual assessments are administered province-wide. The purpose o f the PLAP is to measure student achievement o f the curriculum to determine whether change has occurred over time and to assist in the identification o f schools' and districts' strengths and weaknesses. The Ministry o f Education produces a report after each fioll provincial assessment which consists o f results at the provincial, district, and individual school levels, as well as recommendations for areas identified as those requiring attention. 59 The m ost recent PLAP assessment results are reported and graphed so that statistically significant différences between schools', districts', and provincial results, if any, are easily identified. Estimated means and their 95% confidence intervals (within which the true average percent correct will fidl) are graphed. Overlapping confidence intervals o f provincial, district, and/or school means indicate no significant difference. Less recent assessments, however, only provide the mean percent correct for each school and district. Statistically significant differences, therefore, are unavailable for these evaluations. The available full provincial PLAP assessment results reported for the present study include Reading (1984), Reading (1988), Math (1990), Math/Science (1995), Social Studies (1996), and Reading (1998). Although the Math assessment results are not necessarily indicative o f students' reading achievement, they will, nonetheless, provide some information regarding the academic performance o f students in the treatment school as compared to all students in the school district. Table 2 lists the mean percent correct o f Grades 4 and 7 students in the treatment school (TS) and all Grades 4 and 7 students in School District No. 56 for the six PLAP assessments mentioned. The results o f the Math/Science 1995 and Social Studies 1996 assessments include information regarding statistically significant differences between the school's and district's outcomes. No statistically significant differences existed between the treatment school's Grade 7 students and all Grade 7 students in the district for the Social Studies 1996 or the Math 1995 assessments. As well, no statistically significant differences existed between the treatment school's Grade 4 students and all Grade 4 students in the district for the Math 1995 assessment. Statistically significant differences did exist between both Grades 4 and 7 students in the treatment school and Grades 4 and 60 7 students in the district for the Science 1995 assessment, with the differences favoring the treatment school. The results o f the remaining three assessments appear to somewhat equally favor both the treatment school and the district. About one half o f the treatment school's listed mean percents correct are higher than the district's. However, it is unknown whether these differences are statistically significant. Table 2 Provincial Learning Assessment Program Results Grade 4 Sub|ect/Year Reading 1984 Reading 1988 Math 1990 Math 1995 Science 1995 Socials 1996 Reading 1998 TS 69.25 75.40 44.00 56.80 70.00 — ' 64.60 Grade? SD#56 69.05 72.58 46.00 54.60 58.80 63.40 61.10 TS 66.08 70.38 48.00 48.10 60.00 62.20 52.00 SD#56 72.94 68.48 49.00 49.20 54.80 61.50 60.60 Note. TS = Treatment School; SD#56 = School District No. 56. 'D ata were not available for the treatment school's Grade 4 students on the Socials 1996 PLAP assessment. From this examination and comparison o f the available PLAP results, it appears that no important differences exist between the academic achievement o f Grades 4 and 7 students in the treatment school and all Grades 4 and 7 students in School District No. 56. In making this claim, however, it must be acknowledged that data o f only one assessment measure were examined and with only a small sample o f students. Collection and Treatment o f Data Since 1993, the Canadian Tests o f Basic Skills have been administered to students in the spring o f each school year. Prior to 1993, tests were administered to students in the 61 fall. Completed tests are sent to Toronto, Ontario for scoring. Scored tests are returned and copies o f the results are kept at both the School Board OfiBce and at the individual schools. Although it is mandated by the district that all schools administer the Canadian Tests o f Basic Skills to students in Grades 3, 6, and 9, many schools have chosen, at the elementary level, to administer the test to all students in Grades 3 through 7. The Canadian Tests o f Basic Skills data from the schools and years involved in the present study were entered into an SPSS file by a graduate research assistant at the University o f Northern British Columbia.^ Due to convenience and availability o f the Canadian Tests o f Basic Skills data, four years o f Reading and Vocabulary subtest scores will be analyzed for each o f the five schools, at two points before the treatment reading program's implementation in 1994, and at two points after the program's implementation. The Canadian Tests o f Basic Skills scores from October, 1991 and May, 1993 were chosen as pre-treatment data because they were complete and available for all the schools involved (school district records o f Canadian Tests o f Basic Skills data prior to 1991 are incomplete and inconsistent). As well, sample sizes in the treatment school were largest for these two years. The Canadian Tests o f Basic Skills scores from May, 1996 and May, 1997 were chosen as posttreatment data for several reasons. First, Canadian Tests o f Basic Skills data from 1995 were not available from the School Board OfiBce for any district schools. Second, earlier editions o f the Canadian Tests o f Basic Skills (prior to 1998) were normed in 1989. A ^ This work was funded through a School District No. 91 research grant under the administration o f Peter MacMillan at the University o f Northern British Columbia. 62 new norming sample was used for the 1998 edition o f the Canadian Tests o f Basic Skills, making that year's test scores not useful for comparison with previous years' scores. As well, returned results o f the newer edition which schools in School District No. 91 wrote in 1998 do not include the Vocabulary subtest scores. It is acknowledged that percentile ranks are a less-reliable measure o f achievement than raw test scores (May & Nicewander, 1994). However, the Canadian Tests o f Basic Skills results which are returned to schools and school board offices after being scored do not include students' raw test scores. Also, grade equivalencies are not considered useful or meaningful for the present study's analysis since numbers o f students and their grades vary both between the schools and also between the pre-treatment and post-treatment years. National percentile ranks, however, provide a consistent and meaningful measure and national standard across all schools involved and for both testing periods. Therefore, in spite o f their lower reliability, national percentile ranks will be reported as opposed to raw scores or grade equivalencies in analyzing data for the present study. A final consideration which was made in the present study's treatment o f data concerns the high percentage o f Aboriginal students in School District No. 91. The Canadian Tests o f Basic Skills Reading and Vocabulary subtest scores of Aboriginal students and non-Aboriginal students will therefore be analyzed separately to determine whether the two groups o f students respond in a similar manner to the treatment reading program. The benefit o f this analysis is that the results o f Aboriginal students will not be buried in the results o f the greater group. Aboriginal students are identified by consulting school records, since Aboriginal ancestry is an ethnic variable which the Ministry o f Education requires all schools to collect. Students in School District No. 91 are 63 designated as Aboriginal if they identify themselves, or if a family member identifies them, as having Aboriginal ancestry. O f the sample for the present study, approximately 24% o f the students fi'om the treatment school are o f Aboriginal ancestry, and approximately 17% o f the students fi’om the control schools are o f Aboriginal ancestry. As a preliminary analysis, the Canadian Tests o f Basic Skills Reading and Vocabulary subtest scores o f the four control schools, both pre- and post-treatment, will be compared using analyses o f variance (ANOVA). The results will determine whether the four control schools can then be combined to form one control group for further analysis. If the results o f these ANOVAs determine that it is viable to combine the control schools into one control group, independent samples t-tests will be conducted to compare the Reading and Vocabulary subtest mean scores o f the treatment and control groups in the pre-treatment years and then again in the post-treatment years. Didependent samples t-tests will also be conducted to compare the Reading and Vocabulary subtest mean scores of both Aboriginal students and non-Aboriginal students in the treatment and control groups. Effect sizes using Cohen's d statistic (Cohen, 1992) will also be calculated for the mean difference scores o f the treatment and control groups between the pre- and post­ treatment years for all students as well as for Aboriginal and non-Aboriginal students as separate groups. Ethics Throughout the present study, ethical guidelines concerning the policies o f the School Board and the University of Northern British Columbia were strictly followed. The Canadian Tests o f Basic Skills data are collected atmually by individual schools and the school district as part o f regular and ongoing evaluation procedures, and approval was 64 obtained from the. School Board to analyze these previously collected data from all schools and years deemed necessary for the study (see Appendix C). Principals o f the schools involved were informed o f the study and its intentions regarding the use of their school's data (see Appendix D). Permission to analyze the results o f Aboriginal students as a separate group was granted by the five Aboriginal Band Councils which operate within the School District's boundaries (see Appendix E). As well, whole school data as opposed to specific classroom data were reported, and the anonymity and confidentiality o f the schools were preserved by not using school names. 65 RESULTS The general problem statement o f the present study asks whether or not implementation o f the reading program in the treatment school has been successful in increasing students' reading abilities as measured by the Reading and Vocabulary subtests o f the Canadian Tests o f Basic Skills (CTBS) This section summarizes the study's results as they correspond to the two specific questions related to this problem statement: 1. Will there be an equal difference between the treatment school's (Precision Teaching/Direct Instruction reading program) and the control group schools' (combination o f other reading programs) mean CTBS Reading subtest scores in the pre-treatment years and in the post-treatment years? 2. Will there be an equal difference between the treatment school's precision Teaching/Direct Instruction reading program) and the control group schools' (combination o f other reading programs) mean CTBS Vocabulary subtest scores in the pre-treatment years and in the post-treatment years? These two questions will also be investigated for Aboriginal students and non-Aboriginal students as two separate groups in order to determine whether the treatment reading program is effective for both. Preliminary Analysis As a preliminary analysis, the CTBS Reading and Vocabulary subtest scores o f the four control schools, both pre- and post-treatment, were compared using analyses o f variance (ANOVA) As might be expected, these tests, followed by Scheffe's multiple comparisons, yielded one significant difference between the highest- and the lowestperforming school on the Reading subtest, F(3, 1095) = 3.25, p < .05, and on the 66 Vocabulary subtest, F(3, 1093) = 3.80, p < .05, and only at the pre-treatment level (see Table 3). No significant differences existed between any o f the four control schools in the post-treatment years on the Reading subtest, F(3, 1214) = 1.92, p > .05, or on the Vocabulary subtest, F(3, 1214) = 1.22, p > .05. Table 3 Analysis o f Variance Summaryfor Comparison o f the Control Schools Source Pre-treatment Reading NPR Pre-treatment Vocabulary NPR Post-treatment Reading NPR Post-treatment Vocabulary NPR Between Within Total Between Within Total Between Within Total Between Within Total Sum o f squares 8.457.35 950.378.20 958.835.60 9,473.59 907,820.80 917,294.40 4.697.62 987,877.90 992,575.50 2.696.65 891.915.90 894,612.50 df 3 1.095 1.098 3 1.093 1.096 3 1.214 1.217 3 1.214 1.217 Mean square 2.819.12 867.93 F P 3.25 .02* 3.157.86 830.58 3.80 01* 1,565.88 813.74 1.92 .12 898.88 734.69 1.22 .30 Note. NPR = national percentile rank. *p < .05. As well, Levene's test for equality o f variances, the standard test o f this measure available through the SPSS computer program, indicated equal variances between the four control schools in both the pre-treatment years, F(3, 1095) = 2.75, p > .01, and the post­ treatment years, F(3, 1214) = 0.78, p > .01. An alpha level o f .01 was used as recommended by Milliken and Johnson (1992, pp. 22-23). Because the results indicated no significant differences in variance and no apparent outliers, the four control schools were combined to form one control group in order to compare its CTBS results to those 67 o f the one treatment group. In the analysis that follows, however, the results of the four control schools are presented both separately and combined. It was considered useful to present the scores o f each control school in order to provide evidence that the combined score was not a result o f an individual control school's score that varied greatly from the other three control schools' scores. Main Analysis For the present study's main analysis, a descriptive comparison o f the treatment and control groups' means, standard deviations, and mean difference scores will be presented. As well, a statistical comparison o f the two groups will be provided through a summary o f independent samples t-test results. Finally, effect sizes (Cohen's d) will be reported for the treatment and control groups' mean difference scores from the pre­ treatment years to the post-treatment years. Descriptive Comparison o f the Treatinent and Control Groups Mean scores and standard deviations o f the Reading and Vocabulary subtests for all students for the pre-treatment and post-treatment years will be summarized. Mean scores and standard deviations o f Aboriginal students, however, will not be reported in this study or presented in the tables o r figures due to confidentiality agreements between the researcher and the five band councils involved. Instead, mean difference scores from pre- to post-treatment for Aboriginal and non-Aboriginal students will be reported. All students - Reading. The numbers o f subjects, mean national percentile ranks, and standard deviations for the Reading subtest are presented in Table 4. Results are shown for the one treatment school and for the four control schools, both individually and combined. The treatment school was the only group to exhibit a national percentile rank 68 gain from pre- to post-treatment (+6.21). All four control schools exhibited a decrease in national percentile rank, ranging from -2.17 to -6.23, and with a weighted mean decrease o f -S.05. For greater clarity, graphic representation of these results is provided in Figure 1. As is indicated, the treatment group went from considerably below the control group in the pre-treatment years to above the control group in the post-treatment years. Table 4 CTBS Reading Subtest Data School Treatment Control 1 Control 2 Control 3 Control 4 Combined controls Pre-treatment NPR mean difference (post-pre) Post-treatment n 91 167 275 197 460 NPR mean 35.46 43.57 48.07 40.20 42.37 SD 24.72 31.61 27.96 29.08 29.69 n 140 328 188 274 428 NPR mean 41.67 39.89 41.84 38.03 36.39 SD 31.02 28.35 27.43 28.20 29.33 +6.21 -3.68 -6.23 -2.17 -5.98 1,099 43.59 29.55 1,218 38.54 28.56 -5.05 Note. NPR = national percentile rank. 44 42 - : 38s 36 1 ■ 2 34 . 32 30 Pre Post 'Treatment ■Control Figure 1. CTBS Reading subtest national percentile rank means o f the treatment and control groups before and after treatment. 69 A ll students - Vocabulary. The numbers o f subjects, mean national percentile ranks, and standard deviations for the Vocabulary subtest are presented in Table S. The treatment school and the four control schools all exhibited a decrease in national percentile rank from the pre-treatment to the post-treatment years. However, the treatment school exhibited the smallest decrease (-2.17), while the four control schools' decreases ranged from -2.70 to -8.39, with a weighted mean decrease o f -5.99. This decrease is also evident in Figure 2. However, it is apparent that the treatment school went from slightly below the control group in the pre-treatment years to slightly above the control group in the post-treatment years. Table 5 CTBS Vocabulary Subtest Data Pre-treatment School Treatment Control 1 Control 2 Control 3 Control 4 Combined controls NPR mean difference (post-pre) Post-treatment n 91 166 275 197 459 NPR mean 42.01 47.51 48.13 40.29 43.14 SD 26.38 29.99 27.43 28.66 29.26 n 140 327 188 274 429 NPR mean 39.84 39.12 41.61 37.59 37.39 SD 28.90 25.34 27.44 27.01 28.29 -2.17 -8.39 -6.52 -2.70 -5.75 1,097 44.54 28.93 1,218 38.55 27.11 -5.99 Note. NPR = national percentile rank. 70 44 42 40 38 36 34 32 30 Pre Post -Treatment -Control Figure 2. CTBS Vocabulary subtest national percentile rank means o f the treatment and control groups before and after treatment. Aboriginal situants - Reading. The numbers o f Aboriginal students and their mean difference scores on the Reading subtest from pre- to post-treatment are presented in Table 6. The largest gain in national percentile rank from the pre-treatment to post­ treatment years was exhibited by the treatment school (+8.89). Two o f the control schools also exhibited gains (+1.02 and +4.10), and the remaining two control schools exhibited decreases. The four control schools' weighted mean difference was -3.74. As is also indicated in Figure 3, the treatment group went from considerably below the control group in the pre-treatment years to above the control group in the post-treatment years. 71 Table 6 CTBS Reading Subtest Difference Scores o f Aboriginal (AB) and Non-Aboriginal (Non-AB) Students School Treatment Control I Control 2 Control 3 Control 4 Combined controls Pre-treatment (n) Post-treatment (n) AB 17 40 31 38 57 Non-AB 74 127 244 159 403 AB 38 72 23 62 81 Non-AB 102 256 165 212 347 166 933 238 980 NPR mean difference (post - pre) Non-AB AB +7.77 +8.89 -6.50 +4.10 -4.24 -20.26 -0.35 -6.55 -5.64 +1.02 -3.74 -4.38 Note. NPR = national percentile rank. I I Pre Post Treatment 'Control Figure 3. CTBS Reading subtest national percentile rank means o f Aboriginal students in the treatment and control groups before and after treatment. Non-Aboriginal students - Reading. The numbers of non-Aboriginal students and their mean difference scores on the Reading subtest from pre- to post-treatment are also presented in Table 6. The treatment school was the only group to exhibit a national percentile rank gain from pre- to post-treatment (+7.77). All four control schools exhibited a decrease in national percentile rank, ranging from -0.35 to -6.50, and with a weighted mean decrease o f -4.38. As is also indicated in Figure 4, the treatment group 72 went from considerably below the control group in the pre-treatment years to above the control group in the post-treatment years. 2 I Pre Post 'Treatment ■Control Figure 4. CTBS Reading subtest national percentile rank means o f non-Aboriginal students in the treatment and control groups before and after treatment. Mean difference scores of Aboriginal students (+8.89) and non-Aboriginal students (+7.77) on the Reading subtest were similar in the treatment school (see Table 6). As well, mean difference scores o f Aboriginal students (-3.74) and non-Aboriginal students (-4.38) on the Reading subtest were similar in the control schools. As is indicated in Figure 5, both Aboriginal and non-Aboriginal groups in the treatment school exhibited similar substantial gains from pre- to post-treatment in comparison to Aboriginal and nonAboriginal groups' performances in the control schools. 73 ÜAB □ Non-AB -10 Treatment Control Figure 5. CTBS Reading subtest mean difference scores o f Aboriginal (AB) and nonAboriginal (Non-AB) students in the treatment and control groups. Aboriginal students - Vocabulary. The numbers o f Aboriginal students and their mean difference scores on the Vocabulary subtest from pre- to post-treatment are presented in Table 7. The treatment school and the four control schools all exhibited a decrease in national percentile rank from the pre-treatment to the post-treatment years. The treatment school exhibited the second-smallest decrease (-1.84), while the four control schools' decreases ranged from -1.46 to -13.07, with a weighted mean decrease o f -4.92. This decrease is also evident in Figure 6. 74 Table? CTBS Vocabulary Subtest Difference Scores of Aboriginal (AB) and Non-Aboriginal (Non-AB) Stucknts School Treatment Control I Control 2 Control 3 Control 4 Combined controls Pre-treatment (n) Post-treatment (n) AB 17 39 31 38 58 Non-AB 74 127 244 159 401 AB 38 72 23 62 81 Non-AB 102 255 165 212 348 166 931 238 980 NPR mean difference (post-pre) AB Non-AB -1.84 +0.49 -1.46 -10.77 -13.07 -5.50 -4.56 -1.59 -3.94 -4.49 -4.92 -5.29 Note. NPR = national percentile rank. Pre Post ■Treatment ■Control Figure 6. CTBS Vocabulary subtest national percentile rank means o f Aboriginal students in the treatment and control groups before and after treatment. Non-Aboriginal students - Vocabulary. The numbers o f non-Aboriginal students and their mean difference scores on the Vocabulary subtest from pre- to post-treatment are also presented in Table 7. The treatment school was the only group to exhibit a national percentile rank gain from pre- to post-treatment, although this gain is 75 approximately zero (+0.49). The four control schools' decreases ranged from -1.59 to -10.77, with a weighted mean decrease o f -5.29. This decrease is also evident in Figure 7. However, it is apparent that the treatment school went from slightly below the control group in the pre-treatment years to above the control group in the post-treatment years. 2 I Pre Post •Treatment •Control Figure 7. CTBS Vocabulary subtest national percentile rank means o f non-Aboriginal students in the treatment and control groups before and afrer treatment. Mean difference scores o f Aboriginal students (-1.84) and non-Aboriginal students (+0.49) on the Vocabulary subtest were similar in the treatment school (see Table 7). As well, mean difference scores o f Aboriginal students (-4.92) and non-Aboriginal students (-5.29) on the Vocabulary subtest were similar in the control schools. As is indicated in Figure 8, the treatment school's Aboriginal group exhibited a small decrease in its Vocabulary subtest mean scores, while the non-Aboriginal group exhibited a small increase, although these differences are approximately zero. Both Aboriginal and nonAboriginal groups in the control schools exhibited a decrease in their Vocabulary subtest mean scores. 76 i/i mAB □Non-AB I -10 Treatment Control Figure 8. CTBS Vocabulary subtest mean difference scores o f Aboriginal (AB) and nonAboriginal (Non-AB) students in the treatment and control groups. Statistical Comparison o f the Treatment and Control Groups Because the present study involves only two groups, the treatment group and the control group, statistical analysis was conducted using independent samples t-tests to compare the treatment and control groups' Reading and Vocabulary scores as two separate variables in the pre-treatment years and then again in the post-treatment years.'* In the cases where Levene's Test for equality o f variances indicated that the homogeneity o f variance assumption was not tenable (p < .01), the t-test for unequal variances was ^Because there is a known correlation between the Reading and Vocabulary subtests o f the CTBS (King-Shaw, 1990), scores o f the treatment and control groups on these subtests as a combined variable were first compared using a multivariate analysis o f variance (MANOVA). The results indicated a significant difference between the treatment and control groups in the pre-treatment years, F(2, 1183) = 5.46, /K .O l, and no significant difference between the groups in the post-treatment years, F(2, 1353) = 0.99, />>.01. However, it was considered more useful and relevant for the present study to determine whether the treatment reading program had a similar effect on each o f the Reading and Vocabulary subtest scores. Therefore, further statistical comparisons o f the treatment and control groups were conducted using a univariate analysis o f each separate subtest. 77 employed. The criterion for statistical significance was set at an alpha level o f .05. The t-test results indicate a significant difference between students in the treatment group and students in the control group on the Reading subtest in the pre-treatment years, r(l 12.42) = 2.97, p < .01. No significant difference exists between the treatment and control groups on the Reading subtest in the post-treatment years, /(1356) = -1.22, p > .05. As well, no significant difference exists between the treatment group and the control group on the Vocabulary subtest in the pre-treatment years, /(1186) = 0.81, p > .05, o r in the post-treatment years, /(1356) = -0.53, p > .05. Independent samples t-tests were also conducted to compare the CTBS Reading and Vocabulary subtest mean scores o f both Aboriginal students and non-Aboriginal students in the treatment and control groups. When considering Aboriginal students, the t-test results indicate a significant difference between the treatment and the control groups on the Reading subtest in the pre-treatment years, r(41.16) = 3.40, p < .01. No significant difference exists between the groups on the Reading subtest in the post-treatment years, /(274) = -0.42, p > .05. As well, no significant difference exists between Aboriginal students in the treatment group and the control group on the Vocabulary subtest in the pre-treatment years, /(181)= 1.15,^ > .05, or in the post-treatment years, /(274) = 1.03,/? > .05. When considering non-Aboriginal students, the t-test results indicate a significant difference between the treatment and the control groups on the Reading subtest in the pre­ treatment years, r(89.44) = 2.17,p < .05. No significant difference exists between the groups on the Reading subtest in the post-treatment years, f(1080) = -1.82, p > .05. As well, no significant difference exists between non-Aboriginal students in the treatment 78 group and the control group on the Vocabulaiy subtest in the pre-treatment years, /(1003) = 0.21, p > .05, or in the post-treatment years, /(1080) = -1.79, p > .05. Effect Size Efifect sizes (Cohen's d) were also calculated for the mean difference scores o f the treatment and control groups between the pre- and post-treatment years. Effect size was calculated as the control group's national percentile rank difference score minus the treatment group's national percentile rank difference score divided by the weighted average o f the control group's pre-treatment and post-treatment standard deviations. According to Adams and Engelmann (1996), "For educational purposes, an intervention that changes the performance o f students by l/4th o f a standard deviation (an effect size o f 0.25) is educationally significant" (p. 35). Very few educational approaches, in their opinion, achieve this criterion. Adams and Engelmann go on to claim that an effect size o f 0.50 is rare in educational research. According to Cohen (1992), when testing the difference between independent means, an effect size o f 0.20 is defined as small, an effect size o f 0.50 is defined as medium, and an effect size o f 0.80 is defined as large. An effect size o f 0.39 was found when comparing the Reading subtest mean difference scores of all students in the treatment and control groups. This is interpreted as a small effect size according to Cohen (1992), and as educationally significant according to Adams and Engelmann (1996). An effect size o f 0.51 was found when comparing the Reading subtest mean difference scores o f Aboriginal students in the treatment and control groups. This is interpreted as a medium effect size according to Cohen (1992), and as rare in educational research according to Adams and Engelmarm (1996). 79 An effect size o f 0.42 was found when comparing the Reading subtest mean difference scores o f non-Aboriginal students in the treatment and control groups. This is interpreted as a small effect size according to Cohen (1992), and as educationally significant according to Adams and Engelmann (1996). Less than small effect sizes were found when comparing the Vocabulary subtest mean difference scores o f all students in the treatment and control groups (0.14), and Aboriginal students in the treatment and control groups (0.13). A small effect size o f 0.21 was found when comparing the Vocabulary subtest mean difference scores o f nonAboriginal students in the treatment and control groups. Summary As the data presented in this chapter indicate, the treatment school has made significant and important gains in its students' Reading subtest scores since implementation o f the treatment reading program. This was also the case for both Aboriginal and nonAboriginal students. The treatment school has made no significant gains, however, in its students' Vocabulary subtest scores. This was also the case for both Aboriginal and nonAboriginal students. The four control schools have made no significant gains in their students' Reading or Vocabulary subtest scores since the pre-treatment years. In fact, in most cases, the control schools have exhibited decreases in their students' mean national percentile ranks. These results and their implications will be discussed in greater detail in the next chapter. 80 DISCUSSION This final chapter summarizes the rationale, methodology, and results of the present study. An interpretation o f the study's findings will be provided, as well as a comparison o f its results with those o f other related research previously discussed in the review o f the literature. The limitations o f the present study will be described and, lastly, implications regarding its relevance for practitioners and recommendations for future, related research will be discussed. Summary of the Study The present study involves the evaluation o f a reading program which was implemented in a pilot school in School District No. 56 in 1994. hiitial implementation was made as a response to the general and growing concern regarding the reading d)ility o f students throughout the school district. The treatment reading program combines the teaching strategies o f Precision Teaching and fluency-building with Reading Mastery and its emphasis on Direct Instruction. Relevant research concerning Precision Teaching, fluency, and Direct Instruction was discussed and analyzed in the present study's review o f the literature. Although available studies investigating the effects o f Precision Teaching and fluency are scarce and often non-experimental in nature, the findings o f those discussed appear to indicate that these teaching strategies are effective in the development o f students' reading skills and general learning. Much research has been conducted on Direct Instruction as a teaching strategy although, again, several of the studies previously mentioned are nonexperimental in nature. Results o f the studies discussed, which included follow-up studies of Project Follow Through, studies involving special-education students, and two meta- 81 analyses, support the view o f Direct Destruction as an effective teaching strategy. To evaluate the effectiveness o f the treatment reading program, the Canadian Tests o f Basic Skills (CTBS) Reading and Vocabulary subtest national percentile ranks o f students in the treatment school were compared to those o f students in four combined control schools both in the pre- and the post-treatment years. Di addition to an analysis involving all students, the scores o f Aboriginal and non-Aboriginal students were analyzed separately to determine whether the two groups responded in a similar manner to the treatment reading program. Results o f the descriptive comparison o f the treatment and control groups' mean national percentile rank difference scores between pre- and post-treatment indicated that students in the treatment school, including Aboriginal and non-Aboriginal students, exhibited substantial gains on the Reading subtest, whereas students in the control group, including Aboriginal and non-Aboriginal students, exhibited decreases. Students in the treatment group, including Aboriginal and non-Aboriginal students, exhibited differences o f approximately zero on the Vocabulary subtest from pre- to post-treatment, whereas students in the control group, including Aboriginal and non-Aboriginal students, exhibited decreases. The mean difference scores between pre- and post-treatment for both the Reading and the Vocabulary subtests were similar for Aboriginal and non-Aboriginal students in the treatment group This was also true for the control group. Didependent samples t-test results indicated a significant difference between the treatment and control groups on the Reading subtest in the pre-treatment years, but no significant difference between the groups in the post-treatment years. This was the case for both Aboriginal and non-Aboriginal students as well. No significant differences 82 existed between the treatment and control groups on the Vocabulary subtest in either the pre-treatment or in the post-treatment years. This was the case for both Aboriginal and non-Aboriginal students as well. Last, effect sizes (Cohen's d) were calculated when comparing the Reading subtest mean difference scores o f all students. Aboriginal students, and non-Aboriginal students in the treatment and control groups between the pre- and post-treatment years. No educationally significant effect sizes were found when comparing the two groups' Vocabulary subtest mean difference scores. Interpretation of the Results The results o f the present study indicate a consistent trend in that the students in the treatment school, including Aboriginal and non-Aboriginal groups, demonstrated significant and important gains in their Canadian Tests o f Basic Skills Reading subtest scores. Gains were not made, however, in students' Vocabulary subtest scores. This may be considered as somewhat o f a discrepancy in light o f the relatively high intercorrelation (r = .80) between the two subtests' scores o f students involved in the study. It is also an unexpected finding that the Vocabulary subtest scores did not increase considering the emphasis which the treatment reading program places on vocabulary development. One may speculate, however, that if the reading program had not been implemented, the treatment school might have exhibited decreases in their Vocabulary subtest scores such as those exhibited by the control group. Even though significant gains were demonstrated in only one o f the two subtests, however, it is reasonable to claim that the treatment reading program has indeed been successful in increasing students' reading abilities. After all, it may be argued that the Reading subtest is a more valid indication o f students' reading 83 achievement because it evaluates both decoding and reading comprehension, whereas the Vocabulary subtest simply evaluates students' knowledge o f a selected group o f words. For both the Reading and Vocabulary subtests and for all students, including Aboriginal and non-Aboriginal groups, the treatment group's scores went from below the control group's scores in the pre-treatment years to above the control group's scores in the post-treatment years. This occurred in every case except for one, where the treatment group's Aboriginal student scores on the Vocabulary subtest remained below those of the control group both in the pre- and the post-treatment years. The results o f the independent samples t-tests consistently demonstrate this same trend. The significant difference existing between the treatment and control groups' Reading subtest scores in the pre-treatment years and the non-significant difference existing between the groups in the post-treatment years provide evidence that, although significantly behind the control group before the reading program's implementation, afrer implementation, the treatment school exhibited comparable reading achievement to that o f the control schools. The educationally significant effect sizes (Cohen's d) which were calculated in every case for the Reading subtest fiirther support this trend. There also appears to be somewhat o f a downward trend demonstrated by the control group in both its Reading and Vocabulary subtest scores since 1991. This slight downward trend serves to further emphasize the reading program's effectiveness in the treatment school, where consistent and substantial increases in Reading subtest scores were demonstrated. Also, because only the treatment group's scores increased and the control group's scores did not, it is reasonable to attribute this difference to the treatment reading program and not some other factor, such as district-wide programs or a difference 84 in overall student population over time due to socio-economic or community changes. Without implementation o f the reading program in the treatment school, it may be speculated that similar decreases in student scores would have occurred. It is also worth noting that, in her research study involving high-school students in School District No. 56 (Nechako), Maksymchak (1998) documented a similar downward trend in Math and Physics achievement. Last, it is also encouraging to note that the treatment reading program is equally effective for both Aboriginal and non-Aboriginal students. In fact, when comparing all the groups, the greatest national percentile rank gain was exhibited by the treatment school's Aboriginal students on the Reading subtest. Two statistical hypotheses were examined in the present study. The null hypothesis is rejected in the first case, because the difference between the treatment and control groups' mean Reading subtest scores in the pre-treatment years and in the post­ treatment years was not equal; H i ; Upte(rXc-t) ^ mxMl(rXc-t) , where |ipre(rxc-t) is the difference between the control group's (c) and the treatment group's (t) mean CTBS Reading subtest scores (r) in the pre-treatment years, and PpwKrxc-,) is the difference between the control group's (c) and the treatment group's (t) mean CTBS Reading subtest scores (r) in the post-treatment years. The null hypothesis is accepted in the second case, because the difference between the treatment and control groups' mean Vocabulary subtest scores in the pre-treatment years and in the post-treatment years was equal: 85 Ho : ^pte(vXc-t) = Mpcat(vXc-t), where |Xp„(vxc-t) is the difference between the control group's (c) and the treatment group's (t) mean CTBS Vocabulary subtest scores (v) in the pre-treatment years, and (ipoiKvxc-t)is the difference between the control group's (c) and the treatment group's (t) mean CTBS Vocabulary subtest scores (v) in the post-treatment years. Comparison o f the Research Comparisons can be made between the present study and the various studies previously discussed in this paper's review o f the literature. One must be cautious in making such comparisons, however, considering that all but one o f the studies discussed involve the evaluation o f only Precision Teaching, fluency-building, or Direct Instruction alone. The only known study which does incorporate all three strategies involves Momingside Academy. Students here experience an average academic gain o f 2.5 years per school-year. Similar gains were exhibited in the two pilot schools in School District No. 56 after 2 years o f the reading program's implementation (Johnson & Layng, 1992). It is difBcult to compare these findings to the present study, however, because different standardized tests were administered and grade equivalencies were reported as opposed to national percentile ranks. Another study by Francis (1991) was similar to the present study in that it evaluated a reading program which incorporated both Reading Mastery and Direct Instruction. It did not, however, involve the strategies o f Precision Teaching or fluencybuilding. Results indicated that students achieved significantly higher scores on both the Reading and Vocabulary subtests o f the Gates MacGinitie Reading Tests after 1 year of the program. Again, a different standardized test was administered than in the present 86 study. It is also interesting to note that Francis' study resulted in Vocabulary subtest score increases, whereas the present study did not. Every study which utilized national percentile ranks as part o f its analysis reported substantially greater score increases than those exhibited by the present study's treatment school. Although intervention periods in these studies were similar to the intervention period o f the present study, different testing measures were employed. National percentile rank increases included 20 points (Beck & Clement, 1991), 25 points (Stebbins, 1977), and 17 points (West et al., 1990). National percentile rank increases exhibited by the treatment school o f the present study ranged from approximately 6 to 9 points. The treatment school's increases, however, may be underestimated considering the slight downward trend exhibited by the control group. The two Project Follow Through foUow-up studies that reported effect sizes as part o f their analyses claimed results that replicated the present study's calculated effect sizes o f 0.39, 0.42, and 0.51 for the Reading subtest. Becker and Gersten (1982) reported effect sizes which ranged from 0.38 to 0.56, and Meyer et al. (1983) reported an effect size o f 0.38. It is interesting to note that the effect sizes reported in the two meta-analyses o f Adams and Engelmann (1996) and White (1988), however, were far greater than these (0.87 and 0.84, respectively). An additional study which replicated the findings o f the present study was conducted by Dowdell (1996). The results o f Dowdell's study determined that students who pre-tested significantly below a comparison group on the Iowa Tests o f Basic Skills were able to match the group's post-test scores after implementation o f a Direct Instruction program. 87 The numbers o f students in the present study possessing low, average, or high academic abilities are unknown. Although students designated as EMH ^ducably Mentally Handicapped) were not included in the study's analysis, many students possessing learning difficulties and/or learning disabilities were included. Because these students' scores were combined with those o f students possessing average and above-average abilities, it is difficult to compare the present study’s findings with those o f the many studies which evaluated the effects o f Precision Teaching, fluency-building, or Direct Instruction on the achievement o f special-education students. Most o f these studies report substantial and significant gains in the reading achievement o f students with learning difficulties or disabilities as a result o f implementing reading programs which emphasize Precision Teaching, fluency-building, or Direct Instruction strategies. Although studies which involve students o f average ability all indicated similar increases in academic ability, it appears that in most cases, the greatest gains are reported in the studies which involve special-education students. This could simply be attributed to the fitct that there are more studies available which involve students with learning difficulties. Another explanation, however, could be that the below-average students are farther behind their average-ability peers to begin with and therefore have greater gains to make. Perhaps, therefore, those students in the present study possessing below-average academic abilities exhibited the greatest national percentile rank gains. This, o f course, is only speculation and cannot be determined without analyzing the scores o f below-average students and average to aboveaverage students as two separate groups. In summary, the present study's results are similar to those o f the studies discussed in this paper's review o f the literature in that they all indicate gains in student achievement 88 as a result o f employing the teaching strategies o f Precision Teaching, fluency-building, or Direct Instruction. The present study is unique, however, in that the treatment reading program involves a combination o f these teaching strategies, and also because students o f all ability levels are involved. As well, no other studies were found which evaluated the effects o f Precision Teaching, fluency-building, or Direct Instruction on the reading achievement o f Aboriginal students. Limitations o f the Study Schools and classrooms from which the data were collected for the present study were previously intact, and random assignment to treatment and control groups was impossible. Selection, therefore, is a threat to the study's internal validity. However, although the subjects in the treatment school were not selected randomly from the population, it is known that they are representative o f the entire sample with respect to the variables of sex, age, and Aboriginal ancestry. Another limitation o f the study's analysis is that it involves a between-groups comparison, as different groups o f students were in attendance from the pre-treatment to the post-treatment years. Because the four control schools involved in the present study have used various components o f the treatment reading program, although only in the occasional situation and usually in an adapted format, it is erroneous to consider them as a pure control group. This is a threat to the study's internal validity and could result in a difrusion effect, where the control schools exhibit partial effects o f the treatment reading program. One could speculate as to whether the true impact o f the reading program in the treatment school was reduced as a result. Perhaps the control schools would have exhibited greater score decreases between pre- and post-treatment with no exposure to the treatment reading 89 program's components. The use o f national percentile ranks as opposed to raw scores in the present study's analysis could also have influenced its results. Percentile ranks are a less-reliable measure o f achievement than are raw scores (May & Nicewander, 1994), and the chances o f a Type n error occurring (failure to reject the null hypothesis when it is false) are therefore increased. One could again speculate as to whether the use o f raw scores in the present study's analysis, if indeed possible, would have resulted in greater increases between the treatment school's pre- and post-treatment scores although, o f course, this would be equally true for the control group as well. Another limitation o f the present study concerns the influence o f extraneous variables which may be considered as "rival plausible explanations" to the research outcomes. For example, teacher styles, student-teacher ratio, homogeneous vs. heterogeneous grouping, numbers of special-needs students, and attendance o f students may all be influential factors. Some o f these factors, however, such as teacher styles (for example, use o f Direct Instruction), student-teacher ratio, and homogeneous vs. heterogeneous grouping, are a part o f the treatment school's program design. It is uncertain whether one o f these program components or a combination of them are responsible for the study's results. However, it remains beyond the scope o f this study to determine whether individual components o f the treatment reading program are more or less critical or responsible for its outcome. One may also argue that the vocabulary or content taught in subjects other than Language Arts could influence test performance. However, the probability o f this occurring may be assumed to be relatively equal for both the treatment and the control groups. 90 The external validity o f the present study includes its replicability. Replicability o f the treatment reading program in other schools is dependent upon a number o f factors. Adequate funding is necessary for inservice and training o f teachers in the teaching strategies involved. Administrators and staff-members must be willing and committed to teaching an alternative reading program and to putting in the necessary time and effort to do so. It is important that the staff is cohesive and shares similar goals towards adopting the program and all o f its components. If these criteria are not met, successful implementation o f the reading program may not be possible, and increases in students' reading achievement such as those exhibited by the treatment school may not be experienced. A final limitation o f the study which must be acknowledged concerns experimenter bias. The researcher’s involvement in teaching the reading program in the treatment school since its implementation has resulted in strong, positive opinions regarding its effectiveness. The results o f the study, however, should remain unaffected by this bias, assuming that the standardized testing was administered in a nearly-identical manner by all teachers involved, and because scoring o f the test was conducted by an independent agency. Implications for Practice The results o f the present study indicate that a reading program which combines the teaching strategies o f Precision Teaching and fluency-building with Reading Mastery and its emphasis on Direct Instruction is effective in increasing the reading achievement o f elementary school students. This is an important and relevant finding, especially because children's reading ability appears to be a current issue o f concern for educators and non­ 91 educators alike, both in central British Columbia and beyond. In School District No. 91 this concern may be legitimate, considering that the national percentile rank means o f students in both the treatment and the control groups o f the present study were consistently below the 50th percentile national norm, even in the post-treatment years. In a relevant study by Mothus (1997), the term "cumulative deficit" is used to describe how students with learning difBculties fiül behind their peers a little more each school year and never seem to be able to catch up. As a result, by the time these students reach high-school, they tend to experience high absenteeism and eventually resort to dropping out o f school. A solution lies in teaching them more in less time in order to end this debilitating cycle. As well, it is crucial to deal with this cumulative deficit well before the student enters high-school, because the gap between the learning-disabled student and his or her peers is smallest in the lower grades. Therefore, Mothus states the need for prevention as opposed to remediation. Her data also indicate a disheartening trend, that the reading ability o f the elementary students evaluated for her study appears to be decreasing. Mothus assumes that this downward trend not only exists with these students but is a district-wide problem. She is referring to School District No. 57 (Prince George), the district next in location to School District No. 91, both o f which, it is probably correct to assume, enroll students of similar experience and ability. Mothus suggests a solution, that "Different and better instruction is required, especially since the problem o f reading disability appears to be increasing" (p. 114). It is believed fi’om the experience o f those involved in the treatment school that one such example o f "different and better instruction" has been discovered, and one which addresses the problem o f cumulative deficit by providing students with the opportunity to leam at an accelerated rate. 92 The treatment reading program has been proven efifective with all students, and not only those possessing learning difficulties or disabilities. It is also equally beneficial for both Aboriginal and non-Aboriginal students. This finding is especially notable considering the Ministry o f Education's (1999) recent report on Aboriginal education results in British Columbia. The purpose o f the report is "to improve understanding o f the performance o f the school system in educating Aboriginal students " (Ministry o f Education, p. 1). Two o f the eight recommendations made by the hfinistry o f Education as a result o f this report are "Use the school accreditation process to carefully monitor the academic and social progress o f Aboriginal students, individually and collectively"; and "Find what works in helping Aboriginal students leam, and share the results" (p. 22). The findings o f the present study are also relevant given the fact that the treatment reading program can be implemented in any school o f any size. It has been experienced by staff-members at the treatment school, however, that successful implementation requires two essential ingredients. Firstly, it is necessary to provide extensive training and inservice in the teaching strategies involved. Secondly, a school must have a dedicated and cohesive staff that has similar goals and that is committed to adopting the program's methodology. Schools that are able to meet these criteria may, as a result, experience increases in students' reading achievement similar to those exhibited by the present study's treatment. Implications For Future Research There are a number o f additional topics which are conducive to further and fiiture investigation and which are unfortunately beyond the scope o f the present study. One such issue regards the relationship between student achievement and the number o f years 93 in the treatment reading program. Do students demonstrate greater progress with each subsequent year o f being in the program? Do t h ^ demonstrate equal gains at all ages? Becker's (1978) longitudinal follow-up study of Project Follow Through indicated that it is academically beneficial to begin Direct Instruction at lower grades and, preferably, in Kindergarten. Adams and Engelmann's (1996) meta-analysis, however, resulted in similar educationally significant effect sizes for studies involving interventions o f less than 1 year and those involving interventions o f more than 1 year. Additional studies o f this nature would allow educators to determine whether there is an optimum age when first exposure to the program is most beneficial and whether there is an optimum number o f years that the program should be taught. Gersten et al. (1986) have examined another interesting issue o f whether certain essential components o f Direct Instruction exist which are necessary for producing the most student progress, while other program components are less crucial. Some program components o f Direct Instruction, for example, include teacher scripts, unison responding, mastery learning, task analysis, homogeneous grouping, corrective procedures, and reinforcement strategies. Results o f Gersten et al.'s research indicated that the teacher performance variables most strongly related to students' academic growth included fi-equent feedback and correction procedures, maintaining a brisk pace throughout the lessons, and ensuring student success rates o f at least 80% for unison group responses. Additional studies which further compare these individual components and their effects on student achievement could prove informative. Many studies have been discussed which demonstrate the benefits o f Precision Teaching, fluency-building, and Direct Instruction in teaching both special-education 94 students as well as students o f average ability. It would be interesting to further compare the treatment reading program's effectiveness with low-ability students vs. students with average and/or above-average abilities. More research, as well, is necessary in determining the effects o f Precision Teaching and Direct Instruction on higher-order thinking skills and problem-solving, since few such current studies are available. Such studies which have been discussed, however, claim similar findings o f increased higherlevel thinking skills as a result o f teaching with these strategies (Johnson & Layng, 1996; White, 1988). Previously-discussed studies have indicated that teaching with Precision Teaching, fluency-building, and Direct Instruction strategies results in higher measures o f students' self-esteem (Beck & Clement, 1991; Stebbins, 1977), and higher levels o f on-task behavior and lower levels o f disruptive behavior (Nfiller et al., 1995). Although teachers in the treatment school have experienced similar results first-hand, further investigation concerning the reading program's effects on students' self-esteem and/or behavior would make relevant research topics. The issue o f parents', teachers', and students' attitudes toward the treatment reading program and its components would be another interesting topic o f examination. Related to this are the results o f two previously-discussed studies which indicated that students were highly-motivated and developed positive attitudes as a result o f increased reading scores (Kessissoglou & Farrell, 1995; Mordecai, 1977). Although generally supportive and positive opinions have been expressed by those involved in our school's reading program, a more valid and reliable investigation could prove worthwhile. 95 Almost all o f the related research which has been conducted investigates either Precision Teaching, fluency-building, or Direct Instruction alone. Therefore, more studies like the present one are needed to examine the combined effects o f these teaching strategies. It would be informative to continue the present study's analysis with the Canadian Tests o f Basic Skills data from the years beyond 1997. As well, a valid indication o f the treatment reading program's long-term effectiveness could be determined through a longitudinal analysis o f the present study's subjects in their high-school years and beyond. 96 REFERENCES Adams, G., & Engeimann, S. (1996). Research on Direct Instruction: 25years beyond DISTAR. Seattle, WA: Educational Achievement Systems. Allington, R , & Cunningham, P. (1996). Schools that work: Where all children read and write. New York: HarperCollins College. Allington, R., & Walmsley, S. (E ds). (1995). Mo quickfix: Rethinking literacy programs in America's elementary schools. New York: Teachers College Press. Axelrod, S. (1992). Disseminating an efifective educational technology. Journal of Applied Behavior Analysis, 25(1), 31-35. Bateman, B. (1991). Teaching word recognition to slow-leaming children. Reading, Writing, and Learning Disabilities, 7, 1-16. Beck, R., & Clement, R. (1991). The Great Falls Precision Teaching Project: An historical examination. Journal of Precision Teaching, 8(2), 8-12. Becker, W. (1978). The national evaluation o f Follow Through; Behavior-theory-based programs come out on top. Education and Urban Society, 10, 431-457. Becker, W , & Camine, D. (1980). Direct Instruction: An efifective approach to educational intervention with the disadvantaged and low performers. Advances in Clinical Child Psychology, 3, 429-473. Becker, W., & Gersten, R. (1982). A follow-up o f Follow Through: The later effects o f the Direct Instruction model on children in fifth and sixth grades. American Educational Research Journal, I9(\), 75-92. Berquam, S. (1985). Precision Teaching - Does it work? B.C. Journal o f Special Education, 9(4), 321-327. Binder, C. (1988). Precision Teaching: Measuring and attaining exemplary academic achievement. Youth Policy, 10(1), 12-15. Binder, C. (1993, October). Behavioral fluency: A new paradigm. Educational Technology, 8-14. Binder, C. (1995). Promoting HPT irmovation: A return to our natural science roots. Performance Improvement Quarterly, 8(2), 95-113. Binder, C. (1996). Behavioral fluency; Evolution o f a new paradigm. The Behavior 97 Analyst, 19(2), 163-197. Binder, C , Haughton, E., & Van Eyk, D (1990). Increasing endurance by building fluency: Precision teaching attention span. Teaching Exceptional Children, 22(3), 24-27. Binder, C , & Watkins, C. (1990). Precision Teaching and Direct Instruction: Measurably superior instructional technology in schools. Performance Improvement Quarterly, 5(4), 74-96. Blackman, D. (1984). The current status o f behaviourism and learning theory in psychology. In D. Fontana (Ed ), Behaviourism and learning theory in education (pp. 3-14). Edinburgh, Scotland: Scottish Academic Press. Bloom, B. (1986, February). "The hands and feet of genius": Automaticity. Educational Leadership, 70-77. Brent, G., DiObilda, N., & Gavin, F. (1986). Camden Direct Instruction project 19841985. Urban Education, 2/(2), 138-148. Camine, D. (1992). Expanding the notion o f teachers' rights: Access to tools that v/ork. Journal of Applied Behavior Analysis, 25(1), 13-19. Camine, D., Silbert, J., & Kameenui, E. (1997). Direct Instruction reading (3rd ed ). Upper Saddle River, NJ: Prentice-Hall. Carver, R. (1990). Reading rate: A review o f research and theory. Toronto, Canada: Academic Press. Cohen, J. (1992). Quantitative methods in psychology: A power primer. Psychological Bulletin, 7/2(1), 155-159. Dickinson, D., Wolf, M., & Stotsky, S. (1993). Words move: The interwoven development o f oral and written language. In J. Berko Gleason (Ed ), The development of language f3td ed., pp. 369-420). Toronto, Canada: Maxwell Macmillan Canada. Dowdell, T. (1996). The effectiveness of Direct Instruction on the reading achievement of sixth graders (Report No. CS 012 513). Chicago: Research Press. (ERIC Document Reproduction Service No. ED 396 268) Engeimann, S. (1992). War against the schools' academic child abuse. Portland, OR: Halcyon House. Engeimann, S., Becker, W , Camine, D , & Gersten, R. (1988). The Direct Instruction 98 Follow Through model: Design and outcomes. Education and Treatment o f Children, 7/(4), 303-317. Engeimann, S., & Camine, D. (1982). Theory o f instruction: Principles and applications. New York: Irvington. Engeimann, S., & Hanner, S. (1988). Reading Mastery III teacher’s guide. Worthington, OH: Science Research Associates. Engeimann, S., Hanner, S., & Johnson, G. (1989). Corrective reading series guide. Worthington, OH: Science Research Associates. Engeimann, S., Osbom, J , Osborn, S., & Zoref, L. (1988). Reading Mastery teacher’s guide. Toronto, Canada: Science Research Associates. Farr, M. (1987). The long-term retention o f knowledge and skills: A cognitive and instructionalperspective. New York: Springer-Verlag. Francis, B. (1991). Matching reading programs to students’ needs: An examirujtion o f alternate programming using a Direct Instruction program in the regular classroom. Unpublished master's thesis, Simon Fraser University, Burnaby, British Columbia, Canada. Gersten, R., Becker, W., Heiry, T., & White, W. (1984). Entry IQ and yearly academic growth o f children in Direct Instruction programs: A longitudinal study o f low SES c)m]dx&ci. Educational Evaluation and Policy Analysis, 5(2), 109-121. Gersten, R., Camine, D., Zore^ L , & Cronin, D. (1986). A multifaceted study o f change in seven inner-city schools. Elementary School Journal, 86(3), 257-276. Gersten, R., & Keating, T. (1987). Long-term benefits from Direct Instruction. Educational Leadership, 44(6), 28-31. Goodman, K. (1986). What’s whole in whole language? Richmond Hill, Canada: Scholastic-TAB Publications. Grossen, B. (1996). Making research serve the profession. .American Educator, 20 (3), 78, 22-27. Grossen, B. (1997). Overview: The story behind Project Follow Through [On-line]. Available: http://darkwing.uoregon.edu/~adiep/fl/grossen.htm Haughton, E. (1977). A metric dozen o f practicing practices. Unpublished manuscript, Hastings County Board o f Education, Belville, Canada. 99 Howell, K., & Lorson-Howell, K (1990). What's the hurry? Fluency in the classroom. Teaching Exceptional Children, 22(3), 20-23. Idol, L., & Rutledge, M. (1993). Teaching phonics to poor readers: Direct Instruction using sound sheets. Teaching Exceptional Children, 25(4), 58-61. Johnson, K. (1991). About Momingside Academy. Future Choices, 5(2), 64-66. Johnson, K. (1997). Momingside Academy. Behavior and Social Issues, 7(1), 31-35. Johnson, K., & Layng, T.V. (1992). Breaking the structuralist barrier: Literacy and numeracy with fluency. .^OTcr/cow 47(11), 1475-1490. Johnson, K., & Layng, T.V. (1994). The Momingside model of generative instruction. In R. Gardner, D. Sainato, J. Cooper, T. Heron, W. Heward, J. Eshleman, & T. Grossi (Eds ), Behavior analysis in education: Focus on measurably superior instruction (pp. 173-197). Belmont, CA: Wadsworth. Johnson, K., & Layng, T.V. (1996). On terms and procedures: Fluency. The Behavior Analyst, 79(2), 281-288. Jones, C , & Slate, J. (1996). Educators' attitudes toward educational practices based in behavior analysis. Journal o f Research and Development in Education, 50(1), 3141. Kessissoglou, S., & Farrell, P. (1995). Whatever happened to Precision Teaching? British Journal o f Special Education, 22(2), 60-63. King-Shaw, E. (E d). (1989). Canadian Tests o f Basic Skills: Elementary multilevel battery teacher’s guide. Scarborough, Canada: Nelson Canada. King-Shaw, E. (E d). (1990). Canadian Tests o f Basic Skills: Manualfor administrators, supervisors, and counsellors. Scarborough, Canada: Nelson Canada. LaBerge, D., & Samuels, S. (1985). Toward a theory o f automatic information processing in reading. In H. Singer, & R. Ruddell (Eds ), Theoretical models and processes o f reading (3rd ed., pp. 689-718). Newark, DE: Intemational Reading Association. Lindsley, O. (1990). Precision Teaching; By teachers for children. Teaching Exceptional Children, 22(3), 10-15. Lindsley, O. (1992). Precision Teaching: Discoveries and effects. Journal of Applied Behavior Analysis, 25(1), 51-57. 100 Makysmchak, L. (1998). Evaluation o f the effectiveness o f the Nechako Valley Secondary School mathematics program: An interrupted time series analysis. Unpublished master’s thesis. University o f Northern British Columbia, Prince George, British Columbia, Canada. Marston, C , Deno, S., Kim, D , Diment, K., & Rogers, D. (1995). Comparison of reading intervention approaches for students with mild disabilities. Exceptional Children, 62(1), 20-37. May, K., & Nicewander, W. (1994). Reliability and information functions for percentile ranks. Journal o f Educational Measurement, 3/(4), 313-325. Mercer, C , & Mercer, A. (1993). Teaching students with learningproblems. Toronto, Canada; Maxwell Macmillan Canada. Meyer, L., Gersten, R , & Gutkin, J. (1983). Direct Instruction: A Project Follow Through success story in an iimer-city school. The Elementary School Journal, 54(2), 241-252. Miller, A., Hall, S., & Heward, W. (1995). Effects o f sequential 1-minute time trials with and without inter-trial feedback and self-correction on general and special education students' fluency with math facts. Journal o f Behavioral Education, 5(3), 319-345. Milliken, G , & Johnson, D. (1992). Analysis of messy data. New York: Chapman & Hall. Mordecai, J. (1977, May). Precision Teaching in the elementary classroom. Paper presented at the annual meeting o f the Intemational Reading Association, Miami Beach, FL. Mothus, T. (1997). The effects of strategy instruction on the reading comprehension achievement of Junior Secondary School students. Unpublished master’s thesis. University o f Northern British Columbia, Prince George, British Columbia, Canada. Nitti, J. (1990). Utilizing Precision Teaching to measure growth of reading comprehension skills in law achieving students. Unpublished doctoral dissertation. Nova University, Florida. Osbom, S. (1988). Reading Mastery series guide.y^onYàn^on, OH: Science Research Associates. Pennypacker, H. (1994). A selectionist view o f the future o f behavior analysis in education. In R. Gardner, D. Sainato, J. Cooper, T. Heron, W. Heward, J. Eshleman, & T. Grossi (Eds ), Behavior analysis in education: Focus on 101 measurably superior instruction (pp. 11-18). Belmont, CA: Wadsworth. Province o f British Columbia: \finistry o f Education (Curriculum Development Branch). (1990). Language arts english primary-graduation curriculum guide (Ministry o f Education Publication No. 85303). Victoria, Canada: Queen's Printer for British Columbia. Province o f British Columbia: Ministry o f Education. (1994). The Kindergarten to Grade 12 education plan (Ministry o f Education Publication No. 11416). Victoria, Canada: Queen's Printer for British Columbia. Province o f British Columbia: Ministry o f Education, Skills and Training (Curriculum Branch). (1996). English language arts K to 7: Integrated resource package 1996 (Ministry o f Education, Skills and Training Publication No. IRP 038). Victoria, Canada: Crown Publications. Province o f British Columbia: \& iistry o f Education, Skills and Training. (1997). Accreditation resource: B.C. public school accreditation program (Ministry o f Education, Skills and Training Publication No. 21386). Victoria, Canada: (^een 's Printer for British Columbia. Province o f British Columbia: Ministry o f Education. (1999). Haw are we doing? An overview o f Aboriginal education resultsfor: Province of British Columbia Victoria, Canada: Author. Kinder, S. (1994). Helping children develop skills in phonic analysis, sight vocabulary, and handwriting. Unpublished doctoral dissertation. Nova University, Florida. Rosenshine, B. (1976). Classroom instruction. In N. Gage (Ed.), The psychology of teaching methods: The 75th yearbook o f the National Societyfor the Study of Education (pp. 335-371). Chicago: University o f Chicago Press. Rosenshine, B. (1979). Content, time, and direct instruction. In P.L. Peterson & H.J. Walberg (Eds ), Research on teaching: Concepts, findings, and implications (pp. 28-54). Berkeley, CA: McCutchan. Samuels, S., Schermer, N., & Reinking, D (1992). Reading fluency: Techniques for making decoding automatic. In S. Samuels, & A. Farstrup (Eds ), What research has to jqy about reading instruction (pp. 124-144). Newark, DE: Intemational Reading Association. Skinner, B. F. (1968). The technology o f teaching. Toronto, Canada: Prentice-Hall of Canada. Skinner, B. F. (1982). The science o f learning and the art o f teaching. In R. Epstein (Ed ), 102 Skirmerfor the classroom: Selected papers (pp. 207-222). Chicago: Research Press. Skinner, B. F. (1984). The shame o f American education. American Psychologist. 9, 947954. Slocum, T., Street, E., & Gilberts, G. (1995). A review o f research and theory on the relation between oral reading rate and reading comprehension. Journal o f Behavioral Education, 5(4), 377-398. Snyder, G (1992). Momingside Academy: A learning guarantee. Performance Management Magazine, 10(3), 29-35. Solity, J. (1991). An overview o f behavioral approaches to teaching children with learning difiSculties and the national curriculum. Educational Psychology, 11(2), 151-167. Somerville, D., & Leach, D. (1988). Direct or indirect instruction?: An evaluation o f three types o f intervention programme for assisting students with specific reading difiSculties. Educational Research, 30(1), 46-53. Sprick, R. (1992). Myths, misconceptions, and the thief at the door. The Association for Direct Instruction News, 7/(4), 9-13. Stahl, S., Dufiy-Hester, A., & Stahl, K. (1998). Everything you wanted to know about phonics (but were afi’aid to ask). Reading Research Quarterly, 33(3), 338-355. Stebbins, L. (1977). Education as experimentation: A planned variation model. Volume rV-A: An evaluation o f Follow Through. (Report No. AAI-76-196A). Cambridge, MA: Abt Associates. (ERIC Document Reproduction Service No. ED 148 490) Stephens, M. (1993). Developing and implementing a curriculum and instructional program to improve reading achievement of middle-grade students with learning disabilities in a rural school district. Unpublished doctoral dissertation. Nova University, Florida. Tan, A., & Nicholson, T. (1997). Flashcards revisited: Training poor readers to read words faster improves their comprehension o f text. Journal of Educational Psychology, 89(2), 276-288. Vanderhoof District Chamber o f Commerce. (1996). Vanderhoof. Vanderhoof, British Columbia, Canada: Author. Watkins, C (1996). Follow Through: Why didn't we? Effective School Practices,_\^(\), . 57-66. 103 Weikart, D. (1985). Project Follow Through. In The intemational encyclopedia o f education research and studies (Wo\. 3, pp. 4105-4108). New York: Pergamon Press. West, R., Young, K., and Spooner, F (1990). Precision Teaching: An introduction. Teaching Exceptional Children, 22(3), 4-9. White, W. (1988). A meta-analysis o f the effects o f Direct Instruction in special education. Education and Treatment o f Children, //(4 ), 364-374. 104 Appendix A Opinions o f Teachers in the Treatment School 105 Opinions o f Teachers in the Treatment School As part o f the present study, it may be interesting to relay the opinions o f teachers who have been involved in teaching the reading program in the treatment school/ The treatment reading program has been accused by some in the educational community of being rigid, artificial, authoritarian, and reductionistic. Such critics may claim that it stifles teacher and student creativity, promotes passive learning, is inconsiderate o f individual differences, and ignores higher-order thinking skills. Although the program is structured, sequenced, and scripted, the teachers do not sense a loss o f creativity or the need to be denied their individual teaching styles. Direct Instruction demands continuous monitoring and evaluation o f student progress, and constant decision-making is necessary to determine whether concepts need re-teaching and whether additional practice is necessary. The teachers agree with Sprick (1992) who stated, "Anyone who gets bored teaching Direct Instruction is not focusing on the right things—stay focused on the needs o f the students because this provides endless diversity" (p. 12). Teacher creativity and flexibility, therefore, are necessary and important in ensuring that lessons are taught as effectively as possible. Another criticism o f Direct Instruction is that it ignores the development o f higherorder thinking skills in students. The teachers have discovered, however, that students who are fluent at basic skills are better able to modify and adapt their knowledge in more *All sections o f this paper referring to informal observations regarding the reading program and its effects were made available to staff-members involved in its implementation at the treatment school. They agreed that these statements are accurate representations o f their opinions. 106 complex and real-world situations, and they try to provide many opportunities for students to be challenged with creative and higher-order skills throughout the remainder o f the school day. The teachers would also argue that Direct Instruction, with its underlying philosophy that all children can leam and its creed that "if the student has not learned, the teacher has not taught" (Adams & Engeimann, 1996, p. ix), is an extremely humanistic and child-centered teaching approach. Proper student placement is a major priority so that they receive the best possible instruction at their level, data is collected and analyzed daily for each student, and students are involved in their own learning through Precision Teaching strategies. Students are hardly passive, but respond and interact frequently and in varied ways during each lesson. As well, the teachers all work hard to provide warm, cooperative, positive, student-centered classrooms. Another highly-debatable issue concerns the use o f homogeneous grouping during reading instruction. Many within the educational community argue that same-ability grouping is detrimental to the self-concepts o f older students with weaker abilities who are placed with younger students. From the teachers' experience, they believe the reverse to be true. Prior to the program's implementation, when reading was taught to same-aged students of mixed abilities, students were always well-aware o f who their weaker peers were, regardless o f the program used. This common knowledge can have negative and lasting effects on students' self-esteem. With homogeneous grouping, however, these students are successful, and the teachers have all experienced the positive effect which success has on students' self-images. It also needs to be noted that students spend a considerate proportion o f each day with their same-aged peers in other subject areas. As 107 well, the homogeneous groups, in general, do not span more than three years difference in age due to students' differing emotional and physical characteristics at these ages. One could also argue that Precision Teaching and fluency-building strategies do much to promote the development o f self-esteem in students. Much has already been said about the rewards o f students monitoring their own academic development. As well, building fluency in small steps provides frequent reinforcement and assurance regarding their progress. It has been the teachers' experience that when students are fluent readers and writers, they are generally able to complete tests and assignments quickly and accurately, and when students experience success in school and have earned that success through hard work, they often also experience positive feelings about themselves and school as a result. The teachers have also noticed that their students appear to possess better work habits and improved cooperative learning skills (students do daily fluency and peercoaching exercises with assigned partners). As well, most students enjoy the reading program and are motivated, positive learners. The behaviour o f many students who previously displayed resistant or apathetic attitudes has improved. Some attribute this to the predictable and structured methods which provide necessary security and stability for many students, and to the frequent reinforcement contingencies which contribute to a more motivating and positive atmosphere. As well, the staff at the treatment school has become more cohesive as a result of working toward a common goal. As teachers, their instructional skills, data collection techniques, and evaluation strategies have improved. With each subsequent year since first beginning the program, the teachers have noticed less time and energy spent in 108 review, since students appear to be retaining previous learning. As well, each year fewer students are on modified programs and more students are working at or above grade level. Students no longer "fall through the cracks", for everyone is placed in the most appropriate learning environment possible. Another important result o f implementing the reading program in the treatment school is the support and positive feedback received by the vast majority o f students' parents. They are pleased because their children, in general, are happy and experiencing success in school. Such positive opinions were evident as a result o f the treatment school's 1995 accreditation, a mandatory program which all public schools in British Columbia undergo every six years. The purpose o f the accreditation process is to provide schools with the opportunity to examine and identify their strengths and weaknesses and to develop long-term School Growth Plans which consist o f goals aimed at maintaining their strengths and at improving their weaknesses. This is achieved through a lengthy and rigorous process o f collecting and assessing data and evidence which is eventually reviewed by a visiting External Committee (NCnistry o f Education, 1997). As part o f the treatment school's accreditation process, surveys were distributed to students and parents to determine their opinions and attitudes toward various aspects of the school. Questions relating to reading and the reading program were included in the surveys, especially since the reading program had only been in effect for one school-year at the time o f the accreditation. Ninety percent o f primary students responded with a "Yes" to the survey statement, "I am learning to read". Ninety-one percent o f intermediate students agreed or strongly agreed to the statement, "The school does a good job teaching me to read". O f the 50 parents who responded to the question, "How would 109 you rate the way students at your school are learning the basics o f reading and writing?", 68% answered "Good" and 18% answered "Excellent". As a result o f these positive opinions and due to teachers' perceptions o f improved reading achievement, one o f the school's stated strengths in the accreditation's final Internal Report was; "The school offers a coordinated language arts program which allows students to demonstrate success in the acquisition o f basic skills in literacy and expression". The External Committee, as well, agreed with this stated strength after their week-long examination and review o f the school. Many o f the teachers' opinions o f the reading program, however, were not always so positive. It is probably fair to state that the first year o f the program was difiBcult for all who were involved. The instructional methodology was quite unlike that which had been used in previous years and many o f the teachers had received teacher-training based on wholistic and cognitive-developmental philosophies. Some teachers, therefore, began the pilot project with negative and pessimistic attitudes regarding its outcome. Not only did the program's behavioural emphasis contradict their educational belief systems, but incorporating the program was hard work. Almost all o f the teachers received training sessions in the summer o f 1994 in order to leam Precision Teaching charting strategies, fluency-building strategies. Direct Instruction methodology, and to become acquainted with the Reading Mastery curricular materials. For the first two months o f the program's first year, staff meetings were held almost on a daily basis, usually unplanned and spontaneous in nature, to discuss and compare discoveries, agonize over fhistrations, and celebrate student progress. 110 It was not until the end o f the program's first year and when the second year's implementation began that perceptions started to shift. The teachers had gradually become fluent at using Precision Teaching and Direct Instruction strategies and, as a result, were able to incorporate more o f their individual and creative teacher styles into their instruction. The philosophy and theory underlying the program became more clear and reasonable as they grew familiar with its components. Most influential in changing their attitudes, however, was observing student progress. From the teachers' experience and informal comparisons o f student performance, they all agreed that the program was effective. It is interesting to note that the teachers' experience with the program's initial implementation replicates findings o f two earlier studies. In a study conducted by Gersten, Camine, Zoref, and Cronin (1986), teachers from seven schools and in the first year o f implementing a Direct Instruction program experienced feelings o f misunderstanding, philosophical conflict, and even anger at having the program imposed upon them. Some described it as an extremely difficult and painful time due to the overwhelming changes in instructional strategies which they felt forced to make. It was not until the second year of teaching the program, mainly due to seeing student improvement, that these teachers began to alter their opinions and experience positive attitudes towards the implementation. In a second study, Engeimann, Becker et al. (1988) conducted interviews over a two year period and found that, initially, teachers commonly experienced negative reactions to the structure and regimentation o f using Direct Instruction methods, but that their attitudes changed considerably and for the better after several months o f day-to-day Ill observation o f student progress and the program's effectiveness. It can probably be expected, therefore, that educators who are not familiar with Precision Teaching and Direct Instruction will experience negative reactions when first reading about it or observing it in action. Teachers generally become strong advocates o f the program only after using it for an extended period o f time and after experiencing its effectiveness through student performance and progress. Now that teachers at the treatment school have been using Precision Teaching and Direct Instruction strategies for five years, they are comfortable in adjusting components o f the program to better suit their and their students' needs. They believe that it is important, however, to always maintain the integrity o f the program's underlying theoretical philosophy when making any adjustment because it is based on years of proven, empirical, classroom-tested research. Another recent development at the school is the addition o f supplemental activities and materials to promote students' critical thinking and problem-solving skills. The teachers believe that this step is progressive and developmental in nature and has become necessary because many o f their students are now fluent at the essential skills which are prerequisite to more complex and higher-order learning. At the treatment school, teachers have not become stagnant in their instructional methods, but rather attempt to evaluate the programs' effectiveness on a regular basis in order to adjust and supplement them accordingly. It is not their intention, therefore, to use Direct Instruction forever with all students. The teachers have found that the program's sequence and structure is necessary for teaching basic skills and new concepts, for review o f these skills and concepts, and in instructing those students who are behind 112 their peers in academic performance. Once the students are fluent at the basic skills and have a firm foundation o f knowledge and strategies necessary to achieve success at more complex tasks, they are then ready to move beyond Direct Instruction and towards more wholistic instruction that maintains a high level o f interactivity, active participation, and higher-order thinking skills. The teachers would agree with the statement o f Momingside Academy's founder and director: "It eventually becomes necessary for students to progress from having to say something, to having something to say" (K. Johnson, personal communication, April 16, 1998). It is, after aU, their goal as educators to provide opportunities for students to become competent, problem-solving, self-directed, positive, and responsible future members o f society. 113 Appendix B Elementary Teacher Questionnaire 114 Decembers, 1997 ELEMENTARY TEACHER QUESTIONNAIRE Dear Elementary Teachers: I am a teacher at Fort Fraser Elementary School and I am working on my Master’s thesis through the University o f Northern British Columbia. My research study involves examining the effects o f an experimental reading program on students’ reading abilities as measured by the Canadian Tests o f Basic Skills. The experimental reading program was implemented in our school in 1994. I will be comparing the test scores o f students in Fort Fraser Elementary School with the test scores o f students in four other elementary schools in what used to be School District No. 56 (Nechako). As part o f my thesis, I will be describing the various reading programs and strategies used over the past ten years or so throughout our district. As well, I am interested in seeing whether there has been any kind o f general change or shift in use o f reading programs or strategies since the late 80s. This is where I need your help! I would greatly appreciate your filling out the anonymous questionnaire attached and returning it to me at Fort Fraser Elementary via the School District courier pouch or by faxing it to me at 690-7338 by December 19“*. If you need more room to provide your answers, please attach additional pages. Thank you very much for your assistance at this busy time o f year! Please feel free to contact me with any questions or comments you may have regarding this questionnaire and/or my research study. Sincerely, Tammi Rothenbusch 115 ELEMENTARY TEACHER QUESTIONNAIRE (PLEASE RETURN TO FORT FRASER ELEMENTARY SCHOOL) I. GENERAL QUESTIONS 1. How many years have you taught in School District #91 (or #56)? ____________ 2. Which grade/s have you taught? ____________________ 3. Please list all the schools within the original School District #56 in which you have taught: _____________________________________________________________ n . READING PROGRAMS With what types of reading programs have you taught? Please check all that apply. As well, please indicate (to the best of your memory!) the years during which you taught the program/s: Basal reading series Please indicate which series:___________ ___________ Years taught: pre-1990 _____1990-1992 1993-1995 1996-present Literature-based reading series Please indicate which series:______________ Years taught: pre-1990 1993-1995 ____ _____1990-1992 1996-presait Phonics-based program Please indicate which program: _____________________________ Years taught:________ _____pre-1990 _____1990-1992 1993-1995 1996-present Novel studies Years taught: _____pre-1990 1993-1995 _____1990-1992 1996-present “Teacher-made” reading units or program Please describe: __________________________________________________________ Years taught: _____ pre-1990 1993-1995 _____ 1990-1992 1996-present Other Please describe: __________________________________________________________ Years taught:________ _____pre-1990 1993-1995 _____1990-1992 1996-present Has your use of reading programs changed over the past few years? Yes No If yes, please indicate how it has changed, in general: ___________________________ 116 UI. READING STRATEGIES/TEACHING STYLES What types of strategies and styles have you used to teach reading? Please check all that apply. As well, please indicate (to the best of your memory!) the years during which you taught using these strategies and styles; Small groups Years taught: Whole class Years taught: Same-ability grouping Years taught: Mixed-abiiity grouping Years taught; “Traditional”, structured style Years taught: pre-1990 1993-1995 _____1990-1992 _____1996-present pre-1990 1993-1995 _____1990-1992 _____1996-present pre-1990 1993-1995 _____1990-1992 _____1996-present pre-1990__ _____1990-1992 1993-1995 _____1996-present pre-1990 1993-1995 ____ 1990-1992 _____ 1996-present “Holistic”, less-structured style Years taught: pre-1990 1993-1995 _____1990-1992 _____ 1996-present Combination of the “traditional” and “holistic” styles mentioned above Years taught: pre-1990 _____1990-1992 1993-1995 _____ 1996-present Other Please describe: ___________________________________________________ Years taught: pre-1990 ____ 1990-1992 1993-1995 _____1996-present Has your teaching style and/or use of strategies in teaching reading changed over the past few years? Yes No If yes, please indicate how it has changed, in general; ___________________________ 117 Appendix C School Board Approval 118 Box 129 Vandertioof. B.C. V0J3A0 Telephone: (604) 567-2284 Fax: (604) 567-4639 r a \ I- 'v I S*j Serving Fort F r a s e r Fort SL James Fraser Lake Vanderhccf October 30, 1996 Ms. Tammi Rothenbuscb Box 1046 Fraser Lake, B.C. VOJ ISO Dear Tammi: On behalf of the Board of School Trustees. School District No. 56 (Nechako), I am writing a letter in support of your research for your Master's degree. Improving the quality of education is important to all of us and we would be very interested in the results of your research. Anonymity of students and score results are essential in maintaining the confidentiality of the students. Your desire to further your education is to be commended and it demonstrates a commitment on your part to continually improve oneself. We wish you luck on your Master's program and look foniward to your sharing the results. Yours truly. Louise Burgart Superintendent of Schools LB/cp cc: J. Six, Principal 119 Appendix D Letter o f Intent to Principals 120 November 15, 1996 Greetings! (This message applies to ail schools in our district except for Fort St. James Secondary and NVSS.) For those o f you who don't know me, I am currently teaching grade 7 at Fort Fraser Elem. and this is the sixth year that I've taught in the district. I began my Master's program at UNBC a year ago, and I am beginning to plan for my thesis topic. I am interested in evaluating the Reading Mastery program combined with the teaching components o f the Momingside Academy model (fluency, charting. Direct Instruction, etc ) which our school implemented in 1994. I have received permission from the School Board to use available CTBS data with which to conduct such an evaluation, and I will be following strict confidentiality and anonymity guidelines (no names of students or schools will be used, etc.). I hope to use as many schools as is possible for comparison groups. Here's where you come in (!)... I would like to find out whether your school administers the CTBS to all grades or only to those required (grades 3, 6, 9)? For my research, the more grades' results I have, the better (altho I do realize that administering the CTBS to all grades costs $). I would greatly appreciate your answering this question at your convenience (but as soon as possible, please?!). Also, please feel fi’ee to contact me with any questions or comments you may have regarding my potential research study. Thanks, Tammi Rothenbusch 121 Appendix E Letter to Aboriginal Band Councils Research Study Pennission Form 122 December 10, 1997 To Whom it May Concern: I am a teacher at Fort Fraser Elementary School and I am working on my Master’s thesis through the University o f Northern British Columbia. My research study involves examining the effects o f an experimental reading program on students’ reading abilities as measured by the Canadian Tests o f Basic Skills. The experimental reading program was implemented in our school in 1994 in response to a growing concern regariÛng the reading abUity o f its students. The Canadian Tests o f Basic Skills have been regularly administered in our school district for many years. Test results are normally retained by the district as part o f an ongoing evaluation program and according to its ethical guidelines and proc^ures. Previously collected data from this standardized measure will be analyzed to determine the effects o f the reading program’s implementation. The vocabulary and reading scores o f the experimental school will be compared to the vocabulary and reading scores o f four control schools in the district. Permission has been obtained from the School Board to use the Canadian Tests o f Basic Skills data from all schools and years deemed necessary. Throughout the study, confidentiality and anonymity o f schools will be maintained and ethical guidelines concerning the policies o f the school district and the University o f Northern British Columbia will be strictly followed. As there is a high percentage o f Aboriginal students in our district, I am interested in analyzing the scores o f Aboriginal students and non-Aboriginal students separately to determine whether the two groups o f students respond in the same manner to the new program. Investigating the effectiveness o f the experimental reading program for both groups could result in important implications regarding the future education for all our students. As well, the results o f this study will enable us to address the needs o f our students in order for them to learn more effectively. I am therefore requesting the permission o f Aboriginal Band Councils represented by students in the original School District No. 56 (Nechako) to analyze the results o f Aboriginal students as a separate group for my research study. Confidentiality and anonymity of all schools and students will be maintained. As well, I will gladly share the results o f my research with you when it has been completed. I would appreciate your signing the attached permission form. Please feel free as well to contact me with any questions, concerns, or comments you may have regarding this research study. I thank you for your consideration o f this request. Sincerely, Tammi Rothenbusch 123 RESEARCH STUDY PERMISSION FORM I, (name), from the (Band), do give my permission for Tammi Rothenbusch to analyze the Canadian Tests o f Basic Skills scores o f Aboriginal students in School District No. 56 as a separate group for a Master’s thesis research study at the University o f Northern British Columbia. I understand that confidentiality and anonymity o f all Bands, schools, and students will be maintained and that ethical guidelines concerning the policies o f the school district and the university will be strictly followed. Signed: Date: