AN EVALUATION OF THE SUPPLEMENTAL INSTRUCTION PROGRAM IMPLEMENTED IN A FIRST-YEAR CALCULUS COURSE by Vivian Fayowski B.Sc., University of Victoria, 1997 THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF EDUCATION in CURRICULUM AND INSTRUCTION THE UNIVERSITY OF NORTHERN BRITISH COLUMBIA July 2005 © Vivian Fayowski, 2005 1^1 Library and Archives Canada Bibliothèque et Archives Canada Published Heritage Branch Direction du Patrimoine de l'édition 395 W ellington Street Ottawa ON K 1A 0N 4 Canada 395, rue W ellington Ottawa ON K 1A 0N 4 Canada Your file Votre référence ISBN: 978-0-494-28373-8 Our file Notre référence ISBN: 978-0-494-28373-8 NOTICE: The author has granted a non­ exclusive license allowing Library and Archives Canada to reproduce, publish, archive, preserve, conserve, communicate to the public by telecommunication or on the Internet, loan, distribute and sell theses worldwide, for commercial or non­ commercial purposes, in microform, paper, electronic and/or any other formats. AVIS: L'auteur a accordé une licence non exclusive permettant à la Bibliothèque et Archives Canada de reproduire, publier, archiver, sauvegarder, conserver, transmettre au public par télécommunication ou par l'Internet, prêter, distribuer et vendre des thèses partout dans le monde, à des fins commerciales ou autres, sur support microforme, papier, électronique et/ou autres formats. The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission. L'auteur conserve la propriété du droit d'auteur et des droits moraux qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son autorisation. In compliance with the Canadian Privacy Act some supporting forms may have been removed from this thesis. Conformément à la loi canadienne sur la protection de la vie privée, quelques formulaires secondaires ont été enlevés de cette thèse. While these forms may be included in the document page count, their removal does not represent any loss of content from the thesis. Bien que ces formulaires aient inclus dans la pagination, il n'y aura aucun contenu manquant. Canada APPROVAL Name: Vivian Fayowski Degree: Master of Education Thesis Title: AN EVALUATION OF THE SUPPLEMENTAL INSTRUCTION PROGRAM IMPLEMENTED IN A FIRST-YEAR CALCULUS COURSE ^ f\ /r Examining Committee: ----- — ------------ — - ^ ' ~ ^ iii Chair: Dr. Greg Hhlselli, Pioiessor Natural Resources and Environmental Studies Program Canada Research Chair, Rural and Small Town Studies University o f Northern British Columbia Si^ervisor: Dr. Peter MacMillan Associate Professor, Education Program University o f Northern British Columbia Committee Member: Dr. Dennis Procter Assistant Professor, Education Program University of Northern British Columbia C o^m itte^^em ber:*^r. Jennifer Hyndman, Associate Professor Mathematical, Computer, and Physical Sciences Program University of Northern British Columbia // g xtemal EiMminer: Dr. William Owen Assistant Professor, Psychology Program University of Northern British Columbia Date Approved: ABSTRACT Supplemental Instruction (SI) is a voluntary program that incorporates collaborative learning in peer-led, small group settings in order to integrate instruction in learning and reasoning skills with the content of the course with which the SI is paired. Calculus for Non-Majors is the course that forms the basis of this three year study. This study addresses two related questions. First, does SI participation improve student achievement, as measured by course final letter grades? Second, does SI participation improve the pass/fail rate in the course? Prior student success, a combination of ability and motivation, was statistically controlled for in both analyses through the use of incoming grade point average. Gender was also chosen as an independent variable both for increased statistical sensitivity and generalizability. The effect of SI participation in Analysis of Covariance, after prior GPA and gender were controlled for, was statistically significant (p<.0005) and practically significant (d = .48) or the equivalent of a two letter grade improvement. Pass/fail analysis was determined through binary logistic regression with observed statistically significant differences between the first model containing both prior GPA and gender and the second (full) model which also contained SI (chi-square = 41.19, p < .0005). In the full model gender did not contribute significantly (p = .24). The odds of succeeding were 2.7 times greater for SI participants. Overall, SI participants succeeded at a higher rate than non­ participants (73% vs. 51%). These findings are consistent for both genders. Supplemental Instruction is an effective method for boosting success rates in a difficult undergraduate course with concentrated mathematical content. 11 TABLE OF CONTENTS ABSTRACT----------------------------------------------------------------------------------- ii TABLE OF CONTENTS-------------------------------------------- iii LIST OF TA BLES---------------------------------------------------------------------------- v ACKNOWLEDGEMENTS----------------------------------------------------------------- vii CHAPTER I - INTRODUCTION Introduction--------------------------------------------------------------------------Background--------------------------------------------------------------------------- 1 5 CHAPTER n - LITERATURE REVIEW Supplemental Instruction---------------------------------------------------------Discourse - A theoretical framework for Supplemental Instruction Problem statement-----------------------------------------------------------------Research questions-------------------------------------------------------------Limitations----------------------------------------------------------------------- 10 20 25 25 27 CHAPTER m - METHODOLOGY Research population--------------------------------------------------------------M easures----------------------------------------------------------------------------Procedures--------------------------------------------------------------------------Data collection----------------------------------------------------------------Specific procedures-----------------------------------------------------------Ethics------------- 28 29 30 30 32 33 CHAPTER IV - RESULTS Preliminary analyses--------------------------------------------------------------34 Data Cleaning and Screening----------------------------------------------34 Selection and Criterion Variables used inthe Preliminary Analyses- 35 First analysis: One-Way Analysis o f Variance (ANOVA) with post-hoc tests--------------------------------------------------------- 37 Second analysis: Two-Way Contingency Table Analysis Using Crosstabs----------------------------------------------------------- 39 Analysis of Relationships between Final Grades, Supplemental Instruction, Incoming Grades, and G ender-------------------------42 Additional Variables - Gender and Incoming Grade Point Average (G PA)---------------------------------------------------------------------42 Third Analysis: Analysis o f Covariance (ANCOVA)-------------------44 Fourth Analysis: Binary Logistic Regression------------------------------ 46 111 CHAPTER V - DISCUSSION Discussion---------------------------------------------------------------------------Limitations---------------------------------------------------------------------------Implications--------------------------------------------------------------------------Conclusions--------------------------------------------------------------------------- 51 54 55 56 REFERENCES-------------------------------------------------------------------------------- 58 APPENDIX A - ETHICS APPROVAL FORM S------------------------------- 62 APPENDIX B - SUMMARY D A TA --------------------------------------------- 65 APPENDIX C - ANALYSIS OF COVARIANCE, CHI-SQUARE ANALYSIS 71 APPENDIX D - NOTATION----------------------------------------------- ------------ - 76 APPENDIX E - UNBC GRADING SYSTEM --------------------------------- IV 77 List of Tables Table 1: UNBC Grading System -------------------------------------------------------- 77 Table 2: Conversion o f letter grades to numerical values------------------------- 77 Table 3: Means and Standard Deviations-------------------------------------------- 37 Table 4: SI treatment* Success/Failure Crosstabulation-------------------------- 40 Table 5: Post hoc tests fo r Chi-square Analysis-------------------------------------- 40 Table 6: Mean final grades by gender and SI treatment--------------------------- 42 Table 7: ANCOVA Summary (non-transformedfinal grade)--------------------- 45 Table 8: Sequential Binary Logistic Regression M odels--------------------------- 48 Table 9: Variables in the Equation----------------------------------------------------- 49 Table 10; SI DATA: WINTER 2002 - MATHEMATICS 152 (Calculus fo r Non-Majors) Sections A1 and A 2 ------------------------- 65 Table 11: Summary Chart (WINTER 2002)-------------------------------------------- 65 Table 12: SI DATA: FALL 2002 - MATHEMATICS 152 ( Calculus fo r Non-Majors)--------------------------------------------- 66 Table 13: Summary Chart( FALL 2002)------------------------------------------------- 66 Table 14: SI DATA: WINTER 2003- MATHEMATICS 152 (Calculus fo r Non-Majors) Sections A I and A 2 -------------------------- 67 Table 15: Summary Chart (WINTER 2003)--------------------------------------------- 67 Table 16: SI DATA: FALL 2003- MATHEMATICS 152 ( Calculus fo r Non-Majors)--------------------------------------------------- 68 Table 17: Summary Chart (FALL 2003)------------------------------------------------ 68 Table 18: SI DATA: WINTER 2004 - MATHEMATICS 152 (Calculus fo r Non-Majors)--------------------------------------------------- 69 Table 19: Summary Chart (WINTER 2004)-------------------------------------------- 69 Table 20: SI DATA: Fall 2004 MATHEMATICS 152 (Calculus fo r Non-Majors) --------------------------------------------------- 70 Table 21: Summary Chart (Fall 2004)------------------------------------------------- 70 Table 22: Computation Equations fo r Sums o f Squares and Cross-Products in One-Way Between Subjects Analysis o f Covariance---------- 73 VI ACKNOWLEDGEMENTS I would like to thank Dr. Peter MacMillan, thesis supervisor, for all his expertise, guidance, and support throughout the process of writing this thesis. I would also like to thank committee members; Dr. Dennis Proctor, a methodological expert, and Dr. Jennifer Hyndman, a strong advocate for improvement in mathematics education. Special thanks also to Lyn Benn, Coordinator, Learning Skills Centre, and Dr. Lee Keener, former Chair of Mathematics, for their vision on all that Supplemental Instruction could be and their support of the Supplemental Instruction program. And, I would like extend a special thank-you to all the Supplemental Instruction leaders without whom SI would never have succeeded. V ll CHAPTER I - INTRODUCTION Introduction The search for effective methods of teaching mathematics, so that students are better able to master the concepts, ideas, and the language of the discipline, continues to challenge professional educators and researchers alike (Steele, 2001; Sfard, 2000; Chapman, 1995; Wells, 2001a). Attempts to improve mathematics achievement have fallen short of desired outcomes. The need to improve mathematical competency in students has led to extensive research in recent years and an examination of existing practices (Gee, 1996; Sfard, 2000; Steele, 2001; Wells, 2001a). Current trends in educational research point to the need to create discourse through social interaction as a means for improving mathematical knowing (Forman & Ansell, 2001; Gee, 1996; Steele, 2001; Wells, 2001b; Zack & Graves, 2001). In this paper 1 examine current theory on how mathematics may best be acquired and learned within a sociocultural context, through the use of “dialogue” (Gee, 1996; Vygotsky, 1978), and demonstrate that Supplemental Instruction (Center for Supplemental Instruction, 2000) provides a forum for creating discussion or dialogue. 1 then analyze the results of data collected throughout the three year duration of the Supplemental Instruction program offered in a Calculus course at the University of Northern British Columbia (UNBC). Based on the results of the data analysis 1 attempt to answer the question, “Does the Supplemental Instruction program improve mathematical knowing?” In particular, 1 analyze the final grades of students enrolled in the Calculus courses where Supplemental Instruction has been available. This question of improved mathematical achievement is of particular interest for the course. Calculus for Non-Majors, (Mathematics 152) that is being supported with Supplemental Instruction. The majority of students in the course are enrolled in Mathematics 152 because it is a program requirement. Forestry, Commerce, Economics, and Biology are examples of departments that require students take Mathematics 152 as part of their program. An informal poll, through a show of hands in one section of the course, indicates that only three out of one hundred and twenty enrolled students were there for reasons other than program requirements. Typically, many of the students enrolled in this course do not like mathematics, many suffer from mathematics anxiety, and many have fared poorly in their previous mathematics classes. Moreover, this course is notorious for its high failure rate, with 20 to 30% of the students failing every semester at this particular post-secondary institution, (Learning Skills Centre, 2001). Students often delay taking this course until late in their programs. Many students repeat the course several times before achieving an acceptable grade. However, it is important to recognize that the difficulties associated with this kind of course are not unique to this institution. Similar issues arise in calculus at many post-secondary schools across North America (Gee, 1996; Kenney & Kallsion, 1994; Sfard, 2001; Wells, 2001a). In an effort to alleviate the problems associated with calculus in general, and this course in particular. Supplemental Instruction (SI), an academic support program, has been piloted at this institution (Learning Skills Centre, 2001). The University of Northern British Columbia is a relatively new university (official opening 1994). It was established with the intention of providing accessible post-secondary education to people residing in northern British Columbia. Much of the student population is comprised of first generation students (students who are the first generation in their family to attend a post-secondary institution). Aboriginal students, and mature students. Some of the students attending UNBC are a combination of more than one of these groups. Therefore, analyzing the impact of SI is important to the University as it strives to ensure students are succeeding in their academic goals. If students do not succeed in courses such as Calculus for Non-Majors, they will be unable to complete program requirements and may drop out of University. Ultimately, the University will not be seen as succeeding in its mandate to support northern students. If the SI program can be shown to improve student grades in Calculus and to improve mathematical knowing, then the program is likely to receive appropriate funding with the possibility of expansion. It is extremely important to demonstrate the effectiveness of new programs as post-secondary institutions, especially in British Columbia, are told not only must they be accountable; they must do more with less. Furthermore, the Supplemental Instruction Program has already been extended to courses in Computer Science and Economics at UNBC. Thus, it is vital that a comprehensive and thorough analysis be performed on the new program, SI, to determine if it is significantly improving student grades. Analyzing the results of this program will also contribute to ongoing debate surrounding the impact of the SI program in historically difficult courses such as Calculus. There have been limited analyses of the SI program in Calculus since its development at the University of Missouri-Kansas City (UMKC) 30 years ago. Over the last 30 years, SI has expanded to countries around the world, including Australia, Canada, Great Britain, and South Africa (Center for Supplemental Instruction, 2000; Widmar, 1994) resulting in a number of research studies examining the effects of SI on student performance in a variety of disciplines (Center for Supplemental Instruction, 2000). Still, much of the research is limited in scope to programs implemented in American colleges and universities (Arendale, 1994,1999; Blanc, Debuhr, & Martin, 1983; Congos & Schoeps, 1998, Kenney & Kallison, 1994; Kochenour, Jolley, Kaup, Patrick, Raoch, & Wenzler, 1997). Few studies have focused on SI in Mathematics (Blanc et al. 1983; Burmeister, Kenney, & Nice, 1996; Kenney, 1989), and only one of these studies has focused extensively on the impact of SI in a Business Calculus course (Kenney, 1989). However, Kenney categorized SI participants as those students who attended at least 60% of the sessions and limited her study to one instructor, thus losing some generalizability. As such, it is possible that the influence of the instructor and/or the 60% attendance level criterion had a greater impact on the results than the SI program. On the other hand, Blanc et al. (1983) included all students who attend one or more sessions as an SI participant across several disciplines. This study differs from both Kenney (1989) and Blanc et al. (1983) in that the sample includes nine non-majors Calculus classes taught by four different instructors; the instructors all used the same grading procedures, textbooks, and format for assessment; and the students had attended five or more sessions (out of approximately 25 sessions offered by one SI leader) for categorization as an SI participant. In this study I attempt to answer several questions. Do students who participate in Supplemental Instruction earn higher course grades? Do they earn fewer D and F grades? Do the positive results shown in American colleges and universities extend to a small northern university in Canada? Do the results generalize to students enrolled in non-majors Calculus courses? In this thesis, I examine the effects of Supplemental Instruction in the course. Calculus for Non-Majors, offered over a three year period at UNBC, in an effort to answer these questions. Background The Supplemental Instruction program was initiated at UNBC in response to a growing demand for mathematics assistance. In 2001, the requests for mathematics help through the one-to-one tutoring program offered at the Learning Skills Centre (LSC) began to exceed the capabilities of the Centre. Several issues arose. Students were unable to schedule appointments with mathematics tutors in a timely manner. The time delay for seeing a tutor reached a week or more. The LSC lacked space to accommodate hiring additional tutors. Furthermore, hiring additional tutors posed considerable difficulty as there were few highly competent third and fourth year mathematics students available for tutoring. Excellent mathematics students are at premium at all small institutions and are in demand as teaching assistants for tutorials, as laboratory assistants for labs, and for marking. The shortage of qualified mathematics tutors created a serious problem for students requiring assistance in mathematics, where each new concept or topic tends to build on previous content. As a result of these issues, the LSC staff looked to other options for providing mathematics assistance. Several ideas were explored and presented to the Mathematics Program at UNBC. After careful consideration of the merits of Supplemental Instruction, re-establishment of a Mathematics Drop-in Centre, and provision of additional tutorials and targeted workshops, it was decided to pilot the Supplemental Instruction program in the Calculus for Non-Majors course as a large number of students from this course drew heavily on LSC services. By attempting small group, peer facilitated sessions in this Calculus class, it was hoped that more students could be served with less expenditure in terms of cost and human resources. Traditional tutorials had always been available for this course but were poorly attended by students except on occasion just prior to an exam. At this point, a one-hour tutorial was inadequate preparation for the majority of students who were looking for last minute assistance. Mathematics tutorials at UNBC, and many other post-secondary institutions, often have the following structure. Students come to the tutorial and pose questions related to the course. The tutorial assistant is expected to be able to answer student questions in a knowledgeable manner. Generally, the tutorial assistant solves the question posed by the student on the board. Most tutorial assistants, although knowledgeable with the content, receive little or no training in learning strategies. Frequently, the student at a tutorial session is merely a passive observer. Once the decision was made to pilot SI, the next step was to establish the program in the course. This process began by finding model students who had previously taken the course and earned an A or A+ in the course. These students went through a rigorous interview. Those who were finally selected were provided 10 - 15 hours of training. The training was carefully designed to ensure that the leaders have the essential tools to facilitate SI sessions in the manner prescribed by UMKC SI program guidelines. It was not the aim of this program to provide more tutorial time where the tutorial leader answered student questions on the board. The focus of the SI program was to have the students discussing questions and concepts and collaboratively problem solving, with students demonstrating solutions on the board. The SI Leader was there primarily to act as a guide. Supplemental Instruction is introduced early in the first two weeks of classes to provide proactive, rather than reactive, support. The LSC Supervisor and the SI Leaders work together to determine a schedule to provide three 50-minute sessions per week per leader in efforts to provide accessibility for all students enrolled. Scheduling difficulties do occur as it is difficult to accommodate all student schedules. In particular, the Calculus for Non-Majors course typically has students enrolled from a variety of disciplines and they may be in the first, second, third, or even fourth year of their programs. All reasonable attempts to schedule SI sessions so that all students were able to attend at least one session per week, have been made throughout the duration of SI at this institution, including offering evening sessions. However, one or two students each semester reported they were unable to attend any of the sessions. Once sessions commence, SI Leaders monitor student attendance through the use of a sign-in sheet. Students record their name on a new attendance sheet at the start of each month and tick a box for the current date. In keeping with SI guidelines, instructor support was obtained for all Calculus for Non-Major classes prior to implementation of the SI Program. All students were informed of the support available through the SI program and of other mathematics assistance available to them at the university. Other assistance included one-to-one tutoring services offered through the LSC, instructor office hours, and the traditional tutorials offered for this course. All services are free to all students. Tutorials were continued in the Calculus for Non-Majors course during the first three semesters of SI service. However, after three semesters, it became apparent that students were not attending the traditional tutorials even though excellent mathematical students were placed as tutorial assistants for Calculus for NonMajors. Attendance at tutorials was limited to two or three students. These students were provided the option of attending SI, booking an appointment with a one-to-one tutor in their small group format, or access to individual tutoring. Tutorials were not offered in subsequent semesters for this course. Beginning in the fall semester of 2003, assistance for this course was available through SI, one-to-one tutoring, Math Drop-In, and instructor office hours. As a result of practical experience at this institution and using SI guidelines, it was determined that scheduling one SI Leader for every 50 students enrolled in the course provided optimal coverage, while minimizing costs associated with having too many SI Leaders offering poorly attended sessions. Supervision was an essential component of this program. Even with an initial eight-hour training session in group facilitation methods and instruction in SI philosophy and guidelines, leaders can slip back into the familiar tutorial structure where they answer student questions at the board. Ongoing training and monitoring for approved SI practices was essential. SI practices are described in detail in the Literature Review following this chapter. The SI program at UNBC was built on the SI model guidelines as provided by the University of Missouri-Kansas City (UMKC), pioneers of the program. The UNBC Math/Stats Advisor (the current researcher) underwent a prescribed three-day training program at UMKC to ensure that correct practices were undertaken when establishing and monitoring the SI Program at UNBC. Summary analyses were compiled using descriptive statistics based on prescribed SI methods. These summary analyses were used in preliminary assessments of the program and are presented in Appendix B. Analyses were compiled for all sections of Mathematics 152 with SI support and indicated that SI participants were achieving improved final grades and substantial decreases in D and F grades associated with Mathematics 152. However, these analyses were limited to a comparison of final grades of students who participated in SI to those who chose not to participate in SI. The analyses did not provide the rigour necessary to thoroughly establish effectiveness. For example, the summary evaluations did not address factors such as each student’s natural academic ability. They did not provide information on withdrawal or retention of students enrolled in the course. It is the purpose of this study to look at contributing factors that impact final course grade, aspects such as ability and gender, to answer the principal question, “Does Supplemental Instruction improve the final course grades of students enrolled in Calculus for Non-Majors?” CHAPTER n - LITERATURE REVIEW Supplemental Instruction (SI) Supplemental Instruction was developed by Dr. Deanna C. Martin (1973) at the University of Missouri-Kansas City (UMKC) to improve the learning of students in historically difficult post-secondary courses (Center for Supplemental Instruction, 2000). Many academic support programs have been developed to assist students in first-year courses at the post-secondary level. In contrast, the SI program evolved because of a need for an academic assistance program for students enrolled in difficult courses in professional programs such as the School of Medicine, Dentistry, and Pharmacy at UMKC (Martin & Arendale, 1992). The students enrolled in these programs did not show any pre-disposing weaknesses when they enrolled. Most of these students had excellent academic records and scored well on college entrance exams (Martin & Arendale, 1992). However, many students in these programs were encountering academic difficulties with certain courses in the professional programs. The academic rigour of some of the courses exceeded the scholastic foundations of even these well-prepared students (Martin & Arendale, 1992). As a result, the SI program was developed in response to difficulties in high risk courses (courses that are traditionally very difficult), rather than targeting at-risk students. It has since been used extensively with a wide range of graduate, undergraduate, and professional school courses, and in a wide range of disciplines (Center for Supplemental Instruction, 2000; Martin & Arendale, 1992). The guiding principles of the SI program evolved as a result of collaborative learning theory and a need for improved practices that extended beyond study skills. Martin (1973) petitioned for a program that integrated reasoning and study skills with course content; not 10 isolated from it (Martin & Arendale, 1992). Consequently, the SI Program developed with the following guiding principles. Service is attached directly to a specific course. Reading, studying, and problem­ solving skills are offered in the context of the targeted, traditionally difficult course. Instruction in these skills is developed out of student questions and concerns as they occur within the course. Service is proactive rather than reactive. The SI Program is implemented in the first two weeks of class to provide assistance before students earn a critical D or F grade on an assignment or examination. Supplemental Instruction Leaders attend all classes fo r the targeted course. Both the SI leader and the student are hearing the same lecture, creating an immediate point of reference for the students and SI Leader. Furthermore, the SI leader is able to clarify what was said in the lecture, thus avoiding the common pitfall of student misconceptions on what occurred in lecture. The SI Leader, a student who has demonstrated superior academic achievement in the course, is provided with a timely review and often gains deeper insight into the course content upon hearing the concepts explained for a second time. The leader is also able to draw on his/her knowledge of the objectives of the course, thus creating an ideal learning environment for students attending the SI sessions as they strive for success in the course. Supplemental Instruction is not a remedial program. The program evolved as a means to improve student achievement in historically difficult courses. Many of the students attending the sessions are not underachievers or under-prepared. In fact, studies on affect and SI have pointed to the exact opposite. (The affective domain encompasses those II behaviours characterized by feelings, emotions, or values. Affect may be positive or negative; Sax, 1997.) Internal motivation is an integral component of students who participate in the SI Program (Visor, Johnson, & Cole, 1992). Supplemental Instruction Programs are designed to provide a high-degree o f student interaction and mutual support. Supplemental Instruction has relied upon the power of group study for over 30 years and is built on the practice of collaborative learning and interaction through peer study groups facilitated by a near-peer (Center for Supplemental Instruction, 2000). Near-peers are described as students who have previously taken the course but may only be a year or two ahead of the students in the course. The SI program incorporates cooperative/collaborative learning to integrate instruction in learning and reasoning skills with a review of the course content of selected courses (Center for Supplemental Instruction, 2000). The SI program at UNBC is patterned on this model. Participation in SI is voluntary, free-of-charge, and is open to all students in the course (Center for Supplemental Instruction, 2000; Learning Skills Centre, 2001). The SI sessions are structured as small-group, peer-led sessions aimed at improving student confidence and competence in the targeted course (Blanc, DeBuhr, & Martin, 1983; Center for Supplemental Instruction, 2000). Supplemental Instruction has been designed to target difficult courses rather than students who are doing poorly, thus creating an academic support program without a remedial image. Students at all levels of ability are encouraged to attend SI sessions as per UMKC guidelines (Center for Supplemental Instruction, 2000). One of the key elements of SI is extensive SI Leader training in group facilitation practices. For example, the SI Leaders are trained to use proactive and participative activities in the sessions such as ‘think, pair, share’ where students are encouraged to 12 brainstorm ideas, pair up with another student, and discuss their views or approaches to problem solving. The leaders are trained in questioning techniques based on Bloom’s taxonomy (1964). Bloom’s taxonomy of questioning begins with questions that assess knowledge (primarily recall of information such as formulas), comprehension (articulates and understands the meaning), application (primarily performing operations in mathematics), analysis (problem solving), and synthesis (combining concepts for a deeper understanding). Supplemental Instruction Leaders assess skills not only through questioning, but also through the development of quizzes based on Bloom’s taxonomy. These quizzes are not for marks, are often open book, and are generally completed in collaboration with other students. Quizzes provide students an opportunity to practice for tests, thus reducing the test anxiety that often accompanies mathematics tests and helps build confidence. The SI Leader draws on his/her previous knowledge of course goals and what is currently being discussed in lecture to prepare practice questions and tests. Supplemental Instruction Leaders implement strategies in sessions such as generating a table of contents, built on student input. These tables of contents assist students in summarizing the key concepts taught over a certain time period, perhaps to be tested in an upcoming test. Another strategy is to have students generate potential test questions, compile a quiz based on these questions, then do the quiz and discuss solutions. It is these practices that are currently forefront on theories of learning and reflect theory by creating opportunity for discourse in the language of the discipline (Chapman, 1995; Gee, 1996; Kenney & Kallsion, 1994; Linn & Kessel, 1996; Steele, 2001; Wells, 2001b). Many of the guiding principles of SI, although arising out of practice, have a solid theoretical grounding. 13 The SI Leaders at UNBC are model students who have previously attained an A+ or A in the targeted course. Potential SI Leaders are determined in cooperation with the relevant faculty members and are then selected after a rigorous interview process. The SI Leader is encouraged to maintain an active link with the faculty member, thus assisting in overall communications within the course between students, the instructor, and the SI facilitator. The SI Leader is not only selected for proficiency in course content, but also for personable qualities such as friendliness and a desire to help other students. These natural attributes are supplemented by an initial 10-15 hours of training, followed by regularly scheduled weekly or bi-weekly training in teaching and learning strategies (Center for Supplemental Instruction, 2000; Congos & Schoeps 1998, Learning Skills Centre, 2001). Supplemental Instruction leaders “praise participation to facilitate cooperation, information exchange, and verbalization among students” (Congos & Schoeps, 1998, p.6). Supplemental Instruction strategies, bom out of a need for improved learning practices in historically difficult post-secondary courses, have a sound theoretical basis underpinning the methods integral to the SI Program. Supplemental Instruction practices reflect evolving trends for improving learning through collaboration, discussion, and interaction, especially in mathematics. Studies have shown that SI has improved student achievement, most notable in the decrease of D and F letter grades and increased GPA among students who attend SI (Blanc, DeBuhr, & Martin, 1983; Burmeister, Kenney, & Nice, 1996; Center for Supplemental Instruction, 2000; Congos & Schoeps, 1998; Kenney, 1989; Kenney & Kallison, 1994). In 1981, and again in 1992, the U.S. Department of Education validated the Supplemental 14 Instruction Program as an Exemplary Educational Program (Martin & Arendale, 1992). The SI Program is one of only two programs that are officially recognized by the U.S. Department of Education as contributing to increasing student graduation rates (Martin & Arendale, 1992). Blanc, DeBuhr, and Martin (1983) were among the first researchers to examine the effectiveness of Supplemental Instruction and to conclude SI participants earned higher course grades. In their study, Blanc et al. analyzed the impact of SI offered in seven Arts and Sciences courses to 746 students in 1980. The first analysis examined final course grades with three groups, an SI participant group (students who attended one or more sessions), a non-participant group (students who opted not to attend), and a motivational control group (students who wished to attend but were unable to). Subsequent evaluations indicated that motivation alone did not account for improved final grades (Blanc, DeBuhr, & Martin, 1983). However, assignment to a motivational control group may not assure that selected students were truly motivated thus creating some limitations in the study. Blanc et al. also demonstrated significant improvements in reenrollment for the following two semesters. A total of 73.2 % of SI participants versus 60.0 % of non-participant reenrolled two semesters later. A further study undertaken by Kenney (1989) looked specifically at the impact of SI in two sections of Business Calculus, thus reducing the confounding factor of analysis across multiple disciplines. The Business Calculus course being supported through SI had consistently resulted in 30% of enrolled students earning D, F, and W grades. Kenney incorporated a control group for her study on SI impact. One section was provided support by a tutorial assistant (TA) using a content focus in sessions. The other section was 15 supported by an SI leader trained in SI methods. Kenney established stringent guidelines of 60% attendance at tutorials and 60% attendance at SI - sessions that were closely supervised for correct SI practices. These controls minimized the motivational factor that may occur with SI. There were 84 students in the control group; 51 of these met the criteria, and 83 students in the SI supported group; 50 of these met the criteria. Kenney obtained College Board SAT Verbal and prior Mathematical scores for all students and compared the two groups. Kenney demonstrated there were no significant differences in ability between the two groups. Kenney then analyzed the mean final course grades of two groups and found significant differences in final grades. The scale for grades was: A = 4, B = 3, C = 3, D = l, F = 0. The SI group earned a mean final grade of 3.0 and the control group earned a mean final grade of 2.43. Kenney followed up with more complex analyses to account for relationships between final grades and factors such as aptitude and prior mathematics achievement. Using multiple regression analysis, Kenney was able to establish that SI participation is a significant predictor of final course grades in the Business Calculus course. However, Kenney was the SI leader, resulting in a potential threat of experimenter bias in her research. Kenny and Kallison (1994) planned another series of investigations into the effects of SI in entry-level Calculus courses in efforts to improve upon and add to the research. They established similar controls to Kenney’s (1989) analysis but employed two different students to act as TA and SI leader. The SI leader underwent training in SI methods and the TA used traditional content-only focus. The same instructor taught both sections of the Business Calculus course leading to a eommon final exam. Kenney and Kallison reported that the students in the two different classes were equivalent with respect to ability and 16 mathematics achievement levels. They also reported findings that indicate a significant difference in final grades (2.39 vs. 1.96). Kenney and Kallison’s second investigation was virtually identical to the first but compared the performance of two classes of Engineering and Natural Science students. In this analysis, final grades were not significantly different. Kenney and Kallison conjectured that this result may due in part to the higher ability of engineering students. They completed further investigations to determine if lower ability students benefit more from SI intervention but encountered significant interactions limiting the scope of this study. A more recent study undertaken by Burmeister, Kenney, and Nice (1996) demonstrated that SI participants earned significantly improved final grades in all three of College Algebra, Calculus, and Statistics courses. Their research contained data obtained from 45 different institutions in 177 mathematics courses for a total of 11,252 students. They reported that SI participants earned higher mean final course grades and experienced lower rates of withdrawals: algebra (2.21 vs. 1.98); calculus (2.28 vs. 1.83); and statistics (2.49 vs. 2.32). The scale for grades was: A = 4, B = 3, C= 3, D = I, F = 0. A total of 3,631 students (32%) attended SI sessions with reported range of participation at sessions of 5% to 88%. Surprisingly, their study revealed that SI participants earned more D grades than expected but the rate of withdrawal from their respective courses was lower than their non­ participant counterparts. Burmeister et al. identify some limitations in their research. For example, two questions they posed are: How closely did each of the institutions follow the SI model? Are the groups of SI participants similar from campus to campus? Unanswered questions about SI indicate a need for further analysis of the SI program. 17 The Center for Supplemental Instruction has been monitoring the effectiveness of SI since its inception in 1973. The Center compiles and analyzes data submitted by over 100 College and University SI programs annually. A review of the research (2000) used a quasiexperimental design to conduct a longitudinal analysis of SI effectiveness both in courses at UMKC and in course data submitted by other institutions. Again, the scale for grades was: A = 4, B = 3, C = 3, D = 1, F = 0. In all analyses in this study, a student was categorized as a participant if they attended at least one SI session. Chi-square analyses and t-tests were used to determine SI significance for improving course grades, decreasing D, F and W (withdrawal) grades, and improving retention trends. The first analysis included data collected over a 19 year period, in 525 courses, for a total of 19,962 SI participants and 31,368 non-participants. Chi-square analyses demonstrated significant differences in A and B grades with a reported 54.4% of SI participants earning A and B grades in comparison to 42.9% of non-participants. Similarly, the Center reported significant decreases in D, F and W grades amongst SI participants (20.2% vs. 33.8%). The Center also established an overall significantly improved mean GPA value (2.70 vs. 2.43). The Center replicated the studies using the criteria of attendance at 5 or more sessions and concluded there is statistically significant improvement with these comparison measures that favour the SI participants. Similar results are reported by the Center for Supplemental Instruction (2000) on data collected from other institutions. The national data was provided by 270 institutions between 1982 and 1996, composed of 4,945 courses offering SI to over 500,000 students. In the first analysis, the courses were categorized as Business, Health Science, Humanities, Mathematics, Natural Science, and Social Science. The Center reported higher mean final course grades across all disciplines, a significantly higher percentage of A and B final 18 grades, and a lower percentage of D, F and W final course grades. There were 815 courses in the Mathematics category with significant increases in A and B grades and significant decreases in D, F and W grades but a non-significant improvement in mean final course grades (2.17 vs. 2.11). A third study looked at the national data by course. There were a total of 143 Calculus courses supported by SI with significant differences in increased A and B grades, significant decreases in D, F, and W grades, and a significantly improved final grade (2.26 vs. 2.06). Similar results were obtained for 219 College Algebra courses: increased A and B grades (36.4 % vs. 27.9 %), decreased D, F, and W grades (37.5 % vs. 52.7 %), and an improved mean final grade (2.20 vs. 1.91). Similar results were demonstrated for courses in Finite Mathematics and Statistics indicating SI is effective in Mathematics courses offered at a variety of institutions nation-wide. 19 Discourse - A Theoretical Framework for Supplemental Instruction “One does not have to be an educational researcher to agree with this: Mathematics is one of the most difficult school subjects” (Sfard, 2000, p. 1). Recent reform efforts by educators in the United States and Canada have attempted to move beyond direct instruction methods of teaching mathematics to the incorporation of discussion and meaning making in mathematics classrooms (Baxter & Williams, 1996; Chapman, 1995; Forman & Ansell, 2001; Wells, 2001b; Zack & Graves, 2001). Educators have made this move in response to societal need for individuals who are mathematically competent and able to contribute and achieve their full potential in our culture. To accomplish this goal, students must engage in activities in school that educate them in the values and practices that allow them to participate effectively in a democratic society (Wells, 2001a). They need to be given opportunity to develop personal initiative and responsibility, adaptable problem-posing and solving skills, and the ability to work collaboratively with others (Dewey, 1916). This need has led to the recent research examining current education practices and teaching methodologies. Two major learning theories have evolved during the course of this last century and have influenced current conceptions of knowing and coming to know (Wells, 2001a). The first of the emerging theories, Piaget’s theory of “constructivism”, challenged the idea that knowledge is passively acquired. The second theory, sometimes referred to as “social constructivism” (Wells, 2001a) is a Vygotskyan philosophy that argued learning occurs through social interaction in meaningful contexts (Vygotsky, 1978). Wells (2001a) examined Piaget’s theories of constructivism and ways of coming to know. Piaget, on the basis of numerous detailed observations and experiments with children. 20 proposed that the learner’s active, exploratory transactions with the environment gave rise to knowledge. In constructivist processes, if new material is compatible with what is known, then it is easily assimilated. On the other hand, if it is in conflict with what is known, the new knowledge will either be rejected or the existing knowledge will be transformed to accommodate the new (Wells, 2001a). Piaget’s theory led to an emphasis on discovery and supportive learning rather than directive learning in education (Wells, 2001a). Educators have moved beyond Piaget’s emphasis on cognition and discovery processes, and now are also concerned with the cultural context as proposed by Vygotsky (Bunch, 1995; Daniels, 1995; Gee, 1996; Steele, 2001; Wells, 2001a; Zack & Graves, 2001). Vygotsky (1978) placed emphasis on the importance of culture and social interaction in accounting for individual development. Vygotsky argued that through engagement in culturally valued activities, and with the aid of other participants and of the mediating artifacts that the culture makes available, we become who we are. In these particular events, we adapt, extend, and modify both intellectual and material resources in order to solve problems. In other words, individuals come to learn the meaning of the culture by internalizing the meanings and being transformed by them as they learn to speak the language of the culture (Steele, 2001). Vygotsky’s theory emphasized the need to develop communication and interaction for effective learning, and extends to mathematics education. If students are given opportunities to share their reasoning about ideas with others who in turn share their understanding, then culturally acceptable mathematics practices are established (Daniels, 2003; Gee, 1996; Steele, 2001). The SI Program was designed to provide an opportunity for models students, usually third and fourth year students, to share 21 their understanding of the course being supported through SI and to provide an opportunity for discussion of the course. Mathematics conversation can lead to a deeper understanding of the language of mathematics (Gallimore & Tharp, 1988; Gee, 1996; Steele, 2001; Vygotsky, 1976; Wells, 2001b). Through communication, ideas are reflected upon, refined, and remembered. As students learn to speak mathematical language they transform their thinking of the mathematical concepts. The mathematical language comes from the discourse of the culture, and the thought, or understanding of the concept, comes from the individual (Steele, 2001). Students, throughout their elementary and secondary school years, have traditionally been taught mathematics separate from other disciplines. Group work has seldom been encouraged in mathematics classrooms, and transmission-style teaching still tends to be the usual mode of instruction in the mathematics classroom. Teacher-centred instruction can provide the necessary skills to successfully complete a thematic unit such as functions. However, students rarely become competent in mathematical discourse through this type of instruction (Baxter & Williams, 1996; Chapman, 1995; Gee, 1996; Steele, 2001). The creation of mathematical knowledge can be improved by making meaning through processes of social interaction and language (Gee, 1996; Sfard, 2001; Vygotsky, 1976; Wells, 2001b). Gee (1996) defines discourse as composed of ways of talking and listening, acting and interacting, believing and valuing, and using the tools of the discourse to become part of a particular social identity. He also claims that discourses are mastered by enculturation into social practices associated with the discourse —lending itself well to Vygotskyian theory and to other theorists such as Sfard (2000), Linn & Kessel (1996), and Wells (2001a). For example, Sfard (2001) conceptualizes “knowing” mathematics as an ability to participate in 22 mathematical discourse. Linn and Kessel (1996) emphasize the need to provide social support for learners. They state that, “All learning takes place in a social context, so the goal is to structure social interactions to support all learners” (p. 127). Gee (1996) states that enculturation is best accomplished through scaffolded and supported interactions with people who have already mastered the discourse. Scaffolding is a Vygotskyan concept and can be described as the various types of support that teachers/near peers need to provide in the process of supporting students as they learn to think. Scaffolding can be accomplished through directions, suggestions, and meaning making (LeFrancois, 1997; Linn & Kessel, 1996). Linn and Kessel also assert that scaffolding is a method to increase student success in mathematics. Furthermore, it is most effective if it involves tasks within the learner’s zone of proximal growth (LeFrancois, 1997). The zone of proximal growth is another Vygotskyan concept. It is the state of the individual’s current potential for further intellectual development. Vygotsky believed that through the use of scaffolding, the individual may rise to further understanding. This may be accomplished through modeling, feedback, and dialogue (Gallimore & Tharp, 1988). However, mathematics continues to be taught using direct instruction methods, in isolation, with little or no connection to other disciplines. Very few people succeed in becoming conversant in mathematics and there is little opportunity for enculturation into the discourse of mathematics. The goal of the SI program is to create an informal but structured social environment where students are encouraged to discuss course content, clarify and refine ideas, and become conversant with the topics at hand (Center for Supplemental Instruction, 2000). Further, it is the role of the SI Leader to create and support student interactions in the SI sessions, thus providing the recommended scaffolding for learning mathematics. The SI 23 Program guidelines emphasize the need for practice in socially non-threatening environment where students can “safely” make mistakes, where open discussion is a means for clarifying concepts. Supplemental Instruction occurs without formal teaching, in a setting where students know they need to acquire the knowledge to do well in the particular course being supported and have the situation to do so. Current theories on improving mathematical knowing, built on models proposed by both Piaget and Vygotsky (Gee, 1996; Wells, 2001a), indicate that the SI program guidelines are established on a solid theoretical foundation. Supplemental Instruction provides an environment for creating discussion and meaning-making in a socio-cultural context. The SI program at UNBC was designed to provide an opportunity for models students to share their understanding of the course being supported and to provide an opportunity for discussion of the course. The SI leader, through facilitation, interaction, scaffolding, explanation, and breaking down of material into parts promotes learning in a socio-cultural context, similar to what Vygotsky, and recent theorists such as Wells (2001a) and Gee (1996), envisioned. In this study, 1 examine this form of academic support. Supplemental Instruction, to determine if it contributes to improved final course grades of students enrolled in mathematics courses, particularly in Calculus for Non-Majors, offered at the University of Northern British Columbia. 24 Problem Statement Supplemental Instruction requires close supervision and is costly to implement. As a result, there have been extensive administrative requests for proof of effectiveness. Preliminary analyses point to improved student grades in Calculus for Non-Majors (See Appendix B). However, a thorough investigation is essential before statements of impact can be made. This analysis is to determine the effects of the SI program on student final grades in the course. Calculus for Non-Majors. The information obtained will contribute to determining the viability of this program at this university. It will also contribute to the existing literature on the SI program by assessing its impact on students attending a small, northern university in Canada, many of whom are the first generation in their family to attend a post-secondary institution. Additionally, gender will be included in the analyses to determine if gender has an effect on who attends SI, as well as on final grades in the Calculus course. There is a widespread belief that males outperform females in mathematics, particularly in geometry, spatial mathematics, and problem-solving (Chipman, 2005; Kleinman, 1995; Randhawa, 1994). Since Calculus is comprised of all three of these concepts, gender needs to be considered as a factor. The effects of the program will be examined by answering the following research questions through a comprehensive statistical analysis (see Appendix D for a brief description of notation). Research Questions 1. “Do students who participate in Supplemental Instruction sessions, offered to a first-year Calculus for Non-Majors class, achieve higher final course grades than students who do not participate in the Supplemental Instruction sessions?” This question will be answered by considering the following sub-questions in the analysis. 25 a) Do SI participants earn significantly improved final course grades in comparison to non-participants? b) What are the effects of gender on final grades and SI participation? c) Is there significant interaction between gender and SI participation? H ypotheses • /^SlParl ~ a) nonpan ~ ^ nonpart ^ ^ ^ a 'f^SIPart The alternative hypothesis, , states there is a difference in mean final course grades of SI participants versus non-participants. ^o • ~Mom ~ ^ Ha -^GS -f^Gm ^ 0 The alternative hypothesis, H o • M si g ~ M s ! ~ M g M-- ~ H a - M s i g ~ M si ~ M g M- , states gender has an effect 0 ^0 The alternative hypothesis, H ^ , states there are significant interactions between SI participation and gender. 2. “Do any of the factors, SI participation, gender, or incoming GPA, contribute to prediction as to whether students will succeed in the course?” H ypotheses , Ho : bsipar, = 0 H a : b s iP a r t * ^ The alternative hypothesis, H ^ , states SI participation is a predictor of success The alternative hypothesis, H ^ , states gender is a predictor of success 26 . H o 'b ,„ c G P A = 0 H a • ^In cG P A ^^ The alternative hypothesis, , states incoming GPA is a predictor of success Limitations Two important considerations in this study are internal motivation and self-selection. Students attending SI have the internal motivation to access resources for improving their academic success. In addition, the voluntary nature of SI contributes to the issue of self­ selection. Certainly affect plays a role in which students will attend the SI sessions. As a result, students may be favourably or unfavourably predisposed to the SI program. This predisposition may be influenced by factors such as negative feelings to mathematics. Furthermore, a study on the relationship of SI to affect revealed that students who participate in SI are inherently different from those who elect not to participate or participate minimally (Visor, Johnson & Cole, 1992). However, this study is not aimed at answering the question of what role internal motivation and self-selection plays in SI. It has been designed to answer the questions, “Do those students who participate in SI have statistically significant improved final course grades in comparison to those who elect not to participate in SI?” and “Do any of SI participation, gender, or incoming GPA predict whether a student will succeed in the course. Calculus for Non-Majors?” 27 CHAPTER n i - METHODOLOGY Research Population The population of interest consisted of the students enrolled in the course, Calculus for Non-Majors, offered at this institute. Calculus for Non-Majors is a required course for many programs at UNBC and as a result, many students. Students who enrolled in the course were usually majoring in one of the following disciplines: Economics, Business Administration, Forestry, Bio-Chemistry, or Biology. They had varying backgrounds and were in different levels of their programs. For example, they may have been in any of first year to fourth year. However, the class composition semester after semester should typically be very similar in configuration and will be treated as such for the purposes of this study. As a result, factors such as socioeconomic class, age, and marital status will not be considered. The course. Calculus for Non-Majors, being provided with the Supplemental Instruction, was selected for its “gatekeeper” reputation. Gatekeeper courses often are large first year courses with high failure rates. Failure rates in this course are historically between 25 to 30% of the class (ESC, 2001). This figure does not include withdrawals or D grades that often preclude students from further courses in mathematics. Classes are generally large; student enrollment usually exceeds 1 0 0 students per section and may be as high as 160 students. Occasionally, enrollment in a section is low but the number of students registered rarely drops below 50. In addition, mathematics, and in particular Calculus, is considered to be the most difficult of disciplines for many students in education (Forman & Ansell, 2001; Sfard, 2000; Wells, 2001a; Zack & Graves, 2001). Instructors and SI Leaders are also an important part of the research population. The SI Leaders were most often selected on the basis of their success in the Calculus for Non- 28 Majors course as per SI guidelines (Center for Supplemental Instruction, 2000). As a result, only two of the fifteen SI Leaders in the program since commencement, were mathematics majors. The other leaders were drawn from Business, Biology, Forestry, and Biochemistry. The SI Leaders were generally third and fourth year students. Over the past four years, four different instructors taught the course. Two of these were part-time instructors; the other two were full-time mathematics instructors. Of these two, one was a lecturer; the other was an Associate Professor. Measures Final course grades are critical to this analysis as they form the dependent variable in this study. These grades were submitted by the instructor for the course for each offering to the Chair of the Mathematics Department. The Chair viewed and approved the final grades after which they were sent to the registrar’s office. Final grades were recorded as a letter grade with an assigned grade point for each letter as per UNBC calendar guidelines (Appendix E, Tables 1 and Table 2). These intact, normally submitted grades were used in the analysis. The second measure used in this analysis was the incoming grade point average (GPA) of students enrolling in Calculus for Non-Majors. Incoming GPA is vital for controlhng for the academic ability of each student. It was used as a covariate in the analysis. As expected, not all students have a recorded incoming UNBC GPA. For example, many first year students recently graduated from high school did not have a UNBC GPA. Similarly, newly-admitted transfer students did not have a UNBC GPA. To address this issue, another measure of student ability was used, a transfer GPA value. It is formed from either a college transfer GPA or an incoming overall high school percentage. A prioritized 29 system was employed with prior college GPA recorded over high school data. College transfer GPA’s are converted to the UNBC grading system at the registrar’s office when the student is admitted. These converted grades are being used in this analysis and provide optimal comparison to UNBC GPA values. High school percentages were converted to UNBC grade points as per UNBC calendar scales and were only used if the student had no UNBC or transfer GPA. Another measure being used in this research was a tally of the number of SI sessions each student attended throughout the semester. Procedures Data Collection Although there were four different instructors teaching Calculus for Non-Majors throughout the data collection period, they all calculated final grades based on the same grading system. All instructors used the following weighting scheme: 20% for each of three midterms and 40% for the final exam. They assigned similar exercise sets using the same textbook. Assignments were not collected for grades. The textbook changed in edition only over the four year period for which data has been gathered. There are no labs attached to the course. In semesters where there was more than one section of Calculus for Non-Majors, there was a common final. Thus, it was necessary that the course reach the same conclusion and cover the same content. Based on this information, it was concluded the final course grades are indicative of the student’s overall achievement in the course and were consistent across sections over the four year period. Final course grade information was provided on an Excel spreadsheet obtained via an electronic copy from the registrar’s office at UNBC. The incoming GPA measures were 30 obtained from the registrar’s office at the same time. This information was obtained after the Ethics Committee approved the research. In addition to the measures of final grade and incoming GPA, data was collected that counted the number of SI sessions each student attended for the semester of SI support in the course. The data consisted of a tally of attendance for each student by dates attended, maintained by the SI Leader. Attendance sheets were turned in monthly to the SI Supervisor. This method for monitoring attendance seemed to be the simplest, most efficient method that detracted the least from the session time. It is also the recommended practice taught in SI supervisor training at UMKC. The SI Supervisor counted the number of times each student attended SI and recorded this information. This information was stored with the final course grades. The attendance information was also used for preparing summary data (Appendix B). The summary data reports provided some preliminary information on the proportion of students attending SI, the number of sessions offered per semester, average attendance at sessions, and final grades earned by participants and non-participants. They also assisted in short-term planning for services at UNBC. Data was collected for a period of three years after implementation of the SI program beginning with the winter semester of 2002 and ending with the fall semester of 2004 for use in this study. Approximately 900 students, in nine sections, enrolled in the Calculus for Non-Majors course in this period and have recorded a final grade for the course. Of these about half, or 450 students, attended at least one SI session. An additional two semesters of data, comprised of approximately 400 students in three sections prior to SI implementation, were collected and used as controls in the evaluation of the SI program for a total of just under 1300 students. 31 Supplemental Instruction sessions were scheduled so that all students could attend at least one session per week. Eight incidences of inability to attend sessions were reported by students. Specific Procedures Students take this course as part of their program requirements. Therefore, randomization was not possible. Further, it would be unethical to withhold Supplemental Instruction from students to create a control group. Thus, the inability to randomize, combined with the use of intact classes, resulted in a quasi-experimental design (Mauch & Birch, 1998; McMillan & Wergin, 2002). SPSS (1975), a statistical analysis software program, was utilized to examine differences in two groups, SI participants and non­ participants. Students were categorized as either SI participants or non-participants and were further identified as female or male. The dependent variable was the final course grade. A two-factor analysis of covariance (ANCOVA) was performed to determine the extent to which reliable mean differences on the dependent variable, final course grade, was associated with group membership. Incoming GPA, either UNBC GPA or transfer GPA, was used as a covariate to control for internal ability. The effects of gender and SI participation were examined for significant interaction. No significant interactions were noted. A two-tailed procedure was used to improve robustness of the ANCOVA. A further analysis, binary logistic regression, was performed to examine if any of the factors, SI participation, gender, and incoming GPA, predict which students succeed and are retained in the course. Logistic regression was selected for this analysis because it allows one to predict a discrete outcome such as group membership from a set of variables that are 32 discrete, continuous or dichotomous, or a mix (Tabachnick & Fidell, 2001) such as occur in this analysis. Ethics Ethics approval was acquired through the University of Northern British Columbia to research the effects of the SI program for students enrolled in Calculus for Non- Majors at the onset of the program, in February 2002. This process was initiated prior to a completed research proposal since results of this analysis were not only important for improving mathematics education practices but also for institutional planning. Confidentiality was maintained on all data gathered throughout the duration of this project as per ethics guidelines. Students were provided the option of attending SI sessions and were encouraged to take advantage of the many supports provided with the course. These included not only Supplemental Instruction sessions, but also one-to-one tutoring, mathematics drop-in times, and the office hours of the instructor. Copies of the Ethics Approval forms are in Appendix A. 33 CHAPTER IV - RESULTS Preliminary Analyses Data Cleaning and Screening The five variables used in the analyses were thoroughly examined in a data cleaning and screening process. The criterion variable for the analyses are the final grades (obtained as letter grades) assigned to students enrolled in the Calculus for Non-Majors course, collected for the three year period of SI implementation (2002 - 2004) and for one year prior to implementation (2001). Other variables included incoming UNBC grades, college transfer grades, high school percentage grades, and the number of Supplemental Instruction sessions attended. The data were examined for out of range values, plausible means and standard deviations, missing data, univariate and multivariate outliers, and normality. Three univariate outliers were noted; all three were retained in the analyses reported here. Two of the outliers were verified as accurate; one was determined to be entry error and was corrected. Data for one additional multivariate outlier was examined. Since the student withdrew from the course and did not attend SI, the case was deleted. There were 27 cases with no recorded value for incoming GPA of any type. Eleven of these students attended SI. Since this represents 2.1% of the data collected, the researcher determined that deleting these cases would not significantly impact the analyses. The final sample contained data for 1259 students. An examination of the distributions of each variable indicated some positive skew was present in the final grades variable. SPSS tests for normality (Kolmogorov-Smimov) also indicated the final grades variable was non-normal. A skewed distribution is typical for grades, particularly in this course. A transformation, the square root of the final grade. 34 improved the skew. Preliminary analyses were performed on both the non-transformed and transformed final grade variables. Interpretation of results obtained for each are provided in the results chapter. Examination of the other variables gave no indication of non-normality. Selection and Criterion Variables used in the Preliminary Analyses The preliminary analyses were performed on two variables, the final course grades (criterion variable) and level of SI treatment (selection variable). The letter grades were converted to numerical values for use in the analyses. For example, a value of 1 was assigned to a withdrawal grade; a value of 2 was assigned to an F grade; up to a value of 12 assigned for an A+ grade (see Appendix E, Table 2). The use of letter grades resulted in a dependent variable that may be considered discrete. However, Tabachnick and Fidell state, “Sometimes discrete variables are used in multivariate analyses in place of continuous ones if there are numerous categories and the categories represent a quantitative attribute” (p. 6 , 2001). For example, a variable may represent age categories, where say 1 stands for 0 to 4 years, 2 stands for 5 to 9 years, and so on up through the normal age span (Tabachnick & Fidell, 2001). In the case of letter grades, an A+ represents a grade of 90 - 100%, an A represents a grade of 85 - 89.9%, and so on. Tabachnick and Fidell also state, “In practice, we often treat variables as if they are continuous when the underlying scale is thought to be continuous but the measured scale is actually ordinal, the number of categories is large— say seven or more—and the data meet other assumptions of the analysis” (Tabachnick & Fidell, 2001, p.7). Consequently, the final grades variable will be treated as a continuous variable for the ANOVA in the preliminary analyses and the ANCOVA in the comprehensive analyses. 35 The independent variable, SI participation, was categorized into three different levels of treatment. Final grades for 390 students taking Calculus for Non-Majors prior to SI implementation were placed in the first category. This category was coded as 0 for level of treatment. These 390 students had all taken the course the year prior to SI implementation, thus forming a pre-treatment, or “pre-SI”, group (0). The remaining 869 students in the sample who had the option of participating in SI were further partitioned as follows: students who participated in 0 to 4 sessions were coded as I for level of treatment, while students who attended 5 or more SI sessions were coded as 2 for level of treatment. The choice of 5 sessions for level 2 was chosen by the researcher for three reasons. First, there were large observed frequencies for students who attended less than three sessions (n = 118). Second, students will often attend a review session prior to midterms and the final exam. Since there were three midterms and a final exam, one could expect students to attend four review sessions prior to exams and still not fit the category of “SI participant”. By attending five or more sessions, a student could be assumed to be an SI participant. Third, five sessions are approximately 25% of offered sessions for one SI Leader per semester. As a result, 269 students were categorized as SI participants (2). The 600 remaining students were categorized as non-participants (I). Of these 600 students, 468 opted to not attend any sessions at all. Mean final grades and standard deviations were obtained for each group and are summarized in Table 3. 36 Table 3 Means and Standard Deviations SI treatment Number Pre-SI (0) Non-participant (1) SI participant (2) Standard deviation 3.4 Variance 390 Mean final grade 4.9 600 5.4 3.7 13.42 269 6.9 3.5 12.60 11.63 The distributions of both variables, final grades and level of SI treatment, were examined again. As noted previously, the dependent variable, final grades, had a statistically significant skew (z = 6.77). Taking the square root of the final grades resulted in an improved variable, one that no longer had a statistically significant skew (z = 2.38) However, transformations create difficulties in interpretation of results. Given that ANOVA is robust to some departures from normality and the sample size is large (N = 1259) the researcher examined both the transformed and non-transformed variable in the following ANOVA and ANCOVA investigations to determine if the skew was impacting results. First analysis: One-Way Analysis o f Variance (ANOVA) with post-hoc tests An ANOVA F-test evaluates whether the group means on the dependent variable differentiates individuals across treatments (Green & Salkind, 2003; Tabachnick & Fidell, 2001). In particular, the ANOVA was used to assess whether means of the final grades are significantly different among the three different levels of SI treatment. An ANOVA was performed first on the non-transformed final grades data and the three SI participant categories as described above. The F ratio was significant, -^2,1257 ~ 26.779, with resultant p-value < 0.0005, indicating significant differences in final grades across SI participant 37 groups. Post-hoc tests were then carried out to determine where significant differences occurred. Post-hoc tests for ANOVA differ depending on whether the independent variable has equal or non-equal variances. The standard deviations for the independent variable, SI categories, were squared to obtain variances. The variance values were 11.63,13.42, and 12.60, accounting for a difference in variance of less than 2. An and yielded a value of 1.15. ( ratio was examined is the ratio of the largest cell variance to the smallest cell variance.) The F^^ corroborated equality of variance (Tabachnick & Fidell, 2001). In contrast, Levene’s test of equality of variance (a value obtained in the SPSS analysis) was significant, pointing to unequal variance. However, Levene’s is a sensitive test, particularly when the sample size is large. In light of the two differing results for equality of variance, the researcher examined both Dunnet’s C and Tukey’s post hoc tests on the non-transformed final grade variable. The results of both post hoc tests were virtually identical with significant differences (p < .0005) occurring between the pre-SI group (0) and the 81 participant group (2 ) and between the non-participant group ( 1 ) and the 81 participant group (2). No significant differences occurred between the pre-81 group (0) and the non-participant group (1). In other words, students who attended less than 5 81 sessions showed similar trends in their final grades as students who did not have the option of 81. On the other hand, the 81 participant group, those who attended 5 or more 81 sessions, showed significantly better final course grades. Next, the same procedures were carried out on the transformed final grades. The square root transformation is generally the most acceptable transformation to perform on a positively skewed variable although several other transformations were attempted in efforts 38 to obtain the most-improved distribution. Analyses carried out on the transformed final grade variable yielded comparable results to the analyses on the non-transformed variable: . ^ 2,1257 = 28.650, p-value < 0.0005, with statistically significant differences occurring 0 05 between the pre-SI group (0) and the SI participant group (2), and between non-participant group (1) and the SI participant group (2). For ease of interpretation, the non-transformed variable was retained. Second analysis: Two-Way Contingency Table Analysis Using Crosstabs, based on a ^ distribution, with follow-up post-hoc tests A chi-square test ( ^ ) is a nonparametrie test used for analyses that include one or more variables measured on a nominal or ordinal scale. Given the positive skew in the final grade distribution, resulting in some measure due to the ordinal nature of final grades, the researcher deemed it worthwhile to further examine the data using the chi-square test. (See Appendix C for additional explanation on the Chi-square analysis.) A two-way contingency table was selected to evaluate the relationship between the two variables, final grades and level of SI treatment. For this analysis, final grades were partitioned to form two groups. The first column included students who have withdrawn from the course or earned a D or F grade. A D grade often precludes a student from taking a requisite course and was included in this group which can be seen as a “no success” or “failure” group. The second column included all students earning a C- to an A+ grade and was deemed as “success” in the course. The first category, DFW, contained 44.6% of the sample (562 students); the remaining 55.4% (697 students) were in the second category. The following contingency table illustrates the observed and expected frequencies for each level of treatment. 39 Table 4 SI treatment* Success/Failure Crosstabulation SI treatment Pre-treatment (0) Count Observed Expected % Failure 205 174 52.6 Success 185 216 47.4 Total 390 390 Non-participant (1) Observed Expected % 283 268 317 332 600 600 47.2 52.8 Observed Expected % 74 120 27.5 195 149 72.5 SI participant (2) 269 269 The contingency table demonstrated a marked increase in successful grades (72.5%) and a marked decrease in DFW grades (27.5%) in the SI participant group. The Pearson chi-square test was significant, .0 5 ^ 2 = 43.403, p-value < 0.0005, indicating that differences among the groups were not due to chance variation. Strength of relationship values were inspected to determine effect size. Phi values range from -1 to +1 with values close to 0 indicating a weak relationship. The phi value obtained was 0.186 with p-value < 0.0005, indicating a practically significant relationship between SI treatment and success in the course. Post-hoc tests were performed to determine where the significant differences occur, and summarized in the following table. Table 5 Post hoc tests fo r Chi-square Analysis SI Treatment (i) Pre-treatment (0) Pre-treatment (0) Non-participant (1) SI Treatment (j) Non-participant (1) SI participant (2) SI participant (2) P (i,.j) .097 .000 .000 40 There were significant differences between the SI participant group (2) and the pre­ treatment group (0), and there were significant differences between the SI participant group (2) and the non-participant group (1). There were no significant differences between students in the non-participant group (1) and students in the pre-treatment group (0). The SI participant group earned significantly fewer DFW grades (expected I I I , observed 74) and significantly more “successful” grades (expected 159, observed 195). Based on the results of the ANOVA and the Chi-square Two-Way Contingency Table, the researcher determined there was no evidence to differentiate the two levels of SI treatment, (0) and (I). This information also speaks to the selection bias often associated with a voluntary treatment such as SI. One can assume there are equally motivated students enrolled in the course the year prior to SI implementation, yet both ANOVA and analyses point to significantly improved grades of students who have the benefit of Supplemental Instruction. Consequently, for the purposes of the following analyses, the pre­ treatment (0 ) and the non-participant group ( 1 ) categories were combined, resulting in two levels of SI treatment. The first SI treatment group (1) included students from the pre­ treatment group and the students who attended less than 5 SI sessions. The second category, the SI participant group (2) remained the same. This category continued to include students who attended 5 or more SI sessions offered throughout the semester. 41 Analysis of Relationships between Final Grades, Supplemental Instruction, Incoming Grades and Gender The previous analyses were very simplistic and did not take in to account other relationships that occur between the dependent variable and other independent factors such as gender, prior mathematics achievement, or aptitude. The following analyses include gender and incoming grade point average to provide a more complete method of analysis to respond to the research questions posed in Chapter 2. Additional Variables - Gender and Incoming Grade Point Average (GPA) Gender was added as a factor for the following analyses in order to determine any effect gender may have on final grades and in efforts to eliminate any confounding that gender may add to the analyses. Typically, the expectation is that males outperform females in Mathematics (Chipman, 2005; Randhawa, 1994). Gender is equally split within the SI categories. However, females appear to be achieving higher mean final grades in the course in both categories. The following table summarizes descriptive information. Table 6 Mean Final Grades by Gender and SI Treatment SI treatment Gender Number Nonparticipant Overall SI participant group Overall M F 509 481 990 135 134 269 M F Mean final grade 4.75 5.64 5.18 6.59 7.19 6.89 Standard deviation 3.40 3.70 3^ 8 3.49 3.59 3.55 In addition, a covariate variable was included for the purpose of adjusting for student ability. The covariate was the student’s incoming GPA. Of the 1259 students in the sample, 951 students had a recorded UNBC incoming GPA. Transfer GPA was also examined. (All college transfer GPA values are converted to an equivalent UNBC GPA as part of the 42 admission process at UNBC.) Of the 1259 students, 495 students had a transfer GPA. A further 633 students in the sample had incoming grades based on high school percentages. High school percentages were converted to the UNBC scale and included in the transfer GPA variable with priority assigned to the college transfer GPA. Unfortunately, including even two values for incoming GPA resulted in a loss of 35% of the data when performing ANCOVA and logistic regression. To minimize data loss, it was necessary to create a single covariate for incoming GPA. Prior to creating a single incoming GPA covariate several exploratory analyses were performed. The first test was to ensure that the assumption of homogeneity of slopes was met. An ANOVA was used to test the interaction between the covariate and the SI factor to ensure no significant interaction between the covariate and the factor were occurring. Both incoming values, UNBC GPA and transfer GPA, were examined through ANOVA with no significant interactions observed. Thus, the assumption of homogeneity of slopes was tenable. Second, an analysis of Pearson r correlations was examined. Results clearly indicated a strong correlation between final course grades and the UNBC incoming GPA (r = .516). The transfer GPA, although significantly correlated to the final course grades, had a lower correlation (r = .273). The correlation between UNBC incoming grades and transfer grades was .387. Tabachnick and Fidel (2001) suggest that highly correlated covariates should be eliminated, supporting formation of one incoming GPA variable for use as a covariate. As a final test, both covariates were tested separately for significance on final grades. Results of ANOVA were significant for both ( 0 5 -^ 1,945 = 288.7 and 0 5 -^ 1,1121 = 84.3 for UNBC incoming GPA and transfer GPA respectively). Based on the preceding analyses, the 43 researcher formed one variable for the covariate. Because of the high correlation between final grades and UNBC incoming GPA, and a larger portion of sample (approximately 75%) having a UNBC GPA; priority was given to the UNBC incoming GPA. For the remaining 308 students, transfer GPA was used. Another test for homogeneity of slopes revealed no significant interaction between the SI treatment factor and the newly-formed covariate variable for incoming GPA. Third Analysis: Analysis o f Covariance (ANCOVA) In ANCOVA, group means on the dependent variable are adjusted through use of a covariate to what they would be if all subjects scored equally on the covariate. Use of a covariate increases the sensitivity of the test of main effects and interactions by reducing the error term (see Appendix C for additional explanation on ANCOVA). In efforts to determine the impact of SI on final grades, an analysis of covariance was carried out to answer the question, “Do the adjusted group means differ significantly from each other over the levels of treatment?” Assumptions for ANCOVA include a normally distributed dependent variable. As discussed previously, the dependent variable, final grades, is positively skewed and analyses were performed on both the transformed and non-transformed variable. Another criteria for ANCOVA is homogeneity of variance (variability in scores for one variable is roughly the same at all values of another continuous variable; Tabachnick & Fidell, 2001). In the case of ANCOVA, the covariance was evaluated for homogeneity. Using the two-category SI factor and the combined incoming grades variable as a covariate, an ANCOVA was executed on the non-transformed final grade variable. Table 9 summarizes the ANCOVA results. 44 Table 7 ANCOVA Summary (non-transformedfinal grade) N = 1259 Source df Incoming GPA SI treatment Gender SI treatment /Gender interaction 1 1 1 1 Mean Square Error 3186.776 520.125 74.929 10.107 F P 316.897 51.722 7.451 1.005 .000 .000 .006 .316 .202 .040 .006 .001 The results of the ANCOVA were statistically significant for SI ( = 51.722, p-value < .0005). Gender was also statistically significant ( 0 5 ^ 1,1254 ~ 7.451, p < .01). No significant interaction occurred. Levene’s Test of Equality of Error Variances resulted in F3 = 3.577 with p-value = .014, not significant at the customary 0.01 level. Homogeneity of variance was also assessed using an ratio test. The ratio for the largest to the smallest variance is approximately 1 .2 , justifying the assumption of homogeneity of variance and the use of ANCOVA to analyze the impact of SI on final grades. Next, an ANCOVA was performed on the transformed final grade variable. As in the case of ANOVA in the preliminary analyses, the transformed grades resulted in very similar outcomes. SI participation was a statistically significant factor ( ^^Fj .0005). Gender was also significant ( ^^Fj Error Variances resulted in = 56.128, p < = 6 .6 8 6 , p = .01). Levene’s Test of Equality of = 1.949 and p-value = 0.120. |Zi - x J Practical significance was examined using Cohen’s d where d =---------- ', a measure <7 of effect size (Hurlburt, 1998). The overall effect size was .48, indicating the mean final grade of the SI participant group was approximately half a standard deviation higher than 45 the non-participant group. Given an overall standard deviation of 3.6, this indicated that SI participants were earning 1.8 letter grades higher. For example, a student opting to participate in SI might be expected to earn a C rather than the D grade they may have achieved without the benefit of SI; or an A rather than a B+. One can conclude, after holding constant differences in ability, SI participation was a practical and significant factor in final grades. A further breakdown into male and female final grade means suggested males may benefit more than females. Observed Cohen’s d value for males was .54; for females .42. However, an examination of effect size based on gender disclosed a Cohen’s d value of .23, a small effect size, indicating there was less than one letter grade difference between males and females. An examination of the partial eta squared (//^ ) in Table 9 also suggested gender may not be practically significant (effect size close to 0 in both ANCOVA analyses). Fourth Analysis: Binary Logistic Regression Logistic regression is a form of regression that is relatively free of restrictions, and can be applied to determine any, or all, of the following. It can be used to predict the category of outcome on the basis of the independent variables; to determine the percent of variance explained by the independent variables; to rank the relative importance of independent variables; to assess interaction effects; and to assess the significance of any covariate variables. Predictors may be dichotomous, discrete, continuous, or a mix (Tabachnik & Fidell, 2001; Garson, 2005). Binary logistic regression is applied when the dependent variable is dichotomous. By performing this analysis, the researcher was attempting to quantify the contribution of the independent variables to the final grades and to develop an equation to predict the category of outcome, success or failure. 46 In logistic regression, the dependent variable is transformed into a logit variable. As a result, the model produced is nonlinear, with slightly more complex equations used to describe the outcome (Tabachnik & Fidell, 2001). The outcome variable, Y., is the probability of having one of the outcomes based on a nonlinear function of the best linear combination of predictors: K ' where l + e“ is the estimated probability that the ith case (i = 1 , 2 ,...., k) is in one of the categories and u is the linear regression equation: M= with constant +^2 ^2 , coefficients B j, and predictors, X j for k predictors (j = 1 , 2 , ..., k). This linear regression equation creates the logit or natural log of the odds: In That is, the linear regression equation is the natural log of the probability of being in one group divided by the probability of being in the other group (Tabachnick & Fidell, 2001). For this analysis, the same two outcome categories were used as in the Chi-Square analysis, success or failure in the course. Incoming GPA was used as the covariate for the analysis. Gender was included as a predictor along with SI treatment. Analysis on a full model, one that includes all predictors, was statistically significant (%^^ = 224.120 , p < .0005). This analysis was followed up by a sequential logistic regression. In sequential logistic regression, the researcher specifies the order of entry of the predictors into the model (Tabachnick & Fidell, 2001). This procedure allows evaluation of particular factors in terms 47 of their contribution to prediction. For this analysis, both incoming GPA and gender were included in the first model. Supplemental Instruction participation was added to the second model. The difference in the two models was then compared to determine if SI participation contributed significantly to prediction. The following table highlights the results of the two models. Table 8 Sequential Binary Logistic Regression Models Chi-Square Model df P ( % ') -2 Log likelihood ( % ') Model 1 GPA, Gender 182.931 2 .000 1547.910 Model 2 GPA, Gender, SI 224.120 3 .000 1506.721 An examination of the table revealed a significant difference in -2 Log likelihood Chi-square (,0 5 = 41.189) when SI participation was added to Model 1, confirming that SI participation was a statistically significant contributor to prediction of final grades. Next the Goodness of fit statistic was examined. This statistic determines whether the model adequately describes the data. The Hosmer-Lemeshow test is the recommended test for Goodness of fit statistics (Tabachnick & Fidell, 2001). It indicates a poor fit if the significance value is less than .05. In this example, the goodness of fit value was .447, indicating the model was a good fit for the data. The Nagelkerke R Square value (between 0 and 1 ) is a pseudo r-squared statistic and estimates the percent of variance explained by the model. The final model resulted in a value of r^ = 0.218, indicating that SI and incoming GPA explained approximately 22% of the variation in final grades in Mathematics 152. The following table summarizes the results for the variables in the equation of prediction. 48 Table 9 Variables in the Equation Full Model SI GPA gender Constant B .992 1.103 .146 -4.157 S.E. .160 .092 .125 .364 Wald 38.333 144.089 1.365 130.205 df 1 1 1 1 Sig. .000 .000 .243 .000 Exp (B) 2.696 3.014 1.157 .016 Examination of the statistics in Table 12 provided the following information. Supplemental Instruction treatment has a Wald ( ^) value of 38.333 (p< 0.0005) and incoming GPA has a Wald (.osZ^i ) value of 144.089 (p < 0.0005). The addition of SI was significant leading to the conclusion that SI treatment was a valid predictor of success in the course. Gender was found be to not significant (Wald ( i ) = 1-365, p = .243) and as a result, was not included in the prediction equation since it failed to contribute significantly. Using Table 12, the following equation was obtained: = -4.157+ .992%! + 1.103X2 In where X, represents the SI variable and X^ represents the incoming GPA variable. This produced the following equation for Ÿ., where represents the probability of having one outcome or the other: ^ -4 .1 5 7 + .9 9 2 X ,+ 1.103X 2 Y: =■1 , ^ - 4 .1 5 7 + .9 9 2 X ,+ 1.103X 2 ' I l+e It can easily be shown that g ^ = 2.696, the EXP (B), or , coefficient of the SI variable, X j, in Table II. Similarly, the coefficient for incoming GPA was 3.014. These coefficients represent the ratio change in the odds of the event of interest for a one-unit change in the predietor. For example, the odds of a person succeeding were 2.696 times 49 greater as a result of SI attendance. The coefficient for incoming GPA (3.014) substantiated the use of incoming GPA as a covariate for the analyses. The practical results of using the model are its ability to predict outcome. The final model predicted failure correctly 61% of the time, success correctly 74% of the time, for an overall 6 8 % correct prediction of outcome. 50 CHAPTER V - DISCUSSION Discussion In summary, both the ANOVA and ^ analyses point to statistically significant differences in the final course grades of SI participants. In particular, the analysis provided information on the expected grades and observed grades, with a significant increase in successful grades (C- to A+ grades, 72.5% vs. 47%) and a significant decrease in D, F, and W grades (27.5% vs. 53%) among the SI participant group. Furthermore, one can attribute the significant improvements in grades among SI participants to more than motivational differences among students, thus minimizing the self-selection bias often associated with a voluntary treatment such as SI. Undoubtedly, there are equally motivated students enrolled in the course the year prior to SI implementation, students who would seek out assistance through one-to-one tutoring and attend tutorials, yet there were significant improvements in grades among SI participants. The results of additional investigations that incorporate incoming GPA as a covariate and gender as a factor also validate the significance of SI for improving final grades in the course. In response to the first research question, “Do students who participate in Supplemental Instruction sessions, offered to a first-year Calculus for Non-Majors class, achieve higher final course grades than students who do not participate in the Supplemental Instruction sessions?” and corresponding hypotheses, evidence of statistical and practical significant differences in final grades were observed in the SI participant group in the ANCOVA analysis. The mean final course grade for SI participants was a B-. The mean final course grade of non-participants was a C. In other words, after controlling for student ability, SI participants achieved two grades higher than the non-participant students enrolled 51 in the course. These results are comparable to results obtained by Burmeister et al. (1996) who report SI participant grades of 2.28 vs. 1.83 (or C vs. a D grade) in their study of SI in Calculus. Gender was not found to be a practically significant factor in ANCOVA. Nonetheless, gender was found to be statistically significant (p < .01) with females achieving a mean final grade nearly one full grade higher than their male counterparts. This is in sharp contrast to extensive research undertaken by the U.S. National Institute of Education (Chipman, 2005) that points to superior male performance in mathematics. However, more recent research (Gallagher & Kaufman, 2005) suggests that there are minimal cognitive differences between genders in mathematics achievement. Furthermore, most of the differences previously determined may be largely due to affective factors (Gallagher & Kaufman, 2005; Kleinman, 1995; Randhawa, 1994). Girls are stereotypically described as not liking mathematics and not doing well in mathematics. They have traditionally been discouraged from pursuing mathematics-related careers and have been described as lacking self-confidence in mathematics. Gallagher & Kaufman (2005), Kleinman (1995) and Ranhawa (1994) all suggest that creating learning environments where females are encouraged, supported, and motivated would eliminate some of the discrepancies in achievement noted in previous research. The results observed in the analyses support these conjectures. Supplemental Instruction leaders were trained in scaffolding techniques that emphasize process-related understanding through discussion and dialogue in a supportive environment. This type of support structure may have contributed to the successful final grades observed in females in this study. 52 Males experienced a slightly larger improvement in final grades than females (Cohen’s d of .54 vs. Cohen’s d of .42) as a result of SI participation. Males and females participated equally in SI (N = 135 for males; N = 134 for females) suggesting that both genders benefited from SI treatment. There were no significant interactions occurring between SI participation and gender. Given the theoretical underpinning of the SI program and the resultant outcomes generated through inclusion and analysis of gender, it is tenable to conclude that females were not outperformed in the course. Calculus for Non-Majors, and that both genders benefited from SI participation. In response to the second research question, “Do any of the factors, SI participation, gender, or incoming GPA, contribute to prediction as to whether students will succeed in the course?” and corresponding hypothesis, the sequential binary logistic regression results indicated SI participation is a significant contributor to success in the course. The SI participant group also earned fewer D and F grades, addressing the issue of retention. If students are earning fewer D and F grades, then one can assume that they are apt to persist in their studies (Center for Supplemental Instruction, 2000; Burmeister et al. 1996). Gender was not found to be significant predictor of success in the course and was not included in the final prediction equation. Incoming GPA was the strongest predictor of success in the course. Given the results obtained in all the analyses, one can conclude that SI participation is having a positive impact on student final grades in the course, Calculus for Non-Majors, and ultimately improving mathematical knowing among SI participants enrolled in the course. These results validate current theory (Wells, 2001b; Gee, 1996; Pinker, 1994) on acquisition of mathematical knowing through use of scaffolded discourse in a meaningful 53 socio-cultural context. There appears to be something intrinsically different between Supplemental Instruction and tutorials. If there were not, one would expect students would not attend SI sessions just as they have not attended tutorials offered for this course in the past. The results also converge with earlier studies performed by Kenney (1989) on SI offered in a Business Calculus course, and by Burmeister et al. (1996) on SI offered in select mathematics courses (including courses in College Algebra, Statistics, and Calculus). Kenney (1989) used similar methods (t-tests and linear regression) to explore the impact of SI in two business calculus classes and was able to conclude that SI improved final course grades. Burmeister et al. (1996) used t-tests and Chi-Square analysis to conclude SI participants show improved grades and are more likely to remain on track toward their goals. Given the decreased D, F, and W grades observed in the analyses undertaken by the researcher, it is also reasonable to expect improved persistence and retention among students who participate in SI. Limitations Some methodological limitations of this study include the use of final letter grades for the dependent variable and the formation of the covariate variable from two different GPA values. In the first case, a solution may be to obtain final percentage grades for the criterion variable. Given that this study was undertaken at the end of the three year data colleetion period, final percentage grades were no longer available for all classes. In particular, two of the instructors from the four year period were no longer at UNBC. In the second situation, the combined GPA score, a solution might be to obtain prior mathematics achievement scores. Using prior mathematics grades may provide more insight into the effects of SI in mathematics courses. However, these are not readily available and bring 54 about an entire other issue for concern since these prior achievement grades are from a variety of colleges and secondary schools and very likely not comparable. Two other important considerations are internal motivation and self-selection, both of which are extremely difficult to control for. Given that Visor, Johnson, and Cole (1992) have found that SI participants are inherently different, it may be worthwhile to investigate instruments for measuring these affects to further improve analysis of SI impact. However, by using incoming GPA as a covariate, there is a reduction in the impact of the motivational factor since motivation will impact student outcomes in all courses and consequently, their GPA. One other consideration is the decision to categorize students as participants if they attend 5 or more SI sessions. The resultant findings will contribute to the research but still do not determine the optimal criteria to use for categorization as a participant. Implications Some implications for future analyses include an investigation in to the effects of SI in other mathematics courses being supported through SI, particularly at this institution. For example, SI has been offered in Finite Mathematics, another non-majors mathematics course. SI has been provided to students enrolled in two first-year Computer Science courses that are based on Discrete Mathematics topics. Other considerations include an examination of the effect of SI in courses that are not mathematical in nature currently supported by SI at this institution. It would also be very informative to further analyze retention trends among SI participants and to determine if the benefits of participating in SI transferred to other courses. Some questions to consider, “Do students who participate in SI achieve higher grades in other courses?” “Do students who participate in SI develop improved study habits?” “Do students who participant in SI form study groups that continue in to other 55 courses?” An additional area yet to be researched in depth is the impact of the SI leader. Several aspects of SI leadership should be examined since trained SI leaders are essential to successful implementation of the SI program. The SI leader must be able to relinquish control to the students to refrain from traditional TA support practice. They need to support collaborative learning in a safe environment to create the discussion and scaffolding that theorists such Gee (1996), Vygotsky (1978), and Wells (2001) envisioned as ideal learning environments. Further implications of this study include decisions on expanding the program, given the positive results in Calculus. In particular, the mandate of this institution is to provide accessible post-secondary education to Northern, remote, and Aboriginal communities. SI has been shown to improve final course grades of students comprising the distinctive population served by this institution. Conclusions Based on the results of the analyses, one can conclude that SI participants achieve higher final course grades than non-participants, in the course Calculus for Non-Majors, offered at the University of Northern British Columbia. Results demonstrated an improvement of two letter grades in the Calculus course being supported once prior achievement was controlled for. This was a substantial increase in outcome, in particular since the average grade of non-participants was a C and the average grade of SI participants was a B-. Furthermore, SI participants succeeded in the course at considerably higher rates (73% vs. 47%). Additional analysis demonstrated that Supplemental Instruction significantly contributed to prediction of success in this course as did incoming GPA. Gender was not found to be a practically significant factor in final course grades, nor a significant predictor 56 of success in the course. No significant interactions occurred in the analyses. These findings confirm the results obtained in larger, primarily American colleges and Universities and extend them to a small northern university in Canada. They also validate generalization of the effectiveness of SI in non-majors Calculus courses. Reduced D, F and W grades corroborate claims of improved retention. The Supplemental Instruction sessions have provided an avenue for students to acquire the knowledge needed to succeed in the course through guided sessions where students have the opportunity to discuss content, practice problems, and prepare for exams. By attending SI sessions, students are being supported through innovative techniques that emphasize process-related learning through scaffolding and dialogue. These techniques are based on sound underpinning developmental theory that emphasizes the integration of socially meaningful contexts to create supported learning and the use of peers as facilitators to guide students through the processes required to succeed in the course. Some students that may have withdrawn from post-secondary education as a result of mathematics requirements may now persist given their success in Calculus. Moreover, many of these students have developed stronger foundation skills in mathematics that will assist them in completing their program requirements. Some students will acquire an improved disposition towards mathematics as a result of their experience in SI. These findings indicate that Supplemental Instruction, a program developed on sound theoretical practices for learning, has assisted students in their pursuit of education at UNBC. 57 REFERENCES Arendale, D. (1994). Understanding the Supplemental Instruction model. In D. C. Martin & D. Arendale (Eds.), Supplemental Instruction: Increasing achievement and retention (pp. 11-22). San Francisco, CA: Jossey Bass, Inc. Arendale, David R. (1999). Supplemental Instruction/VSI Annotated Bibliography. Retrieved December 19, 2003 from http://www.umkc.edu/cad/si/ Baxter, J. A. & Williams, S. R. (1996). Dilemmas of discourse-oriented teaching in one middle school mathematics classroom. The Elementary School Journal, Vol. 97, No. 1, 19-38. Blanc, R.A., DeBuhr, L.E., & Martin, D.C. (1983). Breaking the attrition cycle: The effects of Supplemental Instruction on undergraduate performance and attrition. Journal o f Higher Education, Vol. 54 (1). Bunch, S. M. (1995). The context of community: The initiation of Graduate Students in to the discourse of mathematics education researchers. Paper presented at the Annual Meeting of the American Educational Research Association (San Francisco, CA). Burmeister, S.L., Kenney, P.A., Nice, D.L. (1996). Analysis of the Effectiveness of Supplemental Instruction (SI) Sessions for College Algebra, Calculus, and Statistics. In Kaput, J., Schoenfeld, A.H., & Dubinsky, E. (Eds) Research in Collegiate Mathematics Education. II (pp. 145-154). Rhode Island: American Mathematical Society. Center for Supplemental Instruction (2000). Review o f Research Concerning the Effectiveness o f SI. The University of Missouri-Kansas City and Other Institutions from Across the United States. Retrieved December 19, 2003 from http://www.umkc.edu/cad/si/ Chapman, A. (1995). Intertextuality in school mathematics: The case of functions. Linguistics and Education, 7, 243-262. Chipman, S.F. (2005). Research on the Women and Mathematics Issue: A Personal Case History. In Gallagher, A.M. & Kaufman, J.C. (Eds.), Gender Differences in Mathematics (2005, pp. 1-24). New York: Cambridge University Press. Congos, D. H. & Schoeps, N. (1998). Inside Supplemental Instruction sessions: One model of what happens that improves grades and retention. Research and Teaching in Developmental Education, Vol. 15, No. I. Congos, D. H. & Schoeps, N. (1999). Methods to Determine the Impact of SI Programs on Colleges and Universities. Journal o f College Student Retention: Research, Theory, & Practice, Vol. I, No. 1. 58 Daniels, H. (1995). Pedagogic practices, tacit knowledge and discursive discrimination: Bernstein and post-Vygotskian research. British Journal o f Sociology o f Education, Vol. 16, Issue 4. Dewey, J. (1916). From Democracy and Education. In Reed, Ronald F. & Johnson, Tony W. (Eds.), Philosophical Documents in Education (1996, pp. 121-134). New York: Longman Publishers. Forman, E. & Ansell, E. (2001). The multiple voices of a mathematics classroom community. Educational Studies in Mathematics, Vol. 46, 115-142. Gallagher, A.M. & Kaufman, J.C. (2005). Gender Differences in Mathematics - An integrative psychological approach. New York: Cambridge University Press. Gallimore, R. & Tharp, R. (1988). Teaching mind in society: Teaching, schooling, and literate discourse. New York: Cambridge University Press. Garson, D. (2005). PA 765 Quantitative Research in Public Administration. Retrieved April 25, 2005, from http://www2.chass.ncsu.edu/garson/pa765/index.htm Gee, J. P. (1996). Social linguistics and literacies: Ideology and discourses (2“*^Ed.). London, UK: Taylor & Francis. Green, S. B. & Salkind, N.J. (2003). Using SPSS fo r Windows and Macintosh (3"^^ Ed.). New Jersey: Prentice Hall Hurlburt, R.T. (1998). Comprehending Behavioral Statistics (2“^ Ed.). Pacific Grove, CA: Brooks/Cole Publishing Company Kenney, P.A. & Kallison, Jr., J.M. (1994). Research studies on the effectiveness of Supplemental Instruction in mathematics. In D.C. Martin & D. Arendale (Eds.), Supplemental Instruction: Increasing achievement and retention (pp. 75-82). San Francisco CA: Jossey-Bass, Inc. Kenney, P. A. (1989). Supplemental Instruction in mathematics: Needs and approaches, critical aspects o f mathematics training and the role o f SI. Unpublished manuscript. Retrieved from http://www.umkc.edu/centers/cad/si/sidocs/pkmathematics93.htm Kleinman, S.S. (1995). Local Discourse Matters: A Case Study of Communication in an Undergraduate Calculus Class. Journal o f Women and Minorities in Science and Engineering Lam Collection, Vol. 2, pp. 91-111. Kochenour, E.O., Jolley, D.S., Kaup, D.L., Patrick, D.L., Raoch, K.D., Wenzler, L.A. (1997). Supplemental Instruction: An effective component of Student Affairs Programming. Journal o f College Student Development, Vol. 38, No. 6 59 Learning Skills Centre (2001). The University of Northern British Columbia, Prince George. LeFrancois, G.R. (1997). Psychology fo r Teaching (9^ ed.). Belmont, CA: Wadsworth Publishing Company. Linn, M.C. & Kessel, C. (1996). Success in Mathematics: Increasing Talent and Gender Diversity Among College Majors. In Kaput, J., Schoenfeld, A.H., & Dubinsky, E. (Eds) Research in Collegiate Mathematics Education. II (pp. 101-143). Rhode Island: American Mathematical Society. Martin, D. C. & Arendale, D R. (1992). Supplemental Instruction: Improving First-Year Student Success in High-Risk Courses. National Resource Center for the Freshman Year Experience, University of South Carolina, Columbia, SC. McMillan, J. H. & Wergin, J. F. (2002). Understanding and Evaluating Educational Research (2“‘*ed.). New Jersey: Merrill Prentice Hall. Mauch, I.E. & Birch, J.W. (1998). Guide to the Successful Thesis and Dissertation (4* ed.). New York: Marcel Dekker, Inc. Pinker, S. (1994). The Language Instinct: How the Mind Creates Language. New York: William Morrow. Randhawa, B. S. (1994). Self-efficacy in mathematics, attitudes, and achievement of boys and girls from restricted samples in two countries. Perceptual and Motor Skills, 1994, Vol. 79, p. I0II -I0I8. Sax, G. (1997). Principles o f Educational and Psychological Measurement and Evaluation. Washington: Wadsworth Publishing Company. Sfard, A. (2000). On reform movement and the limits of mathematical discourse. Mathematical Thinking & Learning, Vol. 2, Issue 3. SPSS Inc. (1975). Statistical Package for the Social Sciences (Version 12.0) [Computer Software]. Chicago, Illinois: SPSS Inc. Steele, D. F. (2001). Using sociocultural theory to teach mathematics: A Vygotskian perspective. School Science and Mathematics, Vol. 101, Issue 8. Tabachnick, B.G. & Fidell, L.S. (2001). Using Multivariate Statistics (4*^ ed.). Needham Heights, MA: Allyn & Bacon. A Pearson Education Company. Visor, J.N., Johnson, J.J., & Cole, L.N. (1992). The relationship of Supplemental Instruction to affect. Journal o f Developmental Education, Vol. 16 (2). 60 Vygotsky, L.S. (1978). The development of higher psychological processes. In Cole, M., John-Steiner, V., Scribner, S., Souberman, E. (Eds.), Mind in Society. Cambridge MA: Harvard University Press. Vygotsky, L.S. (1962). Thought 'and Language. Hanfmann, E. & Vakar, G. (Eds.) Cambridge, MA; The M.I.T. Press. Wells, G. (2001a). The development of a community of inquirers. In Wells, G. (Ed.), Action, talk, & text: Learning and teaching through inquiry (pp. 1-24). New York: Teachers College Press. Wells, G. (2001b). The case for dialogic inquiry. In Wells, G. (Ed.), Action, talk, & text: Learning and teaching through inquiry (pp. 171-194). New York: Teachers College Press. Widmar, G. (1994). Supplemental Instruction: From small beginnings to a national program. In D. C. Martin & D. Arendale (Eds.), Supplemental Instruction: Increasing achievement and retention (pp. 3-10). San Francisco, CA: Jossey Bass, Inc. Zack, V. & Graves, B. (2001). Making mathematical meaning through dialogue: “once you think of it, the Z minus three seems pretty weird”. Educational Studies in Mathematics, Vol. 46, 229-271. 61 APPENDIX A ETHICS APPROVAL FORMS 62 UNIVERSITY O F N O R T H E R N BRITISH C O LU M BIA Research Ethics Board MEMORANDUM To: V ivia n F ayow ski 117 - 3233 McGill Cr., Prince George V2N 214 Peter M acM illan Education Program From : A le x M ichalos Chair, Research E thics Board Date: Ja nu ary 24,2002 Re: E thics R eview 2002.0117.14 B enefits o f S upplem ental In stru ctio n Im plem ented in a U nive rsity C alcu lu s C ourse Thank you for submitting the above noted proposal for review by the Research Ethics Board. Your proposal has been approved pending additional information. The REB requested more specific information regarding the destruction of the data. Specifically, for how long will the data be stored and how will it be destroyed? Would you please provide this information as soon as possible. Please do not start your research until you have submitted this information and you have received approval. If you have any questions, please feel free to contact me. Sincerely, (X I , / l H ^ Alex 0 . Michalos, Chair Research Ethics Board 63 U flb UNIVERSITY O F N O R T H E R N BRITISH COLUM BIA C Research E thics Board MEMORANDUM To: Vivian Fayowski 117-3 2 3 3 McGill Cr. Prince George, BC V2N 2T4 Dr. Peter MacMillan Education Program From; Dr. Alex C. Michalos Chair, Research Ethics Board Date: January 30, 2002 Re: E th ics P roposal 2002.0117.14 B enefits o f Supplem ental In stru ctio n Im plem ented in a U nive rsity C alcu lu s C ourse Thank you for submitting your response to the reviewer’s comments on the above noted proposal. Your response to the comments have satisfied those concerns expressed by the reviewers. Your proposal has been approved and you may begin your research. If you have any questions regarding the above , please feel free to contact me. Sincerely, Alex C. Michalos Chair, Research Ethics Board 64 APPENDIX B - SUMMARY DATA Table 10 SI DATA: WINTER 2002 - MATHEMATICS 152 (Calculus fo r Non-Majors) Sections A I andA2 NON 81 81 # students who did not (%) # students (%) GRADE of class participate or of class participating in (pt) participated in less than 25 - 1 0 0 % 25% of 81 sessions of 81 sessions. 16 1 0 .6 23.4 11 A+ (4.33) 13 8.5 8 .6 4 A (4.00) 19 12.7 1 2 .6 6 A- (3.67) 8 6.4 5.3 3 B+ (3.33) 14 4 8.5 9.3 B (3.00) 9 3 6.4 6 .0 B- (2.67) 7 3 6.4 4.6 C+ (2.33) 7 4 4.6 8.5 C (2.00) 9 6 .0 2 4.3 C- (1.67) 18 11.9 2 4.3 D (1.00) 1 0 .6 31 5 20.5 F (0.00) 1 0 0 151 100 Total 47 Table 11 Summary Chart (WINTER 2002) Total student enrollment for two sections of Mathematics 152 Number of 81 sessions offered in term Total number and percentage of students who attended 81 Total contact hours of 81 participating students Mean number of sessions attended by 81 participants Mean size of 81 sessions Mean 81 Participant Evaluation Rating of Helpfulness of 81 198 232 (43%) 85 960 11 4 5.8 (l=low,6=high) Mean Final Course Grade of 81 Participants Mean Final Course Grade of Non-81 Participants Percentage of 81 students receiving a D or F grade Percentage of Non-81 students receiving a D or F grade_____ 65 2.9 2.3 15 33 Table 12 SI DATA: FALL 2002 - MATHEMATICS 152 (Calculus fo r Non-Majors) NON SI SI # students who # students GRADE did not participate (%) participating in (%) (pt) of class of class or participated in 25 - 100 % less than 25% of of SI sessions. SI sessions ( 6 or more sessions) 8 10 17 5 A+ (4.33) 4 10 13 I A (4.00) 12 9 17 5 A- (3.67) 2 14 3 4 B+ (3.33) I I 20 6 B (3.00) 5 6 II 3 B- (2.67) 10 0 7 0 C+ (2.33) 4 5 I 3 C (2.00) 0 5 6 0 C- (1.67) 12 2 7 9 D (1.00) 17 2 2 2 7 F (0.00) 100 100 77 Total 29 Table 13 Summary Chart (FALL 2002) Total student enrollment for one section of Mathematics 152 Number of SI sessions offered in term Total number and percentage of students who attended SI Total contact hours of SI participating students Mean number of sessions attended by SI participants Mean size of SI sessions Mean SI Participant Evaluation Rating of Helpfulness of SI 108 86 (50%) 54 413 8 2 5.4 (l=low,6=high) Mean Final Course Grade of SI Participants Mean Final Course Grade of Non-SI Participants Percentage of SI students receiving a D or F grade Percentage of Non-SI students receiving a D or F grade 66 3.0 2.2 14 34 Table 14 SI DATA: WINTER 2003- MATHEMATICS 152 (Calculus fo r Non-Majors) Sections A I andA2 SI NON SI GRADE # students who did # students participating in (%) not participate or (%) (Pt) of class participated in less of class 25 - 1 0 0 % than 25% of SI of SI sessions. (7 or more sessions) sessions 7 2 0 .6 11 A+ (4.33) 9.8 A (4.00) 3 9 8 .0 8 .8 2 5.9 A- (3.67) 8 7.1 2 5.9 4 B+ (3.33) 3.6 B (3.00) 2 5.9 6 5.4 B- (2.67) 3 7 8 .8 6 .2 C+ (2.33) 3 5 8 .8 4.5 C (2.00) 3 6 8 .8 5.4 C- (1.67) 3 7 8 .8 6.3 D (1.00) 1 3.0 22 19.6 F (0.00) 5 14.7 27 24.1 34 Total 100 112 100 Table 15 Summary Chart (WINTER 2003) Total student enrollment for two sections of Mathematics 152 Number of SI sessions offered in term Total number and percentage of students who attended at least one SI Total contact hours of SI participating students Mean number of sessions attended by SI participants Mean size of SI sessions Mean SI Participant Evaluation Rating of Helpfulness of SI 147 126 (37%) 54 676 12.5 5.4 5.4 (l=low,6=high) Mean Final Course Grade of SI Participants Mean Final Course Grade of Non-SI Participants Percentage of SI students receiving a D or F grade Percentage of Non-SI students receiving a D or F grade____________ 67 2.63 1.97 17.7 43.7 Table 16 SI DATA: FALL 2003 GRADE (pt) A+ (4.33) A (4.00) A- (3.67) B+ (3.33) B (3.00) B- (2.67) C+ (2.33) C (2.00) C- (1.67) D (1.00) F (0.00) Total MATHEMATICS 152 (Calculus for Non-Majors) SI NON SI # students # students who participating in did not participate (%) (%) 25 - 1 0 0 % of class or participated in of class of SI sessions. less than 25% of ( 8 or more SI sessions sessions) 8 23 6 7 0 0 4 5 4 11 3 4 3 9 4 5 8 3 3 3 1 3 6 7 2 6 3 3 3 9 4 5 2 6 5 6 14 5 18 21 4 11 29 34 35 100 86 100 Table 17 Summary Chart (FALL 2003) Total student enrollment for one section of Mathematics 152 Number of SI sessions offered in term Total number and percentage of students who attended at least one SI Total contact hours of SI participating students Mean number of sessions attended by SI participants Mean size of SI sessions Mean SI Participant Evaluation Rating of Helpfulness of SI 121 67 (50%) 60 683 11 10 5.6 (l=Iow,6=high) Mean Final Course Grade of SI Participants Mean Final Course Grade of Non-SI Participants Percentage of SI students receiving a D or F grade Percentage of Non-SI students receiving a D or F grade____________ 68 2.57 1.54 25 55 Table 18 SI DATA: WINTER 2004 - MATHEMATICS 152 (Calculus fo r Non-Majors) NON SI SI GRADE # students # students who participating in (%) did not participate (%) (pt) 25 - 100 % of class or participated in of class of SI sessions. less than 25% of (7 or more SI sessions sessions) A+ (4.33) 5 9.43 9 8.91 A (4.00) I 1.89 3 2.97 A- (3.67) 5 9.43 5 4.95 B+ (3.33) 5 9.43 8 7.92 B (3.00) 5 9.43 5 4.95 1 B- (2.67) 1.89 6 5.94 C+ (2.33) 6 11.32 10 9.90 C (2.00) I 1.89 8 7.92 C - (1.67) 5 9.43 1 0.99 D (1.00) 9 16.98 14 13.86 F (0.00) 10 18.87 32 31.68 Total 53 100 101 100 Table 19 Summary Chart (WINTER 2004) Total student enrollment for two sections of Mathematics 152 Number of SI sessions offered in term Total number and percentage of students who attended at least one SI Total contact hours of SI participating students Mean number of sessions attended by SI participants Mean size of SI sessions Mean SI Participant Evaluation Rating of Helpfulness of SI 155 111 (50%) 78 946 12.1 8.5 5.5 (l=low,6=high) Mean Final Course Grade of SI Participants Mean Final Course Grade of Non-SI Participants Percentage of SI students receiving a D or F grade Percentage of Non-SI students receiving a D or F grade____________ 69 2.1 1.8 35.9 45.5 Table 20 SI DATA: Fall 2004 - MATH 152 (Calculus fo r Non-Majors) GRADE (pt) A-k (4.33) A (4.00) A- (3.67) B+ (3.33) B (3.00) B- (2.67) C+ (2.33) C (2.00) C- (1.67) D (1.00) F (0.00) Total SI # students participating in 25 - 100 % of SI sessions. (7 or more sessions) 3 4 3 2 3 4 I 5 0 6 3 34 (%) of class NON SI # students who did not participate or participated in less than 25% of SI sessions (%) of class 17.65 8.82 7 7 4 II 18 10.29 1.47 7.35 4.41 4.41 2.94 10.29 10.29 5.88 16.18 26.47 100 68 100 8.82 11.73 8.82 5.88 8.82 11.73 2.94 14.71 7 I 5 3 3 2 0 Table 21 Summary Data (Fall 2004) Total student enrollment (does not include withdrawals) Number of SI sessions offered in term Total number and percentage of students who attended at least one SI Total contact hours of SI participating students Mean number of sessions attended by SI participants Mean size of SI sessions Mean Final Course Grade of SI Participants Mean Final Course Grade of Non-SI Participants Percentage of SI students receiving a D or F grade Percentage of Non-SI students receiving a D or F grade 70 102 61 (6 6 %) 67 791 12 13 2.5 1.8 26.5 42.7 APPENDIX C ANALYSIS OF COVARIANCE, CHI-SQUARE ANALYSIS Analysis of Covariance Analysis of covariance (ANCOVA) is an extension of analysis of variance (ANOVA). ANOVA is used to compare two or more means by evaluating the differences among the means, relative to the dispersion in the sampling distributions (Tabachnick & Fidell, 2001). In ANCOVA, the main effects and interactions of the independent variables (IVs) are assessed after dependent variable (DV) scores are adjusted for differences associated with one or more covariates (CVs). Covariates are variables that are measured before the dependent variable and are correlated with it. The question for both is essentially the same - are mean differences in the DV between groups larger than expected by chance (Tabachnick & Fidell, 2001)? Tabachnick & Fidell classify three main applications for ANCOVA. The first is to increase the sensitivity of the test of main effects and interactions by reducing the error term. In other words, the error term is adjusted for, and hopefully reduced by, the relationship between the DV and the CVs. The second purpose is to adjust the means on the DV to what they would be if all subjects scored equally on the CVs. This second application is often used in non-experimental situations where subjects cannot be randomly assigned. (This is the primary use of ANCOVA in this research paper.) The third use of ANCOVA occurs in MANOVA where the researcher assesses one DV after adjustment for other DVs that are treated as CVs. The statistical operations are identical in all three major applications of ANCOVA. 71 As in ANOVA, variance in scores is partitioned into variance due to differences between groups and variance due to difference within groups. Squared differences between scores and various means are summed and these sums of squares, when divided by appropriate degrees of freedom, provide estimates of variance attributable to different sources (main effects of IVs, interactions between IVs, and error). Ratios of variances then provide tests of hypotheses about the effects of TVs on the DV. In ANCOVA, the regression of one or more CVs on the DV is estimated first. Then DV scores and means are adjusted to remove the linear effects of the CV(s) before analysis of variance is performed on these adjusted values (2001, pp. 275-276). As a general rule, one wants a very small number of CVs, all correlated with the DV but not each other (Tabachnick & Fidell, 2001). The goal is to maximize the adjustment of the DV but with a minimum loss of degrees of freedom for error. One degree of freedom is lost for each CV. As with ANOVA, the statistical test in no way assures causality. In the case of experimental research, causality may be inferred. However, ANCOVA is also used for nonexperimental research. In this case, attribution of causality is not justified (Tabachnick & Fidell, 2001). Following are associated equations for ANCOVA. 72 Table 22 Computation Equations fo r Sums o f Squares and Cross-Products in One-Way Between Subjects Analysis o f Covariance Source Between Groups Sum o f Squares for the DV (Y) Sum o f Squares for the CV (X) t(. E E ^ ( bg ) n kn “ n k f ji e e V kn k f k V E E ^ „ Within Groups (wf) (, . V Sum o f Products E ^ n E x n H /V n e \ ( ^ k = number o f groups; n = number o f subjects per group » . , « = E E ^ ' — -— n e e n \ f H k n e e kn t r « k ^ n / E Y . E>' V n k E x J\ ^ ^ ^ The Chi-squared ) Analysis Tabachnick and Fidell (2000) provide the following description for use of the chisquare analysis. The chi-square test of independence is used to examine the relationship between two discrete variables. For example, one may want to examine a potential relationship between region of province (south, central, and north) and approval versus disapproval of current political leadership. The chi-square is an appropriate analysis. In the ^ analysis, the null hypothesis generates expected frequencies against which observed frequencies are tested. If the observed frequencies are similar to the expected frequencies, the value of will be small and the null hypothesis is retained. If the observed frequencies are sufficiently different, the value of x ^ is large and the null hypothesis is rejected. The relationship between the size of ^ and the difference in observed and expected frequencies can be seen from the following computational formula: where represents observed frequencies, and represents the expected frequencies in each cell. Usually the expected frequencies for a cell are generated from its row and column sum: Cell = (row sum)(column sum)/N When this procedure is used to generate the expected frequencies, the null hypothesis tested is that the variable in the row (say region of province) is independent of the variable in the column (attitude toward leadership). If the fit is good, ^ is small, so one concludes that the two variables are independent. A poor fit leads to a large hypothesis, and conclusion that the two variables are related. 75 value, rejection of the null APPENDIX D - NOTATION H q is the null hypothesis is the alternative hypothesis l^siPart represents the mean final course grade of the SI participant group ^nonpart represcuts the mean final course grade on the non-participant group represents the mean final course grade of the female gender represents the mean final course grade of the male gender represents the interaction between gender and SI participation across final course grades ^siPart is the coefficient of the SI predictor for the logistic regression equation is the coefficient of the gender predictor for the logistic regression equation ^incGPA is the coefficient of the incoming GPA predictor for the logistic regression equation a ^df,n is the F statistic obtained with level of significance, a, degrees of freedom (df), and the number of in the sample, n. APPENDIX E - UNBC GRADING SYSTEM Table 1 UNBC Grading System for years 2002 - 2004 Letter Grade UNBC Grade Point A+ 4.33 A 4.00 A3.67 3.33 , B+ B 3.00 2.67 B2.33 C+ 2 .0 0 c 1.67 cD 1 .0 0 0 .0 0 F W Percentage % (interval) 90 - 100 85 - 89.9 80 - 84.9 77 - 77.9 73 - 76.9 70 - 72.9 67 - 69.9 63 - 66.9 60 - 62.9 50 - 59.9 0 - 49.9 Table 2 Conversion o f letter grades to numerical values Letter Grade Numerical Value for Letter Grade 12 A+ 11 A 10 A9 B+ 8 B 7 B6 C+ 5 C 4 C3 D 2 F 1 W 77