DID WE DO WHAT WE SAID WE WOULD DO? A MULTIPLE STAKEHOLDER EVALUATION OF AN EARLY CHILDHOOD EDUCATION TRAINING PROGRAMME by Patricia McClelland B.A. , York University, 1980 PROJECT SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF EDUCATION © Patricia McClelland, 2003 THE UNIVERSITY OF NORTHERN BRITISH COLUMBIA May 2003 .N VERSITV OF NORTHERN BRITISH COLUMBIA LIBRARY Prince George, BC All rights reserved. This work may not be reproduced in whole or in part, by photocopy or other means, without the permission of the author Did We Do What We Said We Would? II Abstract Did the University College of the Cariboo Williams Lake Campus 2002 First Nations Early Childhood Education Training Programme do what it said it would do? This summative, multiple stakeholder, programme evaluation project summarized the results from the various programme stakeholders and sought to find issues in common to all stakeholders in terms of their assessment of the Early Childhood Education training programme. The results of this project indicated that (a) the programme and instructors were flexible and supportive; and (b) more work needs to be done to address cultural relevancy and appropriate culturally-based teaching resources. Did We Do What We Said We Would? lll TABLE OF CONTENTS Abstract 11 Table of Contents lll List of Tables v Chapter One Statement of the Problem Definitions Literature Review Specific Research Question Significance of the Study 1 3 5 20 21 Chapter Two Design and Methodologies Subjects Instrumentation Procedures 22 22 23 24 Chapter Three Results Quantitative Qualitative 27 27 31 Chapter Four Discussion 36 Chapter Five Conclusion Recommendations Future Research Suggestions Summary 41 45 46 46 48 References Appendix A Original Research Team Correspondence 51 Did We Do What We Said We Would? iv Appendix B Letter of Permission to Conduct Study 54 Appendix C Ethics Approval from UCC and UNBC 57 Appendix D Letter of Information for Participants 60 Appendix E Letter of Informed Consent 62 Appendix F Today's Learnersffomorrow's Educators 64 Simple and Practical Approach(l999) Evaluation Questionnaire Did We Do What We Said We Would? v List of Tables Table 1 Common Quantitative Likert Responses Results 30 Did We Do What We Said We Would? I Introduction General Statement of the Problem The idea for this project has evolved over a ten year period of adapting and delivering purchased or leased Early Childhood Education (ECE) training programmes for First Nations communities on and off reserves in Northern British Columbia. These programmes rely on numerous partnerships for funding, delivery and support; it follows then that these stakeholder voices should have an opportunity to be included in an evaluation of the programme. Programme evaluations contribute to the future considerations in developing and implementing a relevant and appropriate provincial curriculum framework for a First-Nations-specific ECE training programme package. A wide variety of ECE training programme models has been accessed by First Nations groups both here in British Columbia and across Canada. The three basic models of programme delivery currently in use are: (a) established programmes currently in use and purchased by First Nations communities for delivery, (b) established programmes currently in use and adapted by First Nations communities for delivery by the community or post secondary institution, and (c) programmes which are completely developed and delivered by the community itself. First Nations training partners have expressed concern over the ineffectiveness of sending community members away to mainstream ECE training programmes Did We Do What We Said We Would? 2 (Ball & Pence, 1997). First Nations communities continue to be concerned that the ECE theories and methods taught in most universities and colleges, although based in developmental theory and research, do not address the needs and circumstances of individual First Nations cultures (Ball & Pence, 1997). In response to the number and styles of programmes offered for First Nations ECE training, it became evident that there was a need for a standardized self-evaluation tool that would give community partners the opportunity to assess and evaluate training. There has always been the recognition that quality training is a primary indicator of quality child care (van Raalte, 1999). The MA WIW (Tobique First Nations, Big Cove First Nations, and Burnt Church First Nations) Health Council, Incorporated (1999) initiated the Today 's Learners, Tomorrows Educators project as a strategy for assessing programmes from a multiple stakeholder perspective. The project attempted to improve partnership programme delivery by giving students, faculty, community sponsors, and post secondary institutions opportunities to provide feedback about successes and identify areas for improvement in the various ECE training programmes offered. In the project, all identified stakeholders participated in evaluating the programme from their own perspectives using specifically designed evaluation questionnaires. As well, all participants completed a universal questionnaire. The project was carried out in the Atlantic region of Canada under the guidance of the MA WIW Health Council Inc. Did We Do What We Said We Would? 3 This project adds to the existing data by using the evaluation tool developed by the MA WIW Health Council Inc. ( 1999) with the University College ofthe Cariboo (UCC),Williams Lake Campus, Early Childhood Education Programme 2002, which was funded by the Cariboo Chilcotin Aboriginal Training Education Council (CCATEC) to deliver the ECE programme for students from local First Nations communities. Problem Statement In order to improve partnership delivery programmes it is fundamental that all stakeholders have an opportunity to participate in the assessment and evaluation of a delivered programme. In the past, evaluations which addressed only the opinions of the students upon the completion of a training programme have not represented the needs or opinions of the other stakeholders involved in the delivery, support, and funding of programmes. This project provided an opportunity for all of the identified stakeholders connected to the ECE training programme an opportunity to assess the successes of the programme and to identify from their individual stakeholder viewpoints, potential goals for future programmes. Definitions In order to properly interpret the results, key terms must be defmed operationally. In the evaluation document content refers to specific aspects of the programme including theory and practice, resource materials and instructional hours. Did We Do What We Said We Would? 4 The learning environment is understood to relate to the teaching facilities, student academic supports, tutoring and faculty appraisals. Programme design includes the admission criteria, exit requirements and methods of delivery. Support/linkages relate to available community resources, government resources, administrative and financial resources involved in running the programme. Administration in this project includes the physical location, partnering and programme documentation. The term stakeholders represents persons/ groups whose existence or ownership depended upon a commonality. In this instance common groupings exist in the community, institution, faculty and students. Programme overview includes the programme strengths and weaknesses and perceived barriers. (van Raalte, 1999) Did We Do What We Said We Would? 5 Review of the Literature While research literature and subsequent results specific to evaluating Canadian First Nations ECE training programmes is extremely limited, the general research literature of programme evaluation helps to address issues experienced by all programme evaluators. Programme evaluation is increasingly mandated in the current environment of fiscal and academic accountability. In particular, the use of programme evaluation as a measurement tool for quality is growing in the field of ECE training partnership programmes in First Nations communities. This literature review will present (a) an historical overview of programme evaluation designs, (b) a review of Today 's Learners/ Tomorrow's Educator's First Nations ECE Training Evaluation (1999), (c) a look at defining quality programmes, and (d) the use of programme evaluation results. Historical Background Barak (1986) describes the five major stages that lead to the creation of academic programme evaluations. They include the development of the concept of programme, the emergence of the accreditation movement, the emergence of the profession of educational evaluation, the rise of the accountability movement, and the creation of a variety of higher education management approaches. By the mid-nineteenth century, post secondary programmes in higher education were being influenced by a number of factors. Three key factors included Did We Do What We Said We Would ? 6 (a) the development of programmes which were being created in response to identified growing employment needs, (b) the development of a variety of college curriculum and an increase in faculty, and (c) the socioeconomic conditions supporting the continued education of adults (Auclair, 1990). By the end of the 1950s and 1960s, many institutions had created boards to provide rational, relevant planning for specific programmes. Towards the end of the 1970s, the accountability movement demanded that institutions come under tighter public scrutiny because of the inability of funding sources to maintain quality programmes. During this time the role of evaluations increased. External programme evaluation gained greater significance. Momentum for programme evaluation has been growing since its initiation. To address the assessment of student outcomes and programme quality, institutions used a variety of methods including surveys, questionnaires, and interviews. Many programme planners contended that surveys were more influential as criteria for funding higher education. Public post secondary education institutions continued to use programme evaluations as an assessment tool for programme quality. What is Evaluation Evaluation can be defined as "a systematic process to establish the worth or value of something, for example, an educational programme" (Devlin, 1991 , p. 121 ). Did We Do What We Said We Would? 7 The word evaluation originates from the Latin word, valere, meaning strength. Programme evaluators, looking for strengths and goal areas, are usually concerned with the educational and fiscal integrity of the programme including, cost effectiveness, the successful achievement of the established learning outcomes, and the effectiveness ofthe instruction and presentation of the materials. To gather formal answers to these questions, it is important to make a systematic attempt to collect data and evidence. Johnson and Foa (1989) describe evaluation as "the process of providing . .. timely, accurate information that will contribute to decisions about the improvement, continuance and/or expansion of a programme" (p. 159). They additionally counsel that "evaluation does not make decisions, people do" (p. 158). Finally, they state, "evaluations are often initiated on the basis of political, bureaucratic, or competitive agendas. Institutions ... often develop new curricula in response to funding opportunities" (p. 162). This is evident in the area of evaluating First Nations ECE training programmes. The authors of Today 's Learners/Tomorrow 's Educators (1999) were driven by a need to gather information about programmes that were being presented in response to an established community need and an increased availability of funding dollars. Approaches to Evaluation Many models of programme evaluation in education approaches are discussed in the literature, including objectives-based or goal oriented evaluation, Did We Do What We Said We Would? 8 decision-centred evaluation, goal free evaluation, adversary evaluation and transactional or responsive evaluation. The most common and predominant model, objectives-based or goal oriented evaluation, assesses student progress and the achievement of programme objectives. The model stresses the importance of initially specifying programme objectives and standards to be used for evaluating. Evaluators then decide what evidence related to the desired outcomes is to be collected and analyzed in order to make judgments regarding the programme. The key problem associated with this style of programme evaluation is that learners' objectives can be quite different from those of the instructor or programme planner. The decisioncentred evaluation model focuses on the types of decisions that can be improved based on the evaluation results. Planning decisions such as programme objectives; structure decisions such as personnel, facilities and budgets, and implementation decisions such as day to day procedures can all be addressed with this model. Scriven ( 1999) supports the use of goal free programme evaluation which assesses programme effects or outcomes regardless of the programme intent or objectives. All outcomes are considered, positive and negative, intended and unintended. The adversary evaluation model encourages the affirmative evaluator and the negative evaluator to present their programme evaluations to the decision maker. Each evaluator presents their best arguments and the decision maker weighs the evidence and makes an informed decision regarding the programme. This style is considered Did We Do What We Said We Would? 9 useful when evaluating "new, innovative, non-traditional or controversial programmes" (Brack & Moss, 1999, p. 93). Finally, the transactional or responsive evaluation model stresses the importance of evaluating programmes from the perspectives of the various stakeholders. This particular model "emphazises that there are different ways to evaluate programmes and no one way is the right way" (Brack & Moss,1999, p. 99). It often includes a combination of interviews and questionnaires with staff, clients, and anyone associated with the programme. The above brief descriptions of evaluation models provide some insight into the potential variety of approaches used in designing programme evaluations. The importance is not which approach is best, but which style or combination of styles is appropriate for the specific programme being evaluated. The underlying principle in designing evaluations is to be flexible and make the results relevant and useful to those involved in the programme and its subsequent evaluation. As the UCC ECE programme was a partnership venture a transactional evaluation ensured the feedback from all participants. Who Will Participate in the Evaluation Process Knowing the participants for an evaluation is central in guiding and choosing the techniques to be employed. Audiences such as planners, staff who provide the programmes, participants, sponsors, administrators or governing bodies, governments and funding agencies are only a few of the stakeholders who might be Did We Do What We Said We Would? 10 involved in a partnership programme evaluation. The authors of Today 's Learners/Tomorrow's Educators (1999) took this particular issue into consideration when establishing a series of consistent key topic headings such as programme delivery, programme supports, and linkages for all questionnaires and then creating specific questions relevant to each of the stakeholder groups. When considering creating potentially effective programme evaluations to be used by multiple stakeholder groups the question becomes: whose values will prevail in determining educational value, merit and success. Lewis and Dunlop (1990) described programme successes and failures as "relative terms open to subjective interpretation" (p. 1). In their research addressing the factors associated with the perception of programme success and failure, they concluded that programme activities were related to programme outcomes. If evaluation is a process of judging the worth of something, whose values and definitions of worth will apply or will be accommodated? In the case of Today 's Learners/Tomorrow's Educators ( 1999), the creation of stakeholder-specific sections and questions ensured the opportunity for all participants to voice their opinions on programme successes. The Issue of Quality The First Nation!Inuit Child Care Initiative, established in 1995, brought the issue of assessing quality training programmes to the forefront. Did programmes provide a foundation that would serve the distinct needs of First Nations students? Did We Do What We Said We Would? II Did programmes cover all the subject areas applicable to working with children from FirstNations communities? Did programmes address the needs of students being trained? (van Raalte, 1999). These questions highlighted the absence of any means by which to measure components necessary for quality training. Today 's Learners/Tomorrow 's Educators (1999) was not created to measure quality as it pertains to the training of ECE teachers (van Raalte, 1999). While it has been recognized by researchers, providers, and planners that staff education and training is one of the most important factors associated with sustaining high quality child care programmes, it has been also recognized that evaluating ECE training programmes, reflecting on potential changes, and acting on those ideas contributes to programme quality (Doherty, 1999; Gathering Strength, 1996). A quality checklist was developed to assist with the assessment ofECE training as it related to issues identified by current researchers in the field. Moss and Pence (1995) summarized quality child care as "existing to a large extent, in the eye of the beholder and the beholder can be anyone or any group from among a range of stakeholders, each with an interest in early childhood services" (p. 50). Historically, defining quality child care through research studies, assessment instruments, and regulations has been the privilege of a limited group of stakeholders. Discussion, description and evaluation of quality child care has been formulated by experts in government, the ECE profession and academic researchers. The approach has been somewhat Did We Do What We Said We Would? 12 exclusionary in nature. Doherty (1991) argued that, research from the United States, Canada and Bermuda "found that staff who have some college education in any discipline and/or post secondary school training in child development show high rates of positive adult behaviors ... than staff with only a high school diploma or less and are more likely to provide developmentally appropriate activities"( p. 72). The quality checklist helped identify the important components of quality ECE training for purposes of evaluation topics. Bredekamp and Copple (1997) stated that "good advice to students in ECE training programmes is to take stock of the community in which they are working, consider the cultural values and ways of parents who bring their children for care " (p. 6). Ball and Pence (1999) further supported this by asking, "are we willing to take our own advice by yielding a place in curriculum development, programme design and even the training of ECE teachers to knowledge that resides with the community .. . [and] in pedagogical delivery, to yield floor time to respected community members speaking about the needs, practices and goals of their cultural constituency?"( p. 46). The programme director for the Meadow Lake Tribal Council sees quality child care training for First Nations students occurring by "transforming prepackaged didactic processes to an open ended participatory process and engaging communities in dialogues about designing curricula, sharing the floor in delivering courses, and moving over and letting the community determine the desired end Did We Do What We Said We Would ? 13 product of the training" (Ball & Pence, 1999, p. 50). Quality, according to Curtis (1999), is "a relative value based concept that is wholly constructed and subjective" (p. 1). Munton (1995) attempted to deconstruct the concept and to reject the idea of any definition which was exclusionary. Different interest-groups parents, early childhood educators, stakeholders and even children will have a different view on what they call good quality early childhood programmes. Because it is probably impossible to share a common definition of quality, it is more important to move towards a common understanding of what each stakeholder means by the term. It may be beneficial to consider the framework created by Munton ( 1995) which addressed the term quality in terms of indicators of good practice. He noted that to be of practical use a framework should (a) have face validity among the very diverse group of people who might be described as stakeholders, (b) recognize that quality is a multidimensional concept, (c) be able to incorporate the different perspectives on quality held by the different stakeholders, (d) be effective in raising practitioner awareness of quality issues, (e) be relevant to both quality assessment and quality improvement, (f) enable specific aspects of quality to be identified and targeted for improvement, and (g) have the capacity to expand and clarify people' s thinking about quality. As stated previously, quality child care training programmes are often viewed as successful within the mindset of each stakeholder. A programme evaluation tool, then, should address the individual stakeholders concerns and views Did We Do What We Said We Would? 14 of quality. What to Evaluate In conducting programme evaluations, the central issue is deciding what to measure. Verduin and Clark ( 1991) identified six programme aspects to be evaluated: access, relevancy to needs and expectations, quality of programme offered, learning outcomes, impact, and cost-effectiveness. These can also be summarized into four standard elements of educational programmes: (a) contextual characteristics, including characteristics such as programme costs, programme length, days of week, time of day, location and class size; (b) student characteristics, concerned with knowing the characteristics of those attending; (c) programme implementation factors, concerned with the methods and strategies which the planner/teacher has included as a means to the educational end of reaching learning outcomes; and, (d) programme outcomes, including intended and unintended, short and long term, administrative and academic (Potter, 1995). When to Evaluate According to Curtis (1999), this question may cause more of a challenge than any other. One school of thought argues that the process of evaluation should take place at the end of a programme, as only then it can be seen whether the learning outcomes have been achieved. This is often referred to as a summative evaluation (Potter, 1995). This is a retrospective approach using a pre-defined and limited range Did We Do What We Said We Would? 15 of data. Others advocate an alternative process which supports a continuous feedback between evaluators and the field workers (Curtis, 1999). This study used the same summative evaluation instrument as authored by the MA WIW Health Council Inc.(1999). Historical Background ofToday's Learners!romorrow's Educators Self Evaluation Tool In May, 1995, a National Conference on Aboriginal Early Childhood Education Training Programmes was held in Montreal. Participants were asked to explore criteria relating to principles and guidelines for First Nations Early Childhood Education Training Programmes. These programmes had been offered across Canada in many different forms and had built in very little in the way of evaluation processes. Limited data were available to determine whether these programmes had met the needs of First Nations Communities and their respective funding proposals (van Raalte, 1999). For many years, early childhood educators, parents and many individuals from First Nations communities and training councils had been concerned about the quality of First Nations child care training. Did it provide a foundation that would serve the distinct needs of First Nations children? Did it cover all subject areas applicable to working with First Nations children and families? Did it address the needs of the students being trained? Was it being delivered in an inviting and accessible manner? The central concern addressed was Did We Do What We Said We Would? 16 how best to measure the quality and success of the training offered. Concerns about quality child care training in First Nations communities were first voiced after the First Nation/ Inuit Child Care Initiative was established in 1995. Monies were allocated for the development and delivery ofECE training across Canada. As the number of ECE training programmes developed and delivered grew, the need became evident for a standardized self-evaluation tool that gave community members the opportunity to assess their programmes and training (Mic Mac Maliseet Child Care Council, 1999). In 1997, the Mic Mac Maliseet Child Care Council submitted a proposal to establish criteria to assess and evaluate First Nations child care training programmes to Child Care Visions Canada. This evaluation tool addressed a variety of factors that impacted such areas as degree of collaboration among partners, First Nations development and control, cultural relevance, criteria for professional certification, policies and procedures involved in delivery and the philosophical orientation of the development and delivery. The MA WIW Health Council Inc. through the MA WIW Heath Commission, the MicMac Maliseet Child Care Council, and child care consultant Dixie van Raalte, undertook the project. A joint working group was formed and the project began. The initial programme evaluation tool package contained five programme evaluation booklets, one for each of the four key identified stakeholder groups [students, faculty, administrators, communities] and the fifth a universal booklet Did We Do What We Said We Would? 17 containing common qualitative questions for all stakeholders. The package was mailed out to the first sampling to the Atlantic region participants in August 1998. A sixth evaluation booklet was included which asked participants to evaluate the evaluation tools themselves. Results of that initial evaluation were collected and analyzed. Due primarily to the response of students regarding the amount of detail required in such a large form, a second evaluation booklet was designed entitled A Simple and Practical Approach. The intent of this booklet was to enable all stakeholders an opportunity to evaluate a programme in a less time consuming and more direct way (van Raalte, 1999). Students indicated that they were much more comfortable with this new tool. The designers had originally hoped that a second pilot sampling using both the In Depth Evaluation Booklets and The Simple and Practical Booklet would be conducted nationally. Unfortunately, additional funding could not be secured to support this endeavor. However, the project staff subsequently concluded, based on the responses from the single study that "the Atlantic Canada sampling provided sufficient diversity in the design and delivery of programmes within First Nations communities to indicate that the evaluation tool could potentially be used successfully across Canada" (van Raalte, 1999, p. 2) . Use of Program Review Results The increasing popularity of programme evaluation in public institutions supports an earlier finding in which Barak (1982) surveyed a third of the American Did We Do What We Said We Would? 18 public and private postsecondary institutions and found 82% used some form of programme evaluation. In his study of programme evaluation in community colleges, Hoey (1993) believed that an increasing demand for accountability in assessing outcomes has promoted this increase. Hoey's study used a systems approach to explain the various types of programme usage and their possible relationship to certain organizational factors within community colleges. The primary purpose ofHoey's (1993) study was to determine the extent of programme evaluation usage in community colleges in the United States, the degree and the kind of usage of the evaluation results, and the extent to which certain organizational factors were related to the type of usage of program evaluation results within community colleges. Hoey (1993) developed a taxonomy of programme evaluation which included three key items, direct usage, long-term impact and persuasive usage. The first, direct usage, may be measured in terms of decisional outcomes or immediate changes, such as programme budget expansion or programme closure. The second item long-term impact, may be assessed by observing to what extent factors such as student outcomes have improved as a result of programme evaluations. The third and fmal item in the taxonomy was persuasive usage which may be assessed by determining the degree to which programme evaluation fmdings were used politically to influence change in an organization or to convince administrators and funding agencies of the necessity of change. Did We Do What We Said We Would? 19 Hoey ( 1993) used organizational factors to determine if there was a significant relationship between position or stakeholder group and programme evaluation results. He examined components including organizational communication before, during and after programme evaluation, available leadership support for evaluations, the involvement of stakeholders, and the purposes ofthe evaluation. A survey instrument mailed to 253 academic officers at community colleges throughout the United States resulted in a 60% response. Similar to the fmdings ofBanta (1996) on the increase of programme evaluation, Hoey's study found that at the responding colleges, the use of programme evaluation as an evaluative mechanism had increased nearly 25% over a two-year period. Additionally, Hoey's study on the four types of programme evaluation usage indicated that "the clarification of programme goals, strengths, and weaknesses and the extent to which programme evaluation had enabled respondents to persuade others that changes were needed were the most frequently occurring results" (Hoey, 1993, p. 141). Summary Throughout North America, governments at all levels, as well as voluntary agencies, are spending large amounts of money on programmes and projects of different types to support early childhood education. This expenditure must be justified and the programmes must be seen as cost effective. Evaluation is an . Did We Do What We Said We Would ? 20 essential part of any educational programme, as we need to know whether the goals of the programme have been achieved. However, the approach may vary according to the group requiring the information (Curtis, 1999). Investors and outside agencies want to know whether their money is being properly spent and whether the programme is cost effective. For many of the stakeholders, accountability is of prime importance. If the benefits have not been as great as was hoped for, what must be done to amend the programme so that the learning goals can be achieved? Additionally, evaluating ECE training programmes addresses the success and effectiveness of particular teaching strategies. The reflective practitioner may devise a variety of teaching approaches to support student learning and needs to know which is the most successful. Few researchers disagree about why to evaluate; the greater area of discussion appears to be who and what to evaluate for, and how and when to evaluate (Potter, 1995; Devlin,1999; Curtis,1999; Apps,1991). More research needs to be done in this area. Knowledge of various types of successful strategies employed by institutions specific to partnership programmes would be of great benefit to all stakeholders in programme purchase, design and delivery. Specific Research Question Based on the criteria established in the Simple and Practical Approach Booklet of Today 's Learners/ Tomorrow 's Educators Self Evaluation Tool (1999) Did We Do What We Said We Would? 21 has the UCC, Williams Lake Campus Early Childhood Education Certificate programme provided a quality ECE training programme? Significance of the Proposed Study The idea for this project emerged from teaching a specially funded UCC, ECE certificate training programme sponsored by the Cariboo Chilcotin Aboriginal Training Education Council on behalf of the local First Nations bands. This programme was specially funded in order to ensure the training was culturally relevant for First Nations students of the represented bands. Specific courses were designed and implemented to address culture and traditional languages. Having been previously involved in designing and implementing culturally relevant First Nations curriculum, I was interested in seeing a relevant evaluation of these specially-funded programmes. This work also adds to the work done by van Raalt in cooperation with MA WIW Health Council Inc.(1999) on evaluating the success of First Nations ECE training programmes offered across Canada. Did We Do What We Said We Would? 22 II Design and Methodology This project focused on a multiple stakeholder programme evaluation of the University College of the Cariboo Early Childhood Education programme for First Nations students offered January- October, 2002. The programme evaluation sought to address a variety of factors that indicated, in the eyes of all stakeholders, programme success. The evaluation included but was not restricted to areas of jurisdiction, degree of collaboration among the partners, First Nations development and control, cultural relevance, professional certification criteria, programme delivery policies and procedures, and the philosophical orientation of the programme development and delivery. This study replicates a previous study (van Raalte, 1999) conducted in Atlantic Canada. Sampling Procedure The participants in this evaluation were those who were directly and indirectly involved with the programme. This resulted in a non-probability, convenience sampling population of the multiple stakeholders which included the 18 First Nations students originally enrolled in the UCC Williams Lake Campus, 2002 ECE programme. As the study focused on responses from First Nations students, the single non- First Nations-status student who has purchased a seat in the programme was not asked to participate in the survey. Funding stakeholders Did We Do What We Said We Would? 23 included the three Cariboo Chilcotin Aboriginal Training Education Council (CCATEC) members who dealt directly with the students. Community stakeholders included the education representative from each of the bands represented in the programme, a representative from each of the practicum sites utilized, and resource personnel. The instructional faculty and administration stakeholders included the four UCC faculty who taught the programme courses and the UCC administration directly related to the ECE First Nations programme, including the Programme Planner, the Continuing Education Co-ordinator, the Dean of Williams Lake Campus and the Divisional Chair of ECE . Instrumentation This summative evaluation project employed the Today 's Learners/ Tomorrows Educators (1999) survey questionnaire which was developed to evaluate First Nations Early Childhood Education Training Programmes in Atlantic Canada. The domains covered included programme delivery, resources and materials, programme supports/ linkages, theory and practice, and confidence in professional skills. Questions in all booklets used questions to which stakeholders responded on a Likert Scale (strongly agree, agree, disagree and strongly disagree) and qualitative, open-ended questions. Quantitative results from the 1999 pilot project showed that the evaluation tool was user friendly and was able to assess programme strengths and weaknesses. Totaling all the strongly agree categories, Did We Do What We Said We Would? 24 80% of the pilot project respondents would recommend this tool to other communities . Qualitative results from the same pilot project reported that the evaluation tool was very thorough in its identification ofECE training programme components. Programme design, delivery and content was effectively covered and that the information from such a tool would be helpful in improving future ECE training programmes (van Raalte, 1999). This tool, however, has only been recently developed and used and no analysis has taken place to identify the validity or stability over time with this instrument. Procedures Seeking permission. Initially, before the project began, a written letter was obtained granting permission to use the Simple and Practical Approach Booklet from the Todays Learners I Tomorrows Educators from the original research group. (see Appendix A). Upon receipt of this permission, a second letter (see Appendix B) to attain written authorization to distribute the evaluation and use its results for this project was sent to the UCC Williams Lake Regional Campus Dean. Once this written authorization to distribute and use the results was received, the proposal for this project was then submitted both to the Ethics Review Board for the University College of the Cariboo and the Ethics Review Committee of the University of Northern British Columbia (see Appendix C). Permission was received from both UCC and UNBC to proceed with the research. Did We Do What We Said We Would? 25 Informing the participants. Individual participants were provided with a letter of information (see Appendix D) regarding the project and the programme evaluation tool. Participants were requested to sign an informed letter of consent (see Appendix E) regarding the purpose of the project, their participation in the project, and the potential future uses of the project results. The UCC Ethics Review Board additionally requested a statement in the consent letter be included stating a completed and returned questionnaire would assume consent was given. This was added. Distributing the evaluation. The subjects were provided with the letter of subject participation and the questionnaire. Upon their completion of the UCC ECE programme of study in October 2002, they were asked to complete The Simple and Practical Approach evaluation booklet (see Appendix F). Students were each given their own copy of the booklet in their final class and a return envelope addressed with the researcher's name and UCC mailbox location. Faculty and administration had their booklets and a return envelope with the researchers name and mailbox location placed in their faculty mailboxes at the beginning of October 2002. The evaluation booklets for the CCATEC staff and the community representatives were hand-delivered at the beginning of October 2002. Stamped return envelopes with the researchers name and UCC address were provided. As this was a summative programme evaluation, the timing of the completion of this tool coincided with the Did We Do What We Said We Would? 26 completion of the programme. Evaluatioil packages were provided to all stakeholders by October 11 ,2002 in the most economically and efficient manner possible. All stakeholders were asked ro ~omplete the booklets in a space convenient to themselves, answering all questions to the best of their ability in the booklet and to return the completed booklet in the attached envelopes by October 30, 2002 to the researcher's mailbox at UCC Williams Lake Campus. All booklets remained anonymous. However, as each stakeholder would be responding in the appropriate section of the evaluation, stakeholder groups would be identifiable, but not the individual participants. Beginning October 31, 2002 the returned evaluations were opened by the researcher, sorted by stakeholder group and data analysis began. Did We Do What We Said We Would? 27 III Results A total of nine (21.4%) completed questionnaires of the forty two distributed were returned. Given my close association with the programme, this was an unexpected response rate. Research, however, has indicated that participantdependent returnable questionnaires have a typically low response rate (Berg, 1995). The respondent breakdown consisted of five out of eighteen students (27%) , three out of nineteen community I faculty members (16%) and one out of seven administrative I funding representatives (14%). This represents a 21.4% overall total response rate. Quantitative Results of Common Response Questions To address the overall low n (n = 9) and each individual stakeholder group low n; (students n = 5, faculty/administrators n = 3 and funders n = 1), the three stakeholder response categories were collapsed into a single response category. There were simply too few faculty and administrative responses received in order to do any statistical analysis beyond category descriptions between the stakeholder groups ( D. Broderick, personal communication, March 25, 2003). The common Likert Scale response categories were collapsed from the four categories of strongly agree, agree, disagree and strongly disagree to two categories of agree and disagree. The data collapse was justified because fme resolution of a low n analysis Did We Do What We Said We Would ? 28 was not warranted (Broderick, 2003). The responses to common quantitative questions on programme delivery, resource and materials, programme supports, theory and practice and confidence in skills were examined by binomial probability distribution Calculating the frequency distribution by using a measure of entropy or chi square was rejected because of the low n ( Broderick,2003). The binomial probability distribution is used when the measure is dichotomous (2 outcomes) and both outcomes are mutually exclusive and exhaustive. The sum of the probabilities for the events occurring equals one. In analyzing the received data the question "What is the probability of r successes given nand p?" was asked. As the study drew from a non probability convenience sampling, and such a small n response was received, the use of the binomial probability distribution provided an opportunity to see how the collapsed and recorded data reflected as probabilities. The exact probabilities indicated statistical significance in the topic areas of programme delivery and practicum settings meeting practical requirements with p = .07 or 70% probability exceeding p > .05 or 50%. Educational and programme significance was reached in the areas of co- ordinator support, instructor support and transferability of classroom materials with all respondents choosing the agree section. Additionally eight of the respondents marked the agree section in the areas of administrative support and preparation for work in daycare settings. The topic questions which addressed practical settings reflecting First Nations culture and preparation for Did We Do What We Said We Would? 29 work in First Nations daycare programmes achieved an exact probability of p = .16 or 16%, which, while not statistically significant, does indicate a small level of programme success significance with 6 respondents marking the agree section. The final topic of resource and materials was the only category that most respondents marked the disagree section. This identified lack of available resources and materials attained p = .24 or a 24% percent rate of probability. - ---------------------------------. Did We Do What We Said We Would? 30 Table 1 COMMON QUANTITATIVE LIKERT RESPONSE RESULTS A* D** Exact Probability*** Reflection of First Nations Culture 7 2 .0703 Adequate funding for student needs 7 2 .0703 Adequate teaching facilities 7 2 .0703 Sufficient First Nations materials available 4 5 .246 Materials accessible 4 5 .246 Topic: Programme Delivery Topic: Resource and Materials Topic: Support support from co-ordinator 9 .0019 Support from instructors 9 .0019 Support from admin 8 1 .017 Topic: Theory and Practice Classroom material transferable to practice 9 Practicum setting meets practical requirements 7 2 .0703 Practical experience reflects First Nations culture I values 6 3 .164 Prepares students for daycare settings 8 I .017 Prepares students for First Nation Daycare settings 6 3 .164 .0019 Topic : Confidence in Skills Notes N = 9 , total respondents *A: agree **D : disagree *** p >.05 Did We Do What We Said We Would? 31 Qualitative Results The universal questionnaire component encouraged answers that related to the strengths and weaknesses of the programme. Respondents praised specific programme components and indicated areas for future considerations. It was in this section of the survey that the participants had the opportunity to answer in their own voice citing specific examples. The three prominent themes that emerged were cultural relevancy, practicum, and resources and materials. Focus on cultural relevancy. Stakeholders stated that the programme delivered lacked in culturally relevancy. In all aspects of the programme development and delivery, from curriculum materials, resource materials, practicum experiences to staffing, cultural relevance was discussed. One of the most recurring issues was the lack of relevant First Nations ECE training resources, and although this was not a fault directed at those specifically delivering the programme it did impact the participants. The following excerpts represent some of the concerns expressed: • Hard to integrate the areas where materials and relevant curriculum models are not culturally relevant • Instructors lacked knowledge of specific local First Nations groups • Culturally relevant practicum sites were not available • More in depth knowledge and training of the history of each nation, beginnings and future goals so students know what to expect and see where things are Did We Do What We Said We Would? 32 coming from • Local communities did not have the opportunity to participate in planning curriculum • Readings and textbooks did not reflect First Nations culture Although the programmes were not as culturally relevant as participants would have liked, respondents did acknowledge the areas in which cultural relevancy was demonstrated : • Cultural relevancy was encouraged in assignment choice • Although far away instructors did offer cultural relevant practicum sites • Multiculturalism was presented in all courses • Students were encouraged to invite local and relevant speakers for the programme, many were First Nations elders • The time spent at the UBC Longhouse Child Care promoted exemplary First Nations Child care in action • The use of materials from Bkejwanong Children's Centre, although it promoted the Cree culture was interesting and useful • Instructors attempted to empower students to take control and participate in their own learning activities and view concepts from their own context The responsibility of ensuring cultural relevancy fell to both the students and faculty. The students live their culture and by their very presence in the classroom included it in all aspects of the programme. The faculty strove to expand on cultural relevance as part of the design of the programme through accessing minimally available Did We Do What We Said We Would? 33 materials and resources. Practicum component. All participants valued the practicum components of programme. It was viewed as the most stimulating, enjoyable and valued aspects of the ECE training. Students appreciated the opportunity to apply and integrate what they had learned in the classroom in practical situations while working with mentor teachers in real life experiences. The practicum experiences also encouraged the continued development of self confidence and professional skills. • The practicum never seemed like I was going to school , !felt like I was really working there • People treated me with more respect when I went back to my reserve as a student teacher • The lab school we ran was fun, I wish we could have done more of it • When I was in practicum all the class stuff made sense • We tried hard to find appropriate (within traveling distance) First Nations child care programme for practicum , there are not many in the area The use of non-aboriginal child care practicum sites, although supporting the professional training, contributed to the lack of cultural relevancy. This same lack of availability appears in the following section of relevant resource and materials. Access to resources and materials. It is difficult to implement First Nations philosophies, develop First Nations centred programmes and have students in First Nations practicum sites if the resources and not available. Perhaps one conclusion Did We Do What We Said We Would? 34 to emerge from the data might be the realization the First Nations training materials are lacking and the development of First Nations resources and materials created by First Nations people is essential if these programmes are to be offered on the cultural level that stakeholders seem to desire. Comments regarding resources and materials included: • More use of First Nations instructors and resource materials would have made the courses more interesting • How come all the videos and books don 't include First Nations people or ideas ? • The only training materials that showed First Nations people were the Metis programmes in Manitoba, the Cree in Ontario and the Inuit. These are not our people. Strengths and Weaknesses. The universal questionnaire additionally asked questions regarding the strengths and weaknesses of the programme. Among the identified strengths of the UCC training programme were (a) a consistent classroom for the entire programme; (b) the mutually trusting, supportive, inviting learning environment; and (c) the ability of instructors to use a variety of teaching and 1 assessment strategies to meet the needs of all students abilities and skills. Some comments regarding the strengths of the programme were as follows: • Being able to deliver a strong programme with minimal dollars • Good worfPng relationship with all partners • The teachers let lJS do oral exams instead of written ones Did We Do What We Said We Would? 35 • The teachers were real people, not just talking heads • We laughed a lot • All the field trips were better than learning from books Although the respondents praised many aspects of the programme, they did point out areas that would benefit from further considerations if the programme was to be offered again. These included: (a) more funding dollars are needed in the areas of child care for students, travel, and accessing relevant resources and materials; (b) instructors need to enhance their knowledge of First Nations philosophies and ideologies to deliver programmes to First Nations students; (c) students need to have more time to complete the course work . The January to October full-time course frame was too stressful on students and their families, both emotionally and fmancially; (d) the inclusion of daily practicum experiences; (e) the development of more culturally-relevant training materials and local community resources; (f) the participation of First Nations representatives in the planning of the programme delivery; (g) some long term planning for ongoing training so students (and bands) could plan ahead rather than scramble at the last moment to fill a one time offered training opportunity. Did We Do What We Said We Would ? 36 IV Discussion This programme evaluation project sought to address a variety of factors that indicated programme success in the eyes of all stakeholders, including but not restricted to the degree of collaboration/supports among the partners, First Nations development and control, cultural relevance, professional certification criteria, programme delivery policies and procedures, and the philosophical orientation of the programme development and delivery. Patterns and Common Themes in Responses Programme delivery. Both the quantitative and qualitative results indicated satisfaction with the physical plant and classroom location. It would appear that having a consistent classroom setting for the entire time frame of the programme encouraged the development of a positive and supportive community. Students indicated in the universal questionnaire component that Teaching on reserve for elder visibility would have been a better location. The Royal Commission Report Gathering Strength ( 1996) states "Offering studies in the community or closer to home has been one way to improve access [to educational programmes]" (p. 510). The faculty response indicating Overall time frame - summer I fall are difficult times for placing students as a weakness of the programme represents a key obstacle. While it is acknowledged that access to funding dollars often dictates Did We Do What We Said We Would? 37 programme delivery time frames, the availability of community supports - particularly when accessing practicum settings- needs to be identified before setting programme time frames. Degree of collaboration/supports among partners. Overall the questionnaire responses suggested that the students, faculty and administration/funders were satisfied with the programme and their participation in it. Although as the administrative/funding stakeholder respondent indicated there was no community involvement in programme development all respondents indicated that a mutually supportive environment existed throughout the programme. All respondents viewed the instructors as beingflexible in meeting their needs. Patterson 1973; cited in (Purkey, 1996) stated that "The atmosphere created by good interpersonal relationships is the major condition for learning" ( p. 61). Cultural relevancy, First Nations development and control. The Royal Commission Report Gathering Strength (1996) covered the issue of ownership and development of training programmes delivered in or to First Nations communities. It suggested that where non-Aboriginals continue to dominate the areas of formal education, community control must be a dimension of the programmes delivered to First Nations communities. The respondents all identified this as an area for improvement. Students indicated a wish for the programme to be more reflective of their own particular First Nations culture(s) as well as needing more access to Did We Do What We Said We Would? 38 information regarding First Nations child rearing practices, relevant ECE teaching materials and resources, visibility of elders and the want to be on-reserve. The issue of language has always been an integral part of the building of a culturally relevant programme. "The incorporation of Aboriginal language in early childhood programmes has been a focal point for the drive to ensure that learning in such settings has a distinctly Aboriginal character" (Gathering Strength, p. 450). It is apparent that the stakeholders and research literature support the fact that cultural relevancy can be enhanced through the development of appropriate First Nations resources, and the use of Aboriginal language in programmes can benefit programme delivery. It is worth noting that all stakeholders agreed that the programme would have benefited from the hiring of First Nations instructors for the courses; however, as this did not occur, all stakeholders seemed satisfied with the use of First Nations representatives as guest speakers and workshop providers. Faculty indicated the need for more training related to First Nations culture and child care methods. The involvement of Elders in the educational process for First Nations was identified in the evaluation. The 1996 Gathering Strength document said, "The legacy of elders is precious, unique and irreplaceable. In the models of Early Childhood Education adopted by some communities, elders have been able to resume their own role in intergenerational teaching" (p. 452). Faculty indicated they were open in acknowledging they were not as culturally aware as they needed to be. It would Did We Do What We Said We Would? 39 follow then, that they were part of the issue of cultural relevancy, and because First Nations students are likely to receive a significant amount of their education instruction from non- Aboriginal instructors, there is a need to educate nonAboriginal teachers to the ways of Aboriginal people. Volume 3 of the Royal Commission Report states, "Values reinforced by the teachers, the inclusion or exclusion of Aboriginal materials and perspectives in the course, the type of interaction in the classroom and the relationship between teachers, students and community stakeholders will all effect the success of the programme" (p. 498). In continuing to address cultural relevancy, the potential key for faculty in developing programmes for First Nations communities is their ability to draw upon the resources at hand, from the people in the community and the students themselves. Allowing students the opportunity to draw and reflect upon their own cultural knowledge and experiences as resource material facilitates the construction of new knowledge that benefits all who participate in the learning environment Faculty did indicate that students were encouraged to and did invite many First Nations elders into the classroom as guest speakers. This possibly contributed to enhancing the cultural relevancy for all involved. In keeping with the evaluation results cultural relevancy was additionally addressed in the administrative response. Lack of collaboration between the First Nations communities and the college in the development of the delivered programme possibly contributed to the lack of cultural Did We Do What We Said We Would? 40 relevancy as well. This may have provided more consistency to the programme and relevant resources and materials. Professional certification criteria. Respondents indicated they felt the students were prepared for employment in the childcare field in on-and off-reserve programmes. This is seen as the most essential element of any Early Childhood Education Training. According to the Canadian Child Care Federation (1999), Studies in professional practicum provide opportunities to observe exemplary models in a wide variety of ECCE (Early Childhood and Care Education) settings; implement acquired knowledge, skills and attitudes in supportive environments; analyze field experiences through reflection, self assessment, and feedback from ECCE professionals; and synthesize the practical and theoretical elements of one's learning. (p. 2) The questionnaire responses indicated that practicum opportunities occurred on-and off-reserves, this provided students with a wider range of programme investigations. Students indicated that the professional development trip to a large urban centre to observe additional programmes and programme support agencies was a highlight of the programme. This possibly contributed to further knowledge attainment of how programmes interpreted various philosophies from a first hand concrete perspective. Faculty also indicated the onsite lab school experience as a strength of the programme again, enhancing the ideas of guided practicum experiences. Did We Do What We Said We Would? 41 IV Conclusion The intent of this project was to build on the original Atlantic Canada study (1999) and contribute to the ongoing evaluation of First Nations Early Childhood Education Training Programmes through the use of a multiple stakeholder evaluation tool. The Today 's Learners I Tomorrow's Educators (1999) evaluation tool was developed to assess purchased and/or developed training programmes and included the opportunity for respondents to reflect and comment on many issues pertaining to the training programmes. The particular tool used in this project-The Simple and Practical Evaluation- requested stakeholders to respond to a series of five universal qualitative questions and a series of quantitative specific stakeholder questions. A total of nine out of the forty two questionnaires were returned from the various stakeholder groups which included students, faculty I instructors, community supports, administration and funding partners. Limitations of Design Threats to Internal and External Validity Internal validity. The results from this study may have been affected by certain standard internal validity limitations including history, subject selection, .instrumentation, subject and experimenter effects and statistical conclusion. The threat of history defmed as "extraneous incidents or events affecting the results that occurAuringthe research" (Macmillan & Schumacher,1997, p. 184) must always be Did We Do What We Said We Would ? 42 considered whether it comes from the world at large or from the classroom. In this particular case a funeral for an elder meant the cancellation of the final class, which in turn meant that surveys to students were hand-delivered and a self-initiated return of them was anticipated. Secondly the selection of subjects may have posed certain limits. As this was a non- probability convenience sample group ,difficulties arise in generalizing the data to other populations. This was also an issue in the pilot study (van Raalte, 1999). Subjects, for their own reasons, may have wanted to have this programme viewed as successful and responded more positively to questions than may have been the reality reflected in the programme itself. As this questionnaire survey evaluation tool is still in a somewhat developmental phase,results may vary due to misinterpretation or poorly written questions. Language may have also created a barrier as some of the respondents operate in English as a Second language with Shuswap, Carrier and Chilcotin being their first language(s). An attempt to provide translation of this evaluation proved difficult as many of the terms and phrases do not exist in the first languages. As this evaluation was administrated by an instructor of the programme, researcher bias may have influenced the completion of the tool by the students and instructors in particular. The original study cited low response rates and this also occurred with this project. Although internal validity can be difficult to control, it is possible that the results achieved are reasonable. External validity. This evaluation is keyed for a specific type of ECE Did We Do What We Said We Would? 43 training programme. As the project was limited to a specific population at a specific institution, generalizability and external validity may not be noteworthy. It will, however, add to the data collected in the original project and provide general recommendations to be considered by other ECE First Nation partnership programmes. Overall Conclusions Through data collection and analysis of the results, students, faculty, community and administrative partners reflected upon their experiences within the ECE training programme. Their responses revealed many aspects about the programme. Students respondents saw the programme as flexible in design and delivery. While they would have preferred the courses taught on reserve, the consistent and stable classroom space seemed acceptable. Students contributed to the cultural relevancy of the programme by arranging workshops and guest speakers from their home communities. They also viewed the field trip to a large urban centre as a strength of the programme. This adds to the need for concrete, hands-on learning opportunities to observe various programme philosophies in action . Faculty members provided an inviting and respectful learning environment and, although challenged to access and include culturally relevant programme content, they were able to ensure the training programme met the provincially Did We Do What We Said We Would? 44 standardized programme outcomes as set out by the British Columbia Community Care Facilities Licensing Regulations (2002). The lack of cultural relevancy was the common concern for all stakeholders. It was clear from the collected data that participants would like to eventually have programmes that reflect a stronger cultural relevance as opposed to having cultural content added to the course material . The desire for cultural relevancy, along with other issues and concerns voiced by the respondents is supported by both the Record ofProceedings from the 1 National Conference on Aboriginal Training Programmes in Early Childhood Education (Montreal, 1995) , Gathering Strength ; The Report of the Royal Commission on Aboriginal People(l996), and Today 's Learners Tomorrow's Educators (1999). At the same time, stakeholders indicated that the programme was as culturally relevant as it could be. There are simply not enough First Nations instructors, books, videos, learning materials and licensed child care centres to assist with the cultural relevance of Early Childhood Education training programmes. Respondents did seem to feel that the students would be able to fmd work in community and on reserve programmes and be able to put theory into practice. It is understood that delivered programmes and practicum experiences in local communities and on reserves enhances the learning experience for students, Did We Do What We Said We Would? 45 instructors and community members. Students were able to broaden their knowledge of other programmes, centres and access resources by connecting to larger and more diverse communities. This group of stakeholders supported many of the findings from the original Atlantic Canada study. While the simple answer to the study question is yes, the University College of the Cariboo did do what they said they would; the more difficult challenge is how to improve on the delivery for future programmes. The various stakeholders identified what components are missing in the programme. The next step is to explore initiatives that will enhance future programmes from the perspective of all stakeholders. Recommendations The results from this project indicate support for the recommendations that emerged from the original study. They include the following • Stronger shared beliefs and understandings at the beginning of any First Nations ECE training programme. • Continued levels of communication between partners throughout the programme time frame to enhance emotional, fmancial, and social support. • A strong foundation of cultural relevancy upon which to build relevant ECE training programmes, not just add on pieces of information. Curriculum needs to be developed in partnership with the local community and ECE professionals. Did We Do What We Said We Would? 46 • Monies need to be allotted to the development of First Nations training materials (videos, texts, articles etc.) by First Nations peoples. • Encouragement of First Nations instructors in the classroom settings. • Enhancement of training for non-Aboriginal instructors in the history, philosophies, language and culture of First -Nation peoples • Ongoing professional development opportunities to further students in the field and establish an adequate number of trained personnel to meet the needs of communities Possibilities for future research. Future studies in the area of First Nations ECE programme evaluation could include (a) the comparison of on- and off-reserve delivered programmes; (b) a comparison of purchased programmes versus locally developed programmes; and (c ) the adaptation and use of the Today 's Learners, Tomorrows Educators (1999) evaluation package as a cumulative programme evaluation as opposed to summative programme evaluation. Summary. It appears that the time has come to put into practice the strategies and solutions that have come from previous reports, as well as what has come from this programme evaluation. First Nations peoples have made it clear what is missing in the programmes that have been delivered to their communities. It is now the time for initiatives that will enhance the education of First Nations instructors, the development of First Nations ECE training resources, as well as the development of Did We Do What We Said We Would? 47 ECE training programmes created by First Nations peoples. On a personal level, as a researcher this project has highlighted the importance of the presentation of an evaluation tool, including attention to the clarity of the questions asked. I have also learned the necessity of creating a means for ensuring higher returns in order to have a stronger data set to draw from. As an ECE professional and instructor I continue to support the concept that education, at any level, should be a collaborative, cooperative experience that can best be realized by places, policies, programmes and processes which are specifically designed to invite development of all involved. Thomas Reevesl989; cited in Johnson and Foa (1989) states "evaluation does not make decisions, people do." 48 Did we do what we said we would? References American Psychological Association. (200 1).Publication Manual of the American Psychological Association (5th Ed.), Washingtion D.C: Author Apps, Jerold W.(1991) Mastering the Teaching ofAdults. Malabar Florida: Krieger Publishing Co. p.123 Auclair, L. A. (1990). "Historical development of departmental leadership in American higher education." In Sheila Bassoppo-Moyo. The Relationship between Department Chairs and Academic Program Review. AcademicLeadership.Retrieved March 3 2002, from http://www.academicleadership.org Ball.J.,Pence.A.(1997). Beyond Developmentally Appropriate Practice: Developing Community and Culturally Appropriate Practice. Young Children International : March 1999. p. 46-48. Ball, J., Pence,A.(l999). A postmodernist approach to culturally grounded training in early childhood care and development Australian Journal of Early Childhood. Vol.25.No.l.21-25. Banta, T.W., Rudolph, L.B., Van Dyke, J. & Fisher, H. S. (1996). Performance funding comes of age in Tennessee. Journal of Higher Education, 67,23-45. Barak, R. J. ( 1986). A perspective on the antecedents, present status, and future developments of academic program review in higher education. A paper presented at the annual meeting of the Association for the Study of Higher Education. San Antonio, TX. (ERIC Reproduction Services No. ED268887).In Sheila BassoppoMoyo. The Relationship between Department Chairs and Academic Program Review. Academic Leadership. Retrieved March 3 2002, from http://www. academicleadership. org Bredekamp,S., Copple,C. (Eds) (1997). Developmentally Appropriate Practice in Early Childhood Education Programmes. Revised edn. Washington, DC: National Association for Young Children.p.27 Chelimsky,E., Shadish, W. R. (Eds.). (1997). Evaluation for the 21st Century: A Handbook. Thousand Oaks, CA: SAGE Publications, Inc. Education Review. Retrieved March 5 2002, from, http://www.ed.asu.edu/edrev/reviews/rev6.htm 49 Did we do what we said we would? Curtis,A.(1999).Indicators of Quality in Early Childhood Education and Care Programmes. Retrieved Feb 25 2002,from http://www. worldbank. org/children/nino/basico/Curtis. Devlin, L. E.( 1991) . Program Planning in Adult Education . Certificate in Adult and Continuing Education. Course Manual. University ofVictoria 199l.p120-153. Doherty- Derowski, Gillian (1994) Quality Matters: Excellence in Early Childhood Education Programs Don Mills, Ontario: Addison Wesley Publishers Limited: 11,70-79 Frechtling,J., Sharp-Westate, L.(l997).User-Friendly Handbookfor Mixed Method Evaluations.National Science Foundation RED 94-52965.Retreived March 15 2002 from http://www. ehr.nsfgov/EHRIREC/pubs/NSF9 7-153/STARTHTM#TOC. Gribbons ,Barry. Herman, Joan (1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5(14). Retrieved March 10 2002: http:/I ericae.net/pare/getvn.asp ?v=5&n= 14. Hoey IV, J .J. ( 1993 ). The utility of academic program review: An examination of organizational factors and their impact on usage of program review results at singlecampus community colleges in the United States. In Sheila Bassoppo-Moyo The Relationship between Department Chairs and Academic Program Review. Academic Leadership. Retrieved March 3 2002, from http://www.academicleadership.org Hoey IV, J.J. (1995). Organizational factors in program review. In R. J. Barak & L.A. Mets (Eds.), Using academic program review (pp. 37-47). New Directions for Institutional Research, no. 86. San Francisco: Jossey-Bass. Johnson, K., Foa.L. (1989) Instructional Design: New Alternatives for Effective Education and Training. New York: MacMillan Publishing. pp139-162 . Macmillan, J. H., Schumacher, S.(1997) Research in Education: A Conceptual Introduction. 4 ed NewYork: Addison Wesley Longman Publishers Inc p.184 Miles, M.B., and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed. Newbury Park, CA: Sage. P.4 50 Did we do what we said we would? Moss,P., Pence, A. (Ed.)(1995 ). Valuing Quality in Early Childhood Services Teachers Columbia University :College Press. pp. 50, 117-172 Patton, M.Q. (1990). Qualitative Evaluation and Research Methods, 2nd Ed. Newbury Park, CA: Sage.p.27, 40-43 . Potter, G. (1994). Instructional Design in Adult Education. Certificate in Adult and Continuing Education. Course Manual. University of Victoria .1994 p.196. Purkey, W.W., Novak, J.M. Inviting School Success, 3rd Ed. Belmont CA: Wadsworth. p3 Report of the Royal Commission on Aboriginal Peoples (1996).Gathering Strength,1996 (Vol3). Department of Supply and Services , Canada Communication Groups, p .440- 510. Scriven, M. (1999). The methodology of evaluation. In: R.W. Tyler, R.M. Gagne & M. Sriven, Perspectives of curriculum evaluation. Chicago: Rand McNally. Retrieved March 11,2002 from http://ericae.net/pare/Home.htm Van Raalte.D.(1999). Today 's Learners/ Tomorrow 's Educators. Sponsored by the Mawiw Council of First Nations (Big Cove/Burnt Church!Tobique).Canadian Child Care Federation. Verduin, J.R. Jr. & Clark, T.A.(1991). Distance Education. The foundations of effective practice. San Francisco: Jossey-Bass. In Maurice de Voider, ed., From Penny Post to Information Super-Highway: Open and Distance Learning in Closeup. Leuven, Belgiwn: Uitgeverij Acco, pp. 136-43. Retrieved March 11 2002 from http:/lwbweb4. worldbank. org/DistEdlhome.html Did We Do What We Said We Would? 51 Appendix A Original Research T earn Correspondence Patricia & Paul From: To: Sent: Subject: "Patricia & Paul" "Patricia McClelland" March 12, 2002 1:18PM Fw: Today's Learners, Tomorrow's Educators ---- Original Message - --From: Dixie vanRaalte To: Patricia & Pauj Sent: Tuesday, March 12, 2002 12:11 PM Subject: Re: Today's Learners, Tomorrow's Educators Hi Patricia, I do nt have a problem with you using it. There is a copyright to ChildCare Visions and Mawiw Council Inc. so I think you need to cite the source and those it is copyrighted to in your writing. It was developed to use, so enjoy. Keep in touch and let me know about your work etc. Regards, Dixie Patricia & Paul wrote: Dixie ( and Eve) Thank you for the resource Today ' s Learners , Tomorrow ' s Educators . It was and is extremely useful . I would like to use it as the jump off for my Master ' s Project . I am currently teaching the ECE programme at the University College of the Cariboo in British Columbia . The programme delivery has been purchased through the Cariboo Chilcoltin Aboriginal Training Education Council for five of the local first nations bands in the area . I would like to seek permission for using the assessment survey tools including 1 . How does this programme measure up 2. Universal Questionnaire 3 . Student Questionnaire 4 . Community Questionnaire 5 . Faculty Questionnaire 6 . Adrninistration Questionnaire 7 . A simple and practical Approach Again thank you for all your assistance to date. If you would like I will forward you a copy of the results at the completion of the project. Patricia Ori ginal Message From: " Dixie vanRaalte " To : Sent : Thursday, January 31 , 2002 1 : 58 PM Subject : Today ' s Learners, Tomorrow ' s Educators Greetings Patricia , Thank you for enquiring about our resource. This manual wa s developed as a way for people to evaluate curriuclum being developed , purchas ed our currently used . If you are still interested in this resource , please e - mail your complete address to my assistant at eve@nb . sympatico.ca and she will put a package in the mail . I am away working in a community next week ( northern Quebec) so Eve will look after this for you . Appreciate your interest in our previous project and if you should have any further 04/02/2003 questions about the material, please do not hesitate to e-mail me . Thanks, Dixie Lee vanRaalte Past Coordinator Today's Learners, Tomorrow ' s Educators 04/02/2003 Did We Do What We Said We Would? 54 APPENDIXB Letters of Permission to Conduct a Research Study April2002 To Rob Anderson, Dean University College of the Cariboo Williams Lake Campus From: Patricia McClelland Co-ordinator ,2002 Early Childhood Education Programme University College of the Cariboo Williams Lake Graduate student UNBC I am writing this letter to attain your permission to conduct the following evaluation survey research for my Master in Education project at the University of Northern British Columbia. Study details There has always been the recognition that quality training is a primary indicator of quality child care ( van Raalte, 1999). The Today 's Learners/ Tomorrow's Educators (1999) project was initiated to examine a strategy for assessing programmes from a multiple stakeholder perspective. The project attempted to improve partnership programme delivery by giving the students taking the courses, the faculty teaching the courses, the communities I sponsors funding the courses and the post secondary institutions administrating the programmes, feedback about knowledge, skills, and approaches which determined successes and identified areas for improvement. In Phase 1 of the original project, all identified stakeholders participated in evaluating the programme from their own perspectives using specifically designed evaluation questionnaires. As well, all participants completed a universal questionnaire. In phase 2 of the original project a Simple and Practical Approach format was developed and used. This format included a universal questionnaire section and a section for each of the stakeholder groups. The original project was developed and implemented in the Atlantic region of Canada under the -guidance of the MA WIW Health Council Inc. This project, using the Simple and Practical Approach format, will add to the existing data by using the evaluation tool with the University College of the Cariboo, Did We Do What We Said We Would? 55 Williams Lake Campus, Early Childhood Education Programme 2002 , which has been funded by the Cariboo Chilcotin Aboriginal Training Education Council for students from the local First Nations communities. The results will answer the question: Has the University College of the Cariboo, Williams Lake Campus Early Childhood Education Certificate programme 2002, provided a quality ECE training programme according to the Simple and Practical Approach ofToday's Learners/Tomorrow's Educators Evaluating First Nations Early Childhood Education Training Programmes (van Raalte, 1999)? Thank you for our consideration and written approval. Patricia McClelland mcClelland@hotmail.com Attachments 1) Quick and Simple Evaluation Booklet g --;:::.::::;:: HE • .NIVERSITY OLLEGE FTHE ARIBOO illiams Lak e mpu s , · ~ ~ Otiver ~ hamsLake. B: S Hfl.! e o o ~ 12SO ~ ~ t F·c: t250 • ~ ~~ ~ ~ 2-: : :·c ~~ ~ ~ Ms. Patricia McClelland The University College ofthe Cariboo 301-383 Oliver Street Williams Lake, British Columbia V2G 1M4 June 18, 2002 Dear Patricia: This letter is in response to your request to conduct an evaluation survey research for your Masters in Education project at the University of Northern British Columbia. Given use of the written procedures in your letter, you have my permission to conduct the study. I believe it will be a worthwhile project and I would be interested in knowing your results. Yours truly, Rob Anderson Dean sm G:ISMOON\ Rob Anderson\McC/ellandjunl8 2002.doc Did We Do What We Said We Would? 57 Appendix C Ethics Approval from UCC and UNBC The University College of the Cariboo Ethics Committee For Research and Other Studies Involving Human Subjects Certificate ofApproval Williams Lake Early Childhood Education Patricia McClelland SPONSORING AGENCIES n/a CERTIFICATION The protocol describing the above-named project has been reviewed by the Committee and the experimental procedures were found to be acceptable on ethical grounds for research involving human subjects. J 7 . Sharon S1mpson Interim Chair, Research Ethics Committee This Certificate of Approva l is valid for the above term provided there is no change in the experimental procedures. uflsc UNIVERSITY OF NORTHERN BRITISH COLUMBIA Research Ethics Board MEMORANDUM To: Patricia McClelland RR #5, Nazko Road, Quesnel, V2J 3H9 From: Alex Michalos Chair, Research Ethics Board Date: May 17, 2002 Re: Ethics Review EP 2002.0430.59 Did We Do What We Said We Would? A Multiple Stakeholder Evaluation of the University College of the Cariboo, Williams Lake Campus 2002 ECE Certificate Training Programme Thank you for submitting the above noted proposal for review by the Research Ethics Board. Your proposal has been approved and you may begin your research. If you have any questions, please feel free to contact me. Sincerely, Alex C. Michalos, Chair Research Ethics Board Did We Do What We Said We Would? 60 Appendix D Participant Information Sheet Dear -------------------- Date This study involves a questionnaire designed to comment on Early Childhood Education Training programmes for First Nations students. The study is being conducted by myself, Patricia McClelland , a graduate student from the University of Northern British Columbia, and has been approved by the University ofNorthern British Columbia and the University College of the Cariboo. As part of my research I am conducting a summative programme questionnaire of the success ofthe University College of the Cariboo , Williams Lake Campus 2002 Early Childhood Education training for First Nations students using the Simple and Practical Approach Booklet of the Today 's Learners! Tomorrow 's Educators Evaluation (1999) The questionnaire will seek to address a variety of factors that indicate programme success in the eyes of all stakeholders including , but not restricted to areas of jurisdiction, degree of collaboration among the partners, First Nations development and control, cultural relevance, professional certification criteria, programme delivery policies and procedures and the philosophical orientation of the programme development and delivery. Participation in the study should take no more than 50 minutes and is strictly anonymous within each stakeholder group. All responses are treated as confidential, and in no case will responses from individual participants be identified. Rather, all data will be pooled and published in generalized form only under the three broad categories of student, faculty and administration. Participants should be aware, however, that the research is specific to the UCC ECE Williams Lake 2002 programme so there is a small possibility that responses could attributed in general to individual stakeholder groups. If you agree to participate , I would like you to be aware of the following details. No individuals have reported adverse reactions to participating in this evaluation to date. No credit or monetary compensation will be received by any stakeholder or researcher. Participation is voluntary, refusal to take part in the study involves no penalty or loss of benefits to which participants are otherwise entitled, and participants may withdraw from the study at any time without penalty or loss of benefits to which they are otherwise entitled.Given the nature of the questionnaire, I cannot foresee any unusual risks to the participants. No deception is invo}ved, and Did We Do What We Said We Would ? 61 the study involves no more than minimal risk to participants (i.e. the level of risk encountered in daily life. ) You may respond or not to any inquiry without prejudice. You may withdraws from the process at any time. Your name will not be used in the study. I cannot guarantee total anonymity since my project is a questionnaire regarding a specific programme . You will have the opportunity to receive a copy of the completed project upon request. The evaluation booklets will be kept in a secure locked filing cabinet at the University College of the Cariboo, Williams Lake Campus for a period of no less than 5 years or until they are no longer needed and will be shared with no one other than my project supervisors, Dr. Paul Madak and Dr. Colin Chasteauneuf. At the end of this storage period , the booklets will be shredded at the campus. The material generated from this questionnaire may be presented at conferences. The material generated in the questionnaire will become part of a written project that satisfies the requirement of my master project and/or other publications deriving from this research. The nature of there publications would most likely be , but not limited to , scholarly research destined for an academic/professional audience. Upon completion, publications will be available at the UCC office for your perusal. I do not expect to benefit monetarily from this project in any direct way, but the possibility of a commercial publication at some point is ont out of the question. If participants have further questions about this study or their rights , they may contact the principal investigator , Patricia McClelland at work 250 392 8184 at any time. If participants wish to lodge a complaint or concern about this research they may contact Office of Research , UNBC , 3333 University Way, Prince George B.C. V2N 4Z9 or telephone 250 960 5820 and I or UCC Chair of the Research Ethics Committee- Human Subject 250 371 5734 or use the Human Subjects Feedback Form. I would like to thank you for your consideration of my request . If you choose to have your questionnaire included in this process, please complete the attached informed consent form. Sincerely Patricia McClelland M.ED. Graduate Student UNBC Did We Do What We Said We Would? 62 Appendix E For w --.: Informed Consent by Subjects to Participate in a Research Project or Experiment THE ..... UNIVERSITY COLLEGE OF THE CAR! BOO THE UNIVERSITY COLLEGE OF THE CARJBOO 900 McGill Road Box 3010 Kamloops, BC V2C SNJ Telephone (250) ~ Note: The University College and those conducting this project subscribe to the ethical conduct of research and to the protection at all times of the interests, comfort, and safety of subjects. This form and the information it contains are given to you for your own protection and full understanding of the procedures, risks and benefits This consent form, a copy of which has been given to you, is only part of the process of informed consent. It should give you the basic idea of what the research is about and what your participation will involve. If you would like more details, feel free to ask at anytime. Please take the time to read this carefully and to understand any accompanying information. I have been asked by Patricia McClelland of the Early Childhood Education Division/Department of The University College of the Cariboo, telephone number 250 392 8184, to participate in a research project entitled: Did We Do What We Said We Would? which encompasses the details as they appear in the attached participant information letter and informed consent letters. My signature on this form indicates that I understand the information regarding this Did We Do What We Said We Would? 63 research project including all procedures and the personal risks involved and that I voluntarily agree to participate in this project as a subject . I understand that my identity and any identifying information obtained will be kept confidential.- - - - - - - I understand that I may refuse to participate or withdraw my participation in this project at any time without consequence. My involvement or non-involvement in this project is in no way related to my employment contract or to my status as a student I understand that I may ask any questions or register any complaint I might have about the project with either the chief researcher named above or with Rob Anderson Dean of Williams Lake UCC Campus , telephone number, 392 8000. The University College of the Cariboo. Ifl have any questions or issues concerning this project that are not related to the specifics of the research, I may also contact the Chair of the Research Ethics Committee - Human Subjects, Dr . Jeff McLaughlin,, telephone number,371 5734, or use the Subject Feedback form. I have received a copy of this consent form and a Subject Feedback form. Name: (Please Print) Address: Participant's signature - - - - - - - - - - - - - - - - - - Date Investigator and/or Delegate's signature _ _ _ _ _ _ _ _ _ _ __ Date ************************************************************************* ******************* Did We Do What We Said We Would ? 64 Appendix F Today 's Learners Tomorrow 's Educators Simple and Practical Evaluation Questionnaire Booklet (1999) . ~ ! ~ I ' Evaluating First Nations Early Childhood Education Training Programs (A Simple and Practical Approach) Section A Universal Questions 1. What are/were the strengths of this Early Childhood Education Training program? 2. What are/were the weaknesses of this Early Childhood Education Training program? 3.How was cultural relevancy reflected in this Early Childhood Education Training program? 4. How can cultural relevancy be improved in Early Childhood Education Training programs delivered to First Nations communities? 5. What are some positive experiences with this particular Early Childhood Education Training program? Section B Stakeholder Questionnaire Student Questionnaire Program Delivery 1. Overall, this Early Childhood Education Training program reflected/reflects First Nations culture and value. 0 agree I , strongly agree U disagree strongly disagree 2. Funding for this Early Childhood Education Training program was/is adequate to meet the needs of students. n agree 0 strongly agree 0 disagree n strongly -. disagree ~ 3. Teaching facilities for this Early Childhood Education Training program met/meets students learning needs. !_ [] strongly agree agree D disagree strongly disagree I Quality of Instructor 1. The Early Childhood Education Instructors were/are qualified . n strongly agree L I ' agree - - , disa!.!ree ~ strongly disagree 2. The Early Childhood Education Instructors were/are effective in including First Nations culture and values into my Early Childhood Education Training program. '~ _ i strong Iy agree agree disagree _ ' strongly disagree 3. The Early Childhood Education Instructors were/are flexible in meeting the needs of students. L..J strongly agree agree - , . disagree l_ strongly disagree ,' Resources and Materials 1. Sufficient resource materials for completion of course work for my Early Childhood Education Training program were available. ,- , strongly disagree [] disagree strongly agree = 2. Resources and materials needed for the completion of course work were/are accessible. -_ , stronrrly aaree e o ~ agree '---' ~ ~ ee :--; stronalv L..__J e .. disaaree ::- Support 1. It was/is possible to develop a relationship of mutual support and respect with the Early Childhood Education Training instructor. D agree [J strongly agree ~ c...._ ~ ee 0 strongly disagree 2. It was/is possible to develop a relationship of mutual support and respect with the other Early Childhood Education Training students in this program. 1 1: ·- - disagree agree ~ l_j strongly disagree 3. It was/is possible to develop a relationship of mutual support and respect with Early Childhood Education Training administrators . _ strongly agree - ; disagree agree strongl y disagree 4. It was/is possible develop a relationship of mutual support and respect with Early Childhood Education Training administrators from the community. !- • strongly agree - , disagree agree 1 - strongly disagree Theory and Practice !.Classroom material was/i s easily transferable to the practical setting. 1 _ i strongly agree ---, : agree : disagree 3 --, .,_ _ i strongly disagree 2. The Early Childhood Education Training practicum setting met/meets the practical requirement. D strongly agree D agree strongly disagree ~ disa£ree '----' ~ 3. The Early Childhood Education Training practical experience reflected/reflects First Nations culture and values. 0 strongly agree 0 disagree [j agree L.! strongly disagree Confidence in Skills 1. This Early Childhood Education Training, prepared/prepares students to meet the needs of the children in the daycare setting. i : strongly agree D agree - ~ disa£ree .... LJ strongly disagree ,.-- 2. This Early Childhood Education Training. prepared/prepares students to meet the needs of children in First Nations daycare setting. ~ stron£lv a£ree 1.--- -· - - ' agree ;-'---- disaaree := [J strongly disagree 3. This Early Childhood Education Training. prepares;prepared students for employment in nonNative daycare settings. ==:J strongly agree agree . disagree L strongl y disagree 4 This Early Childhood Education Training. prepares ·prepared students for employment in Nati ve daycare settings outside their own community. ~ I ' strongly agree agree di sagree 4 ~ strongl y disagree Section B Faculty Questionnaire Delivery 1. The location of teaching facilities for this Early Childhood Education Training program met/ meets needs of faculty. [J strongly agree ---, ~ n strom!lv disagree 0 disagree agree ~ ~ ~ 2. The location of teaching facilities for this Early Childhood Education Training program met/ meets the needs of the students. I strongly agree D disagree D agree : disagree - :· stromdv ._... ..... 3. The type of teaching facility for this Early Childhood Education program met/meets the needs of both students and faculty. D strongly agree CJ strongly disagree 1 - U , agree 4. This program allows/allowed faculty to be flexible in the implementation of the Early Childhood Education program taught. strongly agree i_ 0 agree I- disagree ; ' ~ strongly disagree 5. The methods of grading and evaluation were/are reflective of the work submitted/ to be submitted by the students. U strongly agree r-- L- U : ; disagree '--- agree ~ strongly disagree 6. There is/was adequate orientation for faculty to teach First Nations students. LJ strongly agree agree disagree '! strongly disagree 7. First Nations culture and values were/are incorporated into the curriculum content. ~ strongly agree ,- , agree disagree ' 1 strongly disagree 8.Faculty invested/invest personal time researching First Nations culture and material to be able to incorporate First Nations material into the curriculum content. 1 ' strongly agree agree - 5 i disagTee ~ -I strongly disagree Resources and Materials 1. Sufficient resource materials for completion of course work for my Early Childhood Education Training program were available. [] strongly agree 0 disagree 0 agree .- L 1 strongly disagree 2. Resources and materials needed for the completion of course work were/are accessible. n strongly agree D agree [J disagree strongly disagree Support 1. It was/is possible to develop a relationship of mutual support and respect with the Early Childhood Education Training instructor. 0 strongly agree n L......J aaree e C disagree L.J strongly disagree 2. It was/is possible to develop a relationship of mutual support and respect with the other Early Childhood Education Training students in this program. n strongly agree U disagree ~ ____. stron!!lv .... .. disaQ:ree ...... 3. It was/is possible to develop a relationship of mutual support and respect with Early Childhood Education Training administrators. ~ strongly agree - , , ~ disagree agree strongly disagree 4. It was/is possible develop a relationship of mutual support and respect with Early Childhood Education Training administrators from the community. LJ strongly agree r-1 L..J agree ~ ee strongly disagree Theory and Practice !.Classroom material was/is easily transferable to the practical setting. n strongly agree agree 1 disagree stronglv disacrree ~ e ·6 Theory and Practice 1. Students were/are able to grasp the theoretical component of this Early Childhood Education program. D strongly agree D agree U disagree [J strongly disagree 2. Students were/are able to integrate theory with practice. U strongly agree 0 disagree D agree D strongly disagree 3. The Early Childhood Education Training practicum arranged for students was/is able to meet their needs . [] strongly agree 0 agree U disagree D strongly disagree 4. The Early Childhood Education Training practicum setting reflected/reflects First Nations culture and values . n strongly agree rf ·'- _ 1 - .,-_ - ~ disa!!ree agree :l strongly disagree 5. Early Childhood Education Training practicum settings were/are well supervised. • strongly agree ~ - . disagree agree .- : stronulv disu(Jrec e ::: L..-....J ~ Employability 1. There is opportunity for graduating students to be employed in non-Native daycare settings. - ,..__ - ! stron!!lv a!!ree ........ agree disagree disa!..!rce ·-- - . stronulv - -' ... 2. There is opportunity for graduating students to be employed in Native daycare settings outside their own communities. ~ strongly agree i_ - . di sagree agree 7 strongl y disagree Section B Administrative Questionnaire Administration for Early Childhood Education Training programs may have been handled by Institutions or Communities or both. Please answer the questions that are applicable to your participation in the administration of the Early Childhood Education Training program Funding 1. Funding for the Early Childhood Education Training program met/meets the needs of the students. LJ strongly agree D disagree ~ agree ~ strongly disagree 2. Funding for the Early Childhood Education Training program was/is sufficient for program delivery. ,- i strongly agree L,__l - .. - ----, agree L__j disagree I i strongly disagree 3. Funding for the Early Childhood Education program was/is sufficient for resources and materials . ..., I ~ strong y agree ! i disagree ·- - agree .'_ _ strongly disagree 4. Funding for the Early Childhood Education program was/is sufficient for faculty salaries. '~ - · strong Iy agree agree disagree strongly disagree 5. Funding for the Early Childhood Education program met/meets travel needs for both students and faculty. -- - !- ! strongly agree - i :- , disagree agree 8 ! 1 strongly disagree Faculty Qualifications 1. Faculty employed for the Early Childhood Education Training program were/are qualified Early Childhood Education instructors. n strongly agree ,- - ~ ! i disagree agree strongly disagree 2. Faculty employed for the Early Childhood Education Training program were/are qualified to teach First Nations students. [] strongly agree n disagree C agree o.......J ~ 0 strongly disagree 3. The faculty selection process was/is acceptable in locating qualified Early Childhood Education instructors. D strongly agree ! ,r-; _ , agree I ' ~ disagree ~ [J strongly disagree Delivery and Development 1. Institutional administration participated/participate in the development of the Earl y Childhood Education Training program. : ! strongly agree n agree 1 - : disagree strongly disagree 2. Community administration participated1participate in the development of the Early Childhood Education Training program. ·- i stronglv agree --..I -- - - . disagree agree :_ strongly disagree 3. Institutional administration participatedJparticipate in the deli very of the Early Childhood Education Training program. ; ! strongly agree agree disagree strongly disagree 4. Community administration participated/participate in the delivery of the Early Childhood Education Training program. - , strongly agree agree disagree 9 strongly disagree 5. The following supports and linkages were/are provided for this program and the enrolled students. (Please circle all that apply) Financial Community Supports Linkages a. Tuition b. Living allowance c. Travel expenses d. Child care expenses e. Other: a. Liaison person b. Peer support network c. Elders/advisors d. Counselling services e. Child care services f. Other: a. Human Resources (HRD) b. Employment opportunities c. Federal government d. Prov./Terr. Government e. Other: Cultural Relevancy I. This Early Childhood Education Training program reflected/reflects the culture and values of First Nations communities. 2. The resources and materials for this Early Childhood Education Training program were/are culturally relevant. 3. Adequate First Nations resources and materials were/are available for faculty and students. Supports and Linkages 1. There was/is a supportive relationship between the partnering institution and the program administration. 2. There was/is a relationship of mutual support and trust betvveen administration and faculty. 3. There was/is a relationship of mutual support and trust between administration and students. 10 -. Provincial Standards Qualifications 1. This Early Childhood Education program met/meets provincial/territorial standards. 2. Credits from this Early Childhood Education Training program are/were transferable among provinces/territories . 3. Credits from this Early Childhood Education Training programs were/are transferable among other institutions (ie: college, university) Employability 1. Graduating Early Childhood Education students were/are employable in non-Native daycare centers. 2. Graduating Early Childhood Education students were/are employable in Native daycare centers outside their own communities. Financial Questions 1. What is/was the overall cost of implementing this Early Childhood Education Training program? 2. In what ways isiwas this Early Childhood Education Training program cost-effective? 3. In what ways is/was this Early Childhood Education Training program not cost-effective? 11