Effects of Outdoor Education Programs on Children in California

Study Design & Work Plan

April 1, 2004

The American Institutes for Research (AIR) proposes to conduct an independent evaluation of the California Outdoor School Administrators (COSA) outdoor education programs. The purpose of the study is to measure potential impacts that outdoor education programs have on at-risk children in California. As described by California Assembly Bill Number 1330, Chapter 663, the Outdoor Environmental Education Program is designed to foster stewardship of the environment and an appreciation of the importance of the wise use of natural resources. The program serves primarily at-risk youth and underserved demographic groups. The study is designed to evaluate various impacts of the outdoor education programs and will be conducted within the timeframe between May 1, 2004 and January 31, 2005.

Conceptual Framework

The conceptual framework (or logic model) for this study (see Appendix A) was developed based on conversations with key advisors, interviews with staff from elementary schools that participated in outdoor programs, and recent research studies. As shown in the logic model, we will examine impacts of outdoor education programs on students’ environmentally responsible behavior and personal and social skills (such as self-esteem, self-awareness, cooperation, teamwork, etc.). Furthermore, we will attempt to examine impacts on students’ knowledge and understanding of the science concepts that comprise the outdoor education program curriculum.

Research Questions

The specific research questions that we will address in this study are:

How does participation in outdoor education programs foster students’ stewardship of the environment and their appreciation of the importance of the wise use of natural resources?

How does participation in outdoor education programs impact students’ personal and social skills (e.g., self-esteem, self-awareness, cooperation, teamwork, etc.)?

How does the science instruction received through the outdoor education program curriculum increase students’ knowledge and understanding of science concepts?

Work Plan

In this section we propose a plan of work to address the research questions described above. The discussion that follows is organized by task. This work plan concludes with a proposed timeline for the individual tasks.

Task 1: Collection of program information

The California Department of Education (CDE), in consultation with AIR, will choose three COSA outdoor programs as the target programs. These programs should be geographically diverse, serve at-risk sixth grade students during week-long residential programs, and use curricula that are aligned with California’s content and performance standards for science.

Our first task is to collect background information about various aspects of these three outdoor education programs, including program budgets, facilities, staffing, curriculum, program intensity, etc. We will collect the information directly from the outdoor education programs and/or the County Offices of Education. The information collected in this task will be used to contextualize our findings about the effects of these programs on students.

Task 2: Selection of treatment and control groups

Each of these three outdoor programs will recruit one elementary school to participate in this study. Once participating schools are recruited, the treatment group (6^th grade classrooms of program participants) and control group (6^th grade classrooms of non-participants receiving the intervention at a later date – utilizing a “delayed treatment design”) will be randomly selected as matched pairs within the same feeder school for each COSA program.

Task 3: Completion of instrument development

Data collection will include the use of surveys, interviews, and observations. Surveys will be used to collect information about student outcomes and will be administered with students, parents, and classroom teachers. Interviews and observations will be conducted with teachers and outdoor program staff to collect information about the outdoor education curriculum and activities that students are engaged in as well as their perspectives on student outcomes. Appendix B summarizes the potential constructs to be measured by each of these data collection instruments and the potential respondents for each instrument.

In Task 3, data collection tools will be finalized and pilot-tested. (Initial development of instruments will begin in April, through a proposed contract with the Sierra Club, beginning April 1 through April 30, 2004). AIR will complete development of three versions of surveys (for students, parents, and classroom teachers) and interview protocols designed to measure the student outcomes of interest (e.g., stewardship of the environment and appreciation of the importance of the wise use of natural resources, in addition to other academic, personal, and social outcomes). Our goal will be to minimize the burden on respondents, by keeping survey and interview protocols as short as possible while ensuring adequate data to address the key research questions. Finally, we will pilot test these data collection instruments in an effort to ensure data are accurate and valid.

Task 4: Data collection

The student survey will be administered before and immediately after the treatment group participates in the outdoor program. We will administer the survey again at approximately 2-3 months after the treatment group’s participation to examine longer-term program effects. Data collection will be conducted with the control group at the same three points in time.

The parent and teacher surveys will be administered before the treatment group participates in the program and at approximately 2-3 months after their participation. We will not administer these surveys immediately after the treatment group’s program participation, because we expect that it takes at least a few weeks for parents and teachers to start observing tangible changes in students after their program participation. In addition, we intend to minimize the burden on parents and teachers by limiting the number of survey administrations.

Finally, interviews (and observations, to the extent feasible) will be conducted while the treatment group participates in the outdoor program.

Appendix C illustrates the anticipated timeline for data collection.

Task 5: Data analysis

After data collection is completed, we will record survey results as electronic data files. Once data entry is complete, we will analyze both immediate- and longer-term impacts of the program. Differences in student outcomes and rates of change between participants and non-participants (among treatment and control groups) will also be examined for statistical significance. Furthermore, survey data from students, parents, and teachers will be linked for each student, and we will investigate consistency of responses among them.

Qualitative data from interviews and observations and any other relevant information will be used as context for the quantitative data on student outcomes.

Task 6: Final Report

We will prepare a comprehensive final report by January 31, 2005. The report will include a table of contents, executive summary, presentation of findings and analysis, and recommendations.

Timeline

The timeline is based on a start date of May 1, 2004 in order to complete all of the activities by January 31, 2005. The schedule for the tasks described above is shown pictorially in Exhibit 1.

Exhibit 1: Timeline for study tasks

Study Tasks	Timeline: 5/04 – 1/05
Study Tasks	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Jan
Task 1:
Collection of program information
Task 2:
Recruitment of elementary schools
Selection of treatment and control groups
Task 3:
Instrument development
Pilot Test
Task 4:
Data collection
Task 5:
Data entry
Data analysis
Task 6:
Report writing

Appendices

Appendix A: Logic Model

Appendix B: Potential Constructs to be Measured by Data Collection Method

Appendix C: Timeline for Data Collection