Challenges and Opportunities in Evaluating Two MSP Projects

Authors: Jane Butler Kahle, Yue Li, Sarah Woodruff

The PennSTI used a pre/post-project evaluation design with no control group of either teachers or students. Rather, changes in teaching and learning strategies, attitudes, interest, and content knowledge of both teachers and students were evaluated across the three years of each teacher's participation. Two different types of analyses were performed for the MSPinNYC project: pre/post and comparison group designs. First, comparisons of TRT teacher responses and those of their students before and one year after the teacher taught in an institute were made. Second, comparisons were made between teacher responses and student responses of participating (TRT) teachers and non-participating teachers in the same school one year after TRT teachers had taught in an institute.
The evaluators recommended and both projects adopted the E & A Center's student and teacher questionnaires to assess inquiry-related teaching and learning practices as well as interest and attitudes. Validity was established by panels of scientists/science educators and mathematicians/mathematics educators. Cronbach alpha subscale reliabilities for the questionnaires are shown in Tables 1 and 2.

Table 1. Reliabilities for PennSTI Teacher and Student Questionnaires.

Table 2. Reliabilities for MSPinNYC Teacher and  Student Questionnaires.

It is important to note that the instrument may not be as appropriate for MSPinNYC students as it is for PennSTI students. The PennSTI focused on inquiry teaching/learning while the MSPinNYC stressed both inquiry and other forms of instruction.

Teaching and Learning subscales used a Likert type scale with rating categories ranging from Almost Never (1) to Very Often (5). These two subscales have parallel items on the teacher version and on the student version. Student responses may be used to assess any potential teacher bias. (The instruments are fully described in another submission).  Demographic information was collected that was specific to each project but with commonalities such as gender, race, year in school, age, and class.  This practice has allowed the evaluators to compare and contrast the projects, as appropriate, and to provide comparison points for project personnel.  This paper focuses on changes in teaching and learning strategies, a foundation of the MSP initiative.