EVALUATING LEADERSHIP PREPARATION PROGRAMS
Survey Development & Field Testing
Survey Development
The conceptual foundation for the survey research uses Kirkpatrick’s four-level program evaluation outcome model (Kirkpatrick, 1998) to cluster measures. Kirkpatrick proposed that the four levels of program evaluation are reaction to the experience, learning gains, the application or transfer or learning to practice, and organizational impact. Consequently, the leadership preparation outcomes to be measured for all graduates are 1) quality ratings of the program, 2) reported learning about leadership learning, 3) leadership career intentions, 4) advancement into leadership positions, 5) leadership practices, 6) school improvement work, and 7) improved school climate and student, parent and teacher engagement (a summary of current measures is included in Appendix A).
The survey was developed through an iterative process beginning with a review of research literature (M. T. Orr, 2003) and pilot research (M.T. Orr & Hung, 2002). A survey instrument was constructed pairing research on leadership preparation and research on leadership practices and school improvement, and then piloted again (M. T. Orr & Barber, 2007). The resulting conceptual model is included in Appendix B. The survey instrument was refined further when used as the basis of research on innovative leadership preparation programs for the Wallace Foundation (Darling-Hammond, Meyerson, LaPointe, & Orr, 2007). In that study, 200 graduates of innovative preparation programs were compared with a national sample of 571 principals on their preparation experiences, learning outcomes (and for those who were principals) their leadership practices and school improvement work. The research findings from this study and another piloting of the survey by the Evaluation Taskforce (M. T. Orr, Silverberg, & LeTendre, 2006) lead to further survey refinement and administration (by reducing redundancy, sharpening and clarifying survey items and adding measures). The survey was then administered to more programs and the results compared and presented at national conferences (M. T. Orr & Pounder, 2007). Using the study findings, the survey was refined again, adding more measures on principals’ leadership practices.
Currently, almost all survey items have proven to be robust (the exception is reports on the number of hours and weeks of an internship or field experience, which we still include despite these weaknesses). Survey items aggregate well into conceptually guided scales, showing strong factor loadings and reliability coefficients (see Orr & Pounder, forthcoming for scale statistics).
Survey Administration Process Development
During its on-going development, fielding and evaluation process, the survey has been reviewed by the 20-30 members of the Evaluation Taskforce. This oversight has helped to strengthen the survey quality, both conceptually and methodologically. It has also enabled us to collectively share in survey fielding and to develop a process for groups of programs to field the survey as well.
Throughout the survey development process, Evaluation Taskforce members have made the survey available to the leadership preparation field generally through conference presentations on research findings and training sessions on evaluation research at the UCEA and AERA conferences (2003-present), and news updates through the LTEL-SIG semi-annual newsletters. The taskforce provides several resources for evaluation support—participation letter for sites; generic information for IRB applications; the survey; an SPSS codebook; background information on scale attributes and item loading. The taskforce also experimented with making the survey available through an on-line survey hosting service, but it proved to be expensive and cumbersome to analyze the results and aggregate across institutions.