The following article was written by Dana McCurdy, program evaluation director, and Brian Edwards, research writer, for Partners in School Innovation.
Principals, instructional coaches and teacher leaders invest in professional learning for their staff in the hope that it will yield better instruction and, ultimately, better outcomes for students. Like all good investors, these educational leaders want to know whether their efforts are paying off. However, assessing the value of professional learning is not as simple as checking stock prices.
Measuring the impact of professional learning is both an art and a science. But this does not mean that it has to be done by a professional external evaluator. With appropriate planning and an analytical mindset, internal staff can generate powerful knowledge about professional learning in their school or district.
Working together to grow
For professional learning – and the evaluation of it – to be meaningful, a certain ethos must be present in a school or district. Teachers and leaders must have a results-orientation; they have to care about improving in measurable ways and using their skills to increase student learning. They have to embrace both quantitative and qualitative data as tools to help them grow. And there must be trust among teachers and leaders.
If all of these factors are in place, teachers will be open to trying new instructional strategies, being observed by their colleagues and administrators, and working together to improve.
We have found that one of the best ways to foster these cultural conditions is to include teachers in the design of professional learning and the evaluation of it. When teachers help collect, analyze, and interpret data from professional learning, it is far more likely that they will trust and reflect on the data and adopt the behavior that the professional learning is intended to instill.
In addition, bringing more minds to bear on the evaluation increases the likelihood of crafting good questions and producing information that can be used to improve professional learning. Involving the staff also reinforces trust; teachers see that administrators are most interested in increasing adult and student learning, not in discovering problems with individual teachers’ instruction.
Required skills and tools
Effective evaluation of professional learning requires more than a results-oriented, data-driven, and trustful culture. It also requires certain skills and tools. The good news is that most principals, instructional coaches, and teacher leaders already have them or can acquire them relatively easily. In other words, education leaders should have confidence that they can competently assess professional learning in their school or district.
The first step is to articulate a “theory of change” regarding professional learning. A theory of change is a description of how and why a desired change is expected to happen (Clark and Taplin, 2012). It maps out long-term goals for professional learning and the learning activities that are assumed to lead to those outcomes. Typically, a theory of change has three components:
1.) The ultimate goals that leaders have for their students – i.e., the changes they hope to see in student outcomes, achievement or learning.
2.) The preconditions or intermediate outcomes that must exist for the long-term goals to be realized – i.e., the changes that leaders hope to see in their staff, often expressed as increases in knowledge, skills and practice.
3.) The professional development, coaching, and collaboration that are intended to achieve the desired preconditions (Brest, 2010).
Having a theory of change helps a school or district avoid the common mistake of offering disconnected, one-time professional development sessions and instead offer a series of professional learning activities focused on a small set of aligned goals.
When creating a theory of change for professional learning, education leaders need to consider how they will measure each component of their theory. These measures will be central to assessing the impact of professional learning on staff and students. Incorporating measurement activities in the theory of change from the outset, and collecting data as the professional learning unfolds, make evaluation more efficient and accurate than if the process is done as an afterthought.
A theory of change can be thought of as an idea about a chain of events: Professional learning activities 1, 2 and 3 will lead to intermediate outcomes A, B and C, which will lead to our ultimate desired outcome, D. Most schools are fairly certain about their desired outcome – for example, a 10-percent increase in reading scores – and about the professional learning activities they want to try – for example, monthly professional development sessions in literacy instruction and individual coaching.
Often, the most challenging part of developing a theory of change is identifying the intermediate outcomes and deciding how to measure them. Ideally, intermediate outcomes indicate whether change is occurring at the pace needed to achieve the ultimate goals. Thomas Guskey’s five-level model for evaluating professional development (2000) helps educators identify and measure intermediate outcomes.
According to Guskey, each of the five levels in his model is important in its own right, but each level also builds on the preceding levels. In addition, if the five levels are reversed, they can be used to plan professional learning in the same way that backward mapping is useful for curriculum planning.
The Guskey model can be used to address the middle section of a theory of change – the preconditions needed to achieve long-term goals for students. To apply the Guskey model in a school or district, education leaders should start by developing one or two questions for each level. Keeping the number of questions small simplifies the evaluation and ensures that administrators and staff are not burdened with too much data to analyze within a reasonable timeframe.
Then, for each question, a school or district should determine the best method of data collection. It would be wise to start by identifying methods that are already in place, such as classroom walk-throughs or post-session feedback surveys.
Data collection methods should include both quantitative and qualitative data. To continue with the example of improving literacy instruction, school leaders could survey teachers and ask them to rate the usefulness of professional development sessions on a five-point scale and to describe how they plan to apply the instructional techniques taught during those sessions. Then the leaders could observe classrooms two weeks after the session to see which techniques are being implemented and how well.
Having a sound theory of change and the ability to apply Guskey’s model will help education leaders evaluate professional learning tremendously, but these leaders may still face challenges, including the following:
- Limited time. Systematically evaluating professional learning adds to educators’ already-full plates. The best solution to this challenge is to prioritize: Focus on the most important goals from the theory of change and evaluate the activities designed to meet those goals. Whenever possible, monitor activities with tools or processes that are already in place or that can serve a dual purpose, such as classroom observations, reviews of meeting agendas or student assessments.
- Resistance from staff. Leaders may face resistance from staff when evaluating professional learning because evaluation activities, such as classroom observations and analyzing student test scores, can leave teachers feeling vulnerable. Leaders need to reassure staff members that evaluations are done to help the whole school improve, not to discover individual teachers’ challenges. It can help greatly to distribute responsibility for the evaluation of professional learning beyond the individuals who conduct performance evaluations. In addition, open communication with collective bargaining representatives about the purpose of evaluating professional learning may ease concerns.
- Limited skill with data. Some educators may feel intimidated by the need to work with quantitative data because they are out of practice or do not have experience analyzing survey or interview data. One way to overcome this is to utilize free web-based survey tools that automate data analysis and provide summaries of results as soon as surveys are completed. Observation data can be analyzed easily by tallying the number of classrooms engaged in new skills and practices. Interview data can be quickly analyzed by a group of people identifying the most common themes in interview transcripts or notes.
- Narrow perspective. When educators evaluate their own professional learning, they may misinterpret results because of possible blind spots: implicit bias, a fixed mindset about themselves or their students, or simply by being too close to the situation. Or they might make conclusions that are not warranted by the evidence. For these reasons, we believe it is prudent to periodically bring in someone who is external to the school or district to help interpret results. The external party could be a seasoned educator from a neighboring school or district, a professional external evaluator, or a coach – anyone who will ask challenging questions about an evaluation’s assumptions and conclusions.
A success story
One of the schools that we have worked with recently, an urban K-8 school, has markedly improved its system of professional learning, and the evaluation of it, in just three years. These efforts have led to more effective instruction and increases in student achievement.
In three years, math performance among Latino students increased from 6 percent on grade level to 37 percent, matching the district-wide average. Among English learners, reading performance rose from 0 percent on grade level to 32 percent, and math performance increased from 0 percent to 41 percent. African American students also made great strides, showing a 10-percentage point gain in reading and math during the same three-year span.
The school developed a theory of change for whole-school improvement, with professional learning playing a key role. Now, professional development sessions, coaching and collaboration among grade level teams are aligned and serve a common end goal.
To evaluate the intermediate outcomes of these professional learning efforts, the instructional leadership team considers data from teacher questionnaires, classroom walk-throughs and students’ benchmark assessments.
This model demonstrates that distributing ownership of evaluation beyond the principal, utilizing both quantitative and qualitative data, and measuring intermediate outcomes among teachers can lead to great improvements in student outcomes without the addition of resources.