Rating Teacher Performance Using Observation Rubrics–“Why the Variation??”

Last Wednesday Courtney and I did a half-day session on Inter-Rater Reliability training for a group of 34 principals, superintendents, and teachers.  The administrators came from various regional districts, but the eight teachers were all from the same school system–Galway Central School (GCS). And, they came as future peer evaluators! I’ve written about teacher leadership in the past, and kudos to GCS for promoting such leadership within their school culture. Honestly, is there any other way to grow our profession and student achievement without such leadership?

Back to our inter-rater reliability session, the focus was for people to use the Danielson Frameworks for Teaching to observe and rate classroom instruction. Our goal was to make everyone aware of the necessity for collecting objective, detailed evidence to support teacher performance ratings (Highly Effective, Effective, Developing, and Ineffective). We used videos from the New Teacher Center to practice collecting and tagging evidence to rate the various instructional components within the 2011 revised Danielson Rubric (At the New York State Education Department Network Team Institutes, we are using the 2007 Danielson rubric in evaluator calibration sessions which goes into greater specificity than the 2011 version.).

Over the course of three hours, we watched two videos each 20-minutes in length. Audience members were directed to collect evidence, tag the appropriate Danielson component, and then rate the level of performance within each component. We then had people work in small groups to discuss observations and ratings. All data were collected and put into an Excel document to show the level of consistency among the 34 raters. Below is a chart of our second video results. Vertical columns rank the level of performance from ineffective (1) to highly effective (4), and horizontal rows represent components within domains two (Classroom Environment) and three (Instruction).

As you can see from the data, there was much greater variability within domain three. When the data were shared with the group, one of the Galway teachers asked, “Why is there so much spread with ratings from ineffective to highly effective in some components?” Great question. We explained that this is new work Race to the Top is asking of New York State educators. I asked the audience, “How many of you had inter-rater reliability training or reviewed classroom observation videos with rubrics in your districts or leadership programs?” No one raised their hand. Courtney and I explained that before Race to the Top, such attention to teacher observation and inter-rater consistency were sorely lacking in this state and across the nation, and that the work we are now doing in this area is exciting and promising for the field and the students. We also assured our hard-working group that consistency and efficiency grow over time as one becomes more familiar with the process.

We ended our day with some key thoughts and next steps. I suggested that people return to their districts and conduct similar sessions with colleagues. What a great thing to do at a faculty, department, or grade level meeting! If we hope to diminish the fear factor regarding teacher observation that legitimately resides within many schools, then we must develop people’s knowledge and comfort with the teacher observation process. We also have to remind the press and public that publishing teacher performance records greatly compromises the Race to the Top reform effort, sowing seeds of distrust, fear and resentment. The Annual Professional Performance Review through student achievement and teacher observation data are best used to grow teachers’ capacities to enhance student achievement–not to politicize and badger public education.

Led Zeppelin, Cool Hand Luke, and RttT Network Team Institute Communications

Last week we had our May Network Team Institute Training. By day four, most everybody was totally spent. It was hard work. Good work, but hard. Our focus was on Student Learning Objectives (SLOs) and Teacher Evaluation, with the following objectives defined in the four-day agenda:

• Understand how teacher observation and SLOs support learning for ALL students and target the learning of sub-groups [ELL, SWD, etc.]
• Understand the expectations for the first calibration of inter-rater reliability
• Gain essential insight for implementing district SLO processes
• Understand best practices for turn-keying training to local districts
• Collaborate with colleagues

With state-wide budget votes and school board elections smack in the middle on day two, and the hoopla of state testing issues still lingering in the air, people were a little testy–particularly during SLO training. Perhaps Kate or Ken had anticipated as much when they opened the institute with the following quote from Ronald Heifetz‘ Leadership Without Easy Answers, “The difference between an adaptive problem and a technical one is key. There are problems that are just technical. I’m delighted when a car mechanic fixes my car, an orthopedic surgeon gives me back a healed bone, or an internist gives me penicillin and cures my pneumonia. That’s a key question: is this a problem that an expert can fix, or is this a problem that is going to require people in the community to change their values, their behavior, or their attitudes? For this problem to be solved, are people going to need to learn new ways of doing business?” What a great reminder of how complicated reform is, particularly when done right. Race to the Top poses adaptive problems in addition to the technical ones. We can efficiently work through the technical issues, but adapting to change is another story.

Race to the Top Network Team members that descend on the Capital Region every six weeks or so are a good lot. We have gotten to know one another over the past 250+ hours of Network Team  Institute time, and the people doing this important reform work care deeply and want to do the right thing for their schools and regions. With stakes high for teachers and principals, we sometimes get stressed out when answers from presenters aren’t neat and tight. With so much of this work highly contextual, the myriad of possible scenarios take too much time to belabor during a Network Team Institute session–particularly when it comes to Student Learning Objectives. Unfortunately, that doesn’t always sit well when you have 100+ people in one room. During our second day, people began asking context questions that, for the most part, are resolvable using the NYSED APPR Guidance document. In any event, a need to move on with the agenda led to what could have been a Led Zeppelin (Communication Breakdown)/Cool Hand Luke (Failure to Communicate) moment. Fortunately, SED and the Regents Fellows partnered together to manage the situation promptly and efficiently.

Kate respectfully reminded people there was much work yet to be done during the training, and that answers to all questions posed would be addressed the following day. Ken later reiterated Kate’s message and offered to meet with anyone who wished to discuss the SLO contextual questions further (three chose to meet with Ken after his talk). By honoring those individuals wanting their particular scenarios immediately addressed while also keeping to the day’s agenda, the SLO session progressed as scheduled with all parties receiving what they needed.

In spite of the tremendous change being pushed at breakneck speed, the quality of material being rolled out at Network Team Institutes and the savvy opening day presentations by John King, Ken Slentz, and Kate Gerson are making all this change adaptable. Certainly not easy, but doable just the same. Most encouraging, when emotions rise and rigidity increases, SED reminds us all to take a deep breath and “Don’t let perfect get in the way of good.”

Celebrating A Great Teachers Academy

This past Friday was the culmination of a unique 28-hour professional development experience involving teachers and administrators from three distinctly different schools in size and setting. The event went off spectacularly. One administrator commented, “I’ve been through plenty of professional development before, but this was not something administrators and teachers typically receive.” Other participant comments were: “I learned how to write quality essential questions and use them every moment of the day as I work with students. I now constantly think in terms of essential questions and the desired results when teaching or planning;” “This Academy has helped me see how Common Core State Standards, Teacher Evaluation, assessments, and Student Learning Objectives fit together in the Annual Professional Performance Review (APPR);” and, “I’m thinking about using performance assessments, which is something I never did before when planning math lessons.” There are more data yet to be analyzed, including a post-survey set which we will use to revise the academy for our next iteration. Meanwhile, today’s feedback suggests the Academy was effective, which is something we all had hoped for when we first came together last October to explore an idea for bringing coherence to Race to the Top for teachers and school principals.

In a previous blog entry, I wrote about the perceived disconnect by educators regarding Race to the Top deliverables. As a colleague of mine commented, the layers of reform ranging from Common Core State Standards, Response to Intervention, Data-Driven Instruction, and Annual Professional Performance Standards are akin to a seven-layer cake with each distinct and separate from one another. Not the message or perception we want educators and administrators to have in this busy period of reform. Ironically, my co-facilitator and I were thinking the same thing when we invited a core group of National Board-Certified Teachers (NBCT) to explore the possibility of developing an Academy that revealed the interdependencies within Race to the Top while also cultivating teacher and principal leadership. Preliminary participant comments suggest we realized our goals.

To gain both credibility and professional rigor for such an academy, we sought NBCTs and teacher leaders who walked the talk, balancing the daily challenges of classroom instruction with instructional leadership within and outside their buildings. Self-directed professionals sought out as experts because of their pedagogical skill sets and knowledge of what matters in this era of reform. We hit the jackpot with four NBCTs representing secondary ELA, primary reading, special education, and middle level mathematics. We also got financial support from the Greater Capital Region Teacher Center, which has done so much for educators over the past three decades. At a preliminary meeting, we brainstormed what the region’s educators and principals needed to successfully implement RttT components. In particular, we were looking for ways to cultivate teacher leadership. Over the course of a many meetings, we resolved our focus on six modules and described the program as follows:

A 28-­‐hour, six module pilot program integrating teacher artifacts, classroom video analysis, and cognitive coaching for participants to develop teacher leadership capacity by fully understanding the integrated nature of New York State Teaching Standards, Danielson Frameworks, Annual Professional Performance Review process, and Student Learning Objective protocols described in New York State’s Race to the Top Program. Master teachers will facilitate sessions over a ten-­‐week period, and participants will develop knowledge products that include a participant portfolio, videotaped classroom instruction with analysis and reflections, an Understanding by Design unit aligned with the Common Core Learning Standards for literacy, and individual participant goals using the Student Learning Objective format. Educators and building principals will also observe and rehearse the best practices and artifacts shared by Academy facilitators across the six modules. Each participant will receive a set of books, binder, and flash drive containing best   practices and program modules.   

Our pilot was ambitious, and we did not have time for the creation of videotaped classroom instruction we described in the description. However, we accomplished so much more than what could be described on paper. The participants’ attitudes and comfort level regarding Race to the Top underwent a signficant shift over the past three months, and our team is eager to launch a second edition this summer at the end of July. However, the summer Academy will not be a pilot but instead will be open to all teachers and principals interested in becoming teacher and principal leaders in Race to the Top components. After all, anything is possible when you bring a highly talented group of teacher leaders and NBCTs together.

PS. Happy Mother’s Day to all moms.

Subject Area Teachers and Common Core Literacy Standards: “Just Do It”

Bo Jackson, the first professional athlete selected to both the National Football League and Major Baseball League All Star Games, was a spectacular sports phenomena in the 80’s. In addition to racking up yardage on the football field and clobbering balls out of stadiums on the ball field, Jackson helped Nike popularize their slogan, “Just Do It.” “Just Do It” is a powerful message so apropos to those uncomfortable with change, fearful of failure, or distrusting of outside interests. And when it comes to Common Core Literacy Standards for teachers who don’t teach english, “Just do it” is especially salient.

On Tuesday I worked with a group of 25 mostly middle and high school non-ELA teachers on Integrating Common Core Literacy into the Content Areas. We also had two principals in attendance (I love having principals present for teacher professional development–they send a powerful message that the work being done matters). I blogged about developing the workshop a few weeks ago, and now that the session is over, I have some analyses and reflections to share. First and foremost, the group hung in there and worked hard studying the standards, reading the assigned tasks, working together in groups, and brainstorming strategies and sharing observations.

We began the day with a powerful video of Taylor Mali’s Poem “Totally like whatever, you know” read by Ronnie Bruce on  Typography about language. My favorite line is, “Speak with conviction,” which captures the essence of why we’re doing Common Core in this country. We want to develop students as thinkers, confident thinkers, thinkers who have something important to say and who can back up their words with data, evidence, and conceptual understanding. We want students who can tolerate frustration when wading through challenging text; who understand good writing requires graphical organization and iterations of drafts before reaching the beloved final draft; and who read and write for learning as well as pleasure. Regardless of the discipline, students can and must use language to convey their thoughts, passions, ideas,….and be able to argue and support their claims with the content of their subject. Beginning with Mali’s poem was an exquisite hook for the session.

After sharing the necessary essential questions, conducting a Know-Need to Know, and the obligatory directions to restrooms and exits, we reviewed the brief history of the standards and their structure. Besides the Taylor Mali poem, we looked at the unsettling research on high school graduates’ readiness for college and careers via data from the American Diploma Project. This was followed by a review of the 6-12 Standards for Literacy in History/Social Studies, Science, and Technical Subjects and a brainstorming activity by disciplines on strategies for meeting the ten reading standards (we repeated this for writing later in the day). The strategies teachers came up with were self-affirming and justified the message, “Just Do It.” The anxiety many had at the start of the day was eased as they settled into the standards and closely read what the standards were asking of students.

The day progressed through Lexiles, qualitative and quantitative measures of text complexity, a Susan Tovani exercise on purposeful reading from her book, I Read it but I Don’t Get it, and two close reading tasks worth elaborating on here. To help teachers understand what close reading looks and feels like, teachers were organized by content areas and then given passages to read. Science teachers read a passage from Frederick Douglass’ famous speech, What to the Slave is the Fourth of July, and social studies/history teachers were asked to read about Eurasian Water Milfoil in the Southeast Exotic Pest Plant Council Invasive Plant Manual (Both passages were truly fascinating). Art and special ed teachers could choose from either passage. Teachers had to read the passage and circle words they were unfamiliar with, and jot in the margins things they thought were extremely interesting or important, and additional questions they had. After a very quiet 25 minutes of intense reading, teachers identified the standards the readings aligned with and then shared the vocabulary and notes with the main group. We later used the same texts to explore writing options.

At day’s end, we had thoroughly reviewed all reading and writing standards. We discussed the increasing rigor from grade to grade (a great structure for differentiation), learned how to access informational texts and associated Lexiles, looked at follow-up writing activities tied to the text passages, and brainstormed next steps. When all was said and done, the group left feeling good about the Common Core Literacy Standards realizing, when it comes to integrating Common Core Literacy into their subject areas, “They can do it!”

Oh, and before I forget, remember to thank a teacher this coming week to celebrate Teacher Appreciation Week.