Will New Pre-K Accountability Metrics in D.C. Enhance—Or Undercut—Pre-K Quality?

Like many in D.C.’s family-heavy Ward 4, Sam Chaltain sends his children to charter schools. His older son attends Latin American Montessori Bilingual, and his younger son will follow in a few years. This is just one of the area’s charters; it also boasts E.L. Haynes, Capital City, and several others that rank among the District’s very best, according to D.C.’s Public Charter School Board’s accountability rating system. While Chaltain loves the pre-K experience his oldest is having, he’s concerned that coming policy changes could jeopardize it in the future.

That’s because D.C.’s Public Charter School Board (PCSB) is deciding how to expand its Performance Management Framework (PMF) to pre-K providers on September 23 (public comments are welcomed through August 28). The proposed changes would allow charters that offer pre-K to choose between two systems for scoring these programs. As the chart below shows, Option 1 ties 60 percent of a pre-K program’s PMF score to reading and math growth between the beginning and the end of the pre-K year. Option 2 reduces that number to 45% if the program opts to include a social-emotional learning assessment as part of its accountability system.

Source: PCSB Proposal for Early Childhood Performance Management Framework

These weightings concern Chaltain. “It’s naïve to the last 12 years of federal policy and what happens when you only incentivize certain things and leave other things to be optional,” said Chaltain in an interview. “Most public schools that I enter, I just see reading and math happening all the time. To me, that’s the issue [with the proposed PMF changes]. It’s just a question of balance.”

Chaltain’s concerns are valid. No Child Left Behind unintentionally encouraged test preparation and “teaching to the test” in the grades included in states’ standardized testing programs. All too often, these practices led to a narrowing of the curriculum, significantly reducing time for non-tested content areas like social studies and science in the early grades, especially in schools serving struggling children. Would the new PMF lead to the same consequences?

The updated PMF formula would apply to lower elementary charter programs as well, under a slightly different formula. For the K–2 years, charter schools can choose to have 80 percent (without a social-emotional learning assessment) or 70 percent (with an SEL assessment) of their PMF scores tied to reading and math progress or achievement.

Progress, sometimes known as growth, is a measure that provides data on how much progress students have made between the beginning and end of the school year in a specific area. Achievement is a summative (end of school year) measure that provides data on the attainment level of students. Acknowledging developmental differences of young children, the framework for grades K-2 can measure achievement or growth. If a student does not meet the achievement target on the summative assessment, then the student must show adequate progress toward the target. (Read more here.)

Source: PCSB Proposal for Early Childhood Performance Management Framework

Chaltain comes at the policy changes from two perspectives: as a father and education policy expert who writes the blog “Of, By, For: In Search of the Civic Mission of K-12 Schools” for Education Week. He recognizes the challenge of improving quality and maintaining accountability, but worries about its emphasis on academic subjects. “When you disproportionately weight one assessment almost to the exclusion of everything else, you change it from a diagnostic tool into an accountability tool.” Research on assessment does indicate that using one assessment for a purpose it was not intended, such as accountability, could invalidate the results for both purposes. You wouldn’t use a life vest to jump out of an airplane.

For example, if a school leader felt pressured by the PMF to increase the school’s early grade math and reading scores, he could in turn pressure his teachers to focus more on those subjects – or worse, to prepare only for the test, which sometimes measure only a limited range of basic skills. Young children might not be appropriately challenged or given time to play or explore new topics. Not only is this is one of the quickest ways to get kids to dislike school from a very young age, but it also means we’re not going to get quality assessment data that help the teacher know how well her students really understand a concept. Nor do we get an accurate picture of how well a school is education its students. 

For pre-K, at least, the plan includes measures beyond student progress in reading and math. The Classroom Assessment Scoring System (CLASS) accounts for 30 percent of the schools’ rating. Lisa Guernsey and Susan Ochshorn write about the CLASS in their paper Watching Teachers Work. The CLASS is a research-based tool used by trained observers who watch and record how teachers interact with children in their classrooms. The tool measures teacher-child interactions within three domains (emotional climate, classroom organization, and instructional support) and along multiple dimensions, including quality of feedback and concept development, among others.

In an interview yesterday, Sara Mead, DC Public Charter School Board member and former director of the Early Education Initiative, pointed out that even if programs choose to not assess students’ social-emotional growth, the requirement to use data from the CLASS will capture the extent to which teachers provide a positive emotional climate, provide appropriate feedback, and engage students in rich conversation.

PCSB School Quality and Accountability Specialist Erin Kupferberg is confident that the proposed PMF would improve both accountability and quality in D.C.’s charter-based pre-K classrooms. “It is an improvement on our previous accountability plans...Before, every school’s goal was different. [By making] this pre-K PMF similar to the PMF in use for grades 3 and above, it gives parents the ability to compare between quality choices,” Kupferberg said.

It also encourages charter schools that serve PreK-8^th grade, as some schools do, to ensure they are working to meet the needs of their youngest students as well as the older students, points out Jack McCarthy, president and CEO of AppleTree Institute for Education Innovation and AppleTree Early Learning Public Charter School. These charter schools will be measured by both the Early Childhood PMF and the Elementary/Middle PMF.

The Early Childhood PMF was developed in collaboration with local charter school leaders. “I do think it is an important step in the right direction,” says McCarthy. He believes looking more closely at student learning and developmental outcomes will help to define what high-quality early childhood really is.

It’s important to note that the new PMF is not a mandate forcing 3- and 4-year old students to sit for grueling rounds of standardized “bubble tests.” It allows schools considerable flexibility in their choice of assessment. Most assessments for children this age are observation-based, meaning that they are based on what a teacher has watched a child do. (Has the child been able to hold a crayon and press with enough firmness to form letters or produce other signs of early writing? Can the child provide answers when asked open-ended questions about a story?) These assessments are given once in the fall and again in the spring. “We began with a list of assessments our schools were already using,” Mead said. The Board evaluated those assessments for quality; the approved list includes any that made the cut.

D.C. charters have been assessing their pre-K students for years. The new PMF would incorporate this data into PCSB’s annual report on the quality of each of D.C.’s charter schools. The report groups schools into three tiers, with the top-performing schools in Tier 1 and the weakest-performing schools in Tier 3. Parents can use that information to help guide their decision in choosing a school. The reports are also a helpful resource when it is time for a school’s charter to be reauthorized, which happens every five years. The PMF, along with other factors, could be used to determine whether a school’s charter is reauthorized or not. However, “no school closure policies have been established for early childhood based on the PMF,” said Mead.

This tiered-rating system is similar to states’ quality rating and improvement systems (QRIS) that are being implemented across the country to provide parents with an easier way to judge the quality of preschool programs and child care centers they are considering for their children. QRIS, though, do not typically incorporate student progress or achievement data in centers’ ratings and the idea is controversial in traditional early childhood circles. Instead, QRIS often include measures of the learning environment and teacher interactions with children (like the CLASS) as well as measures like class size, student-to-teacher ratios, meal provisions, the presence of family engagement programs, and whether the center conducts student assessments. The measures focus on inputs rather than outcomes. This is the case for   the QRIS for DC non-charter child care programs proposed by the Office of the State Superintendent for Education (OSSE).

“We are the only [charter school] authorizer in the country doing this,” says Mead. A few states are considering including student outcomes in QRIS to rate child care centers. Mead notes that Louisiana is one.

Source: DC Public Charter School Board

(Note: The above slide from the Early Childhood PMF Task Force meeting on July 8, shows the differences between DC’s Office of the State Superintendent of Education’s (OSSE) QRIS proposal and the Public Charter School Board’s proposal for early childhood.)

“If you look at 3^rd- and 4^th-grade scores on D.C.’s CAS, a majority of students still are not ready to read. This is about making sure that children are ready for school,” said PCSB Director of Communications Theola Labbé-DeBose.

It’s also about making sure that the gains children make in early education programs are followed by high-quality instruction, developmentally appropriate curriculum, learning environments, and opportunities in the early grades. The K-2 PMF leaves out the measure of teacher interaction and downplays the weight for social-emotional development, provided that school leaders choose to assess that learning domain at all.

Given all this context, here are some preliminary suggestions based on our work here at the New America Foundation:

• We think the charter school board should encourage K-2 schools to emphasize social-emotional assessment and teacher-child interactions.
• We also think the board should reconsider the weights assigned to reading and math in the PreK-3^rd grades and allow for a more balanced framework.
• DC PCSB’s focus on progress rather than endpoint achievement is the right way to go, and should be the case for PreK-12^th grades.
• Finally, while we think that outcome data should play a role in whether a school’s charter is reauthorized or not, it should never be the only factor.

Perhaps it’s also worth noting that arguments over the quality of public pre-K (in charter or traditional schools) are a second-order affair. In other words, we can only argue over how we rate public pre-K providers once we’ve invested resources in establishing pre-K in the first place. While it’s surely cold comfort for those unhappy with however the PMF is ultimately structured, these are privileged problems to have.

To comment on PCSB’s proposed early childhood PMF, click here.

(Interested in more analysis of D.C.’s pre-K program? Don’t miss Conor’s column in this Sunday’s Washington Post on the secret hero of D.C.’s nigh on universal pre-K system.)