Data Collection and Use

Problem: For most early childhood teachers, "data" is not a friendly word. If they have any experience at all with data collection and analysis, it is usually negative—tedious evaluations of students, or data that are often used as what some call “accountability sticks” to evaluate teaching practice.

But as efforts focus on improving the quality of early learning programs and strengthening the quality of adult-child interactions, community leaders need to know how and whether their programs are working, and how kids’ learning and developmental needs are changing. What is more, data can also help to make the case for scaling up programs that work and help to inform policy questions.

Solution: Some school districts have taken important early steps to put the infrastructure in place to collect data. This information will allow them to understand the students they serve and how they are being taught, as well as to track progress.

Districts are also working to partner with researchers, teachers, and community organizations to improve the validity of the data and to encourage its use to help inform decision-making.

All three communities in the Starting Smart and Strong initiative have embarked on a developmental evaluation that aims to test new approaches to improving teaching and learning for young children, identify factors that make for a strong early learning system, and understand the best way to support expansion, leadership, and collaboration.¹ Leaders in all three communities are working closely with evaluators to understand how this work is unfolding, so they can make adjustments without having to wait years for evaluation conclusions.

One approach called lean data is meant to overcome the problem of lack of available data to measure social impact. Instead of working to build data capacity up front, pre-analysis, this approach encourages working with existing data while at the same time building capacity to gather more. Clare Nolan of Engage R+D, who is leading the evaluation efforts for Starting Smart and Strong, calls this “one of the most effective ways to work in communities.” Nolan says starting with what is available and presenting data in ways that educators value “makes people hungry for more data and builds appetite for the harder work of improving data systems.”

All three communities have also administered the Early Development Instrument (EDI), which provides a measure of children’s health, development, and kindergarten readiness across a community and is designed to help local policymakers target resources to improve conditions for young children and track change over time. The EDIs are filled out by kindergarten teachers in the fall about each student and measure school readiness in five areas known to affect well-being and school performance: physical health and well-being; social competence; emotional maturity; language and cognitive skills; and communication skills and general knowledge. The survey is administered every two to three years.

Instead of working to build data capacity up front, pre-analysis, the lean data approach encourages working with existing data while at the same time building capacity to gather more.

Results are not interpreted at the individual child level, but instead are reported to school districts and community partners in aggregate. Results for each community are just now being analyzed, but leaders are working with researchers from UCLA to identify geographic patterns in the data and to understand how they relate to other indicators.

In Franklin-McKinley, for example, results show that over half of kindergarten students are vulnerable (29 percent) or at risk (25 percent) across all developmental domains.² Juan Cruz, the district’s superintendent, called the results “eye-opening,” and told us the survey helped the district see new pockets of need and make decisions about where to direct resources.

Leaders intend the EDI results to provide a baseline that can inform planning and improvement over the long term both at the community level and for the Starting Smart and Strong initiative as a whole.

The districts have learned the importance of building in evaluation and data collection from the beginning so there is a baseline with which to compare information over time. Project leaders have had to backtrack in Fresno, as eagerness to begin the Language Project meant that not as much baseline data as needed were gathered.

Oakland’s school district became a participant in Quality Counts, the county’s quality rating and improvement system. Classrooms in the district's 15 Childhood Development Centers and 13 state preschool programs have shown significant improvement since their initial review several years ago. This year seven of the sites received a 5-star rating, the highest available. Higher ratings bring in additional dollars for the district.

Oakland Starting Smart and Strong also supported a parent engagement study conducted by local parents through the East Bay chapter of Parent Voices, a group that advocates for affordable, quality child care. In addition to revealing the power of data collected by the community, the study identified key information that policymakers can use to meet critical needs. The study, for example, found that informal child care in the city is growing, in part because of unmet child care needs and unstable family circumstances often tied to a lack of affordable housing. “Parents said they chose Family, Friends, and Neighbor care because there was no other choice,” authors write.³ The study also found that parent leadership and building ecosystems and “safety hubs” for families with young children are essential.

The study found that informal child care in the city is growing, in part because of unmet child care needs and unstable family circumstances often tied to a lack of affordable housing.

The Oakland Starting Smart and Strong collaborative is working with local researchers to better understand the disparities in health and education outcomes for boys of color and has commissioned several studies and data-tracking projects. This work will help to develop more targeted strategies to serve these boys of color and close the striking achievement gaps that exist between them and their more advantaged peers.⁴ The district is already piloting some of this work⁵ for African American and Latino boys attending preschool in West Oakland through additional foundation support.

Similar efforts to use data are underway in Franklin-McKinley. In addition, the district is trying to work with teachers to help them better understand and see the value of data collection and use in early childhood classrooms. This is a change from the past, when evaluation data were primarily used for accountability. Instead, the district has worked to make teachers partners in data collection and use, which they say is paramount to the data’s validity and to their district’s ability to use it meaningfully to improve teacher practice.

“When we’re all looking at data together, that’s when we get powerful movement for our kids,” said Tweety Yates, a research assistant professor in the Department of Special Education at the University of Illinois at Urbana-Champaign. She is facilitating professional development sessions with Franklin-McKinley teachers.

The facilitators in Franklin-McKinley say that they aim to integrate discussions of overall systems change into every session, so teachers understand what it is they are a part of and why they are taking part in this learning. They also try to build on and emphasize teachers’ expertise and ability to validate or dispute data about their own classrooms.

“When we’re all looking at data together, that’s when we get powerful movement for our kids."

“Principals and directors really talked with us about why we are doing this,” one participating teacher said of the data collection. “They presented it not as an evaluation tool at all, but a tool to help us create classrooms that are enriching for kids. It took off some of the pressure.”

In addition, the district is participating in an effort to get the different large data systems that track children’s progress in the region to talk to one another. Franklin-McKinley has been working with the Santa Clara County Office of Education to build a comprehensive early learning dashboard that may soon be available to other districts in the region. Students in Franklin-McKinley’s early learning programs have California Department of Education-assigned Statewide Student Identifiers⁶ that will follow them into their K–12 schools to enable more seamless sharing of data, like developmental evaluations and assessments, attendance records, and vision, hearing, and dental screening results.

All three districts have learned the value of community-based research and on relying on those who know the community well, like teachers, to evaluate and collect data. They acknowledge that these methods are, in some ways, a departure from traditional research methods, and some would say they taint the validity of data collected. On balance, though, especially in combination with other methods, these communities believe this has resulted in the collection of information that leaders say is more accurate, culturally relevant, and likely to be used.