Introduction: The Housing Data Challenge

Housing is foundational to every aspect of health and well-being. It anchors communities, provides families stability, and structures other key aspects of our lives. It is also a major economic driver, accounting for 16 percent of U.S. gross domestic product (GDP).¹ But housing in the United States is too expensive, in short supply, and deteriorating in quality. Our supply shortage ranges anywhere from 1.5 million to 5.5 million homes depending on who is counting.² The homes that do exist are increasingly unaffordable: Estimates show that 49 percent of all renters spend more than 30 percent of their income on rent and utilities,³ and homeownership is increasingly out of reach, notably for first-time and low- or moderate-income buyers.⁴ All the while, the estimated cost of needed repairs in occupied housing is nearly $200 billion.⁵

The recently enacted 21st Century ROAD to Housing Act is a bright spot, showing bipartisan momentum through a comprehensive piece of federal legislation that incentivizes new construction, cuts burdensome regulations, and eases rules for manufactured housing, among a host of other provisions, to increase supply and lower costs. But the data needed to understand which provisions are successful, in what parts of the country, and for whom, is often out of date, unreliable, or simply non-existent.    

The fact remains that the United States lacks reliable, granular, and coordinated housing data—data needed to build a comprehensive and accurate picture of housing conditions, needs, and outcomes—and track housing as a dynamic system over time. And these housing data gaps have had real consequences, including:  

• Many families facing housing instability and homelessness, including those informally evicted outside the court system, living in cars or extended-stay motels, or doubled up with family or friends, are not visible in collected data, meaning many of the most vulnerable are the least prioritized and often excluded from assistance altogether. 
• Policymakers, without reliable data, often rely on one-size-fits-all housing solutions, such as housing developer tax credits to increase affordable housing, when in many places, allowing manufactured housing or accessory dwelling units would be more efficient. 
• State and local policymakers cannot enforce consequences for negligent property owners, because this ownership is rarely tracked, leaving millions of renters living in substandard conditions with little recourse.
• The public cannot hold decision makers accountable for improving housing conditions or even push them to specify housing-related goals, because the data needed does not exist, is unreliable, or only becomes available years later.  

By contrast, a robust housing data system would enable us to assess the health of housing markets at a given point in time, understand and develop real-time solutions to address the most pressing housing challenges, and hold leaders accountable for improving housing affordability and stability in communities across the United States. 

The Future of Land and Housing program at New America presents a roadmap for building this housing data system, detailing the key components needed to move us from the fragmented and decentralized system we have today to one where the coverage and quality of data needed to ensure that housing is advancing stability and well-being for all.

Housing data in the United States mirrors the housing system writ large, which is highly decentralized and fragmented. Housing decisions are divided across all levels of government and private actors. Local governments shape where and how housing can be built through land-use rules and permitting; states design and administer housing programs and establish regulatory frameworks; and federal agencies influence housing production and preservation through financing, subsidies, and regulation. The result is a collection of geographically distinct housing markets that are highly local, across 3,000 counties and 19,000 municipalities. Within each market, private actors—including developers, financial intermediaries, investors, and property owners—make decisions about housing construction, financing, ownership, and maintenance. This housing ranges from purely market-based to fully subsidized, all within a broader ecosystem of federal, state, and local policies and programs. 

At the federal level alone, housing policy and assistance are overseen by a range of agencies, including the U.S. Department of Housing and Urban Development (HUD),⁶ government-sponsored enterprises (including Fannie Mae and Freddie Mac), the Department of the Treasury, the Department of Veterans Affairs, and the Department of Agriculture.⁷ Outside the federal government, housing data is generated and collected by a constellation of state and local agencies, nonprofit organizations, banks, lenders, courts, and other private actors, all of whom serve different purposes and end users. 

This fragmentation results in housing data that is stored in hundreds of places, subject to different definitions, methodologies, and reporting requirements, and covers different periods and levels of geographic granularity, making data-sharing, integration, and aggregation difficult and, in many cases, impossible.⁸ What’s more, no single entity, public or private, is tasked with overseeing and making sense of the housing data we have. What emerges from this fragmented and decentralized housing data system is a patchwork of partial views of housing, but never a complete picture.

Despite these challenges, much of the housing data produced in the United States is robust and high-quality. In particular, the U.S. Census Bureau is a global leader in conducting high-quality, nationwide surveys, producing a wealth of publicly available housing data at varying levels of geography. This survey data provides a critical backbone for decision-making, supplying trusted, standardized statistics on a wide range of housing issues, from the physical condition of our housing stock in the census and HUD’s American Housing Survey to new construction permits in the Survey of Construction. 

In addition to Census data, the United States collects a wide range of housing data, including surveys conducted by federal agencies, federal administrative data, mortgage and other financial data, state and local administrative data, and private market data. These include:

• Federal surveys conducted by the U.S. Census Bureau, HUD, and other federal agencies primarily focus on household and housing unit characteristics, financing information, building permits and construction, and other data. Census surveys include the American Community Survey, the American Housing Survey, the Building Permits Survey, the Survey of Construction, and the Household Pulse Survey.  
• Federal administrative data from HUD, the U.S. Treasury, and other federal agencies primarily focus on federal housing assistance, including data on homelessness and public subsidies, such as the Housing Choice Voucher Data Dashboard, Homeless Management Information System (HMIS), the Low-Income Housing Tax Credit database, and the point-in-time and housing inventory count, among other sources.   
• Mortgage-related and other financial data collected from banks and other financial institutions by federal agencies as a result of consumer protection legislation or through the Federal Reserve. This includes the Federal Housing Finance Agency (FHFA) House Price Index, Uniform Appraisal Dataset, and National Mortgage Database; Home Mortgage Disclosure Act (HMDA) data; and Consumer Financial Protection Bureau (CFPB) Rental Housing Payment Data, among other sources. 
• State and local administrative records, including deed transfer records, property tax records, eviction and foreclosure court records, building inspection and code violation records, historic preservation designations, environmental hazard records, zoning maps, and state and locally run public subsidies for housing. 
• Private data from vendors that collect consumer, finance, and market data, including ATTOM Data Solutions (covers ownership, mortgage transactions, foreclosure, and climate/hazard risk), BatchData (includes assessor and parcel data), Cotality (collects from mortgage servicers; covers home prices, property taxes, foreclosures, and investor activity), and InfutorData and Data Axle.⁹ Zillow and other real estate companies also use their proprietary listing information to produce estimates of home values and rental prices. 
• Derived datasets created by researchers and nonprofit organizations by integrating multiple data sources. Examples include the National Housing Preservation Database (NHPD), Eviction Lab’s Eviction Tracking System, the National Zoning Atlas, and the Lincoln Institute of Land Policy’s Who Owns America database.