SDSU Researchers Watered Down a Police Racial-Profiling Study

When a long-awaited study on whether the San Diego Police Department engages in racial profiling finally dropped in late November, the results were unsurprising: It found that black and Hispanic drivers were more likely to be searched, though they were less likely to actually have contraband items, and that minority drivers were more likely to be subjected to field interviews.

When it came to the overarching question of whether officers and SDPD as a whole showed racial bias, however, San Diego State researchers were restrained: Though they found significant differences in the way minorities and white drivers were treated, researchers were careful to point out that such differences “are by no means unique to the SDPD” and that findings only “suggest” that implicit, or unconscious, bias “may exist” among officers.

But a draft copy of the study obtained by Voice of San Diego through public records requests was far more aggressive. City officials fought to avoid releasing the drafts publicly and said their disclosure “would likely increase community tension and discontent,” but copies were provided by San Diego State University, which conducted the study.

During the process of revising the study from a draft to the final version that was presented to the City Council, harsh language was softened and some findings were taken out entirely.

Among the changes:

• In more than two-dozen instances, the word “bias” was replaced with “disparities.”

• An early draft recommended the department stop making traffic stops for minor violations unrelated to public safety, and instead simply issue citations by mail—something, researchers noted, other departments are exploring.

• Also cut from the final draft was the finding, via police survey, that the majority of officers felt they wouldn’t benefit from additional training in fair and impartial policing.

• The final version of the study found that black drivers were more likely than white drivers to be stopped in only one of the San Diego Police Department’s nine divisions, northeastern, which includes the largely white neighborhoods of Mira Mesa, Rancho Bernardo, Rancho Peñasquitos and Scripps Ranch. A draft version reached a different conclusion, finding evidence of racial disparities in three divisions, not just one, and, in aggregate, all police divisions located north of Interstate 8, often considered to be San Diego’s racial and economic dividing line.

On that last point, Joshua Chanin, professor of public affairs at San Diego State University and the study’s lead researcher, said the final change was made because the researchers decided to use a different threshold to determine whether a finding was statistically significant.

Put simply, researchers initially considered a finding statistically significant—meaning the data showed evidence of racial profiling—if there was at least a 90 percent chance the finding was true. This was later changed to 95 percent.

Findings for southeastern and northern divisions were just a few points away from 95 percent: In the Northern division, the likelihood that racial disparities in traffic stops weren’t due to chance was 93.4 percent. In the southeastern division, it was 92.3 percent, and for all divisions located above the I-8 divide, 94.2 percent. This was noted in a draft of the study provided to the city on Oct. 27:

“[O]ur analysis of combined stops from all five divisions above Interstate 8 shows that when compared to Whites, Black drivers are 15 percent more likely to be stopped during daylight hours, when driver race is visible, than after dark, when driver race is obscured,” it read.

But by the final draft the finding was the opposite: “Analysis of the aggregated data from [divisions north I-8] shows no statistically significant difference in the daylight-darkness stop patterns of Black and White drivers.”

The 95 percent threshold is considered a standard in scientific research, and Chanin said the goal was to produce a study whose findings were “unassailable” by the city and police department.

City Councilman David Alvarez said he wishes researchers had found a way to explain that some findings were on the cusp of statistical significance.

“It’s even more important when you’re dealing with the public who aren’t statisticians and who need to get a narrative and a description of what the findings were because that’s how you’re transparent about information,” he said.

Chanin said his team believed the revisions were necessary in order to persuade SDPD to take the study seriously.

“The argument that won out,” he said, “is if we want this to make any sort of meaningful impact, it needs to be something that is, to the extent possible, unassailable in terms of its orientation.”

After a tense period that included San Diego Police admitting they stopped following their own policies to guard against racial profiling, sustained complaints from minority communities of being unfairly targeted by police, high-profile cases of police misconduct, a Justice Department review and the officer-involved shootings deaths of at least two unarmed minority men, city officials in 2015 decided to act: Then-Councilwoman Marti Emerald tapped Chanin and his team to research whether people of color were really being pulled over by SDPD at a disproportionate rate.

The study employed a method called the “Veil of Darkness.” Here’s how VOSD described it in 2015:

The Veil of Darkness assumes two things. First, that it’s more or less the same people driving on a given street between 5:30 and 9 p.m. They’re coming home from a 9-to-5 or heading out for the night shift. Second, it assumes than an officer can better observe a driver’s skin color when the sun is up. So, researchers can compare traffic stop data from 5:30 to 9 p.m. in July, when it’s light out, to the same timeframe in January, when it’s dark. If more people of color are pulled over in that area during the summer, one can assume race is at play.

In addition to examining traffic stops, the researchers also looked at what happened after a car was stopped. They found that citywide, black and Hispanic drivers were more likely to be searched following a traffic stop and less likely to be found with contraband.

The study got its first hearing at a meeting of the City Council’s Public Safety and Livable Neighborhoods Committee on Dec. 7. There, members of the public raised concerns that city officials had pressured Chanin and his team to tone down their findings. On Dec. 1 we asked the city for copies of all drafts of the study. State law requires public agencies to turn over drafts that are “retained in the ordinary course of business”—meaning, if a draft exists in electronic form, like an email attachment, it must be disclosed.

The city denied the request, and said no electronic copies existed. Lea Bernard-Fields, the city’s public records administrator, said there was only one hard copy of a draft, but it had been “extensively annotated by an individual staff member” and therefore was not subject to disclosure.

Nearly three months later, after VOSD put in a similar request with San Diego State University, Bernard-Fields followed up to say an unmarked hardcopy of a draft had been located, but releasing it “would likely increase community tension and discontent in an environment already fraught with friction over the issue of vehicle stops.”

San Diego State University ultimately provided emails between Chanin and city staff that included two drafts of the report.

The earliest draft of the study provided by SDSU—the version emailed to the city on Oct. 27—is far different in tone than the final draft. Between that draft and the final draft, there are small changes, like “significantly more likely” being changed to “more likely.” The line about how findings suggest that implicit bias “may exist” initially said findings “suggest that implicit bias exists” among SDPD officers “to an extent that … is on par with other departments nationwide that have come under federal supervision for such bias.” And, in more than two-dozen instances, the word “bias” was changed to “disparities.” For example, the recommendation that the police department “[a]cknowledge the existence of racial/ethnic bias” became “[a]cknowledge the existence of racial/ethnic disparities.”

Chanin said other academics who study policing strongly recommended changing “bias” to “disparities.” Experts I spoke to agreed with the change.

“Sometimes words have a generally accepted meaning in society, but a far more specific definition in academic disciplines,” said Seth Stoughton, a law professor at the University of South Carolina and former police officer.

Lorie Fridell, a criminology professor at the University of South Florida and a leading expert on racial bias in policing, said she would have made the same recommendation. “It’s not difficult for social scientists to measure disparities [in traffic stops],” she said. “What’s very difficult is to identify the causes of that disparity.”

In an earlier draft of the study, researchers urged the police department to cut back on its use of traffic stops—specifically for equipment violations like broken brake lights or tinted windows. Doing so could improve community relations and officer safety, the draft said.

“We note that other police departments are currently in the process of reconsidering their approach to traffic stops in this way by directing officers to focus on the violations most related to safety, such as speeding and the running of red lights,” the draft said.

Researchers recommended officers record the car’s license plate number and that the department implement a system to issue the driver a warning or a “fix-it” ticket by mail.

But the department pushed back against the recommendation, Chanin said. He and his team were left with a choice: Keep a recommendation that would never be implemented, or cut it to “foster goodwill and enable some of the other [changes] that we felt were possibilities.”

That was the only significant change due to pressure from the city, Chanin said.

But even the toned-down report had little impact. At the Dec. 7 committee meeting, Police Chief Shelly Zimmerman acknowledged the study’s findings, but when asked repeatedly by Emerald whether people of color were sometimes treated differently by her police officers, Zimmerman responded only that “every human being has bias.”

Zimmerman did say that her department was committed to taking a “proactive approach” to combating bias.

In February, the City Council voted to accept the report, but declined to implement any of its recommendations. Alvarez and Councilwoman Georgette Gomez, who replaced Emerald, were the only “no” votes. They both argued that the study’s recommendations, particularly that the department improve its data-collection efforts, warranted further discussion. The study had recommended that the department, at the least, collect basic information on the officer making the stop. “Without these data, it is impossible to know, for example, whether black drivers were treated differently by white cops,” Chanin said.

In an exchange with Alvarez at the meeting, Chanin said the department could easily collect such data under its existing system. Alvarez said he was disappointed recommendations went ignored.

“We need to have this information,” Alvarez said at the meeting. “It does start with the data. It’s a shame that we wasted so many people’s time … hoping to actually get real action and instead what we get is saying this report is before us—there it is—and we’re not doing anything about it.”

Chanin said that regardless of any changes made between the first and final drafts, “there’s clear evidence in this report that there is a difference in the way that black and brown people are treated than whites.”

Like Alvarez, he questions why the city asked for the study, if only to shelve it.

“Here we are, two years removed from the initiation of this process and I don’t know if things have changed,” he said.

This article originally appeared in the Voice of San Diego.