Equity by Design

The Equity by Design project is a product of the Open Technology Institute’s desire to explore innovative ways of addressing discriminatory harms caused by technological products and services. To develop this report, we consulted technologists and tech policy professionals at non-profits, major tech companies, policy organizations, and tech startups. The information and advice they offered were integral to shaping the project and crafting recommendations that work within the technology industry’s production process framework. These recommendations are meant to contribute to the conversation on incorporating equity into product design processes, with a particular focus on people and the critical role they can play in bringing diverse perspectives to the development of technological tools and services.

Why the Tech Industry is Failing on Equity: Process and Personnel

While the creation of technological tools and services has provided society with numerous benefits, not every community has had the same positive experience during the development of such innovation. There are many documented examples of marginalized communities being negatively impacted by technological change: a major social media platform's algorithm cropped photos to mainly display lighter-skinned people; a major tech company's contractors targeted homeless people of color to develop facial recognition technology; and another major tech company allowed landlords and real estate brokers advertising on their platform to exclude people based on their race, family status, or disabilities. As of 2020, 98 percent of all web content fails to meet the Web Content Accessibility Guidelines, which is the international standard for making web content accessible to people with disabilities. Additionally, automated tools can perpetuate patterns of discrimination, which can cause acute harm when they are used to make consequential decisions in areas such as criminal justice, employment, and housing.

As of 2020, 98 percent of all web content fails to meet the Web Content Accessibility Guidelines, which is the international standard for making web content accessible to people with disabilities.

Current initiatives to address the failure to establish equal experiences for all users tend to rely on identifying and prohibiting individual variables associated with race, gender, and other protected classes. This is not enough. The problems present are not merely technical in nature—they are also process and personnel challenges. This project explores how to address these issues in the development stage. Conversations around the impact of technology often center on digital tools and services themselves rather than the people who produce them. Relatedly, product team members are not always encouraged to pursue a human-centered approach in their process. As a result, efforts to address the failures of technology tend to focus on technical solutions. Technology can help people better navigate the world, but only people can shape the world's realities.

People, and their very human experiences, perceptions, and choices, are at the heart of many of the challenges—and solutions—to more equitable design. Simple adjustments to coding or datasets alone will not address the fundamental problem of bias that ignores the experiences, realities, and histories of marginalized communities. Understanding and addressing the causes of bias in technology development is important, and product managers and technologists need well-developed procedural guidance to integrate equity and human-centered design into the production process. Such guidance will help reform the systems creating inequitable technology and prevent the harms that result from building technological products without user diversity in mind.

While the lack of workforce diversity in tech has long been discussed, its contributions to inequitable outcomes are often understated. According to U.S. Census Bureau 2019 estimates and the U.S. Equal Economic Opportunity Commission's Diversity in Tech Report, many marginalized communities, particularly Black and Hispanic/Latinx people, are underrepresented in the tech industry. Black and Latinx people account for 31 percent of the U.S. population, but only 15 percent of the U.S. tech industry. Only 1.4 percent of Silicon Valley startups are run by people of color. A separate analysis illustrated that women accounted for only 22 percent of the workforce at major tech companies. The lack of workforce diversity in the technology sector results in lost benefits for companies and an undue burden and discomfort for employees from marginalized communities.

Product team members who fail to see the impact of certain design choices and hidden bias in data sets likely play a significant role in the discriminatory functionality of technological products and tools. There cannot be a discussion of integrating concepts of equity into the technological production process without acknowledging the harms caused by a lack of diversity in the workforce. Equity means recognizing wants, needs, and circumstances particular to a community and providing them with resources reflective of that recognition. To do this effectively, it is vital to consider a wealth of voices and perspectives, and robust diversity facilitates that. This is beneficial for all parties: it aids companies producing technological products in avoiding discriminatory incidents, and more importantly it protects users from discrimination.

There cannot be a discussion of integrating concepts of equity into the technological production process without acknowledging the harms caused by a lack of diversity in the workforce.

For example, consider facial recognition software that struggles to identify the faces of individuals with darker skin tones. At best, this software is suboptimal; at worst, it is a danger to communities of color. Developers commonly sell this kind of software to law enforcement agencies and other entities. Given the history of law enforcement employing surveillance tools to overpolice people of color and the resulting harms, misidentifications caused by this system's inaccuracy would create catastrophic results for these communities. In this instance, the company should have established an internal process to reflect on equity concerns, with the opportunity for external equity experts to review the product.

Creating a shared language is an important start to establishing equity by design practices. All involved need to understand both the stakes and the goals. For that reason, we consulted multiple technologists and technology policy professionals to gather a diverse array of perspectives. Those consulted came from civil society organizations, major tech companies, and tech startups. In our conversations, we walked through a company’s typical production process, from the inception of an idea or a problem statement to quality assurance or user acceptance testing, allowing us to highlight multiple entry points where product teams could inject conversations around equity and equality. The information and advice these experts offered was integral to shaping the project and crafting pragmatic recommendations for developers.

Many well-intentioned technology company staffers are positioned to mitigate the risks of inequitable outcomes, but lack access to the tools they need to do so. Much of the rules and guidance offered to tech workers are unactionable or prohibitively burdensome. For instance, compiling datasets for training algorithms is incredibly difficult and expensive. So, while it may seem intuitive to ask product teams to fulfill specific data collection goals in pursuit of producing unbiased data sets, they can't do this without adequate time and resources. This does not address the problem. Rather than outlining policy goals and principles with the expectation that employees will develop technical solutions, we hope through this project to bridge theory and practice by providing feasible guidance to achieve those policy goals.

We acknowledge that many companies and researchers are aware of the problems presented, and we are aware that solving them can be complicated. However, the objective for this project is not to lecture product team members on what they must do. Rather, we focus here on the role that people and human-centered approaches can play, and hope this report and our subsequent recommendations and observations will add to the conversation on integrating equity into product design. We believe it is critical to incorporate perspectives from people with diverse backgrounds and lived experience, and encourage product teams to think intentionally about how they can rely on these approaches to achieve equitable outcomes. Ultimately, teams should be able to see the influence of human decisions and the power of human-centered solutions. Further, developing technological products and tools with an equity lens also elevates the goal of optimal and safe user experiences across different demographics. Achieving this means building a better product.

This report will bridge the gap between policy and practice through two discussions. The first section will focus on incorporating diverse people and their perspectives into the production process for technological tools and services, and what touch points exist to ensure equitable outcomes. The second section will focus on the role of experts and consultants who need to be engaged to promote equitable outcomes. The report is directed at those engaged in the governance and production of technological tools and services, but is also meant to begin a conversation lifting up the values of equity and equality in tech development.

Prioritizing equity in the development of technological tools and services means keeping the gravity of technology’s human impact in mind and having the determination to strategically address shortfalls. Doing this effectively requires that companies educate their product teams on the relationship between lived experience and identity, and instruct them on the social and historical contexts the product may influence. If product teams keep this in mind throughout production, then they should only release a product when it produces optimal and safe user experiences across different demographics. Discrimination is not just a bug in the system that needs to be fixed, and equity is not a limitation to technological advancement. A discriminatory tool is a broken tool, and equity is a goal with unlimited solutions.

Discrimination is not just a bug in the system that needs to be fixed, and equity is not a limitation to technological advancement.

The following recommendations focus on pragmatic ways to integrate equity principles into the production process from the inception of an idea to the release of a technological tool or product.

Recommendations

• Formulate ideas and problem statements to account for identity-based experiences.
  Idea and problem statements are the starting point for developing most technological tools or services. Well-crafted questions are much more likely to elicit high-quality answers, and an idea or problem statement that is not formulated to account for identity-based experiences is unlikely to produce equal experiences across different demographics. Key facets of identity that developers need to consider include race, ethnicity, gender, socioeconomic status, age, sexual orientation, religion, and disability. These different facets of a user’s identity may impact their wants, needs, and how they interact with the world.

  Some product teams assess the utility of their idea and problem statements through solicited customer feedback. When idea or problem statements are user-validated, product teams should examine whether the users selected for validation reflect the diversity of the product's target population. This formulation stage is also an important time to assess how a product will affect users across diverse populations. Questions to ask can include: (1) What communities are likely to benefit from the identified problem being solved, and which are not? (2) Is a product team regularly solving problems useful to specific communities? (3) Are certain groups regularly neglected in the sector that the development of a tool or service will impact?

  After idea and problem statements are evaluated through these questions and other similar ones developed by production teams, the identification of patterns can help the team avoid future blindspots. If inequities such as bias or unfairness present themselves, evaluate how the needs of neglected communities can be incorporated into the idea and problem statements, or whether a separate statement needs to be developed to identify the separate but related problems in those communities that need to be solved.

  For example, around 2016, Airbnb, a platform that facilitates users renting or renting out properties for short-term stays, faced a rising tide of discrimination complaints from users of color routinely denied stays by hosts. While designed in an engaging and modern way, the platform did not meet the needs of its users of color. To address the problem, Airbnb initiated a policy where hosts could not see guests' profile photos until after they accepted bookings. The platform then sought input from a plethora of racial justice organizations and initiated a user study "to understand when and where racial discrimination happens on [the] platform and the effectiveness of policies that fight it." While features like profile photos may create a sense of social community or allow hosts to get an idea of who was staying in their home, they also create conditions for hosts to deny guests because of their race. Validation of these ideas by diverse users would likely reflect some of these concerns. When product teams consider diverse voices at the formulation stage, it is easier for them to produce equitable experiences across different demographics in the tool they develop.

• Refine product roadmapping to identify equity concerns.
  Developing a product that works well across different demographics requires a people-centered approach: brainstorming and collaboration distinctly with the goal of incorporating diverse perspectives. After product teams formulate an idea or problem statement, and if product roadmapping is already a part of their process, they should ensure that they are including people from diverse backgrounds and perspectives, and evaluate equity concerns at this stage. If such roadmapping has not been incorporated, teams should conduct a live-product roadmapping meeting to reveal potential equity concerns. Ideally, this meeting should include the design, product, development, quality assurance (QA), compliance, and policy teams. Teams should think through the nature of potential problems, review risks and how to address them, and ensure that everyone has the tools they need to address risks. This is also a good time to ensure the metrics used to evaluate the success of a product, like user definition and objectives and key results (OKRs), are inclusive.

  User definition in tech development is done so that product teams can work with the voice of the client in their heads, and it should account for the diversity of users within the targeted subset. However, an industry product team member interviewed for this project commented that product teams tend to build products with themselves or their friends in mind. While age and income are objectively designated in user definition, from our consultations it appears that other identity facets like race, gender, or ability may not typically be considered. This fails to paint a full picture of the needs, challenges, and wants of users. If this is true for a large swath of the industry, the implications are significant. Namely, a product team’s conceptions of a user will be informed by their own experience, and those perceptions will shape the design, development, and production of a product. This could potentially leave many communities out of the thought process.

  Product teams should expand the identity categories for user definition. Product managers consulted for this project expressed fear or discomfort with this idea because of sensitivities around certain areas of identity and the potential for the process to go wrong. However, it is important to unpack that unease and determine what solutions exist to address those concerns. Additionally, at least for certain facets of identity, these fears ring hollow. For instance, even if tech companies do not keep records of their users' race, their algorithms are likely detecting proxy variables. Thus, where appropriate and with proper use limitations, they should be able to test their algorithms for harmful treatment on the basis of race. The refusal to do so only undermines their commitment and ability to properly address the needs of all communities. Failing that, product teams should go through undefined identity areas category by category, noting how different facets of identity might affect the wants and needs of a user.

  OKRs, key metrics to define success based on the idea or problem statement, are intended to be ambitious objectives, making them a great place for product teams to establish accountability early by setting enterprising equity goals. Product teams can define OKRs with equity in mind by measuring active equity integration attempts. For instance, teams should hold a series of meaningful stakeholder engagements with civil and human rights experts and include an evaluation of how the product will affect or be received by communities of color.

• Data should be critically evaluated to offset systemic inequities.
  Data is a vital resource in the production of technological tools and services. However, it is only a measure of the world around us, reflecting its realities and failings. This is particularly true of historical and predictive data. Discrimination against vulnerable communities has been pervasive in many sectors of our society. The results of this discrimination are often reflected in data, and without explicit actions to counteract these facts, tools built on or trained using such data will perpetuate patterns of discrimination. Teams can avoid replicating these failings by seeking to update data to reflect current trends or events and ensuring they ask the right questions of data.

  An illustrative example can be found in criminal justice software. A criminal justice consultancy developed a criminal sentencing algorithm to determine recidivism rates. ProPublica, a nonprofit news organization, conducted an evaluation of the algorithm’s racial fairness. The study found that the algorithm was correct at the same rate for white and Black people, but the error it did make tended to favor white people, resulting in their undeserved release, and disfavor Black people, resulting in unfair and inappropriate incarceration. Though intended to be an objective tool, the algorithm perpetuated well-documented patterns of racial discrimination in our criminal justice system. Alongside considering how to use data, it is important to evaluate when to use data. Sometimes it may cause the least harm to use an alternative data source, and sometimes it is most equitable not to use data at all.

• Users selected for interviews, journey mapping, testing, and research should reflect overall user diversity.
  User interviews, journey mapping, testing, and research are intended to reveal how users will interact with a product, to understand the accuracy of assumptions, and to garner as much reference information as possible. Trying to incorporate a lens of equity in tech during data collection may be challenging, but selecting users who reflect overall user diversity will provide a more insightful and reliable pool of data. Where the current user base is not diverse, product teams should commit to seeking out diversity. Insightful and reliable data gives production teams a better chance at building a great product. It also provides users across demographics with a product that considers their voices, needs, and perspectives. These points also hold true for user acceptance testing. Though not conducted universally, it is a last step in the production process. Like other kinds of data gathering, it can be burdensome. Still, it should be considered as a last chance to catch unforeseen discriminatory failures.
• Take steps to avoid repeating the same mistakes.
  Just as developing a product that works well across different demographics requires brainstorming and collaboration distinctly with that goal in mind, so does evaluating equity failures. A live meeting can flesh out the failure and its causes, and provides a space for collaborative problem solving. Like the roadmapping meeting, it should include members of the design, product, development, QA, compliance, and policy teams. It should be aimed at understanding the nature of the problem and reviewing its root causes to ensure that everyone has the tools to address similar issues in the future.

  Product teams should also consider developing a system for assessing and managing discriminatory failures identified internally or through external feedback. People working in technology production know, perhaps better than anyone, the stories accumulated data can tell; tracking equity failures systematically can help them turn an isolated problem or mistake into an account elucidating a pattern or process flaw that needs to be fixed.

Prioritizing equity throughout the production process means being deliberate about incorporating perspectives from people of diverse backgrounds, and considering equity as an essential component of the technological production process. It requires product teams to consider a tool or service “functional” only when it produces optimal and safe user experiences across diverse demographics. This may sound daunting, but technologists have accomplished much more challenging feats. The real work will be in the small everyday decisions they make, and how they process them.

Prioritizing equity throughout the production process means being deliberate about incorporating perspectives from people of diverse backgrounds, and considering equity as an essential component of the technological production process.

Integrating equity into the technological production process requires first that the product teams garner feedback from members of their own team and other experts to identify equity issues. Further, engaging a broad range of perspectives, and doing so with an equity lens, helps product teams develop tools and services in a fluid and intuitive process with unbiased outcomes. The prior sections in our Equity by Design project covered the landscape of inequitable technological products and how product teams can incorporate equity goals into the production process. This section will center on the importance of engaging a broad range of perspectives, including those of external experts and consultants, to mitigate harms and promote equitable outcomes.

Through various consultations for this project, it was clear that most production team members care about building the best product with the potential to create the most good for all impacted communities. However, it was also clear that divisions within a product team could exist, where competing priorities could make production with a singular purpose a challenging task. Further, the conversations also raised the importance of team leaders recognizing that they needed to include third-party perspectives to fully identify and address equity concerns. While some larger companies may utilize informal feedback from civil society organizations to raise concerns around disparate impact and other harms, many companies appear to only rely on their internal team members to draw out concerns. However, the ideal approach is to foster an environment that facilitates internal team members sharing experiential expertise, as well as to engage external organizations who have developed large bodies of work on these issues and are eager to collaborate with companies to ensure that equity is front and center.

Recommendations

• Commit to seeking out representatives of those voices missing from your project.
  As discussed previously, the lack of diversity in the tech industry makes it unlikely that a representative of every community will be able to participate in making decisions. However, product team leaders should think about how they can account for, and find representatives of, the missing voices. This does not mean speaking for people of other communities, but rather seeking out advocates of diverse perspectives.

  For example, say a team of apparently non-disabled people is producing tools for a hospital to determine how to ration care. When the idea of using “likely speed of decline” as the preeminent factor for determining which patients receive care is raised, the team lead should make a decision to find advocates for the disabled community and question how this may harm them. The team lead should also create an environment where any team members who have relevant personal experience feel supported to raise concerns throughout the production process.

• Foster an inclusive environment where those impacted by the technology your team is developing feel empowered.
  It can be daunting to be the sole team member, or one of only a few, from a particular community. However, just as selecting diverse users provides a more insightful and reliable pool of data, diverse voices at work provide perspectives and viewpoints which can help companies look at problems, solutions, and possibilities in another way—ultimately leading to more robust results. While no one should expect people to be experts on their entire community, the value of their insights should be respected. For teams to reap the benefits of workplace diversity, they should take steps to foster inclusive environments. Part of this comes down to trust, but trust is built and requires work.

  Some tools that may help build trust include creating alternative feedback mechanisms to protect and account for different comfort levels among team members, and allowing them to voice ideas and critiques anonymously. Teams may also consider implementing an escalation system where members can anonymously alert a member of the compliance team or other appropriate staff about concerns. There is a category of discriminatory outcomes that result from the absence of certain perspectives or lack of consideration. However, creating these spaces for commentary may improve the product, avert negative fallout, and most importantly, prevent someone from experiencing pain or harm.

• Consult with external organizations who specialize in equity.
  Despite a company’s development and retention of policies and compliance teams to address the issues raised through our Equity by Design blog series, product teams should make it a part of their process to seek input from external civil society organizations specializing in racial justice, civil rights, human rights, or other equity-based work. If there are concerns about how a product will impact a particular community, teams should seek input from organizations or experts that advocate for that community. Product teams should ask these experts to review products’ potential equity harms because experts at these organizations spend every day at work addressing equity issues, and are in a unique position to offer insight. There should ideally be two check-ins: the first early enough to consider and utilize any guidance offered, the last after a product is workable, but before it is too late to make changes.

  For members of product teams, the guidance and critique garnered through consultations with experts should serve as an educational experience where they internalize learning and apply it to future products. This is particularly true for product teams developing tools or working in industries with a fraught history, as well as teams working during a sensitive cultural moment.

A human-centered approach in product design can contribute to the development of technological tools and services that are equitable in their impact. While there are technical aspects to addressing the many issues raised by our Equity by Design project, one major approach that tech companies must undertake is to review the personnel and processes that lead to the creation of technological products and services. The impact of these products and services on society is profound, and the fact that humans designed, developed, and produced them is remarkable. Consequently, product teams can wield immense societal power. As a result, their experiences, perceptions, and choices are central to producing products that can benefit all of us.

The authors would like to thank members of the Open Technology Institute team for their unconditional support and subject matter expertise throughout the development of this report including Sharon Bradford Franklin, Chhaya Kapadia, Nat Meysenburg, Sarah Morris, Lauren Sarkesian, Ross Schulman, Spandana Singh, along with Austin Adams and Lisa Johnson for communications support. Additionally, we would like to thank the many external civil society and industry experts who provided feedback.