Case Study: Amazon

Amazon is an American technology company that offers services such as e-commerce, cloud computing, and streaming. The company was founded in 1994 by Jeff Bezos¹ and is considered the largest online marketplace in the world in terms of revenue and market capitalization, and the largest internet company in the world in terms of revenue.² The company currently ranks fourteenth for global internet engagement on Alexa rankings.³ In 2019, Amazon accounted for 37.7 percent of all U.S. e-commerce sales.⁴

Amazon deploys recommendation systems across a number of its services, including its e-commerce platform and its streaming platform. Given the large scale influence the introduction of Amazon’s recommendation system has had on online commerce, this analysis will focus on Amazon.com, the company’s e-commerce platform. Amazon generates revenue from its e-commerce platform through product sales, targeted advertising, subscriptions (e.g. Amazon Prime), and the fees sellers pay in order to be able to sell on the platform. The longer a user spends on the platform, the more products they will see and potentially buy and the more ads they will see. Given that the company’s recommendation system is designed to understand and predict user interest and behaviors, and make recommendations based on these insights, it is an integral tool for driving user purchases and increasing and maintaining user attention and engagement on the platform. Therefore, the recommendation system is an important contributor to revenue generation on the platform. In addition, some researchers and commentators have argued that Amazon uses its recommendation system to promote its own brands over others, thereby further increasing its profits, maintaining its market dominance, and raising antitrust concerns.⁵

Amazon’s e-commerce recommendation engine is powerful, and it engages with users at every stage of their journey on the website. It can therefore influence everything from what products a user sees to which items they eventually buy. Despite the extensive influence this system has over user behaviors and purchasing decisions, Amazon has provided little transparency around how these algorithmic systems are designed and how they operate. This is especially concerning given that numerous researchers and journalists have highlighted how the platform’s recommendation engine has suggested products that are misleading, false, or conspiracy-theory based. This lack of transparency therefore makes it challenging to understand why these recommendations are made, and how to prevent them going forward. In addition, the platform offers users a limited set of controls around whether and how they would like their platform experience to be shaped by such algorithmic decision-making processes.

Amazon first deployed a recommendation system across its e-commerce platform almost two decades ago.⁶ Before deploying this system, Amazon made product recommendations to users based on human curation and best-seller lists. However, according to Amazon, this approach was found to be inherently biased and did not sufficiently provide recommendations to users with niche interests.⁷ The company later developed and deployed an algorithmic system that matches a user’s purchased and rated items with similar items, and then combines these similar items into a list of recommended products for a user.⁸ This unique approach to making recommendations became known as “item-based collaborative filtering” or “item-to-item collaborative filtering.”⁹

According to a 2001 paper authored by three Amazon employees, the company opted to use the item-to-item collaborative-filtering model rather than a traditional collaborative-filtering model, or models such as content-based models. This is because its algorithm’s online computation grows at a rate that is not connected to the growth in the number of customers and items in the product catalog. As a result, this model is able to generate real-time recommendations, scale to large data sets, and produce recommendations that are more likely to be of interest to users. This model is also less computationally expensive than the other models previously outlined.¹⁰ When a model is computationally expensive it requires a considerable amount of resources to complete. These resources include the overall run time, processing power, and memory usage required to complete a function. In addition, researchers and journalists have found that the item-to-item collaborative-filtering model helped the company recommend niche items to shoppers in a compelling manner, thus increasing their potential to gain revenue on slow-moving inventory.¹¹ Further, given the vast size of Amazon’s product catalog, the use of this algorithmic approach also helped the company address the issues of which recommendations to present and in which order (known in data science as the “learning to rank” problem), and how to ensure that there are a diversity of products in each recommendation set.¹²

The introduction of the item-to-item collaborative-filtering model has served as a significant targeted marketing tool and has transformed the e-commerce space tremendously.¹³ It has also enabled the company to generate a personalized shopping experience for each user in a novel manner. Amazon asserts that providing a personalized experience will enhance users’ overall experience with the platform. However, in doing so, the company also strives to increase the time a user spends on the platform, as well as the company’s average order value (the average amount spent every time a user places an order on the platform), and overall revenue the company generates from each user.¹⁴ The company also sells advertisements, and therefore users who spend a longer time on the platform will see and potentially click on more ads, thus driving revenue through ad impressions (views) and clicks.

Amazon’s recommendation system relies on numerous explicit and implicit data points. Users are able to provide explicit feedback data to the platform by rating items from one to five stars, through both public and private ratings. The private ratings are not shared with other Amazon customers, nor do they impact the average customer review for an item. These private ratings are used to refine the recommendations that the user receives.¹⁵ Although Amazon publicly notes how it uses the private ratings, the company has not indicated whether and how its recommendation systems rely upon the public ratings and feedback that users provide on items they have purchased and sellers they have purchased from.¹⁶ Each action that a user takes on the platform provides implicit feedback data to the company, which it uses to refine its recommendations in real time.¹⁷ These data points include a user’s browsing and purchase histories, which items a user added to their virtual shopping cart, which items a user rated and liked, and what items similar users have browsed and purchased.¹⁸ For new users, there is less implicit and explicit data available to the company. As the user continues using the Amazon platform, however, the company is able to collect a vast amount of additional data, which can then be used to refine recommendations.¹⁹

Today, Amazon uses recommendation algorithms to offer different categories of recommendations to users at a range of junctures on its e-commerce platform. The company, however, provides little transparency around these recommendation system use cases. This limits the understanding and agency that users have over these tools. Several engineers, journalists, and researchers, however, have identified some of the categories of product recommendations that the company’s recommendation system generates. These categories are broken down below:

1. Recommended For You: When a user visits the Amazon.com webpage and²⁰ logs in, they will see a tab on the main toolbar that is tied to their account (e.g. Gabby’s Amazon.com). Once they click on this they are presented with a range of product recommendations across multiple categories.²¹ For example, this can include recommendations specific to Amazon Books, apparel, or electronics.
2. Frequently Bought Together: In order to increase average order value, the company makes recommendations on items that have been purchased together in the past. These product recommendations aim to convince customers to purchase an additional item (known as up-selling) and/or purchase a different product or service (known as cross-selling) by providing suggestions based on items a user has added to their shopping cart or below items that they are currently exploring on the website.²²
3. Similar Items: The company also makes recommendations for products similar to ones that a user has viewed recently. These recommendations are based on a user’s browser history and the recommended items typically vary in terms of shape, size, and brand.²³
4. General Browsing History: The company will make product recommendations based on a user’s browsing history in case they want to access and purchase something they have previously demonstrated an interest in.²⁴
5. Items Recently Viewed: The company also makes recommendations based on the items that a user recently viewed. Amazon has the same goal in making these recommendations as with the Similar Items recommendations, in that they want to suggest products based on a user’s recent browsing history.²⁵
6. Related To/Based On Items You Viewed: These categories of recommendations suggest products that are similar to items that a user recently viewed. For example, if a user searched for hangers on the platform, these recommendations would suggest hangers of different shapes, sizes, brands, etc. These recommendations are also based on a user’s recent browsing history.
7. Customers Who Bought This Item Also Bought: This category of recommendations suggests items that have been purchased collectively by users in the past. This category of recommendations is similar to the Frequently Bought Together section and it similarly aims to increase average order value through up-selling and cross-selling.²⁶
8. Recommended Items Other Customers Often Buy Again: This category of recommendations suggests items that similar users often purchase multiple times.
9. New Version of This Item: This category of recommendations is based on the assumption that users like to upgrade items, such as electronics, that they purchased. As a result, this category of recommendations informs users when a new edition of an item they purchased is available.²⁷
10. Recommended For You Based on a Previous Purchase/Inspired By Your Purchases/Inspired By Your Shopping Trends: These categories of recommendations make product suggestions to a user based on a recent purchase they made. After a user makes a purchase on the platform, they are directed to an order details page. On this page, the user will receive further recommendations for items that can be paired with the initial order. For example, if a user purchases an iPad on Amazon, they might receive recommendations for iPad covers on the subsequent order details page. These recommendations also appear on the homepage. This category of recommendations aims to encourage users to make a second purchase by offering a relevant cross-sell offer.²⁸
11. Best-Selling in Different Categories: This category of recommendations features top-selling items across the different categories of products on the platform. It is based on the notion that an item that has been widely purchased by other users is validated as worthwhile. In addition, these recommendations aim to help users identify popular products and make purchases from new categories of products that they have not made purchases from before. This produces a range of up-sell and cross-sell opportunities for the platform.²⁹
12. Popular in Brands You May Like: This category recommends tems that are popular and are sold by brands or sellers that a user may be interested in.
13. Off-site Email Recommendations: Amazon also sends users product recommendations via email. Users can opt out of receiving these marketing emails, and can also select certain categories of items (e.g. beauty, books, Amazon Echo) for which they would like to receive marketing emails. These emails vary in their content and focus, but contain similar categories of recommendations as those available on the platform.³⁰

In the early 2000s, Amazon’s product recommendation system relied on both automated and human decision-making. Initially, the recommendations put forth on the website relied more on automated decision-making, whereas the email recommendations users received were generated by humans. These recommendations were produced by Amazon employees who were tasked with using software tools to target users based on their purchasing and browsing history. These employees were also assigned a certain product and were responsible for identifying similar items to recommend along with the original item.³¹

In the early 2010s, journalists noted that the company relied on a range of metrics when constructing and deploying its email recommendations. These included email open rates, click rates, opt out rates, as well as revenue-prioritizing metrics. The company’s use of revenue-prioritizing metrics and related marketing and targeting techniques meant that if the recommendation system determined a user should receive a recommendation email for best-selling shoes and best-selling books, the company would only send them the email with the highest average revenue-per-mail. In this way, the company aimed to avoid spamming users and therefore maximize purchases. According to Sucharita Mulpuru, a Forrester analyst, the conversion rate and efficiency of Amazon’s recommendation emails were high, and considerably more effective than the on-site recommendations a user receives.³² Mulpuru shared that based on the performance of other e-commerce sites, one could estimate that Amazon’s on-site recommendation conversion is approximately 60 percent, suggesting a much more staggering rate of success for email recommendations.³³ The company also offers users the option to receive marketing newsletters by traditional mail.

The introduction of a robust recommendation system has radically transformed Amazon as a business. Between 2011 and 2012, the company integrated its recommendation system across all stages of the purchasing process, beginning with the product discovery stage, when a user first identifies an item of interest, and ending with the checkout stage. In the second fiscal quarter of 2012, the company reported a 29 percent increase in sales (approximately $12.83 billion). This was a significant increase from the $9.9 billion in sales that the company reported during the same quarter the previous year.³⁴ According to the 2013 data released by McKinsey & Company, recommendation systems drive 35 percent of purchases at Amazon.³⁵

As outlined, recommendation algorithms are used widely on the Amazon platform. They have the power to significantly influence what products a user sees, and can subsequently drive purchases. The company, however, has provided little transparency around the data it uses to train these recommendation systems and how they contribute to the company’s determinations on how effective its recommendation system is. Therefore, although the recommendation system has impacted the company’s success as a business, it is difficult to know how this system impacts user behaviors and whether its recommendations are equally as refined for all of its users, across demographics. This therefore also prevents researchers from identifying and understanding any sources of bias. In addition, the results of these automated decision-making processes are not always positive. For example, over the past few years, journalists and researchers have identified numerous instances of the platform’s recommendation engine-making suggestions for products that are misleading, false, or conspiracy-theory based.

For example, in March 2019 WIRED reported finding that Amazon recommended a number of anti-vaccine books in the best-selling categories of various health sections on Amazon Books, including the Epidemiology, Emergency Pediatrics, History of Medicine, and Chemistry categories. Many of these books were marked as #1 Best Sellers in their respective categories.³⁶ In addition, the outlet also found that in the Oncology category, one of the books marked as a Best Seller made recommendations on cancer treatment that are contrary to the consensus of medical experts, such as consuming juice instead of undergoing chemotherapy. Further, when WIRED journalists searched for “cancer,” a misinformation-filled book titled The Truth About Cancer was one of the first product recommendations that appeared. The outlet found that the book had 1,684 reviews and had a 96 percent five-star rating.

Also in March 2019, NBC News reported that the company’s recommendation engine was recommending QAnon: An invitation to the great awakening in its Hot New Releases section of Amazon Books. The book outlined a number of the common beliefs held by the QAnon conspiracy theory movement,³⁷ including that Democrats murder and eat children and that the U.S. government manufactured both AIDS and the movie Monsters Inc.³⁸ The book rapidly climbed to the top 75 books sold on Amazon that month. The company did not respond to questions about how its algorithmic recommendation system had been used to make these product suggestions, and they did not clarify whether the recommendation of the book in the Amazon Books section meant that the book had also been recommended through its other recommendation mechanisms on the platform.³⁹ This raised significant concerns regarding the lack of oversight over how product recommendations are made, and the subsequent consequences that could result from recommending a harmful and misleading product such as this book.⁴⁰ Further, cases like these have raised concerns around popularity-based recommendations on the platform, as researchers and academics have suggested that popularity is a weak metric for quality, and it homogenizes results.⁴¹

In 2015, Amazon introduced the “Amazon Choice” label to indicate highly-recommended items across the platform. According to the Amazon website, these recommendations are based on highly-rated, well-priced products that can be shipped immediately. However, there is little transparency around how these signals are weighted and how the final recommendations are made, and the company has not shared any information around the performance of its recommendation system and how and whether it is able to deliver personalized recommendations to users. ⁴²

The lack of transparency around how the company uses algorithmic decision-making is also concerning, as there have been numerous instances indicating that the company’s recommendation algorithms can be taken advantage of. The rise of the QAnon book in Amazon’s top books sold list, as described above, is an example of the ease with which the company’s algorithms can be gamed. Although the QAnon movement has a relatively small base, through coordinated purchasing behaviors, the group was able to generate a spike in book sales and therefore foster its rise in Amazon’s rankings. The group also coordinated subsequent reviews of the book, giving it five stars and prompting the platform’s algorithms to continue recommending it. As a result of this, the book, which peddles fringe ideas, was able to gain significant viewership. The book’s sales peak may not have lasted long, and the boost in viewership may not have resulted in a significant number of subsequent purchases, but it did succeed in making the book and its ideas seem mainstream.⁴³

Such coordinated inauthentic activities extend far beyond small conspiracy theory-based groups who are seeking to amplify their ideas. Because of the vast influence that Amazon has over users’ consumption habits, sellers have a strong incentive to appear on the first page of product search results and recommendations. In addition, according to WIRED, Amazon reviews appear to significantly influence the company’s ranking and recommendation algorithms⁴⁴ (although the company has not confirmed this⁴⁵). Further, the company has stated that it accounts for profitability when ranking products and making recommendations on its service. This means that items that could earn more revenue would be ranked higher, and therefore there is an incentive to engage in additional black hat operations in order to secure a place in these rankings.⁴⁶

As competition on the platform has risen significantly, it has resulted in the growth of a vast black market industry⁴⁷ where sellers can purchase “black hat” services that can manipulate the Amazon platform to help the sellers gain an advantage on their competitors. The services that they advertise include helping sellers appear at the top of a product search result page, promoting a seller’s products on the platform such as by removing negative reviews, and taking advantage of loopholes on the Amazon platform to raise a product’s overall sales ranking.

For example, for the book The Truth About Cancer discussed above, Reviewmeta.com, a site that seeks to help customers evaluate whether online reviews are legitimate, has suggested that over 1,000 of the existing reviews were suspicious based on the time, language, and reviewer behavior.⁴⁸

As rules-violating sellers have gained more of an advantage of the platform, traditionally rule-abiding sellers have come under increased competitive pressure and often opt to use these services as well. Typically, Amazon users rely on reviews as an indication of an item’s quality and reliability. However, the rise of this black hat industry has made it easy for sellers to purchase services that result in the production of persuasive positive or negative reviews.⁴⁹ Amazon has stated that it deploys a team of investigators, automated technology, and machine learning technology to prevent and detect inauthentic reviews at scale, and to enforce its policies against actors who violate their policies.⁵⁰ However, many outlets and researchers have found that although these black hat activities are a violation of Amazon’s terms of service, the platform’s enforcement of these rules has been weak,⁵¹ perhaps because of the sheer size of the company’s product catalog.⁵² In addition, there is little transparency around the scope and scale of these enforcement efforts. The platform has also made no indication that it will intervene to change these recommendations, such as by flagging items as misleading, or downranking their presentation in its overall recommendation and rankings. In addition, the company has in the past removed items such as two books that contained misleading claims on how to fight autism for violating its content guidelines,⁵³ however it is unclear how widely and consistently these rules are enforced.The company has removed some misleading content from its streaming service, after facing pressure from the media and legislators in the United States,⁵⁴ but, as highlighted, fewer consequential actions have been taken on its e-commerce platform.

Amazon’s Frequently Bought Together recommendation category has also raised some concerns. A 2017 investigation by a team at Channel 4 News in the United Kingdom found that when a user searched for a common chemical that was used in certain food products, the Amazon recommender system would suggest other items the user could buy, which collectively could be used to make black powder, a chemical explosive. The system also recommended other items, such as ball bearings, which could be used as shrapnel in homemade explosives.⁵⁵ In response, Amazon said it was reviewing its website to ensure that all products were being “presented in an appropriate manner.”⁵⁶ Although merely purchasing these items is not illegal, this recommendation raised concerns around whether users could easily access and purchase items to cause large scale harm and destruction with relative ease and whether Amazon’s algorithmic systems were enabling or even encouraging these actions. In addition, in the United Kingdom, there have been some successful prosecutions against individuals who have bought items that can be combined to make a bomb.⁵⁷

Finally, Amazon’s use of recommendation algorithms has also raised significant concerns around privacy of user data. In 2019, the company began testing a program that relied on machine learning tools and its broader recommendation engine to identify brands and products that a user may be interested in purchasing, based on data the company had collected on the user. Based on these inferences, Amazon sent users free samples of products to test, and this enabled the company to get new brands or brands of interest directly in front of users. The program raised a number of concerns among users, however, who were seeing first hand and offline manifestations of the company’s collection of their personal data. Later in 2019, the company announced it will discontinue the program in 2020.⁵⁸

As discussed, Amazon has faced significant controversy over its recommendation system, issues that are compounded by the fact that the company does not provide meaningful transparency around how its recommendation system is structured and how it makes decisions. The company does offer its users some limited controls over how this system impacts their individual experiences on the platform, however.

For example, when a user is logged into their Amazon account, they have access to a page titled Recommendations. The page explains that when making recommendations, the company examines the items a user has purchased, the items a user has proactively indicated they own, and items they have rated. The page also explains that the company makes recommendations by comparing a user’s activity with the activity of other users. Further, the page states that a user’s recommendations will constantly change, based on whether they purchase or rate a new item, and whether the interests of similar users change. The company also asks that users indicate what items they are interested in by adding products to their Wish List or Shopping Cart in order to improve the personalization of their recommendations.⁵⁹

In addition to this explanatory page, users can access a page titled Improve Your Recommendations. This page gives users a list of all of the items that they have purchased, the videos they have watched on Amazon Prime Video, the items they have marked as “I own it,” the items they have rated, the items they have marked as “not interested,” and the items that they have marked as gifts. On this page, users have the option to select “I prefer not to use these for recommendations” for any item, rate each item from one to five stars, and mark items as a gift.⁶⁰ Further, if a user is unsure why they have been recommended a particular product, they are able to see an explanation of why the item was recommended to them, such as because of a previous item rating, wish list addition, and so on.