How did we Pick the Products? (And Why aren’t We Naming Them?)

In order to design a handbook that would be valuable to testers, we evaluated a selection of sample products using the Digital Standard. This experience allowed us to write out and then refine the steps required to perform each test, and develop further procedures as necessary. Crucially, we also developed specific criteria for a product to pass or fail the test, so that various testers were using the same method of measuring success.

Ultimately, we tested three products, varied enough to cover a wide range of features offered in common smart devices and selected based on the unique privacy and security threats that the potential failure or compromise of a product could pose. Since not all indicators in the standard fully apply to every product, we also selected a range of products to make sure we covered every indicator at least once. For example the Known Exploit Resistance test suggests examining the browser component of connected devices, and while this is important for tablets or even some smart fridges which ship with web browsers, a large number of IoT devices simply don't include a browser feature.

We opted to test products offered by larger and well-known brands, rather than by startups or small manufacturers. We did this in the hope that larger companies may have put more resources into legal documents like terms of service and privacy policies, and that they also may have longer track records with things like patching known vulnerabilities. Since we were not conducting a comparative analysis of similar products, having as much information as possible on a single product was an important part of being able to decide the pass and fail conditions of the testing processes.

In the interest of keeping focus on the testing process itself, rather than on the relative quality of any particular product that we were testing, we opted to not name the manufacturer, make, or model number of any of the products we tested. Unlike groups such as Consumer Reports that use the standard for comparative analysis to make recommendations within a product category, we used each product in our testing handbook as a stand-in for its larger product class, and a demonstration of how one might test that variety of product. While our results reflect our actual findings, they are only presented as examples of how one generic product's test results may look if tested with our handbook.