Why was this Testing Handbook Necessary, and Who is it For?

While security researchers do end up testing a variety of products, their tests are often created to test for specific vulnerabilities found in one product or product line, rather than a comparative evaluation of similar products. The Digital Standard creates a framework for comparative analysis of IoT products by doing what its name suggests—standardizing these types of tests under a single rubric. The goal is that consumer groups, manufacturers, students, and interested hobbyists alike will be able to use the standard to conduct their own testing of products or product types.

However, as it exists, the standard does not provide interested testers all of the information they need to implement the testing protocol. Although many of the tests include procedural overviews which offer helpful guidance as to how a tester might evaluate a specific indicator, they are often broad and incomplete; in many cases leaving it unclear as to what the steps for evaluation may be.

In earlier Open Technology Institute (OTI) projects about the Digital Standard we spoke to many stakeholders in civil society and the private sector who were interested in using the standard for product development or evaluation. The most common response we received was that there was not enough procedural guidance for non-experts to run the tests. Implementing the standard would require testers to possess expertise in a wide range of areas of technology privacy and security policy, and to expand upon the standard’s existing procedural overviews. Stakeholders wanted some sort of handbook or guide that would empower them to use the standard as imagined by the developers.

OTI started the testing handbook project to provide such a guide. In it we build upon the existing procedural overviews in the Digital Standard by providing detailed step-by-step directions on how to perform the tests. In many cases our work involved expanding the procedures listed in the overview, as well as fleshing out the discussion of more sophisticated best practices or technical approaches that could be used as part of testing. Following some of the testing processes requires familiarity with legal or technical language. Others require testers to choose between various approaches to running the tests or evaluating the results. Nonetheless, our hope is that the step-by-step instructions in the testing handbook will make it easier for other interested testers, including consumer groups, hobbyists, and the product manufacturers themselves, to begin conducting their own product testing.