How does the Handbook Score Products?

One of the key choices that we had to make was developing a rating system for use when reviewing the products. Although a numerical or other type of complex rating process can provide valuable nuance, we decided that it would be too subjective for testers. Ultimately we concluded that products could receive “Pass,” “Partial Pass,” “Fail,” and “Not Applicable” grades based on their performance on each indicator. Not Applicable ratings are reserved for cases where the product does not include the functionality or feature being evaluated in the test. For example, some tests may specify examining a browser component, and many IoT products do not have one; for that product all browser component tests would be marked Not Applicable. Partial Pass ratings are used when an indicator contains multiple requirements and a product does not fulfill all of them. For example, a test on privacy policies could include a list of required best practices, some of which are met by the product’s policy, which would result in a Partial Pass grade.