Authentication

See the test in action:

• Smart lock
• Smart pressure cooker
• Smart baby monitor

Notes:

• This test features two separate criteria by which to assess the test’s success. Each criteria features a unique set of indicators and a related methodology.
• Thus far our development of methodologies to assess these criteria is based on analysis of Android applications. This decision is mainly based on prior experience with Android development and workflows, as well as a wider range of available research, documentation, and tooling for the Android ecosystem.

Criteria: A product has an authentication system that corresponds to the sensitivity of the user data it manages.

1) If a product supports user accounts, it has an authentication system for accessing those accounts.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Look for all options available to you when the app starts.
• Note if you are allowed to perform any setup, registration, configuration, preference setting, without first being required to create (or login) with an account.
• Look to see whether any information about user accounts or device settings are revealed before being required to authenticate.
• If the product does not support user accounts, mark NA.
• If the product supports user accounts, and requires authentication to perform any actions before it reveals information about the device, mark PASS.
• If the product supports user accounts but lacks a user authentication system, mark FAIL.
• If the product allows you to perform any actions before authentication, mark FAIL.

2) If a product is packaged with an account with default credentials, those credentials are unique to the instance of the product.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Look at the instructions that came with the product, and look for a section about logging in for the first time. Note if any credentials are supplied with the documentation. If not printed in the documentation, some products’ default credentials can be found on the box.
• As you examine the credentials, try to assess whether they are unique. For example, if the product’s credentials use common words or phrases such as, "default," "admin," or "12345678" you can conclude they are not unique.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product supports accounts and passwords, but does not use default credentials at all, and the user must select and enter their own unique set of characters, mark PASS.
• If the product uses default credentials, but they are unique to the device, mark PASS.
• If the product uses common default credentials, mark FAIL.

3) If a product has an authentication system, the user must authenticate each time they want to use the product.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Close out of and stop the app and then restart it several times, and note if you are required to re-enter your password and/or otherwise reauthenticate.
• If the product does not support user accounts and password-based authentication, mark NA.
• If you are required to reauthenticate upon restarting, mark PASS.
• If you are not required to reauthenticate upon restarting, mark FAIL.

4) If a product has an authentication system, it requires at least two pieces of information to authenticate users.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create a new user and note if user authentication requires at least two pieces of information (e.g. “username” + “password”). Note all required pieces of information.
• If the product does not support user accounts or any authentication system, mark NA.
• If the authentication system requires at least two pieces of information, mark PASS.
• If the authentication system does not require at least two pieces of information, mark FAIL.

5) For products that handle sufficiently sensitive data, users can choose to use multi-factor authentication.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Determine whether the product handles sensitive data. Examples include health information, location, live or recorded audio or video as well as personal messages.
• If the product does not handle sensitive data, mark NA.
• If the product supports accounts, create an account and determine whether there are settings to enable multi-factor authentication, so that in addition to a password, a user can require authentication through a text message, token, or other method.
• If the product handles sensitive data, and multi-factor authentication is available, mark PASS.
• If the product handles sensitive data, but there is no authentication mechanism at all, mark FAIL.
• If the product handles sensitive data, and multi-factor authentication is not available, mark FAIL.

6) For products that handle sufficiently sensitive data, users can choose to use multi-factor authentication whenever the product is activated, or when a device is unrecognized.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create an account and if there are settings to enable it, set up multi-factor authentication.
• Examine the settings option for multi-factor authentication, and see whether there is an option to require multi-factor authentication every time the product is re-started or when a device is unrecognized. If yes, select that setting.
• Completely close the app and restart it multiple times on one device, and note if multi-factor authentication is required for authentication each time.
• Completely close and restart the app multiple times on multiple devices, and note if multi-factor authentication is required for authentication on each device each time.
• Determine whether the product handles sensitive data. Examples include health information, location, live or recorded audio or video, as well as personal messages.
• If the product does not handle sensitive data, mark NA.
• If the product handles sensitive data and requires you to use multi-factor authentication on each login from all devices, mark PASS.
• If the product requires you to use multi-factor authentication on new devices, but does not require multi-factor authentication for subsequent authentication requests on the same device, mark PARTIAL PASS.
• If the product does not offer multi-factor authentication, or does not permit users to select a setting under which it is required for each login and when authenticating from new devices, mark FAIL.

7) If the product uses a password/passphrase for authentication, it requires that passwords are at least 8 characters long.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create or edit an existing account.
• Try passwords such as "a" and "a1b2c3."
• Note minimum password lengths required by the app.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product requires passwords to be at least eight characters long, mark PASS.
• If the product does not require passwords to be at least eight characters long, mark FAIL.

8) If the product uses a password/passphrase for authentication, the password/passphrase may be at least 20 characters long.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create or edit an existing account.
• Try passphrases of 20 characters such as "i love long passphrases."
• Note whether any maximum password length is enforced.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product allows passwords of lengths of at least 20 characters or more, mark PASS.
• If the product limits password lengths to below 20 characters, mark FAIL.

9) If the product uses a password/passphrase for authentication, it requires that passwords are reasonably complex.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create or edit an existing account, attempting to update the password.
• Try passwords such as "aaaaaaaa" and "12345678."
• Note if the app requires complexity in the password (e.g. special characters, capital letters, mixing numbers and letters, etc.).
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product requires some password complexity (e.g. requiring mixing numbers and letters, but not requiring special characters), mark PARTIAL PASS.
• If the product requires several forms of password complexity, mark PASS.
• If the product enforces no forms of password complexity, mark FAIL.

10) If the product uses a password/passphrase for authentication, it allows all reasonable characters as input.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create or edit an existing account.
• Try passwords such as ")a!aaaaa$a%" and "p 4 5 5 w 0 R d !" Try using a variety of the standard special characters that appear on keyboards.
• Note if the app limits the use of any special characters.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product allows all special characters on a standard keyboard, mark PASS.
• If the product allows some special characters, but places limits, mark PARTIAL PASS.
• If the product does not allow any special characters from the standard set on a keyboard in the password, mark FAIL.

11) If the product uses a password/passphrase for authentication, it is compatible with popular password managers.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create and then sign out of an account in the app.
• Add the credential for the app to a password manager.
• Try to log in using the functionality of the password manager.
• Note what, if any, issues you have.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product allows the use of a password manager, mark PASS.
• If the product does not allow the use of a password manager, mark FAIL.

Criteria: A product that has an authentication system resists attempts to break it.

Methodology for Assessing Each Indicator

1) The product allows users to be notified via an out-of-band medium when account security settings are changed.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create an account and set up authentication.
• Look at the documentation that came with the product, and attempt to locate any instructions for enabling notifications out-of-band.
• If this is not documented, look through the app settings for security or notifications.
• Note whether or not the app allows email, SMS, or push notifications to be used as notification when changes occur or account credentials need to be reset.
  • If the app does allow this option, select it.
• Go into settings, and change your password or another security setting. Then note if the app uses an out-of-band medium—such as by sending you a text or email—to send a notification of the change.
• If the product does not support user accounts with security settings, mark NA.
• If the product automatically sends out-of-band notifications, without user input, mark PASS.
• If the product allows user-selectable out-of-band notifications, mark PASS.
• If the product does not allow for or send out-of-band notifications, mark FAIL.

2) To change a password/passphrase/pin, a user must enter the previous password/passphrase/pin, or have access to a secondary system that is used to reset it.

• Note: This indicator contains two opportunities for pass/fail.
• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create an account and set up a password.
• While logged in to the app, attempt to change the password.
• Note whether or not you are required to enter the old password, or use an email account, phone number, or other second factors in order to reset the password.
• Next, log out and close out of the app. Reopen the app and look for a “forgot my password” link or button that enables you to reset the password. Determine whether the password reset option requires use of a secondary system such as an email account or phone number that the user has previously provided to the app as associated with that user.
• If the product does not support user accounts and password-based authentication, mark NA.
• If a product supports user accounts and password-based authentication, first assess what happens when the user is logged in:
  • If the product requires either entering an old password, or confirmation via a secondary system to change a password while logged in, mark PASS.
  • If the product does not require either entering an old password, or confirmation via a secondary system to change a password while logged in, mark FAIL.
• Next assess what happens when the user is not logged in:
  • If the product requires a secondary system such as an email account or phone number to reset passwords while logged out, mark PASS.
  • If the product does not require a secondary system such as an email account or phone number to reset passwords while logged out, mark FAIL.

3) The product notifies users when account security settings have changed.

• Obtain a copy of the manufacturer's application for testing, and install it on a testing device. For these tests, it is sufficient to simply install the app directly from the relevant official app Store.
• Create an account including setting a password and any other security settings like multi-factor authentication.
• Change the password or other security setting.
• Note whether you are notified of the change, and if so how (e.g. by email or text).
• If the product does not support user accounts and password-based authentication, mark NA.
• If the product notifies users when account security settings have changed, mark PASS.
• If the product does not notify users when account security settings have changed, mark FAIL.

4) If the product has an authentication system, it also has a system to prevent brute-force/dictionary attacks.

• Create an account including establishing a password.
• Attempt to log in to the app at least ten times using different incorrect passwords.
• Note any limitations the app places on password attempts, including any messages that appear indicating how many more attempts you may make.
• If the product does not support user accounts and password-based authentication, mark NA.
• If the app limits incorrect login attempts, mark PASS.
• If the app does not limit incorrect login attempts, mark FAIL.