Transparency Reporting

Criteria: The company is transparent about its practices for sharing user data with the government and other third parties.

Note: All of the indicators in this test are predicated on the idea that the relevant company publishes a transparency report of some kind. If the company does not publish a transparency report, it will fail all of these indicators. Knowing this ahead of time may save testers the time and effort of reading through each indicator.

See the test in action:

• Smart lock
• Smart pressure cooker
• Smart baby monitor

Methodology for Assessing Each Indicator

1) The company lists the number of requests it receives by country.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• In the transparency report, see if the tabulations of government requests are broken out by requesting country or grouped together.
• If the numbers of requests are not broken out by country, see if the transparency report states elsewhere that the company has received no requests or only received requests from one government.
• If the company received no requests, only received requests from one named country, or breaks out requests by requesting government, mark PASS.
• If the company does not break out requests by requesting government, or make clear that all requests were from one particular government, mark FAIL.

2) The company lists the number of requests it receives for stored user information and for real-time communications access.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• In the transparency report, look for how the company distinguishes the requests it receives for user information.
• If the transparency report counts requests for stored user information (which may also be referred to as past data) separately from user information collected in real time, mark PASS.
• If the transparency report does not distinguish between stored and real-time user information, mark FAIL.

3) The company lists the number of accounts affected.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• In the transparency report, look for a column of numbers detailing how many accounts were implicated in requests for user information, as distinguished from the sheer number of requests received.
• If the transparency report lists the number of accounts involved in requests for user data separately from the number of requests received, mark PASS.
• If the transparency report does not list the number of accounts affected by requests for user data, mark FAIL.

4) The company lists whether a demand sought communications content or non-content or both.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• Look in the transparency report for a distinction between requests for content and requests for non-content data.
  • Non-content information may be referred to as “metadata” and is information about a communication such as time and date, length, or the parties to the communication.
  • In some cases, the distinction between content and non-content may not be relevant to a given product. For example, some products and services will not generate communications content information. In that case, the company should specify that any requests received for user information were for, e.g., non-content information.
• If the transparency report lists the number of requests for both content and non-content types of data, or if it lists only numbers for one type along with a note that the company does not collect the other type, mark PASS.
• If the transparency report does not distinguish between requests for non-content and content, mark FAIL.

5) The company identifies the specific legal authority or type of legal process through which law enforcement and national security demands are made.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• Look for whether the transparency report separates numbers of requests by what legal authority the request was made under.
  • Commonly this will take the form of the header of a column or row (depending on how the table is formatted) listing a type of authority such as “subpoena” or “National Security Letter.”
  • Note that this information may be in a different table than the primary transparency report table, which may only list high-level aggregations.
• If the company groups requests by the specific legal authority under which they were made, mark PASS.
• If the company does not group requests by specific legal authority, mark FAIL.

6) The company includes requests that come from court orders.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• Read the transparency report looking for indications that the company has included requests for user information that were made under the authority of an established court.
  • Some terms to look for include “subpoena,” “warrant,” and “discovery.”
• If the transparency report includes requests that were made pursuant to a court order, mark PASS.
• If the transparency report does not include such requests, mark FAIL.

7) The company lists the number of requests it receives from private parties.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• Look for indications in the transparency report of requests from parties that are NOT courts or other government entities.
• If the transparency report includes statistics regarding requests made by private parties, mark PASS.
• If the transparency report does not include requests made by private parties, mark FAIL.

8) The company lists the number of requests it complied with, broken down by category of demand.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• If there is a transparency report, see if it includes statistical information regarding requests for user data.
• Look in the transparency report for the number of requests that it received in each category with which it complied (as opposed to those it refused to comply with for any reason).
• If the company lists the number of requests it complied with in addition to listing simply those it received, mark PASS.
• If the company does not list how many requests it complied with in addition to how many it received, mark FAIL.

9) The company lists what types of government requests it is prohibited by law from disclosing.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• Look in the transparency report for a description of the types of government requests the company is prohibited by law from disclosing.
  • Some laws prohibit the recipients of demands for customer data from even revealing the existence of the demand itself (sometimes called a ”nondisclosure order” or “gag order”). This indicator is testing whether the company informs the public about the fact of such a gag order.
  • This information will likely not be in table format along with the rest of the numbers in a transparency report, so look for it in the written language surrounding the report.
• If the transparency report includes information on what types of government requests it is prohibited from listing, mark PASS.
• If the transparency report does not include information about requests it is prohibited from including in its numbers, mark FAIL.

10) The company reports this data at least once per year.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• Look in the transparency report for indications of how often the company releases a report.
  • Ideally, look for links to older reports and determine if they have been released annually or more frequently.
  • Also look for a commitment from the company to release transparency reports on a given timetable
• If the transparency report has been released in the past on at least an annual basis, or if the company has only released one transparency report, but commits to doing so annually, mark PASS.
• If the company has released a transparency report on a regular time table in the past, but has not always done so precisely on an annual basis, mark PARTIAL PASS.
• If the company has released transparency reports in the past but not on an annual basis, mark FAIL.

11) The data reported by the company can be exported as a structured data file.

• Locate the company’s transparency report, if it produces one.
  • The transparency report will likely be located in the same section of the website as the other legal documents such as the privacy policy and terms of service, however some companies place them in their own section.
  • A web search can also help find the transparency report if it is not in an obvious location on the company’s website.
• Look in the transparency report page for a means to download the data that makes up the transparency report as structured data.
  • Structured data is data that is formatted to be read and analyzed easily by a computer.
  • Most commonly, these files will be in a comma-separated values (csv) file, but they may also be in “json” format or even as Microsoft Excel files.
• If the transparency report is available to download in a structured data format, mark PASS.
• If the transparency report is not available for download in a structured data format, mark FAIL.