Getting Up to Speed

Today, an internet service plan is among the most important purchases that an American household can make, so when your internet isn’t working correctly, it’s frustrating. And even if you run a speed test on your home broadband connection, it’s often hard to figure out what’s wrong. Perhaps you’re a consumer wondering how Measurement Lab (M-Lab) differs from speedtest.net; perhaps you’re an internet service provider that wants to implement good practices and help your customers understand their internet connectivity. Or maybe you’re a researcher, and just want more information about how broadband measurement works.

In any case, you’ve come to the right place.

To help resolve some of this confusion and help you understand what exactly goes into good broadband measurement, OTI has released Getting Up to Speed, a policy paper that examines different ways of measuring broadband internet performance. Our research comes at a time when the Federal Communications Commission (FCC) has taken important steps to improve the availability of broadband performance information to consumers through the Open Internet Transparency Rules and the new broadband labeling program. Across the Atlantic, European regulators are also considering transparency guidelines in their implementation of the EU’s new network neutrality rules.

In this paper, we review efforts to empirically measure and quantify internet performance, and discuss best practices for using internet performance tests and performance data for consumer education on broadband issues. We also consider these best practices in a brief case study of the FCC’s Measuring Broadband America program.  Broadband performance is affected by complicated factors that are not always apparent to consumers, and there are a number of different approaches for measuring broadband performance. Not all tests work in the same way, nor do they reflect the same aspects of network performance. As such, internet performance data drawn from different methodologies are like apples and pears–complimentary perhaps, but not interchangeable.

We find that useful performance data requires a consistent, reproducible methodology that provides full transparency to the data’s underlying assumptions, strengths, and limitations. That means that even if methodologies between two given  tests differ, a researcher or consumer could easily understand what those differences mean.  For example, you’d want to know if the speed test application you’ve been using is potentially throwing out some data points, painting a rosier picture than the speeds you are actually getting.  Sometimes the problem with your connection is the result of traffic bottlenecks at the point where the networks that carry content throughout the internet connect to your home internet provider.  To help you distinguish between different types of congestion, internet performance measurement should therefore be able to replicate the end user experience with regard to those interconnection points.

We hope this paper functions to encourage regulators and internet providers to better integrate performance measurement considerations into their consumer-facing communications, and to inform consumers about what constitutes rigorous and transparent broadband measurement.

Our methodology best practices, at a glance:

1. Data should be collected using a consistent and reproducible methodology.

2. Measurement methodology should accurately reflect the experience of the end user, and uphold standards of transparency and openness by providing precise specifications for measurement and analysis.

1. Measurement should capture performance over interconnection points and at peak hours,

2. Methodology should allow for third-party oversight and verification,

3. All methodological and analytic choices should be available in full transparency,

4. Open software measurement clients and back end (the measurement application) should be open source, and

5. Methodology in analysis and processing of the data should be open.

3. Standardized disclosure formats should include baseline best practices for broadband measurement so that customers can confidently gauge their own connectivity against what is expected and what others receive.

Read the full paper here.