M-Lab Tool, Shaperprobe, Reveals Traffic Shaping Among Major ISPs

Developed by Georgia Institute of Technology researchers Partha Kanuparthy and Constantine Dovrolis, a new measurement program called ShaperProbe is the first of its kind to detect traffic shaping over end user connections to the Internet. Through a large sample test deployment on the Measurement Lab (“M-Lab”) platform, the researchers found that several major ISPs are actively shaping end user traffic over their Internet connections.  The research conducted by Kanuparthy and Dovrolis achieved three objectives: 1) the development of an active end-to-end mechanism for detecting traffic shaping, 2) the deployment of ShaperProbe technology over the M-Lab platform and resulting analysis, and 3) the subsequent modification of the ShaperProbe detection algorithm for passive deployment on the traffic of any TCP-based application.  Viewed in the context of recent debates surrounding truth-in-labeling requirements and network capacity, the findings raise concerns about the reliability of ISP advertised speeds and statements about network congestion.

What is Traffic Shaping?

Traffic shaping is one technique among several that an Internet Service Provider (“ISP”) may use to control the speed of data traffic on its network.  The paper illustrates the concept of traffic shaping as a bucket on a network link with a given capacity measured in bits per second (“bps”) filled with “tokens”, each representing an available bit.   When the bucket is empty of tokens, the link will transmit packets at a shaped rate to accommodate the reduction in capacity, holding packets in queue until more tokens are available.  As the paper explains, “[t]raffic shaping is typically implemented using token buckets, allowing a maximum burst of traffic to be serviced at the peak capacity of the link, while any remaining traffic is serviced at a lower shaping rate.”  Thus, in the context of Kanuparthy and Dovrolis’ paper, three components are considered: the size of the bucket, the speed at which tokens are regenerated, and the actual capacity of the network link.

Why do ISPs Traffic Shape?

In explaining why an ISP might utilize traffic shaping, the paper notes three reasons: 1) to accommodate the exceeding by a user of a service rate or connection speed for a limited burst size (exemplified in Comcast's “PowerBurst” technology), 2) limit the service rate of a customer or to limit the service rate consumed by a specific application, or, 3) enforce an upper bound limit of service rate. 

The ShaperProbe research found traffic shaping utilized more frequently by cable providers than DSL providers. While the research itself does not definitively address the question of why this distinction occurs, the paper notes that this may be because DSL can be configured to impose strict speed or capacity limits on subscribers more directly given the differences in network architecture. On cable networks bandwidth is shared by all customers on the same neighborhood loop. Digital subscriber line service (“DSL”) operates in a slightly different manner.  While consumers receiving DSL service do not share bandwidth on the copper line from their house to a provider's local point of presence ("POP"), which is often a Digital Subscriber Line Access Multiplexer (“DSLAM”), bandwidth can be shared by upwards of a 1000 subscribers from that connection to more high capacity connections deeper in the network.  

Implications for Consumers and Policymakers

The ShaperProbe research primarily underscores the need for better transparency among ISPs. Although the research does not indicate that traffic shaping practices of many of the providers are nefarious, they have the effect of substantially obscuring what actual data rates consumers can expect on their broadband service.  The research found for example that in at least one instance, a company was actively shaping in 80% of the tests conducted without any explicit mention or explanation of traffic shaping to consumers in its Service-Level Agreement (“SLA”). 

OTI has called for disclosure of all traffic management techniques in its truth-in-labeling proposal.  The FCC, in its rules for protecting an Open Internet, included obligations on providers to disclose their network management techniques and policies to consumers.  But those rules have yet to be implemented.  Moreover the FCC to has yet to move forward on broader rules imposing mandatory truth-in-labeling requirements on ISPs to ensure that consumer's can make decisions about Internet services with adequate information about actual speeds and network management policies so they are not solely reliant on the often-overhyped “up-to” advertised speeds.

In addition, interestingly the paper finds no significant traffic shaping on the part of AT&T. And to the extent research found traffic shaping on AT&T connections, the shaping did not fluctuate based upon time of day. Though it is difficult to take away any sweeping conclusions from this observation, it is interesting to look at this in the context of the company's claim that its network is so congested to the point it requires bandwidth caps to address “a dramatic increase in the amount of data that is sent and received over its wireline broadband networks.” Thus you would expect to see traffic shaping increase during peak usage periods, but that is not what the data suggests. As Ars Technica notes, “[i]f the concern was congestion, one might expect more of a focus on an actual congestion throttling system,” or traffic shaping, as “[m]onthly data caps don't actually discourage people from using the Internet during the busiest times...”

Indeed, the broader implication for debates surrounding congestion and network management practices seems to be that there should be skepticism toward the industry's tendency to place the burden of congestion, which is in many cases implicitly tied to that provider's business decisions, on consumers rather than on the ISPs themselves to be more upfront about actual service speeds or conversely invest more heavily in expanding capacity through upgrading their infrastructure.  Capacity problems cannot be solved by network management alone; investment in scalable infrastructure is an inescapable component of meeting consumer demand.

As important, the research also discredits the popular notion for speed testing that the throughput of a connection for a few seconds is sufficient to quantify the notion of “speed” of a broadband connection.  Because link speeds change when ISPs are utilizing traffic shaping, the user must know the sustained rate (i.e., the “shaping rate”), the base rate (i.e., the “link capacity”), and also the burst duration before the base rate drops.  A ten-second or shorter speed test will estimate either the base rate or a combination of base rate and shaping rate.  This finding implies that, going forward, longer testing times are required to ensure that all three components can be gathered to adequately account for traffic shaping.

The research presented by Kanuparthy and Dovrolis is an important step in better understanding ISP's network management practices.  Much more research of this kind is needed to better understand the technological realities of network traffic and management as they compare to industry claims.  At the very least, ShaperProbe tells us that even if we have reason to trust claims advanced by ISPs, the public and policymakers need independent means to verify them.   Measurement Lab was created to help fill that gap, providing a platform for empower users and the public with greater transparency and serving as a resource for researchers to develop additional measurement tools. 

You can read more about the research of Kanuparthy and Dovrolis here and here.

For more information on Measurement Lab, visit www.measurementlab.net