March 15, 2010
What's Wrong With The FCC's Consumer Broadband Test?
The FCC recently published some tools to let consumers measure some internet characteristics.
The context is the FCC's "National Broadband Plan". I guess the FCC wants to gather data about the kind of internet users receive today so that the National Broadband Plan, whatever it may turn out to be, actually improves on the status quo.
The motivation is nice but the FCC's methodology is technically weak.
There are several goals to which the National Broadband Plan ought to aspire:
- That consumers have a subjective sense that their use of the internet is fast and without unacceptable delays. I picked a subjective standard here for reasons to be discussed later in this note.
- That reliability of consumer access is high and that the time for providers to detect, diagnose, and repair problems is low (and not expensive to providers.) It seems that these matters of reliability are routinely ignored, yet they are of paramount concern, particularly as the internet becomes more and more a part our health and safety systems; it will be a sorry day if someone picks up their internet based VoIP phone to call 911 and the link (or some necessary ancillary service, such as DNS) is down for an extended repair.
- That consumers' have a real foundation to believe that their use of the net is private and not being used either to generate marketing data about them.
This note will address only the first of these goals.
The first thing that is wrong is that the FCC's tools are not well focused with regard to exactly what parts of the internet they are measuring. And second, the measurements that are taken are too vague to be of more than anecdotal value.
I've drawn up a simple diagram to illustrate.
This is a simplified diagram, it is intended to focus on that part of the net of concern to the National Broadband Plan. In particular it looks at the part of the net that represents the "internet" product sold by today's Internet Service Providers (ISPs). The arrows in this drawing are interfaces where these clouds join, they are not communications lines.
This diagram shows things as connected clouds because that more accurately represents the things that make up the way that user's connect to the internet. The basic parts of the diagram are these:
- User Network: Many users today, and probably nearly all users in the future, will have networks, often wireless, within their homes. The quality and traffic of those networks will have a substantial effect on consumer's perceptions of net quality (and ISPs will bear increasing non-reimbursed costs when their customers have troubles in his part of the net.) However, except with regard to the maintenance issue, the user's home network cloud ought to be considered neither as part of either the National Broadband Plan or of the FCC's Consumer Broadband Test.
- User Access Link and
User's ISP Cloud: I have shown the provider ISP's
path as two parts. First is the part that runs from the
router of "modem" at the consumers home or office to the provider's
first IP router. The second part is the provider's internal
"backhaul", i.e. the IP network inside the provider. It is
important to consider these two parts separately.
- User Access Link: This is the part of that today's ISPs advertise to consumers; this is the part about which the claims of umpteen megabits/second download are made. In general the User Access Link is the IP "hop" between the user's home modem or router and the first IP router within the ISP. Often this "link" is composed of several communications technologies. For example what appears to the consumer to be an Asymmetrical DSL link (ADSL) might be composed in full or in part of ATM or other non-IP switching technologies that exhibit many of the congestive and impairment behaviors found in IP networks. There may be MPLS paths that simply do not show up in "traceroute". Moreover, the User Access Link may have an IP Maximum Transmission Unit size that is less than the 1500 bytes that is presumed by a considerable amount of end-user network applications and protocol stacks; that difference can have a substantial negative impact on some forms of network traffic (video) and almost none on others (VoIP). The User Access Link should not be considered as a private path that is not shared with other users' traffic.
- User's ISP Cloud: This is that portion of the ISP that carries traffic to and from customers User Access Links. Some resources that are critical to user perception of network speed may be located here, most particularly domain name system (DNS) resolvers, web caches, email servers, and the like. For small ISPs the "ISP Cloud" might be as simple as a small Ethernet at the provider's facility; for larger ISPs the "ISP Cloud" might be an national or international network of substantial size and power.
- Internet: This is the vast landscape of the internet except for those content providers with which the ISP entered into special traffic exchange arrangements.
- Private Peering to large
content providers: This is often where the largest of the
large network traffic sources and sinks are to be found. This
is the land of Google/YouTube and of content distribution
networks. Content to/from users might be able to flow via the
internet to those places but in order to provide faster access and
to give the large content providers better control over the quality
of their products both ISPs and large providers often prefer to
create these kinds of special peering relationships. This is
a game for big players; small ISPs and smaller content providers
are often not able to play at these tables.
(Please note that I am using the word "peering" in a way that may be different from its use in settlement-free peering between ISPs.)
The portions of interest to the FCC's National Broadband Plan are the part between "A" and "B" and between "A" and "C". These are shown inside the yellow box.
So what does all of this have to do with the National Broadband Plan in general and the FCC's Consumer Broadband test in particular?
First of all, we must recognize that a user's perception of network quality and speed is a complex function that involves the entire path between the user and the remote service.
Many protocol stacks and applications can degrade badly even if one seemingly small aspect changes. For example, the speed with which domain name system (DNS) queries are answered is often a major, or even the dominant, component of how quickly web pages are fetched and rendered. Indeed with the increasing number of "analytics" web bugs and links to "share" content the number of DNS queries involved in a page fetch can be quite surprising. And DNS responsivity is a matter that involves more than mere bandwidth.
Other applications degrade for other reasons. VoIP is often made incomprehensible by even small amounts of packet reordering, something that can occur quite often as a result of certain wireless technologies, load-balanced pathways, or routing behavior. And applications that use large packets, applications such as high quality video, can be badly affected by fragmentation of packets due to link MTU values of less than about 1500 bytes.
There are many characteristics that play a part. Among these are Quality of Service (QoS) handling, queuing disciplines and drop policies in routers, and congestion handling in protocol stacks. Moreover there are an increasing number of protocol "accelerators" that try to obtain better user performance by abandoning the protocol etiquette algorithms that are built into well implemented TCP stacks. Those accelerators may create local benefits to their users, as long as the number of such users is small, but they damage the experience of other users.
The National Broadband Plan tends to be involved only with the "User Access Link" part of my drawing. Yet the FCC's tests tend to lump all the parts of the drawing into one number thus masking the contribution of each part.
A national broadband build-out that does not deal with the entire system will be a waste of time and money. A user whose ISP has a magnificent broadband User Access Link but inadequate backhaul and connectivity to the internet at large is a user who is going to be dissatisfied.
Thus for the FCC's tests to be meaningful they need to do two things:
- They need to isolate and separately report the attributes of the User Network, the User Access Link, the User's ISP Cloud, and the degree of private peering to large content providers.
- The attributes that are measured need to be much deeper than "bandwidth" and "latency" and "jitter". I would recommend that the FCC look at the way that tools like PathChar and Pchar construct a detailed hop-by-hop analysis of network paths. Those tools require many thousands of packets over many tens of minutes for each hop in a path. In my own work I began (but never completed) a project to design a protocol to enable the fast and inexpensive measure of paths characteristics for proposed packet flows. That work is visible on the net at http://www.cavebear.com/archive/fpcp/fpcp-sept-19-2000.html.