Performance Benchmarks - FAQs

Performance Benchmark

The Performance Benchmark was born out of one main requirement; to create greater transparency within the broadband market.

Currently the average consumer is at a loss in a sea of acronyms, stats and technical jargon without really knowing or understanding what it is they are getting for their money. Through this project the FCC and SamKnows hope to level the playing field and make the industry more accessible, open and accountable.

We're on the lookout for US broadband consumers to join our community, if you're interested please click here and fill out our short registration form.

Still curious? Check out the FAQs below.

Performance Monitoring FAQ

• What is the "FCC Samknows Broadband Community"?

• Why are you doing it?

• What tests are performed?

• Which ISPs are you including in the tests?

• What is the hardware unit based on?

• How is this different to speed test results being published?

• How does the installation of the unit change my network layout?

• What effect will this have on my monthly download cap?

• Will this affect my usage of my Internet connection?

• How does this affect my security and privacy?

• Can Community members see their own individual results?

• How are the ISPs involved?

What is the "FCC Samknows Broadband Community"?

Originally launched as the Performance Monitoring Network the FCC SamKnows Broadband Community is a unique system that aims to gather and report statistical data on the performance of broadband providers across the globe. Being able to compare broadband on price, headline speeds (which are still the subject of much controversy!), anecdotal discussion and speed tester results just isn't enough - the Community, with your help, aims to put the information back in the public's hands.

We'll be deploying our rather lovely SamKnows “White boxes”*, an asset to any home, filled with some clever technology to volunteers all around the country. Once connected to your home network the unit will perform a series of tests at regular intervals during the day, every day of the year, with little to no involvement from you, simple! The results of these tests are then fed up to our reporting engine and combined with the results of others on the same ISP to form a national view of how that ISP is performing. We're even working towards future deployments that will allow us to drill down to regional and/or product performance.

This project goes far beyond running just speed tests; our White Boxes are lean, mean, broadband testing machines periodically checking latency, packet loss, DNS query times and failures, web page loading times, as well as the obligatory suite of speed tests. Additional tests are also in development too. The goal is to provide a statistically sound measurement of overall broadband quality.

*It’s actually a black router, but the airline industry have claimed the term “Black Box”!

Why are you doing it?

We’re striving for great transparency within the broadband market – it’s at the heart of the project and has been at the back of our minds for years. There's simply nothing else like it out there anywhere (to the best of our knowledge anyway!). The statistics will provide a whole new means of looking at broadband services, leveling the playing field and making the industry more accountable and work harder for the consumer.

Presently one might look at a 16Mbps headline speed and assume that it means the service will be good for online gaming. An incorrect assumption is being made here - raw speed is far less important than many think. Latency and packet loss are far more important for online gaming. Our network will measure both of these factors (and many more) across all of the ISPs – greater insight into how the ISP’s perform, more insightful information for the consumer.

Just like Batman (kind of) we've taken it upon ourselves to do this because nobody else has! We think it'll generate some fantastically interesting data. You're very welcome!

What tests are performed?

At present we are performing the following tests:

Data speed (being a combination of the Download Speed and Upload Speed)

Data usage

Download speed

Upload speed

Latency

Jitter

Availability

Packet loss

DNS query resolution time

DNS query failure rate

Web page loading time

Web page failure rate

Single & Multi-threaded web based download speed test (HTTP)

Single & Multi-threaded Web based upload speed test (HTTP)

Multi-stream download speed test

Consumption of the connection – the amount of data sent and received

Availability of the connection – when it is unavailable

Tests 1-6 above are performed against common US-based Internet destinations. The speed tests, are performed against a cluster of our own test servers, which we change periodically and load balance to ensure sufficient capacity.

We are always developing new tests and can remotely upgrade the existing units with new testing profiles once they are ready for production use. Pretty comprehensive I think you'll agree!

Which ISPs are you including in the tests?

We are looking for volunteers for all ISP's to participate. Note that we will only publish statistics for an ISP when we have a reasonable sample size or can be certain that our results are not merely an anomaly.

What is the hardware unit based on?

We're using the Linksys WRT160NL router. The "L" at the end of the model number indicates that this particular version was targeted specifically at developers who want to customize it with their own Linux installation - which is exactly what we have done.

The small device sports 5 Ethernet ports (1 for the WAN connection, 4 LAN connections) and two detachable wireless antennas. This is backed up by a 400Mhz processor, a whopping 32MB RAM and 8MB flash. Fortunately, this is more than ample for the tests we run. It also means that the devices consume between 3 and 4 watts of power!

For those interested, we're running a WRT-derived Linux operating system on the device. We're using tried and tested tools for performing all of our tests, all wrapped up in our own software package that we install on them.

How is this different to speed test results being published?

It's completely different! The only similarity is that our testing suite incorporates speed tests, amongst many other things.

Speed testers were written with the aim of helping individual users diagnose speed problems with their connections. Whilst results taken en-mass from them are useful for drawing very general conclusions, their usefulness does reduce when you consider:

They consider speed alone, and only for your specific machine. They tell you nothing about how a VoIP call would work, or how good your web browsing experience would be.

Environmental factors (e.g. using a wireless connection, having other Internet users on the network, etc) all impact results.

They're run on an ad-hoc basis. If you want to compare results at different times of the day then you need to run it yourself every single time.

That's not to take anything away from speed testing websites; they're a very useful tool for diagnosing an individual connection's maximum upload/download speed. But can we draw meaningful and accurate statistics when looking at more than just speed? Not to the extent we’re aiming to achieve with our project.

How does the installation of the unit change my network layout?

We do ask that our volunteers connect their wired computer(s) through the ports on the rear of the Linksys device and then connect the Linksys's WAN port (aka "Internet Port") to their existing router. By doing this the unit can monitor if there is Internet traffic when it is supposed to be running tests. If there is Internet traffic then the tests will be deferred until a quieter time.

Note that we do not ask that users reconfigure their wireless settings at all. Our software on the Linksys device can passively monitor your existing wireless network and defer tests automatically if required.

It should also be noted that whilst the Linksys device is clearly branded as a router by Linksys, our customised software sees it run purely as a network switch. Therefore your network will remain logically the same even after connecting through it (i.e. no IP addressing information will change; it will not attempt to act as a DHCP server, etc).

Please see the installation document for a guide on how to connect the unit.

What effect will this have on my monthly download cap?

Our units download no more than 2GB per month and upload no more than 2GB. The amount that's downloaded is speed dependant (so a slower connection will use less traffic than a faster connection).

If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile (we can run smaller speed tests on a per-device if necessary).

Will this affect my usage of my Internet connection?

No. Our tests are designed to be non-intrusive. The vast majority of the tests that take place throughout the day are very minor and would have no effect on normal Internet activity.

Speed tests are obviously more intensive, but before running these our units will check how busy your wired and wireless networks are. If they are transferring more than 200Kb/s then the tests will be deferred for 1 minute and then the process is attempted again and repeated for up to 5 minutes.

How does this affect my security and privacy?

The unit operates as a bare-bones router and exposes absolutely no services to the Internet. It's similar to installing a print server or a NAS on to your home network - it's a cut down device that serves a very specific purpose.

We should point out that assuming the unit is installed as per the instructions, all network traffic will be flowing through it. However, the unit simply acts as a network switch and does not look at any of the packets flowing across your network. It only monitors traffic volumes for the purposes of deciding when to run (or not to run!) the tests and to measure consumption.

Testing information uploaded from the unit to our servers contains no information about you whatsoever. Furthermore, all such communications are encrypted, ensuring that results cannot be tampered with en-route.

Your individual unit's test results will be available to you alone. Your unit's results will also be aggregated with others from the same ISP to form a larger average set of results that can be viewed publicly.

We have absolutely no intention of doing anything that may adversely affect your privacy or security. If you have any concerns please feel free to contact us to discuss them.

SamKnows, on behalf of the FCC, is collecting and storing broadband performance information, including various personally identifiable information (PII) such as the street addresses, email addresses, online usage patterns, and broadband performance information, from those individuals who are participating voluntarily in this test. SamKnows will not release, disclose to the public, or share any PII with any outside entities, including the FCC, except as is consistent with the Privacy Act of 1974, Public Law 93579 (5 U.S.C. 552a(b)(5)). For more information, see the SamKnows privacy policy. The broadband performance information that is made available to the public, including the FCC, will be in an aggregated form and with all PII removed, in compliance with subsection (b)(5) of the Privacy Act of 1974, as amended (5 U.S.C. 552a), and the SamKnows privacy policy. The broadband performance information that is made available to the public, including the FCC, will be in an aggregated form and with all PII removed, in compliance with subsection (b)(5) of the SamKnows privacy policy. The FCC is soliciting this information under authority of the Broadband Data Improvement Act of 2008, Pub. L. No. 110-385, Stat 4096 § 103(c)(1); American Reinvestment and Recovery Act of 2009 (ARRA), Pub. L. No. 111-5, 123 Stat 115 (2009); and Section 154(i) of the Communications Act of 1934, as amended.

Can Community members see their own individual results?

You know it! Once signed up and operational you will be sent an email with a link to your statistics page. If you don't receive the email, please contact us and we'll sort this out for you.

How are the ISPs involved?

We are running this independently of direct ISP involvement but we do have their support. For the avoidance of doubt, no ISP’s are providing funding, hardware, services, broadband connections or any other support to the system. We're incorruptible!

We have however approached a handful of ISPs to run the idea past them. These conversations have revealed no more than is contained within these FAQs and gave away no detailed specifics on the testing methodology.