Our Performance Monitoring Network is a unique system that aims to provide statistical data on the performance of broadband providers in the UK. Until now people have only been able to compare broadband on price, headline speeds (which are the subject of much controversy!), anecdotal discussion and some ad-hoc speed tester results.

We've been deploying small hardware-based monitoring units to volunteers all around the country. Once connected to the volunteer's home network the units perform a series of tests at regular intervals during the day, every day of the year. The results of these tests are fed up to our reporting engine and combined with the results of others on the same ISP to form a national view of how that ISP is performing. Future deployments will allow us to drill down to regional and/or product performance.

This project goes far beyond running vanilla speed tests. The units are checking latency, packet loss, DNS query times and failures, web page loading times, VoIP call quality as well as the obligatory suite of speed tests. Additional tests are also in development too. The goal is to provide a statistically sound measurement of overall broadband quality.

On August 2nd 2008 we published our first Performance Monitoring report.

» Download the report (PDF, 4.5MB)

Whilst we'd love to be able to purchase hundreds of broadband connections to run a purely scientific test, this is just financially impossible.

So instead we've asked people interested in the project to consider housing one of our small performance monitoring units on their home broadband connection. The units consume very little power and do not adversely affect your normal Internet usage.

We are not presently seeking new volunteers - we have a backlog of over 1300 people who have signed up already!

This idea has been at the back of my mind for years. There's simply nothing else like it out there anywhere (to the best of our knowledge anyway!). The statistics will provide a whole new means of looking at broadband services.

Presently one might look at a 16Mbps headline speed and assume that it means the service will be good for online gaming. An incorrect assumption is being made here - raw speed is far less important than many think. Latency and packet loss are far more important for online gaming and VoIP (Voice over IP). Our network will measure both of these factors (and many more) across all of the ISPs.

We've taken it upon ourselves to do this because nobody else has and we think it'll generate some fantastically interesting data.

The system itself has been live and operational since February 2008 on some pilot connections. Since June 1st 2008 we have had over 250 units deployed and actively returning us data.

At present we are performing the following tests:

1. Latency (ICMP based)
2. Packet loss (ICMP based)
3. DNS query resolution time
4. DNS query failure rate
5. Web page loading time
6. Web page failure rate
7. VoIP call quality (MOS)
8. VoIP call jitter, delay, packet loss
9. SMTP relaying speed
10. Web based download speed test (HTTP)
11. Web based upload speed test (HTTP)
12. Non web based download speed test
13. Non web based upload speed test
14. Single stream download speed test
15. Multi-stream download speed test
16. Single stream upload speed test
17. Multi-stream upload speed test

Tests 1-6 above are performed against common UK-based Internet destinations. The speed tests, VoIP tests and SMTP relaying tests are performed against a cluster of our own test servers, which we change periodically and load balance to ensure sufficient capacity.

We are always developing new tests and can remotely upgrade the existing units with new testing profiles once they are ready for production use.

The requirements we ask our volunteers to meet are as follows:

1. You currently use a router to connect to the Internet (not a USB modem).
2. You have a stable broadband connection (i.e. it doesn't disconnect frequently). Note that this is just referring to the connection - not the speed.
3. You are not a heavy downloader. We'd classify anything above 30GB per month as being too heavy for us to gather useful results.
4. You have a spare power socket near your existing router (or wherever you plan to connect the unit. Keep in mind that a network cable must run between the unit and your router though! We supply a 1m cable).
5. You need to be on one of the ISPs that we're targeting. See this FAQ item for details of these.
6. You are not an employee or a family member of an employee of one of the ISPs being monitored.

NOTE: As stated in this FAQ, our units download approximately 2GB per month and upload around 500MB. If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile.

In addition to the above we ask that volunteers agree to a set of terms and conditions that are broadly summarised by the points below:

I agree:

1. Not to unplug the unit or your ISP's router unless I'm away for an extended period of time.
2. Not attempt to reverse engineer or alter the unit.
3. To notify Samknows if and when I choose to change ISPs.
4. To return the unit to Samknows should you I longer wish to be involved (Samknows to pay reasonable postage costs).
5. To connect the unit in the way described in the documentation.
6. To keep Samknows updated with valid contact details (i.e. email and postal address).

Initially we're focussing on the following ISPs:

1. Sky
2. TalkTalk
3. BT
4. Orange
5. Tiscali
6. Virgin Media (cable and ADSL)
7. Zen Internet
8. Plus Net
9. Entanet (and their resellers)
10. Be Unlimited / O2
11. AOL
12. Karoo (by popular demand!)

Note that if we will only publish statistics for an ISP when we have a reasonable sample size or can be certain that our results are not merely an anomaly.

We're using the Linksys WRT54GL router. This device has proven extremely popular with enthusiasts since 2004 when it was first released. The "L" at the end of the model number indicates that this particular version was targeted specifically at developers who wanted to customise it with their own Linux installation - which is exactly what we have done.



The small device sports 5 Ethernet ports (1 for the WAN connection, 4 LAN connections) and two detachable wireless antennas. This is backed up by a 200Mhz MIPS processor, a whopping 16MB RAM and 4MB flash. Fortunately, this is more than ample for the tests we run. It also means that the devices consume between 3 and 4 watts of power!

For those interested, we're running a WRT-derived Linux operating system on the device. We're using tried and tested tools for performing all of our tests, all wrapped up in our own software package that we install on them.

When we first began developing this at the start of the year this was the first major question we asked ourselves. There are most definitely benefits to each, but a hardware solution clearly suited our needs better.

Few people leave a computer running 24/7, and in order to get an accurate set of time-series based test results we do need the tests to run continuously. Simply having tests running 9-5 or for a couple of hours in the evening would be completely unsuitable for us, as it'd potentially miss issues that occurred at other times of the day. Our hardware Linksys device is designed to run 24/7 and consumes so little power that its effect is negligible.

Secondly, installing software on people's computers is fraught with problems (besides them turning it off!). Firewalls and anti-virus software may block the tests, and even the end users themselves may uninstall the software if they do not recognise it (after all, there's often more than one user of the family computer). Again, this is not an issue with the hardware device (unless you run an outbound firewall on your router, which few do).

Thirdly, there is the perception of security and performance problems to address. If we were to install software on a persons machine and they then experienced a problem with the computer, their logical conclusion would be that the software caused it. Correct or not, this creates an additional support overhead that we'd rather avoid. The hardware solution is completely independent on the users' computers.

Fourthly, what about non-Windows based computers? The presence of Apple computers in the home is on the rise nowadays and cannot easily be ignored. A software solution could turn to Java to resolve this platform issue, but they'd find themselves having to write many of the most basic tests themselves. Our hardware solution uses tried and tested Linux/Unix applications to perform the core work of nearly all of our tests.

Finally, how do you deal with more than a single computer on the network? With the software solution it's just not possible. You simply cannot detect network traffic beyond the local machine, meaning that other computers on the network could quite easily disrupt results. A recent survey by Point Topic found that over 40% of people had more than one computer at home accessing the Internet - hardly a figure that can be ignored.

This last point was the one that swung it for us. By using the Linksys device we could monitor the volume of Internet-destined traffic flowing through it and defer our tests accordingly. We want to be sure that our results are not being disrupted by outside factors.

It's completely different! The only similarity is that our testing suite incorporates speed tests, amongst many other things.

Speed testers were written with the aim of helping individual users diagnose speed problems with their connections. Whilst results taken en-mass from them are useful for drawing very general conclusions, they usefulness dwindles when you consider:

1. They consider speed alone, and only for your specific machine. They tell you nothing about how a VoIP call would work, or how good your web browsing experience would be.
2. Environmental factors (e.g. using a wireless connection, having other Internet users on the network, etc) all impact results.
3. They're run on an ad-hoc basis. If you want to compare results at different times of the day then you need to run it yourself every single time.

Yes, speed testing websites are a very useful tool for diagnosing an individual connection's maximum upload/download speed. But can we draw meaningful and accurate statistics when looking at more than just speed? Not really, and this is where our solution comes in.

We do ask that our volunteers connect their wired computer(s) through the ports on the rear of the Linksys device and then connect the Linksys's WAN port (aka "Internet Port") to their existing router. By doing this the unit can monitor if there is Internet traffic when it is supposed to be running tests. If there is Internet traffic then the tests will be deferred until a quieter time.

 

Note that we do not ask that users reconfigure their wireless settings at all. Our software on the Linksys device can passively monitor your existing wireless network and defer tests automatically if required.

It should also be noted that whilst the Linksys device is clearly branded as a router by Linksys, our customised software sees it run purely as a network switch. Therefore your network will remain logically the same even after connecting through it (i.e. no IP addressing information will change, it will not attempt to act as a DHCP server, etc).

Please see the installation document for a guide on how to connect the unit.

Our units download no more than 2GB per month and upload no more than 500MB. The amount that's downloaded is speed dependant (so a slower connection will use less traffic than a faster connection).

If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile (we can run smaller speed tests on a per-device if necessary).

No. Our tests are designed to be non-intrusive. The vast majority of the tests that take place throughout the day are very minor and would have no effect on normal Internet activity.

Speed tests are obviously more intensive, but before running these our units will check how busy your wired and wireless networks are. If they are transferring more than 20KB/s traffic to the Internet for more than 30 seconds then the tests will be deferred for 10 minutes and then the process is attempted again.

The unit is a hardware device and as such cannot act as a key logger or install any trojan horse software on your computer. It operates a bare-bones Linux installation and exposes absolutely no services to the Internet. It's analogous to installing a print server or a NAS on to your home network - it's a cut down device that serves a very specific purpose.

We should point out that assuming the unit is installed as per the instructions, all network traffic will be flowing through it. However, the unit simply acts as a network switch and does not look at any of the packets flowing across your network. It only monitors traffic volumes for the purposes of deciding when to run (or not to run!) the tests.

Testing information uploaded from the unit to our servers contains no information about yourself whatsoever. Furthermore, all such communications are encrypted, ensuring that results cannot be tampered with en-route.

Your individual unit's test results will be available to you alone. Your unit's results will also be aggregated with others from the same ISP to form a larger average set of results that can be viewed publicly.

We have absolutely no intention of doing anything that may adversely affect your privacy or security. If you have any concerns please feel free to contact us to discuss them.

Yes, volunteers will have been sent an email with a link to their statistics page shortly after they took delivery of the unit. If you are a volunteer and have not received the email, please contact us for the link.

Below are some samples screenshots of some of the statistics that you will be able to access for your individual connection.

   

These were taken from my home connection (Be Unlimited). Note in the third image the level of DNS problems that have been afflicting the O2/Be network, particularly in early April.

We are running this independently of ISP support. For the avoidance of doubt, no ISPs are providing funding, hardware, services, broadband connections or any other support to the system. To have otherwise would negate the usefulness of this system as an independent resource.

We have however approached a handful of ISPs to run the idea past them. These conversations have revealed no more than is contained within these FAQs and gave away no detailed specifics on the testing methodology.

It is our belief that there is nothing else quite like this out there. Yes, some other companies offer broadband benchmarking services but these are aimed at providing a service to the ISPs - not the consumers (hence they run in conjunction with ISP funding and services).

Ours is the first service that aims to provide a true picture of broadband performance across the UK whilst maintaining complete independance in its operation.

Funding is coming from two main streams - our existing data services and outside investment.

Whilst we don't promote it via the website, Samknows offers a range of data services to businesses (typically web service based). These generate revenue that helps fund the hardware that runs the site and new developments.

In addition to this, LX Capital Limited invested in the website earlier this year with the intention of developing new and exciting projects like this.