Performance Benchmark
Here's how the performance benchmark started up. We sat down one afternoon in 2008 with a nice cup of tea and some biscuits to discuss the information gap that exists within the wonderful world of broadband. Consumers were just relying on either anecdotal information or just having to take for granted what the ISPs were saying. So, we mused, perhaps it was high time that an independent organisation started filling in the gap - someone like SamKnows!
Once we'd devised the tests we needed to answer our questions - stuff like loading time for web pages, latency, VoIP call quality and upload and download speeds - we needed to build the technology. Rather like Hannibal Smith, BA Baracus and the other two A-Teamers, but more handsome even, we hit the lab and built the technology needed to get the performance benchmark show on the road.
With the gear ready to roll, we contacted our friends at theregister.com and PC Pro to see if they could find us the 200 people we needed to make up our panel of volunteers. By the way, if any of you reading this are part of the panel that helped us collate our data, we'd like to thank you from the bottom of our hearts!
Our first report, all about the state of consumer broadband in the UK, was released in August 2008. Holding it in our sweaty little hands, we went round to Ofcom for another cup of tea and a chit-chat. The telecom and TV regulator liked the cut of our jib so much it decided to use our methodology and technology as part of its study into Broadband speeds. So, with the help of another 2,500 people found through Market Researchers GFK, Ofcom compiled its report, the first part of which was released in January 2009.
Hmm, thought the ISP providers, this looks interesting. With their collaboration, we ran an ISP workshop which led to - ta-daa! - our performance benchmark being adopted by the industry.
Still curious? Check out the FAQs below.
Performance Monitoring FAQ
- What is the "Samknows Performance Monitoring Network"?
- Why are you using volunteers?
- Why are you doing it?
- Is it live yet?
- What tests are performed?
- What are the requirements for volunteers?
- Which ISPs are you including in the tests?
- What is the hardware unit based on?
- Why are you using a hardware solution rather than software?
- How is this different to speed test results being published?
- How does the installation of the unit change my network layout?
- What effect will this have on my monthly download cap?
- Will this affect my usage of my Internet connection?
- How does this affect my security and privacy?
- Can volunteers see their own individual results?
- How are the ISPs involved?
- How is this system unique?
- How is this being funded?
What is the "Samknows Performance Monitoring Network"?
We've been deploying small hardware-based monitoring units to volunteers all around the country. Once connected to the volunteer's home network the units perform a series of tests at regular intervals during the day, every day of the year. The results of these tests are fed up to our reporting engine and combined with the results of others on the same ISP to form a national view of how that ISP is performing. Future deployments will allow us to drill down to regional and/or product performance.
This project goes far beyond running vanilla speed tests. The units are checking latency, packet loss, DNS query times and failures, web page loading times, VoIP call quality as well as the obligatory suite of speed tests. Additional tests are also in development too. The goal is to provide a statistically sound measurement of overall broadband quality.
On August 2nd 2008 we published our first Performance Monitoring report.
» Download the report (PDF, 4.5MB)
Why are you using volunteers?
So instead we've asked people interested in the project to consider housing one of our small performance monitoring units on their home broadband connection. The units consume very little power and do not adversely affect your normal Internet usage.
We are not presently seeking new volunteers - we have a backlog of over 1300 people who have signed up already!
Why are you doing it?
Presently one might look at a 16Mbps headline speed and assume that it means the service will be good for online gaming. An incorrect assumption is being made here - raw speed is far less important than many think. Latency and packet loss are far more important for online gaming and VoIP (Voice over IP). Our network will measure both of these factors (and many more) across all of the ISPs.
We've taken it upon ourselves to do this because nobody else has and we think it'll generate some fantastically interesting data.
Is it live yet?
What tests are performed?
- Latency (ICMP based)
- Packet loss (ICMP based)
- DNS query resolution time
- DNS query failure rate
- Web page loading time
- Web page failure rate
- VoIP call quality (MOS)
- VoIP call jitter, delay, packet loss
- SMTP relaying speed
- Web based download speed test (HTTP)
- Web based upload speed test (HTTP)
- Non web based download speed test
- Non web based upload speed test
- Single stream download speed test
- Multi-stream download speed test
- Single stream upload speed test
- Multi-stream upload speed test
Tests 1-6 above are performed against common UK-based Internet destinations. The speed tests, VoIP tests and SMTP relaying tests are performed against a cluster of our own test servers, which we change periodically and load balance to ensure sufficient capacity.
We are always developing new tests and can remotely upgrade the existing units with new testing profiles once they are ready for production use.
What are the requirements for volunteers?
- You currently use a router to connect to the Internet (not a USB modem).
- You have a stable broadband connection (i.e. it doesn't disconnect frequently). Note that this is just referring to the connection - not the speed.
- You are not a heavy downloader. We'd classify anything above 30GB per month as being too heavy for us to gather useful results.
- You have a spare power socket near your existing router (or wherever you plan to connect the unit. Keep in mind that a network cable must run between the unit and your router though! We supply a 1m cable).
- You need to be on one of the ISPs that we're targeting. See this FAQ item for details of these.
- You are not an employee or a family member of an employee of one of the ISPs being monitored.
NOTE: As stated in this FAQ, our units download approximately 2GB per month and upload around 500MB. If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile.
In addition to the above we ask that volunteers agree to a set of terms and conditions that are broadly summarised by the points below:
I agree:
- Not to unplug the unit or your ISP's router unless I'm away for an extended period of time.
- Not attempt to reverse engineer or alter the unit.
- To notify Samknows if and when I choose to change ISPs.
- To return the unit to Samknows should you I longer wish to be involved (Samknows to pay reasonable postage costs).
- To connect the unit in the way described in the documentation.
- To keep Samknows updated with valid contact details (i.e. email and postal address).
Which ISPs are you including in the tests?
- Sky
- TalkTalk
- BT
- Orange
- Tiscali
- Virgin Media (cable and ADSL)
- Zen Internet
- Plus Net
- Entanet (and their resellers)
- Be Unlimited / O2
- AOL
- Karoo (by popular demand!)
Note that if we will only publish statistics for an ISP when we have a reasonable sample size or can be certain that our results are not merely an anomaly.
What is the hardware unit based on?
The small device sports 5 Ethernet ports (1 for the WAN connection, 4 LAN connections) and two detachable wireless antennas. This is backed up by a 200Mhz MIPS processor, a whopping 16MB RAM and 4MB flash. Fortunately, this is more than ample for the tests we run. It also means that the devices consume between 3 and 4 watts of power!
For those interested, we're running a WRT-derived Linux operating system on the device. We're using tried and tested tools for performing all of our tests, all wrapped up in our own software package that we install on them.
Why are you using a hardware solution rather than software?
Few people leave a computer running 24/7, and in order to get an accurate set of time-series based test results we do need the tests to run continuously. Simply having tests running 9-5 or for a couple of hours in the evening would be completely unsuitable for us, as it'd potentially miss issues that occurred at other times of the day. Our hardware Linksys device is designed to run 24/7 and consumes so little power that its effect is negligible.
Secondly, installing software on people's computers is fraught with problems (besides them turning it off!). Firewalls and anti-virus software may block the tests, and even the end users themselves may uninstall the software if they do not recognise it (after all, there's often more than one user of the family computer). Again, this is not an issue with the hardware device (unless you run an outbound firewall on your router, which few do).
Thirdly, there is the perception of security and performance problems to address. If we were to install software on a persons machine and they then experienced a problem with the computer, their logical conclusion would be that the software caused it. Correct or not, this creates an additional support overhead that we'd rather avoid. The hardware solution is completely independent on the users' computers.
Fourthly, what about non-Windows based computers? The presence of Apple computers in the home is on the rise nowadays and cannot easily be ignored. A software solution could turn to Java to resolve this platform issue, but they'd find themselves having to write many of the most basic tests themselves. Our hardware solution uses tried and tested Linux/Unix applications to perform the core work of nearly all of our tests.
Finally, how do you deal with more than a single computer on the network? With the software solution it's just not possible. You simply cannot detect network traffic beyond the local machine, meaning that other computers on the network could quite easily disrupt results. A recent survey by Point Topic found that over 40% of people had more than one computer at home accessing the Internet - hardly a figure that can be ignored.
This last point was the one that swung it for us. By using the Linksys device we could monitor the volume of Internet-destined traffic flowing through it and defer our tests accordingly. We want to be sure that our results are not being disrupted by outside factors.
How is this different to speed test results being published?
Speed testers were written with the aim of helping individual users diagnose speed problems with their connections. Whilst results taken en-mass from them are useful for drawing very general conclusions, they usefulness dwindles when you consider:
- They consider speed alone, and only for your specific machine. They tell you nothing about how a VoIP call would work, or how good your web browsing experience would be.
- Environmental factors (e.g. using a wireless connection, having other Internet users on the network, etc) all impact results.
- They're run on an ad-hoc basis. If you want to compare results at different times of the day then you need to run it yourself every single time.
Yes, speed testing websites are a very useful tool for diagnosing an individual connection's maximum upload/download speed. But can we draw meaningful and accurate statistics when looking at more than just speed? Not really, and this is where our solution comes in.
How does the installation of the unit change my network layout?
Note that we do not ask that users reconfigure their wireless settings at all. Our software on the Linksys device can passively monitor your existing wireless network and defer tests automatically if required.
It should also be noted that whilst the Linksys device is clearly branded as a router by Linksys, our customised software sees it run purely as a network switch. Therefore your network will remain logically the same even after connecting through it (i.e. no IP addressing information will change, it will not attempt to act as a DHCP server, etc).
Please see the installation document for a guide on how to connect the unit.
What effect will this have on my monthly download cap?
If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile (we can run smaller speed tests on a per-device if necessary).
Will this affect my usage of my Internet connection?
Speed tests are obviously more intensive, but before running these our units will check how busy your wired and wireless networks are. If they are transferring more than 20KB/s traffic to the Internet for more than 30 seconds then the tests will be deferred for 10 minutes and then the process is attempted again.
How does this affect my security and privacy?
We should point out that assuming the unit is installed as per the instructions, all network traffic will be flowing through it. However, the unit simply acts as a network switch and does not look at any of the packets flowing across your network. It only monitors traffic volumes for the purposes of deciding when to run (or not to run!) the tests.
Testing information uploaded from the unit to our servers contains no information about yourself whatsoever. Furthermore, all such communications are encrypted, ensuring that results cannot be tampered with en-route.
Your individual unit's test results will be available to you alone. Your unit's results will also be aggregated with others from the same ISP to form a larger average set of results that can be viewed publicly.
We have absolutely no intention of doing anything that may adversely affect your privacy or security. If you have any concerns please feel free to contact us to discuss them.
Can volunteers see their own individual results?
Below are some samples screenshots of some of the statistics that you will be able to access for your individual connection.
These were taken from my home connection (Be Unlimited). Note in the third image the level of DNS problems that have been afflicting the O2/Be network, particularly in early April.
How are the ISPs involved?
We have however approached a handful of ISPs to run the idea past them. These conversations have revealed no more than is contained within these FAQs and gave away no detailed specifics on the testing methodology.
How is this system unique?
Ours is the first service that aims to provide a true picture of broadband performance across the UK whilst maintaining complete independance in its operation.
How is this being funded?
Whilst we don't promote it via the website, Samknows offers a range of data services to businesses (typically web service based). These generate revenue that helps fund the hardware that runs the site and new developments.
In addition to this, LX Capital Limited invested in the website earlier this year with the intention of developing new and exciting projects like this.