RAW DATA: TrueNet March 2015 Broadband Report - TV Streaming Impacts ISP Performance?

(TrueNet is contracted to the Commerce Commission to monitor broadband performance. It uses a network of probes attached to 400 volunteers' internet connections to rate ISPs — Editor.)

The market for television and movies on the Internet (Over-the-Top-Technology, or OTT) suddenly took off this year, and March appears to be the month that demand started to impact supply.  

While Most ISPs would have increased capacity to prepare for the extra demand, it would appear the balance of supply and demand we have seen for many months is no longer present.  In March, both Vodafone and Slingshot had severe DSL performance reductions during the peak demand hours of 8pm to 10pm, while Snap and Spark also dropped significantly.  

Netflix Demand Takes Off

OTT takes a lot of capacity, creating high demand, which requires ISPs to match with increased supply.  Sandvine’s latest Internet and Subscriber Trends publication in January reported that Netflix in the USA dominates 35% with of all traffic.  

In Australia, Netflix traffic is changing very quickly from 15% of all traffic a few weeks ago to this week;  "iiNet has revealed that already, 25 percent of its own network traffic is from Netflix"

Initial subscriber demand may be very high as new subscribers try out the new service on their Apple TV, binging on desirable programs until the cap runs out.  Unlimited service caps are very likely to become the most popular service for OTT subscribers.

Summary of Performance Measures

Performance History

TrueNet measures Time of Day Performance by comparing the best speed during the day with the worst .  TrueNet recommends ISPs target achieving at least 95% of their best speed.  

Performance by ISPs changes with variations in demand and supply.  A strong growth in customers, or usage (say the introduction of TV on the Internet) needs to be met with an increase in capacity. Our measures check to see if this is achieved, and these variations in performance over time are  shown  in  the movements of the animated chart below.

Chart 1 shows performance over the last 3 years by ISP, animated to show how much performance changes with supply and demand.  The best ISPs maintain capacity over a long period of time to ensure demand is met by supply, ie they stay better than 90% most of the time.  The final chart shows the same data in a line format.

Some ISPs are more often below 90% than others.

In Table 1, during March 2015, Orcon is the quickest to download NZ Live Webpages for  Fibre and ADSL, while achieving second equal in VDSL.  

Spark is easily the best at achieving their advertised fibre speeds, reaching 98%, including the peak demand period at 9pm.  

MyRepublic panelists are now included in our 100Mb/s Fibre comparisons with an average performance for downloading Live NZ and Australian webpages. Their 9pm speed is off the pace compared to Spark, Snap and Orcon.

This month the Speed by Time of Day analysis shows greater  variation in the evening busy hour with Vodafone’s Cable, VDSL and ADSL services  being much more affected by congestion during the 8-10pm busy period than in the past month. 

The results from some services are now showing a much higher variation in Min/Max speed (e.g. MyRepublic 100Mb/s, Orcon 30Mb/s as shown in Table 1).   It is possible that new Internet TV services are having a drastic effect on performance, and TrueNet will keep a close watch on the impact of Internet TV on the quality of users’ overall services.  

Our Quarterly Report  on Latency and Domain Name Server Response time (DNS) sees a significant improvement in tests within New Zealand on all technologies excluding cable. This quarter, DNS results show Snap remains best overall  across all regions.

Webpage Download Time

Responsive website browsing is valued by most Internet users, and conversely, slow-loading sites can be extremely frustrating.

TrueNet tests internet browsing by downloading a selection of popular NZ and Australian Live Webpages, measuring the the time to fully download all files on the page.  These pages are changed from time-to-time.

NZ Live Webpages

Chart 2: Orcon continue to stand out with the best results for ADSL, Fibre, and a close second in VDSL.

Slingshot ADSL improved  from last month, Bigpipe slipped back  two places, and Flip is now the worst of the NZ providers.

Australian Live Webpages

Chart 3: Following on with their excellent track record, Orcon was best in Fibre and VDSL.

Flip and Slingshot were the best of the ADSL providers. Ironically, our Australian comparison ISP, iiNet managed to improve to be on par with bottom placed Spark.

Comparing the high speed fibre, and cable providers web page download speed performances with their file download speeds to Australia, shows that having good International file download speeds is helpful, but not sufficient to give top web browsing response times.

Speed (File Download Performance)

For TrueNet's speed tests each panelist's probe regularly downloads a 1MB file from Auckland, Wellington, Dallas and Sydney.

National results are based on the best Auckland or Wellington download speed from each test run to compute a median hourly speed for each panelist's connection.  International tests represent the median hourly speed of each probe from Dallas and Sydney.

International File Download Speed

TrueNet tests download speeds from our servers in Dallas & Sydney to measure performance to our international neighbours.

We ensure the download file is not held in New Zealand (cached), so that the test truly measures international performance. The speed test results selected are taken at 9pm, in the evening busy period, when congestion is likely to be worst.

Fibre and Cable - Chart 4

MyRepublic is included for the first time in this report, with a top speed to Sydney for the 100Mb/s Fibre providers, but slowest to the Dallas server.

Spark has the fastest Dallas speed of the 30Mb/s Fibre services, although reaching under 15Mb/s.

Vodafone Cable is well behind on raw speed, but note the very competitive Australian Live Webpage results above.

ADSL and VDSL - Chart 5

Vodafone ADSL maintained the top speed to both destinations.

Slingshot had the fastest VDSL performance this month. 

Vodafone's performance fell this month, while Slingshot and Orcon were about the same as last month.

Spark's Dallas performance places them second in VDSL and third in ADSL.

NZ File Download speed

Comparing performance by time-of-day is important as it shows the service degradation when everyone is using the internet during the peak hours of 8pm to 10pm.  TrueNet uses the best of a pair of Auckland and Wellington download tests to calculate the median results by hour over the month for each monitored connection.  We take the average of all median results with each ISP for each hour.

Fibre - Cable - VDSL Comparison

Chart 6, compares the performance by Time of Day of the higher speed broadband offerings.

Snap remains the best of our 100Mb/s fibre providers, with Orcon improving this month to be a close second. MyRepublic is well down, averaging just 75Mb/s from our NZ servers.

Vodafone cable has a more pronounced slowdown in the evening compared to recent months, dipping to just 80Mb/s.

The 30Mb/s Fibre providers offer very similar, consistent performance, with Snap, Spark and Orcon all about the same.

TrueNet has fibre panelists on 8 service speeds at present. An alternative, simplified way to look at results is to compare ISP performance against the advertised service speed.

ISPs with 5 or more probes are shown in Chart 7, along with Cable. Vodafone Cable achieves over 100% because the 100Mb/s Cable plan is capable of speeds around 130Mb/s. Note, that at 9PM that headroom disappears, with Spark and Snap achieving better relative speed at that hour.

MyRepublic and Vodafone fibre are below 80% of adverstised speed throughout the month, with some Time of Day variation compared to Spark and Snap.

DSL Performance by Time of Day (ToD)

VDSL File Download Speed - Chart 8

Following on from last month's results, Time of Day performance was down for a number of ISPs.  One possible driver of change is Internet TV launches, placing a lot more traffic on the network during the evening.

Orcon & Bigpipe maintained speed within 95% of peak speed at all times. Spark and Snap experienced degraded 9PM speeds compared to last month, while Vodafone and Slingshot were significantly affected at 9pm.

ADSL File Download Speed - Chart 9

Most ISPs stayed within 95% of peak speeds this month with Orcon and BigPipe again achieving 98%.   As with VDSL measurements, Vodafone, Slingshot and Flip performances were down at 8-9PM compared to last month. iiNet performance was not impacted, and in fact improved from last month. 

Upload Speed Performance

Upload speed is important to users sending fairly large amounts of data through the internet, or loading files to the Cloud.

TrueNet's upload tests send a 1MB file to our Wellington server, and records the results using a similar method to the upload tests, but measured from Wellington only.  Australian panelists connections test to Sydney. 

By Technology

There were small improvements in the 20Mb/s and 10Mb/s fibre services shown in Chart 10. All others remain similar to previous reports. Clearly, the speed of VDSL services is the minimum required for users wanting to upload content at a reasonable speed.

Upload Speed by ISP

Snap 50Mb/s upload exceeds the advertised speed, as it has for several months. Snap's other fibre services all achieve the advertised speed.

The 10Mb/s services are almost identical, irrespective of ISP or technology (fibre, cable).

Quarterly Feature - Latency and DNS

This month  we report on Latency and Domain Name Server (DNS) response times, two measures that impact  download speeds, and the time it takes to download a webpage. DNS response may seem an arcane measure to many users, but when DNS servers stumble, or because they are too far away, searching and browsing the web becomes very difficult.

Latency Performance

Latency is the time it takes for a packet of data to be returned by a remote server to a Volunteer's probe. Latency is important to both responsiveness of the user interface and also speed. Very high latency makes browsing the internet difficult, and may lead to timeouts.

High Latency can make gaming impossible, adding to reaction time, meaning that if someone else has lower Latency they may see a game change, and react well before a slower connection is able to display the change (i.e. you can be shot, and killed in a game before your computer shows the shooter)

TrueNet measures Latency to 6 test locations - Auckland, Wellington, Sydney, Melbourne, Dallas, and the UK

NZ Latency - Chart 12

Fibre latency is unchanged since the last report, and Cable latency has reduced significantly.

For VDSL the trend is towards improved latency, with all results moving closer. Orcon VDSL improved since December to rank best, while Snap latency deteriorated to second place.

Bigpipe ADSL saw improved latency to match the best VDSL results. However, Orcon ADSL latency increased to create a greater spread than last month.

Australian Latency - Chart 13

Fibre showed a general improvement, with a small spread between Orcon the best, and MyRepublic the worst (highest) latency.

Vodafone Cable, however, had a significant increase in latency to Australian test points, not echoed in their DSL service results.  Latency this large (100ms to each city) suggests connections via another country!

VDSL latency has improved since December, with Orcon again taking out the best performance while Slingshot latency almost halved to be one of the best.

US and UK Latency - Chart 14

The distance to the test locations in the US and UK dominates the latency results, meaning there is less variation between ISPs and technologies.

There is a slight increase in latency to the UK but US results are on a par with December. iiNet (ADSL), and Vodafone Cablestand out, just, as having the greatest latency to the UK.  iiNet latency to the US is over 50ms (25%) worse than all NZ ISPs.

Domain Name Server Response Time

Quick DNS Response Times improve the internet browsing experience of users. Very slow DNS responses can cause timeouts, making the internet almost unusable. DNS problems often manifest as a slow response to changing websites or searching, while the responsiveness or speed once on a page seems fine.

TrueNet measures the time taken to receive a response from all DNS servers that an ISP informs us they use (usually two). We record the best result overall for each hourly test as indicated in the charts below.

DNS response is also affected by latency. The regional performance results can give an indication of the quality of broadband performance in areas far from the DNS server. Those ISPs investing in distributed DNS closer to customers will have better performance, all other things being equal.

In Chart 15, Snap remains best overall, best in Canterbury, and 15 msec or less in all other regions.

Slingshot response time improved significantly in Otago, Canterbury, and the Waikato/Bay of Plenty

Orcon improved in Otago

Vodafone improved in Wellington, and Waikato/Bay of Plenty

In Chart 16, Fibre shows a big improvement in Canterbury.

Similarly, VDSL showed an improvement in DNS response times in Canterbury only.

For ADSL, there was an improvement in Canterbury, and Waikato/Bay of Plenty, but DNS response times were slower in Auckland