A number of ISPs had trouble with Google last week, but not Lightwire. Why?

Last Friday a number of New Zealand and Australian ISPs experienced severe packet loss to Google, the issue appeared to impact some ISP’s accessing Google via a public peering exchange’s, but thankfully Lightwire was immune to the issue. All ISPs have network issues from time to time, we have had our share, so this post isn’t about pointing fingers. Rather, we thought it would be interesting to educate our clients as to exactly what element of our network design protected us from being affected in this case. To do that, we need to talk about public and private peering.

What is public peering?

When a group of ISPs and other companies connect in the same place and trade traffic freely. They do this by all connecting, or “peering”, using the root servers provided by the exchange. Examples of public exchanges in Auckland New Zealand are AKL-IX, Megaport Auckland Exchange, APE (Auckland Peering Exchange).

Lightwire peers at AKL-IX and Megaport Auckland, NSW-IX and Megaport Sydney. We also bilaterally peer over these public exchanges with some key players like Microsoft and Cloudflare.

These exchanges are “public” for two reasons:

• The participants on the exchange are not in control of the exchange infrastructure,  ie there is a third party between all traffic provider A and provider B.
• There are normally many networks talking to each other, so if networks A, B, C, and D are all talking to network E, this traffic is aggregated and will all share the same port on Network E’s network, meaning any of networks A, B, C or D could congest network E’s port and there is nothing the other networks could do about it.

Got that? Good. Next, we need to talk about this…

Google Private Network Interconnect (PNI for short)

This is what Google calls its private peering product.

A private peering like the Google PNI has no third party exchange in the middle. The traffic flows straight from A to B, and is always therefore in the control of either A or B.

No other network’s traffic is present, so each network has control and visibility on network congestion, quality of service (QoS), etc. Private peering networks hosted by content providers will sometimes announce more specific routes or more routes than they advertise on the public exchanges.

So, back to last Friday’s packet loss to Google, why wasn’t Lightwire affected?

• The packet loss was experienced by ISPs using public peering exchanges to access Google routes in Sydney, Australia
• The packet loss appears to have been the result of an issue experienced by one of the exchanges.
• Other Exchanges appeared to be affected as well due to the asymmetry between the forward and reverse paths. ISP was sending data over one Exchange and Google was selecting a different exchange back.
• Lightwire was reaching those same Google routes via its own PNI in both directions.

A couple of other benefits to the PNI with Google:

• Capacity management between Google and Lightwire is completely in our control.
• Having no congestion in the path means that jitter will be tightly controlled and kept to a minimum.
• The fault chain is shorter, Lightwire can deal with its own faults, and raise Google’s faults straight to their support team.
• The public peering and transit paths don’t go away, they are now redundant paths for Lightwire

Read more about our business internet services here.