2020.02.01 Carrier issue under investigation (Toll free and local) *PHOTOS* [RESOLVED]

The following is a summary in reverse order of what we know about the ongoing, widespread telecommunication disruption affecting Canada. The effects are felt across the country with many companies unaware of the issue as their toll free or other services failed. In many cases partial failures are reported which is conbtributing to many people remaining unaware of the problem. We are hopeful the news we received that repairs are underway will prevent this problem from extending into Monday. If not, expect many more people to start paying attention as the offices open tomorrow.

2020.02.11 00:15 PST

We are continuing to receive updates. The carriers are making progress, however repairs are about a day behind schedule - most operations are near normal, however it's still possible there will be additional disruption. Please check out the photos below for an idea of the scope of the problems. Multiple fiber cuts, and disruptions caused by weather related events span much of the province:

 More photos below... 

In this photo you can see the road is washed away exposing piping and conduits...

 And in this photo, you can see the entire rail bridge washed away. Our understanding is that this was the scene in Nroth Bend on Saturday when the major interruption occured.  

2020.02.06 09:00 PST

At this time we have been told that the remaining issues are limited to Telus and Bell clients calling inbound to some numbers on our networks. The issue is actually due to remaining issues with the fiber which is a Telus / Bell interconnect issue. We understand Telus and Bell are working hard on the issue and hope to have it resolved soon. We are not aware of any other issues affecting callers to / from any other carrier networks at this time.

2020.02.05 16:32 PST

We are seeing partial restoration, continuing to monitor and follow up.

2020.02.05 15:47 PST

The repaired connection in North Bend has gone offline again.Following up again with carrier partners - they report they are investigating.

2020.02.05 09:15 PST

We consider the issue now resolved, however we will be following up with all carrier partners to determine additional details that would have helped us help our clients be more informed and possibly deal with the issue more efficiently. We understand now that there were two separate fiber cuts. The first resulted in services transfering to the fiber which was then itself cut on Saturday morning. The communication we received was inefficent though, and did not result in us being able to communicate the details to you properly.

2020.02.04 07:45 PST

It's a new day. No new reports since last night. We will continue to collect information and push for answers, but it seems there was not one landslide but two. The first landslide is reported to have occured Thursday last week, which resulted in traffic re-routing. The second landslide and second fiber cut on Saturday were not the first failure. We were told to expect final repairs last night, but haven't yet seen the status report. Will collect more news stories today to help anyone who requires supporting documentation, and will be scheduling follow up discussions iwth our carrier partners to clarify how this sort of failure is even possible.

2020.02.03 08:56 PST

We are still having some reports of intermittent all circuits busy. If attempting calls using cell phones, your results may vary by trying a different cell carrier if one is available to you. A notable resurgence in trouble reports at https://downdetector.ca/status/telus/ would indicate the issue is still being seen by a number of people still.

2020.02.03 07:45 PST

Current tests indicate services are at least partially restored with all reported failures testing as working as of this morning. We also received additional information from one of our carrier partners. We have included this information here. Please note their description of events is in forward chronological order. Please submit any remaining service issues to Support so we can follow up, test or escallate the issue as appropriate. Thanks to everyone for their patience as we worked through these issues! The short story is that the fiber cut involved a landslide and a rail bridge which made the repair exceptionally difficult. What we do not yet understand is how one fiber cut affected services across the country. We will be investigating that over the coming weeks. If you would like detailed information please open a ticket with Support - we will send updates to the related tickets as we receive them.

Checking for service reports at https://downdetector.ca/ would indicate problem resolution is fairly universal across all carriers.

Thank you, BitBlock Systems.

Here is the explanation we received this morning - unfortunately many of these updates did not come when they were issued possibly relating to the Internet issues that accompanied the problem:
 

We are currently experiencing a service degradation affecting our toll-free numbers and voice circuits caused by a Fiber cut affecting our voice and data traffic. Several carriers across Canada are affected by this external incident 

At 8 pm EST  a fiber crew was dispatched to asses the situation. ETA to be on site is 4 hours. 

UPDATE 2/1/2020 10:45 PM EST
Fiber technicians continue to work to repair the fiber cut. Due to unstable road conditions reported in the area, we expect the repair to take longer than usual although we do not have an estimated time for repair.

UPDATE 2/2/2020 00:08 ET
Fiber technicians continue to work to repair the fiber cut. Area near fiber cut is experiencing road closures which will impact the estimated time for repair. 

UPDATE 2/2/2020 00:38 ET
Fiber crew is waiting for access to the affected site. The area is still limited access and consists of a railway bridge crossing. Weather conditions are also impacting efforts to clear the area for access and will delay restoration efforts. Due to this, we are still not able to provide an ETA and anticipate a lengthy restoration time.

UPDATE 2/2/2020 11:30 EST
Update from our partner carriers. They are rebuilding the fiber cable on another link once they get the a piece of equipment that will arrive at 7pm EST. 

UPDATE 2/2/2020 13:48 EST
Update from our partner carriers. They are rebuilding the fiber cable on another link once they get the a piece of equipment that will arrive at 7pm EST.

Update 02/02/2020 14:48 EST
CN NOC clan up was compleated tracks have to be re builded and then will tak 12 hours to repair so they can re gain access and start splicing cables estimated repair time will be 48 hours
There is a 10 G link was brought. Still waiting for that equipment to arrive on site at 7 pm EST

UPDATE 2/2/2020 10:00 PM ET
The extreme weather-triggered landslides in BC that have affected many telecom providers, including ThinkTel, have restoration efforts well underway. Support workers continue work in North Bend and Hope, BC to remove debris created by these landslides in order to access and fix the fibre infrastructure that was severed. This failure impacts the following services: Inbound and outbound calling (DID), Toll-Free services, Hosted PBX, Data circuits, SIP trunking, TDM Access, Voice PRI.

Affected services in eastern Canada are being re-routed and restored as quickly as possible. We continue to seek alternative restoration options with our operations teams and service provider partners and continue to monitor the situation closely.
 

UPDATE 2/3/2020 06:30 ET
The extreme weather-triggered landslides in BC that have affected many telecom providers, including ThinkTel, have restoration efforts well underway. Support workers continue work in North Bend and Hope, BC to remove debris created by these landslides in order to access and fix the fibre infrastructure that was severed. This failure impacts the following services: Inbound and outbound calling (DID), Toll-Free services, Hosted PBX, Data circuits, SIP trunking, TDM Access, Voice PRI.

Affected services in eastern Canada are being re-routed and restored as quickly as possible. We continue to seek alternative restoration options with our operations teams and service provider partners and continue to monitor the situation closely.

 

2020.02.02 13:52 PST

The problem marches on into it's second day. We've been trying to raise awareness through Twitter, but the mainstream media seems to br reporting small facets of the story as if they are disconnected / isolated problems when in reality they all seem to have the same underlying cause.

One of the more comprehensive articles (with some errors) was posted by The Georgia Straight. https://www.straight.com/tech/1354721/rogers-and-bell-cellphone-service-down-across-much-british-columbia-wake-rainstorm

Searching various news sites turns up other mentions:https://globalnews.ca/news/6495369/bc-landslides-cell-service-roads/

https://www.citynews1130.com/2020/02/01/bell-phone-service-outage-affecting-b-c-police-cities-customers/

Global / CKNW mentiioned in a news broadcast on Saturday evening reported that Bell was acknowledging 911 disruption and asking clients to use alternate methods to contact first responders. In the article above E-Comm (911 provider in the lower mainland) confirmed this.

One of the easiest ways to see the scope of the problem is to check out downetector.ca - hundreds of people reporting problems. You can see from the graphs how the problem reports for most carriers dropped off as people went to sleep, and resumed this morning. For the most part, Rogers looks like they might have worked around their issues, but Telus, Bell, etc. are ongoing.

See https://downdetector.ca/

 

As an example, see https://downdetector.ca/status/telus/ (this issue affects many carriers, we aren't blaming Telus, only providing an example):

16:00 PST

From our investigation we have determined the issue is affecting most if not all carriers in Canada and possibly beyond. We have reached every carrier we have direct contact with to ensure they are aware of and are investigating / working toward resolution. You can see correlation on downdetector.ca of an uptick in reports this afternoon across many companies. We will post updates here and on twitter as additional information becomes availalble.

15:15 PST

At this time we have multiple reports which hint at an issue affecting multiple carriers. Our own attempts to contact upstream carriers would indicate the issue seems to affect some toll free as well as some local DID numbers. The issue is at the carrier level and outside our network, however we have opened tickets upstream and made contact to confirm they are aware of the issues. They are under investigation at this time.

Outbound calls do not seem to be affected (unless they are placed to a number which is affected).

The issue seems to affect inbound calls only at this time, and may related to an underlying network issue we observed affecting Internet latency through BC  which occured at approximately 13:12 PST.

Updates will follow.