Pending steps will include a deep investigation of the carriers that failed, and the interdependencies this uncovered (where Carrier A actually seems to depend on Carrier B for some services), and what we can do to better isolate those interdependencies as well as re-striping (or redistributing) phone numbers across the carrier fabric for better resiliency in the event of a carrier failure.
2021.04.27 15:00 PST Carrier routes remain stable, no further reports have been received. We will be working on our own root cause analysis as well as providing some documentation on options to assist mitigation planning for clients who need to strategize to prevent issues like this.
2021.04.27 10:50 PST Outbound routes are restored. Capacity issues should be resolved. The carriers are reporting services are restored, but that the detailed RFO (explanation / reason for outage) will be published in 5 to 10 days. We will be updating our own plans and documentation with some mitigation strategy as a result of what we observed today. Thanks to everyone for their reports and understanding. We understand Rogers wireless customers may still be affected reaching some numbers, however this is likely something they may have to ask Rogers for support with if it continues. We were able to complete our own Rogers tests successfully at this time! More information will be provided as we collect it from the reporting carriers.
2021.04.27 10:03 PDT Inbound services seem to be restored, HOWEVER, callers from the Rogers wireless network report being able to reach some numbers but not others. Test calls from Telus cellular are working. Looking for a cause, but likely an issue on Roger's side. Will continue investigating.
2021.04.27 08:02 PDT We are experiencing a disruption caused upstream carrier failures. We are working with them to resolve the issue - in the meantime we are changing routes to restore outbound dialing capability by routing away from the affected carriers. This should ensure you are able to place calls while we work with our carrier peers to ensure the inbound calling issues are fixed.