Notices about service interruptions we are tracking including 3rd party events, and details / resolution where available.

Please check here to view status updates on known issues. In the event you are experiencing an issue not documented here, or require direct confirmation of resolution, please submit a ticket? For additional information on our ticket system, and support process, please review the details found here.

During ongoing issues, updates may be posted during the issue, with a final resolution posted following the resolution of the issue.

We may monitor issues which are outside of our network or control in this section as they may affect our clients directly as calls to / from impacted areas might be perceived as a problem on our network.

 

 

The facility is undergoing some maintenance to upgrade core network router infrastructure.

The window is scheduled to occur on June 25th, 23:00 PST - June 26th 03:00 PST. During this time frame, Network Engineers will be upgrading router firmware and migrating router network connections to new-generation line cards. This is to help improve network performance and redundancy. 

While the Network Engineering team will make every effort to keep the impact to an absolute minimum. Please expect increased latency and traffic reroutes during the duration of this maintenance.

If you have any questions, please call us or open a ticket.

Thank you for your continued business and support.

Sincerely,

BitBlock Systems, Inc.

 

2020.06.19 RESOLVED.

After several attempts to work around the issue it became apparent the source of the issue was as the calls were leaving the callers network. The static injected on the line was added by their own carrier as they attempted to call other numbers that routed through the affected equipment in their network and at least one bad cable.

There were at least two separate problems within the network of the western ILEC.

There were two separate repairs (one on the night of the 17th (around 0100 AM PST on June 18) and one on the night of (around 0100 AM PST on June 19) that restored service.

Thank you to all our clients who reported the issue. Our own team conducted about 1,000 test calls in developing the data needed to diagnose the issue. We thank our partner carriers for their cooperation and their speed in resolving the difficult issue.

 

2020.06.10 Update. This issue has been acknowleged by upstream carriers as a larger issue.

In effort to speed resolution of the issue we have completed and logged several hundred test calls complete with call recordings. We are currently working with our carrier partners to furhter isolate the issue to help them locate the source of the noise (which seems to be analog in nature) and implement a repair. We will continue to monitor the issue and push toward a resolution. Thank you!

 

2020.06.09 Original Post

We received a few reports of a crackling noise heard while placing calls from outside our system (cell or remote phone system or PSTN) to numbers on our system. In what seems to be a noticable but low percentage of total received calls the caller reports hearing moderate to severe "crackling".

Investigating calls reported which were being enabled for recording showed no evidence of the crackling noise, but a couple reporters managed to capture the sound of the call. The sound appears to be analog in nature, and by analyzing call reports we have determined the issue seems to be on the carrier network and is possibly a TDM / PSTN network.

We have started tickets with upstream carriers to confirm our findings and get the issue repaired.

At this time the issue has not been reported to affect any outbound calls or any inter-office / internal calls.

We will post updates to this article. If you are affected please open a ticket with any details you can provide.

Thank you,

BitBlock Systems, Inc.

 

Due to the high demand being placed on cellular networks (in part, all of us working from home, compounded by call forwarding (which uses two channels for each call), the networks may not keep up with demand. See video or read on...

For the forseeable future, users of all telecom networks can expect to see intermittent congestion, but there are ways to help reduce the issues.

For Clients with SIP trunking, there is a lot of flexibility to how you transition to a work at home model. There are many factors to consider – the solutions presented here will indicate suggestions for methods and could be customized to work with your existing systems. This is not about being opportunistic. It’s about education, and enabling our Clients to continue to operate their businesses where possible from home or other remote locations. We understand that we are only relevant as long as we are helping our clients.

We are aware of an ongoing issue regarding one of our carrier parnters networks. There is a telecom switch in toronto which is affecting some of our Canadian inbound numbers as well as some of our toll free services. [Since resolved].

Many of these options have always existed, but evidensed by our Clients' enquiries, now more than ever we needed a detailed explanation of the options.

At BitBlock, we believe our Clients are the core of our business. We are committed to continuing to deliver the service you expect to assist you in providing your own clients and customers the service required for business continuity.

We are refining contingency plans for our Clients and others who may be trying to alter their work environments as they adopt “social distancing”, including many options to facilitate remote work / work from home options.

To all our Clients…

Our team at BitBlock is prepared to keep your critical systems stable through the current pandemic / health crisis.

Posting this later than desired - our goal will be to get initial notice posted within 15 minutes of detecting / confirming an issue.

The issue is now resolved (14:20PST) the issue affected a single switch, and was rather unusual in that at no point during the issue (excepting the period during the restart / re-registration of phones) did the switch show a markedly decreased call volume.

The following is a summary in reverse order of what we know about the ongoing, widespread telecommunication disruption affecting Canada. The effects are felt across the country with many companies unaware of the issue as their toll free or other services failed. In many cases partial failures are reported which is conbtributing to many people remaining unaware of the problem. We are hopeful the news we received that repairs are underway will prevent this problem from extending into Monday. If not, expect many more people to start paying attention as the offices open tomorrow.

2020.02.11 00:15 PST

We are continuing to receive updates. The carriers are making progress, however repairs are about a day behind schedule - most operations are near normal, however it's still possible there will be additional disruption. Please check out the photos below for an idea of the scope of the problems. Multiple fiber cuts, and disruptions caused by weather related events span much of the province:

 More photos below... 

A network issue has been detected in Greater Vancouver. The issue affects a small percentage of clients, but those affected may have NO service as the issue affects both Telus fiber (seems at this time to include retail and wholesale) as well as DSL services. Initial reports indicate Telus is aware of the issue, and is working on a resolution. At this time we do now have an ETA.

Services should in theory restore on their own however, you could undertake to restart your modem / router periodically in case your service does not automatically restore.

If you need assistance with this process please contact support @ 604.629.8338

Updates will be posted as available.

Thanks!

BitBlock Systems, Inc.

2019.09.16 16:15 PDT Network issue resolved.

Initial cause was a failed Telus router - part of their network core in Vancouver. The router caused partial to complete loss of some Internet connections on multiple ISP's with network services located in the Vancouver area. Affected services included DSL and Fiber and in some cases unrelated services overloaded by the loss of related primary services. Approximately one hour after the issue occured, the problem resolved.

Please report any residual issues for investigation.

Thank you,

BitBlock Systems, Inc.

Effective 2019.08.03, the Federal Telecommunications Institute of Mexo are introducing changes to their numbering plan.

Mexico (country code 52) has traditionally prefixed mobile numbers with a 1 before the subscriber number internationally - so a number starting with 521 was easily identifyable as a mobile number. From August 3rd onwards that one should be omitted.

For example a number published locally as 045 56 1234 5678 would be dialed as :

011 52 1 56 1234 5678 prior to August 3rd.

011 52 56 1234 5678 on or after August 3rd.

Please update your staff as required regarding the changes - while we do not expect any trouble during transition, please report any issues you have to your support contact.

Thank you,

BitBlock Systems, Inc.

 
Contact Us