tag:surevoip.statuspage.io,2005:/historySureVoIP Status - Incident History2024-03-28T15:47:29+00:00SureVoIPtag:surevoip.statuspage.io,2005:Incident/98725532022-05-09T23:52:30+01:002022-05-09T23:52:30+01:00LINX LON1 Subnet change<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>23:52</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>23:30</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>15:59</var> BST</small><br><strong>Scheduled</strong> - Dear all,<br /><br />LINX LON 1 is increasing it's network subnet size to accommodate more peering points on the public peering exchange. This will only impact the IPv4 connectivity only. IPv6 will not be impacted by this change. <br /><br />While no issues are predicted customers may see minor increased latency while traffic is automatically re-routed in the event of BGP reconvergence. This change is being completed out of hours as to keep disruption to a minimum.<br /><br />Thanks,<br />SureVoIP Support.</p>tag:surevoip.statuspage.io,2005:Incident/69827932021-05-12T23:00:48+01:002021-05-12T23:00:48+01:00Edinburgh Core Router Reboots<p><small>May <var data-var='date'>12</var>, <var data-var='time'>23:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>22:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>13:46</var> BST</small><br><strong>Scheduled</strong> - Dear all,<br /><br />We need to reboot our core routers in Edinburgh. We will do this one by one to keep disruption to a minimum.<br /><br />Thanks,<br />SureVoIP Support.</p>tag:surevoip.statuspage.io,2005:Incident/66486982021-03-31T16:10:47+01:002021-03-31T16:10:47+01:00Packet loss between London and Edinburgh<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>16:10</var> BST</small><br><strong>Resolved</strong> - Dear all,<br /><br />The carrier in question had an issue on the London side of our link. They are not expecting it to happen again, but we will of course keep you updated of any issues that affect us. We can disable this link if there are prolonged issues moving forward.<br /><br />For now we are going to resolve this incident.<br /><br />Thanks,<br />SureVoIP Team.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>15:37</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />The carrier in question had a known issue during the time frame we experienced our issues. Once we have that detail, we'll share what we can. In the meantime, thanks for your patience and enjoy the rest of your day.<br /><br />We'll provide another update when we have that detail.<br /><br />Thanks,<br />SureVoIP Team.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>15:09</var> BST</small><br><strong>Monitoring</strong> - Dear all,<br /><br />We're still awaiting more info, but all is looking good. <br /><br />We'll provide another update in 30 minutes.<br /><br />Thanks,<br />SureVoIP Team.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>14:37</var> BST</small><br><strong>Identified</strong> - Dear all<br /><br />One of our two links between sites looks to have been having issues which we're investing with the relevant carrier. We'll provide another update in 30 minutes once we have more info.<br /><br />Things seem to be stabilising.<br /><br />Thanks,<br />SureVoIP Team.</p><p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>14:06</var> BST</small><br><strong>Investigating</strong> - Dear all,<br /><br />We're looking into some packet loss between Edinburgh and London (https://www.surevoip.co.uk/about/our-network) which may be affecting call quality for some customers.<br /><br />We'll have an update in 30 mins once we have identified the issue.<br /><br />Thanks,<br />SureVoIP Team.</p>tag:surevoip.statuspage.io,2005:Incident/62880432021-02-25T23:00:29+00:002021-02-25T23:00:31+00:00NTT second link migration<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>23:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>17:05</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />NTT will be migrating the last of our links to a new router in Telehouse. We have dual links to them (https://www.surevoip.co.uk/about/our-network) so we aren't expecting any connectivity issues, but there will be a Telehouse Engineer in our rack to swap cabling which poses a risk should any other cables be disturbed.<br /><br />Thank you,<br />SureVoIP Support.</p>tag:surevoip.statuspage.io,2005:Incident/61129882021-01-28T22:32:03+00:002021-01-28T22:32:05+00:00NTT link migration<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>22:32</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>22:56</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />NTT will be migrating one of our links to a new router in Telehouse. We have dual links to them (https://www.surevoip.co.uk/about/our-network) so we aren't expecting any connectivity issues, but there will be a Telehouse Engineer in our rack to swap cabling which poses a risk should any other cables be disturbed.<br /><br />Thank you,<br />SureVoIP Support.</p>tag:surevoip.statuspage.io,2005:Incident/58761212020-12-29T09:00:32+00:002020-12-29T09:00:32+00:00SureVoIP Office Closed<p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>09:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>12:45</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>12:44</var> GMT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>12:41</var> GMT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>12:40</var> GMT</small><br><strong>Scheduled</strong> - The SureVoIP office will be closed:<br /><br /> Thursday 24th December 2020<br /> Friday 25th December 2020<br /> Monday 28th December 2020<br /> Thursday 31st December 2020<br /> Friday 1st January 2021<br /> Monday 4th January 2021<br /><br />The SureVoIP office will be open:<br /><br /> Tuesday 29th December 2020<br /> Wednesday 30th December 2020<br /> Tuesday 5th January 2021 onwards<br /><br />During our closed periods our sales, support and billing teams will not be able to assist with any queries.<br /><br />If you require urgent technical support during our closed days however, please see:<br /><br />https://www.surevoip.co.uk/help-support<br /><br />Thank you,<br /><br />The SureVoIP® Team.</p>tag:surevoip.statuspage.io,2005:Incident/53690242020-11-05T21:43:47+00:002020-11-05T21:43:48+00:00SureVoIP API platform upgrades<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>21:43</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>18:16</var> GMT</small><br><strong>Update</strong> - This maintenance has been rescheduled to Thursday 5th November.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>11:46</var> BST</small><br><strong>Scheduled</strong> - We will be undergoing maintenance during this period to deploy various upgrades to the SureVoIP API. <br /><br />There will be some downtime during this window, so please update any number destinations via the API or Admin Portal prior to 8pm.<br /><br />If you have any questions, please contact support.<br /><br />Thank you.</p>tag:surevoip.statuspage.io,2005:Incident/48806382020-08-19T09:33:46+01:002020-08-19T09:33:46+01:00Ethernet - Power outage in Equinix 8/9 data centre - partial Ethernet service outage<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>09:33</var> BST</small><br><strong>Resolved</strong> - Power issue at Equinix were reported resolved at 23:56 BST last night.<br /><br />If you are still experiencing connectivity issues then please contact support for assistance.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>17:30</var> BST</small><br><strong>Update</strong> - Latest update: The data centre engineers confirmed that all network services are now restored and only redundancy is affected currently.<br />Next Action: We'll keep monitoring the services to ensure no further issues as well as we'll continue restoring the full redundancy.<br /><br />Most services have been restored. However, circuits going through TalkTalk and Virgin Media tails are still experiencing issues.<br /><br />Virgin Media advised that this is still under investigation and next update is expected at 18:00. BST<br /><br />We expect that this incident will be resolved this evening and we will provide our next (and hopefully final) update tomorrow morning.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>15:36</var> BST</small><br><strong>Update</strong> - Equinix IBX Site Staff reports that services have been further restored to more customers and increasing numbers of those affected are now operational along with the majority of Equinix Network Services. IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure and estimate full restoration by or before 21:00 GMT/BST. Customers requiring access to the IBX in order to work on their equipment should be advised that site operations are monitoring work visit requests and prioritizing affected customers.<br /><br />Some customer circuits are still affected.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>14:05</var> BST</small><br><strong>Update</strong> - Equinix IBX Site Staff reports that services have been further restored to more customers and IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure. Equinix advised access will be granted and prioritised to the IBX should any customers need it to work on their equipment.<br /><br />Some customers are still experiencing issues.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:19</var> BST</small><br><strong>Identified</strong> - We are aware that there is a major network fault in Equinix 8/9 Harbour Exchange as a result of power loss overnight. This is having an impact on network operators who route traffic through this data centre.<br /><br />Data centre staff are actively working on resolving the UPS power failure and network engineers are on standby to attend affected equipment.<br /><br />A small number of customer Ethernet circuits are directly affected by this.<br /><br />We will share further updates in due course.</p>tag:surevoip.statuspage.io,2005:Incident/47966752020-08-19T06:00:30+01:002020-08-19T06:00:30+01:00Broadband 3rd party planned engineering works<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>06:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>00:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:26</var> BST</small><br><strong>Scheduled</strong> - Tasks:<br /><br />The maintenance is required so that the 3rd party carrier can perform essential development works across their network.<br /><br /><br /><br />Service Impact:<br /><br />There is a maximum anticipated downtime of 20 minutes within the outage window, however, 6 hours have been provided to cover any unforeseen issues. Only a small number of broadband services will be affected by this maintenance.<br /><br /><br />We apologise for any inconvenience caused. If you have questions about this notification please contact support@surevoip.co.uk</p>tag:surevoip.statuspage.io,2005:Incident/48364082020-08-13T08:53:25+01:002020-08-13T08:53:25+01:00DSL/FTTx/Ethernet Outage - Scotland<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>08:53</var> BST</small><br><strong>Resolved</strong> - BT have confirmed that their engineers are currently working on restoring the final remaining circuits.<br /><br />This incident has now been confirmed as resolved by BT and no further updates will be provided.<br /><br />If you are still experiencing an issue with your connection then please contact support by emailing support@surevoip.co.uk, call 0330 445 0000 option 2 or visit www.surevoip.co.uk and click Live Help.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>16:55</var> BST</small><br><strong>Update</strong> - BT have confirmed that restoration works are currently under way and that over 50% of customer circuits are now back up and running.<br /><br />Our next and final update will be once BT have confirmed resolution of the incident.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>12:35</var> BST</small><br><strong>Update</strong> - BT engineers continue to work on rectifying the issues that were caused overnight by the adverse weather conditions in Scotland.<br /><br />We have begun to see large number of service recover, however, all circuits should be considered at risk until resolution of the incident has been confirmed</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>09:28</var> BST</small><br><strong>Investigating</strong> - We are aware of issues affecting some broadband and Ethernet connections in areas of Scotland. This is being caused by an MSO (major service outage) and BT have engineers on site at the moment trying to rectify the issues.<br /><br />As soon as we have more information, we will provide an update.</p>tag:surevoip.statuspage.io,2005:Incident/47966412020-08-13T06:00:38+01:002020-08-13T06:00:38+01:00Broadband 3rd party planned engineering works (AT RISK)<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>06:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>01:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:23</var> BST</small><br><strong>Scheduled</strong> - On of our 3rd party carriers will be performing maintenance work on one of their core nodes that we connect to for ADSL2+ and FTTC services. During this window, we expect sessions delivered through this node to drop off and reconnect via other interconnects within our network. This work does not affect a specific geographic region and due to the way that the carrier balance their sessions, we will see a number of sessions drop off across all of our L2TP endpoints with them. The majority of sessions will not drop, however, we cannot confirm which sessions will drop due to the dynamic nature of the sessions establishing through different nodes within the the carrier network.<br /><br />We apologise for any inconvenience caused. If you have questions about this notification please contact support@surevoip.co.uk</p>tag:surevoip.statuspage.io,2005:Incident/47966162020-08-07T06:01:28+01:002020-08-07T06:01:28+01:00Broadband 3rd party planned engineering works<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>06:01</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>01:30</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:23</var> BST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:23</var> BST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:20</var> BST</small><br><strong>Scheduled</strong> - Tasks:<br /><br />The maintenance is required so that the carrier can perform essential development works across their network.<br /><br /><br />Service Impact:<br /><br />There is a maximum anticipated downtime of 20 minutes within the outage window, however, 4 hours and 30 minutes have been provided to cover any unforeseen issues. Only a small number of broadband circuits will be affected.<br /><br /><br />We apologise for any inconvenience caused. If you have questions about this notification please contact support@surevoip.co.uk</p>tag:surevoip.statuspage.io,2005:Incident/46989892020-08-06T06:00:36+01:002020-08-06T06:00:36+01:00Broadband 3rd party planned engineering works (AT RISK)<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>06:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>01:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>17:34</var> BST</small><br><strong>Scheduled</strong> - The 3rd party network operator will carry out planned engineering works during the maintenance window. Broadband services are considered AT RISK during this time.<br /><br />Further details:<br />The 3rd party network will be performing maintenance work on one of their core nodes that we connect to for ADSL2+ and FTTC services. During this window, we expect sessions delivered through this node to drop off and reconnect via other interconnects within our network. This work does not affect a specific geographic region and due to the way that the 3rd party network balance their sessions, we will see a number of sessions drop off across all of our L2TP endpoints with them. The majority of sessions will not drop, however, we cannot confirm which sessions will drop due to the dynamic nature of the sessions establishing through different nodes within the the 3rd party network.<br /><br />We apologise for any inconvenience caused. If you have questions about this notification please contact support@surevoip.co.uk</p>tag:surevoip.statuspage.io,2005:Incident/40609572020-05-08T22:43:35+01:002020-05-08T22:43:35+01:00Connectivity Issues<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>22:43</var> BST</small><br><strong>Resolved</strong> - Dear all,<br /><br />We are closing this issue now.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>20:35</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />The new router is now in full production and the core network is stable and fully resilient again in Edinburgh. We will leave this in "Monitoring" status for a few more hours and then move to fully resolved.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>19:57</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />A bit later than expected, but we are now going to bring all connections back online.<br /><br />Please expect some disruption whilst this settles.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>16:55</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />An engineer will be onsite to replace the failing router in the next 30 minutes or so. Work will then begin to swap it out, cable it up and load all settings. <br /><br />We'll then post a further update when we're bringing all the different connections back online. There will be some network interruption during that time which will settle down not long afterwards.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>09:34</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />All looks stable now.<br /><br />More updates to follow once this router has been fully diagnosed and/or replaced.<br /><br />Enjoy the rest of your day and apologies again.<br /><br />Thanks.<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>09:07</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />We have now taken this router out of service, which is stabilising the network again.<br /><br />We're leaving this ticket open and the Edinburgh network marked as "Degraded performance" until the router has been fixed.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>08:55</var> BST</small><br><strong>Update</strong> - Dear all,<br /><br />More instability has occured on the router in question in Edinburgh, so we are moving to take this router out of service. We will update you once this is complete.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>06:35</var> BST</small><br><strong>Monitoring</strong> - Stability has returned to the core network. We will leave this open whilst we work with Juniper.<br /><br />Apologies for the interruption to your morning.<br /><br />Thanks,<br />Gavin.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>06:22</var> BST</small><br><strong>Identified</strong> - The issue has been identified as a hardware problem on one of our two core routers in Edinburgh. We are opening a case with Juniper.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>06:05</var> BST</small><br><strong>Investigating</strong> - We are currently investigating connectivity issues reaching the SureVoIP core network.</p>tag:surevoip.statuspage.io,2005:Incident/35609222020-02-27T23:00:31+00:002020-02-27T23:00:34+00:00Firewall changes and router component reboot<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>23:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>09:36</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />Due to staff availability changes, this maintenance has been re-scheduled to Thursday 27th February.<br /><br />Thank you,<br />SureVoIP Support Team.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>17:55</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />As per https://www.surevoip.co.uk/help-support we are giving 7 days notice for a planned outage.<br /><br />We need to make a core firewall change in London and reboot a component on one of the two routers in Edinburgh. We are not expecting any impact from the firewall change in London, but there is a risk. <br /><br />In Edinburgh, the forwarding engine needs to be rebooted. Therefore all traffic flow will be dropped on that router, which will cause a short period of instability.<br /><br />Thank you,<br />SureVoIP Support Team.</p>tag:surevoip.statuspage.io,2005:Incident/35063622020-02-01T15:25:03+00:002020-02-01T15:27:03+00:00Emailing of last months call charges invoice<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:25</var> GMT</small><br><strong>Resolved</strong> - Dear all,<br /><br />Last months call charges invoice was accidentally emailed again to 159 customers before being caught. The correct one has now been emailed. These are always available via the SureVoIP Portal.<br /><br />Apologies for the confusion,<br /><br />The SureVoIP Billing Team.</p>tag:surevoip.statuspage.io,2005:Incident/33446222020-01-03T09:00:32+00:002020-01-03T09:00:32+00:00SureVoIP Office Closure During Festive Period<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>09:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>24</var>, <var data-var='time'>15:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>13:36</var> GMT</small><br><strong>Scheduled</strong> - Dear Customers,<br /><br />The SureVoIP office will be closed from 3pm Tuesday 24th of December and reopen for business as usual on Friday 3rd January 2019.<br /><br />During the closed period our sales, support and billing teams will not be able to assist with any queries.<br /><br />If you require extremely urgent emergency technical support during our closed period please see https://www.surevoip.co.uk/help-support<br /><br />Thank you,<br /><br />The SureVoIP® Team.</p>tag:surevoip.statuspage.io,2005:Incident/33699232019-12-29T12:13:16+00:002019-12-29T12:13:17+00:00Hosted VoIP registrations and Host VoIP calls failing<p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>12:13</var> GMT</small><br><strong>Resolved</strong> - Dear all,<br /><br />We are marking this incident as resolved. There was a Linux kernal panic on this Virtual Machine host caused by a backup run today which we will investigate in due course. We will also revisit some Hosted VoIP failover scenarios.<br /><br />Thank you,<br />Gavin.</p><p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>10:54</var> GMT</small><br><strong>Monitoring</strong> - Dear all,<br /><br />All Hosted VoIP services are now back to normal. You may need to restart your devices to re-register.<br /><br />We are moving this incident to monitoring status and will investigate the cause for the freeze.<br /><br />Thank you,<br />Gavin.</p><p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>10:24</var> GMT</small><br><strong>Identified</strong> - Dear all,<br /><br />It appears one of our two Virtual Machine host servers has frozen in Edinburgh causing alternating Hosted VoIP registrations and Hosted VoIP call failures. We rebooting it now.<br /><br />Thanks,<br />Gavin.</p><p><small>Dec <var data-var='date'>29</var>, <var data-var='time'>10:17</var> GMT</small><br><strong>Investigating</strong> - Dear all,<br /><br />We are currently investigating Hosted VoIP registrations and Hosted VoIP calls failing.<br /><br />Thanks,<br />Gavin.</p>tag:surevoip.statuspage.io,2005:Incident/33601082019-12-27T12:11:58+00:002019-12-27T12:15:00+00:00Edinburgh Core Router failure<p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>12:11</var> GMT</small><br><strong>Resolved</strong> - Dear all,<br /><br />It has been a full 24 hours since the replacement router has been in production with no issues.<br /><br />We are resolving this incident.<br /><br />Thank you, <br />Gavin.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>14:49</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />The replacement router is now in service. We will continue to monitor the Edinburgh network closely and resolve this issue later this evening once we are fully satisfied.<br /><br />Thank you,<br />Gavin.</p><p><small>Dec <var data-var='date'>25</var>, <var data-var='time'>21:14</var> GMT</small><br><strong>Monitoring</strong> - Dear all,<br /><br />One of our two core routers in Edinburgh has been having issues throughout today. It has now been shutdown and taken out of service.<br /><br />All services are unaffected, but we are setting the Edinburgh network to "Degraded performance" status as we only have one router in operation there.<br /><br />Juniper have identified that the issue relates to an internal PR that describes a file system corruption on the main storage component of our MX router, therefore, it has been recommended that a replacement for the midplane on the router is sent. We are awaiting the replacement hardware.<br /><br />We will mark this issue as resolved once the router is fully back in service and monitor the Edinburgh network closely.<br /><br />Merry Christmas all and I hope you've all had a quiet day!<br /><br />Thanks,<br />Gavin.</p>tag:surevoip.statuspage.io,2005:Incident/33105512019-12-11T23:42:10+00:002019-12-11T23:42:10+00:00Configuration updates<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>23:42</var> GMT</small><br><strong>Completed</strong> - Dear all,<br /><br />We have completed our changes early. No services were interrupted at all.<br /><br />Further work will need to be done to make LONAP and all of the LINX peering LANs fully operational, but that will be done during our normal maintenance windows on Thursday nights 10pm-11pm.<br /><br />In summary, success.<br /><br />Thank you,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>21:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:51</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />We will be making some configuration updates to our London core network to correct our peering connections on LINX and LONAP. This work will consist of configuration changes which *may* cause some interruption to services.<br /><br />The work is to be carried out by Juniper ATAC and our network partners Axians.<br /><br />Apologies for the short notice,<br />Gavin and the SureVoIP Team.</p>tag:surevoip.statuspage.io,2005:Incident/32847802019-12-04T23:00:32+00:002019-12-04T23:00:36+00:00Emergency configuration changes<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>19:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>16:16</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />We will be doing emergency maintenance on the new London core routers which is expected to stabilise the internal packet loss we are still seeing at various times throughout the day. This work will consist of configuration changes which may cause some interruption to services.<br /><br />The work is to be carried out by Juniper ATAC and our network partners Axians.<br /><br />Thank you for your understanding and continued patience.<br /><br />Gavin and the SureVoIP Team.</p>tag:surevoip.statuspage.io,2005:Incident/32838042019-12-04T16:04:40+00:002019-12-04T16:04:40+00:00Connectivity issues<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>16:04</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>12:07</var> GMT</small><br><strong>Identified</strong> - Dear all,<br /><br />Improvements have been made to DNS. We are still working on connectivity issues for some customers.<br /><br />Thank you,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:57</var> GMT</small><br><strong>Investigating</strong> - Dear all,<br /><br />Some customers are reporting connectivity issues reaching the SureVoIP network.<br /><br />We are looking into this. It appears to be related to DNS queries within our network for those using the SureVoIP DNS servers.<br /><br />We suspect this is related to Monday's upgrade. JTAC and our network partner Axians are also still working on reviewing everything.<br /><br />Thank you,<br />Gavin and the SureVoIP Team.</p>tag:surevoip.statuspage.io,2005:Incident/32795542019-12-04T10:52:03+00:002019-12-04T10:52:04+00:00Call quality issues<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:52</var> GMT</small><br><strong>Resolved</strong> - Dear all,<br /><br />We're marking this incident as closed. Please contact support if you have any call quality issues.<br /><br />Thanks,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>15:23</var> GMT</small><br><strong>Monitoring</strong> - Dear all,<br /><br />Both Juniper JTAC and our network partner Axians have restored service stability and stopped internal packet loss.<br /><br />Any additional changes will be scheduled unless something major happens. We will now move this incident to monitoring status. There is still work to do to get the upgrade to where we want it to be, but service stability is more important.<br /><br />If you have an issues moving forward please contact SureVoIP Support as usual.<br /><br />My sincere apologies,<br /><br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>14:52</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />Both Juniper JTAC and our network partner Axians have made their changes and are still confirming that the internal packet loss has stopped.<br /><br />I will provide a further update at 15:45 or sooner if I have more information.<br /><br />Please contact SureVoIP Support (support@surevoip.co.uk) for a full incident report of the failures that have happened in regards to the upgrade of our core network if you require it. It will be available later this week.<br /><br />Thanks,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>13:29</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />Both Juniper JTAC and our network partner Axians have confirmed that there is no cabling issue and are validating the configuration and working on changes to stop the internal packet loss.<br /><br />I will provide a further update at 14:30 or sooner if I have more information.<br /><br />Again, I will provide a full incident report of the failures that have happened in regards to the upgrade of our core network to those that require it.<br /><br />Thanks,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>12:33</var> GMT</small><br><strong>Identified</strong> - Dear all,<br /><br />The root cause of the call quality issues has now been identified as an unexpected internal network loop causing sporadic packet loss.<br /><br />Work will be starting to check all physical cabling to resolve this.<br /><br />Apologies for the continued disruption. I will provide a further update at 13:30 or sooner if I have more information.<br /><br />I will also provide a full incident report of the failures that have happened in regards to the upgrade of our core network to those that require it.<br /><br />Thanks,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>11:22</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />Apologies for the delay.<br /><br />We now have Juniper logged in to our new equipment (JTAC) troubleshooting the root cause of the internal packet loss in London. <br /><br />We will have an update at 12:30<br /><br />Thank you for your continued patience.<br /><br />Gavin.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>10:40</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />Our network partners that performed the core upgrade last night are troubleshooting the London network internal packet loss. We will have an update at 11:15.<br /><br />Thank you,<br />Gavin and the SureVoIP Team.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>10:16</var> GMT</small><br><strong>Investigating</strong> - Dear all,<br /><br />We are still experiencing packet loss within the London network and are continuing to investigate.<br /><br />We will have further updates as soon as possible.<br /><br />Thank you.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>09:31</var> GMT</small><br><strong>Identified</strong> - Dear all,<br /><br />We've identified some BGP issues with peering exchanges and are closing them down.<br /><br />We should start to see call quality improve.<br /><br />Thank you.</p><p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>09:13</var> GMT</small><br><strong>Investigating</strong> - Dear all,<br /><br />We are investigating call quality issues reported by some customers.<br /><br />Thank you.</p>tag:surevoip.statuspage.io,2005:Incident/32535452019-12-03T01:01:34+00:002019-12-03T01:01:40+00:00London core network upgrade<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>01:01</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>21:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>11:31</var> GMT</small><br><strong>Scheduled</strong> - Dear all,<br /><br />We will be upgrading our core routers in London (Telehouse East) from the current Juniper MX5 routers to Juniper MX204 routers. This will increase our capacity by a factor of 20.<br /><br />Engineers will be on site from 8pm, with the official start time of 9pm. Network availability will fluctuate between 9pm on Monday 2nd December and 1am Tuesday 3rd December.<br /><br />A reminder of our core network can be found here:<br /><br /> https://www.surevoip.co.uk/about/our-network<br /><br />For the technically minded, details of the current and new routers can be found here:<br /><br /> https://www.juniper.net/uk/en/products-services/routing/mx-series/mx5/<br /> https://www.juniper.net/us/en/products-services/routing/mx-series/mx204/<br /><br />Thank you,<br /><br />The SureVoIP Team.</p>tag:surevoip.statuspage.io,2005:Incident/32648622019-11-28T17:59:55+00:002019-11-28T17:59:56+00:00Network instability<p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>17:59</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>17:01</var> GMT</small><br><strong>Update</strong> - Dear all,<br /><br />We are happy everything is back to business as usual. If you have any connectivity issues, calling issues or any SureVoIP issue at all, please do get in touch with SureVoIP via phone, chat or email.<br /><br />We will mark this fully resolved this evening.<br /><br />Again, apologies for the stress and interruptions to your day, your customers day and your business. <br /><br />Gavin and the SureVoIP Team.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>16:31</var> GMT</small><br><strong>Monitoring</strong> - Dear all,<br /><br />The affected router is back in service and helping stabilise the network. <br /><br />Investigations show that although the router had "frozen", it was still doing various operations, but doing them very badly. This obviously had a major impact on the rest of the network.<br /><br />We will be monitoring the London network very closely. These routers are to be replaced during the scheduled maintenance on Monday night (2nd December):<br /><br />http://status.surevoip.co.uk/incidents/npzhmmlnzp68<br /><br />Thank you for your patience and sincere apologies for the interruptions to your business and customers.<br /><br />Gavin and the SureVoIP Team.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>15:55</var> GMT</small><br><strong>Identified</strong> - We have taken the decision to reboot the router with immediate effect. <br /><br />We will update with further news as soon as we have it.<br /><br />Thanks, <br /><br />SureVoIP Team</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>14:34</var> GMT</small><br><strong>Monitoring</strong> - The affected core router is operational, but unresponsive on the network. We will be rebooting it and bringing it back into service late this evening so as to avoid any further disruption during business hours.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>14:26</var> GMT</small><br><strong>Update</strong> - One of our core London routers is unresponsive and the second one has taken over to stabilise the network.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>14:23</var> GMT</small><br><strong>Investigating</strong> - Dear all,<br /><br />We have been alerted to a period of instability in our London core network which we are investigating.</p>