On Sunday 26th Feb from 21:00 our engineers will be conducting some low risk maintenance on the hyper-visor layer in Wakefield.

Please note that the platform will be at risk from 22:00 until 02:00

If you have any questions please contact Node4 Service Desk.

Background

On Thursday the 2nd March 2017 between 00:00 and 04:00 GMT Node4 engineers will be performing scheduled maintenance on the N4Cloud Zerto DRaaS platform to patch to the latest version.

 

Impact

As patching occurs VPG replication will be paused for approximately 5-10 minutes whilst services are stopped. There will be no impact to live VMs and should there be a disaster recovery situation invoked Node4 will still be able to recover all VM’s as usual

 

Any questions or concerns please contact the Node4 Service Desk on 0845 123 2229 or www.node4.co.uk/support

On Wednesday 22nd February 2017 @ 8PM Node4 engineers will be performing maintenance on the Wakefield ACI cloud platform.

we expect no service disruption, however the platform will be deemed “At Risk” until Thursday 23rd February 01:00

if you have any questions please contact Node4 Service Desk

Tonight from 22:00 our engineers will be conducting some low risk maintenance on the hyper-visor layer in Wakefield.

Please note that the platform will be at risk from 22:00 until 01:00

If you have any questions please contact Node4 Service Group.

 

UPDATE

This is work has been completed successfully.

At 21:04 Node4 NOC has reported 10Gbps circuit down within our network, traffic has re-routed around resilient connections.

We are currently in talks with supplier to ascertain the reason for outage.

21:46 – Supplier is aware of the outage we are currently awaiting an update from the carrier.

23:54 – We have had reports from the supplier that there has been a fibre break. They are working to resolve this more updates follow.

01:10 – Our provider has updated us stated that engineers are due on site to the fibre break ETA 03:00am we have no ETA to resolve once they are on site more updates to follow.

03:40 – Our provider have advised that the engineers are now onsite at the fibre break and are working to fix this issue, further updates to follow.

05:20 – Engineers are currently still working to fix the fiber breaks, further updates to follow.

07:35 – Engineers are still working to fix this fiber break. Our provider is unable to provide an ETA for resolution at this time. Further updates to follow.

08:30 – Engineers are still working to fix this fiber break. No ETA as yet. We will update again in an hour.

09:30 – Provider incident Management team and techs are currently on a conference call, we expect an update imminently.

10:26 – Update from provider: – Engineers now need to gain access to a junction chamber located on the corner of a country road in order to continue working on this fiber break. This will require a traffic management team. Traffic management have been contacted and are on route. further updates to follow.

11:26 – Update from provider: –

All prep work is done, joint identified, specialist fibre engineers/OTDR’s etc on site,  still awaiting access to chamber via traffic management.

13:05 – Having chased our provider, we are informed that the status of this fault remains unchanged. Engineers on site are waiting on an emergency permit to work from the local authority. We are actively pursuing an ETA.

14:24 – Update from our providers: Permit and traffic will be in place from 15:00. Engineers are ready to work as soon as access is provided.

We will seek a further update and push for an ETA on fix at 17:00

17:30 – Our provider states that engineers are still on site, continuing to work to splice the fibres. Unfortunately no ETA is yet available on fix.

18:00 – Update from our provider :

The cable has multiple cuts beyond the chamber due to vandalism.

This means we will have to pull in approx. 650 metres of cable and re-splice both ends.

Current ETA for completion is 8 hours ETA approx. 02:00.

22:30 – Update from provider, Cable will be laid and prepped at 00:00. Splicing and testing will commence between 02:00 and 03:00.

02:20 – Update from provider, Cable preparation has been completed as scheduled. Splicing and testing is now taking place. Further updates to follow.

04:00 – Our provider has advised us that all 96 fibers have been successfully spliced, our monitoring is reflecting this as all circuits are up and passing traffic. Engineers are currently carrying our final checks and a further update will follow.

04:30 – Our provider has advised us that all testing has been completed and services have been resumed to normal operation. If you have any queries regarding this outage or have any outstanding issues please get in touch with our service desk at 08451232229.

Our alerting system has triggered an event on the SDN cloud platform at Wakefield. Therefore Node4 engineers be doing low risk maintenance on the switches tonight at 20:00.

Please note your platform will be at risk until 22:00

If you have any questions please contact Node4 service desk

 

Maintenance Tracker:

00:15 – Engineers on site preparation work commencing

00:55 – Maintenance to start shortly

01:00 – Maintenance now commencing more updates to follow.

01:55 – Maintenance is progressing new core routers are being brought into service.

02:56 – Work is still continuing we are about more than half though further updates to follow.

04:12 – Work is still on going we are drawing to the end of the maintenance. Updates to follow

04:51 – All work is completed now. We are conducting final check before giving all clear. We have observed during some of these changes that reaching certain destinations may have dropped 2 to 3 times for a small period. We apologise for the inconvenience caused by this.

05:08 – Following post checks we have observed some transit issues we are currently working to resolve these

05:20 – The issue should be been resolved following the move the of one of our peer we saw issues to certain destinations this has been rectified now.

05:35 – Following further post checks we are still seeing some certain destinations being unreachable we are currently working through these. We apologies about the extended outage we working to resolve this asap.

05:44 – We have rectified the issues to the further destination we are continuing our post check s to ensure everything is now resolved we will update once complete.

06:16 – Following the post checks we believe this is now resolved. We apologises for the outages caused outside the maintenance window from 05:00 to 5:37 this was an extended maintenance window. This was due to complexity of the routing discrepancies we saw. There were 3 occurrences that needed to be manipulated with the way we route traffic round our network. Unfortunately this was unexpected, these have now been resolved and we do not expect any further issues. We have given an all clear from now.

If you do experience any issues please contact our service desk. We do ask you supply trace routes (WinMTR) in both directions to aid in troubleshooting.

 

Background:

We intend to introduce two new high powered routers into our core network, to increase our resiliency and capabilities at this POP

Services & Locations Affected:

London Telehouse North POP MPLS and N4-TRANSIT services

Impact:

POP is at risk for the duration of the maintenance, no customers should have any direct downtime as a result of this maintenance. We will be re-routing traffic around this POP and moving traffic off LINX London for a small period,
As such customers may notice reconvergance’s around this work including latency changes to certain destinations.

Maintenance Window:

Sunday 22nd Jan 2017 02:00 – 05:00

Preparation works will start 00:00, with service affecting work beginning at 02.00. We expect the work to take no longer than 2 hours however services will be “at risk” for the full maintenance window.

Maintenance Tracking will be available here

Apologies for any inconvenience this maintenance may cause.

Regards
Node4 Technical Support

Background

Node4 engineers have identified a potential SFP issue with one of the uplinks to the Wakefield N4Protect firewall.

As a result engineers will be performing emergency maintenance to replace the affected SFP and a number of others from the same order batch as a precaution.

Impact

No impact to services is expected, as there are multiple uplinks in aggregation bundles to the firewalls,

However, services should be considered “At Risk” for the full duration of the below maintenance windows.

Wakefield DC3 – At Risk – 9th February 2017 22:00 – 23:59 GMT

Northampton DC4 – At Risk – 14th February 2017 20:00 – 23:59 GMT

 

Updates – Wakefield DC3 – 9th February 2017

22:00 GMT – Engineers are on site carrying out preparatory work.

22:15 GMT – Engineers will now begin work to replace the first set of SFP’s.

23:45 GMT – Engineers have replaced the first set of SFP’s and are currently performing checks.

23:00 GMT – Engineers have completed checks successfully following the first part of the maintenance and will now continue to replace the second set of SFP’s.

23:30 GMT – Engineers have replaced the second set of SFP’s and are currently performing checks.

23:45 GMT – Engineers have completed checks successfully following the second part of the maintenance. This maintenance is now completed at Wakefield, with further maintenance scheduled at Northampton as outlined above. If you experience any issues following this maintenance please raise a support case with the Node4 service desk so we can assist.

 

Updates – Northampton DC4 – 14th February 2017

Updates will be posted here during the maintenance.

20:30 GMT – Engineers are on site carrying out preparatory work.

21:30 GMT – Engineers will now begin work to replace the first set of SFP’s.

22:00 GMT – Engineers have replaced the first set of SFP’s and are currently performing checks.

22:15 GMT – Engineers have completed checks successfully following the first part of the maintenance and will now continue to replace the second set of SFP’s.

22:45 GMT – Engineers have replaced the second set of SFP’s and are currently performing checks.

23:00 GMT – Engineers have completed checks successfully following the second part of the maintenance. This maintenance is now completed. If you experience any issues following this maintenance please raise a support case with the Node4 service desk so we can assist.

 

Our Network Operations Engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at or visit www.node4.co.uk/support.

While it is important for Node4 to perform maintenance such as this to ensure the quality of our network, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback.

Regards,

Node4 Technical Support

On Wednesday the 15th February 2017 between 10:00 and 16:00 GMT Node4 engineers will be performing scheduled maintenance on the Node4 Zerto DRaaS Platform.

During this maintenance window Zerto will be upgraded to the latest version of the software and this will be undertaken with Zerto technical support on hand. At certain points during the maintenance window customers may notice a disconnection within Zerto whereby protected VMs will briefly stop synchronizing however should there be a disaster recovery situation invoked Node4 will still be able to recover all VM’s.

Any questions or concerns please contact the Node4 Service Desk on 0845 123 2229 or www.node4.co.uk/support

 

 

We are currently investigating network instabilities affecting our London POP – further updates to follow.

 

## Update 15:00

Incident details: Instability began at 14:10 – resolved by 14:32

Services Affected: MPLS services terminating on an MPLS P/PE router in Telehouse North POP

While we are continuing to investigate this issue we have traced the event to a network loop in an external suppliers network (connected to the MPLS router).